Test bug/next dev (#63)

* ci(deps): Bump actions/cache from 3.3.1 to 3.3.2 (OSOceanAcoustics#1161) Bumps [actions/cache](https://github.com/actions/cache) from 3.3.1 to 3.3.2. - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](actions/cache@v3.3.1...v3.3.2) --- updated-dependencies: - dependency-name: actions/cache dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): [pre-commit.ci] pre-commit autoupdate (OSOceanAcoustics#1162) updates: - [github.com/psf/black: 23.7.0 → 23.9.1](psf/black@23.7.0...23.9.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Feature/add transient noise mask (#49) * added the feature described in issue 1 * Moved code for the signal attenuation mask to new file * Resolved merge conflicts by accepting remote versions of files * Update mask_attenuated_signal.py * renamed utils * Added unit test * Test can now automatically download its data * Update echopype/utils/mask_transformation.py Co-authored-by: Raluca Simedroni <92971445+simedroniraluca@users.noreply.github.com> * Update echopype/utils/mask_transformation.py Co-authored-by: Raluca Simedroni <92971445+simedroniraluca@users.noreply.github.com> * Update echopype/utils/mask_transformation.py Co-authored-by: Raluca Simedroni <92971445+simedroniraluca@users.noreply.github.com> * Update echopype/utils/mask_transformation.py Co-authored-by: Raluca Simedroni <92971445+simedroniraluca@users.noreply.github.com> * Update echopype/utils/mask_transformation.py Co-authored-by: Raluca Simedroni <92971445+simedroniraluca@users.noreply.github.com> * Update echopype/utils/mask_transformation.py Co-authored-by: Raluca Simedroni <92971445+simedroniraluca@users.noreply.github.com> * Update echopype/utils/mask_transformation.py Co-authored-by: Raluca Simedroni <92971445+simedroniraluca@users.noreply.github.com> * Cleaned up mask_transformation * moved files and rerouted importing paths accordingly (#36) Co-authored-by: alejandro-ariza <alejandro.ariza@ird.fr> * Feature/add transient noise mask (#35) * Completed feature #3 transient noise mask --------- Co-authored-by: Mihai Boldeanu <35964395+mihaiboldeanu@users.noreply.github.com> Co-authored-by: Raluca Simedroni <92971445+simedroniraluca@users.noreply.github.com> Co-authored-by: ruxandra valcu <ruxandra.valcu@gmail.com> * Added extra tests for the utils/mask_transformation module to increase coverage * Reverted test_mask.py reformating * Moved impulse and transient noise functions to echopype/clean * Fixed import order * Fixed missing import in clean/conftest.py * Moved content of tests/clean/conftest.py to tests/conftest.py * Refactoring * Running some linters on the code * ci(deps): Bump actions/cache from 3.3.1 to 3.3.2 (OSOceanAcoustics#1161) Bumps [actions/cache](https://github.com/actions/cache) from 3.3.1 to 3.3.2. - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](actions/cache@v3.3.1...v3.3.2) --- updated-dependencies: - dependency-name: actions/cache dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): [pre-commit.ci] pre-commit autoupdate (OSOceanAcoustics#1162) updates: - [github.com/psf/black: 23.7.0 → 23.9.1](psf/black@23.7.0...23.9.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix unused imports in conftest * Update echopype/mask/mask_attenuated_signal.py Co-authored-by: Raluca Simedroni <92971445+simedroniraluca@users.noreply.github.com> Update mask_transformation.py Cleaned up mask_transformation Delinted Removing pytest req Refactoring Refactoring and cleaning imports Fixed twod test Fixing some future merge conflicts Fix unused import Added attenuation mask to init * Added channel selection to signal attenuation mask --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Mihai Boldeanu <mishu_boldy@yahoo.com> Co-authored-by: Mihai Boldeanu <35964395+mihaiboldeanu@users.noreply.github.com> Co-authored-by: Andrei Rusu <andrei.rusu@beatfactor.net> Co-authored-by: Raluca Simedroni <92971445+simedroniraluca@users.noreply.github.com> Co-authored-by: Andrei Rusu <beatfactor@users.noreply.github.com> Co-authored-by: alejandro-ariza <alejandro.ariza@ird.fr> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Feature/add attenuation mask (#50) * added the feature described in issue 1 * Moved code for the signal attenuation mask to new file * Resolved merge conflicts by accepting remote versions of files * Update mask_attenuated_signal.py * renamed utils * Added unit test * Test can now automatically download its data * Update echopype/utils/mask_transformation.py Co-authored-by: Raluca Simedroni <92971445+simedroniraluca@users.noreply.github.com> * Update echopype/utils/mask_transformation.py Co-authored-by: Raluca Simedroni <92971445+simedroniraluca@users.noreply.github.com> * Update echopype/utils/mask_transformation.py Co-authored-by: Raluca Simedroni <92971445+simedroniraluca@users.noreply.github.com> * Update echopype/utils/mask_transformation.py Co-authored-by: Raluca Simedroni <92971445+simedroniraluca@users.noreply.github.com> * Update echopype/utils/mask_transformation.py Co-authored-by: Raluca Simedroni <92971445+simedroniraluca@users.noreply.github.com> * Update echopype/utils/mask_transformation.py Co-authored-by: Raluca Simedroni <92971445+simedroniraluca@users.noreply.github.com> * Update echopype/utils/mask_transformation.py Co-authored-by: Raluca Simedroni <92971445+simedroniraluca@users.noreply.github.com> * Cleaned up mask_transformation * Update echopype/mask/mask_attenuated_signal.py Co-authored-by: Raluca Simedroni <92971445+simedroniraluca@users.noreply.github.com> Update mask_transformation.py Cleaned up mask_transformation Delinted Removing pytest req Refactoring Refactoring and cleaning imports Fixed twod test Fixing some future merge conflicts Fix unused import Added attenuation mask to init * Added channel selection to signal attenuation mask --------- Co-authored-by: Mihai Boldeanu <mishu_boldy@yahoo.com> Co-authored-by: Mihai Boldeanu <35964395+mihaiboldeanu@users.noreply.github.com> Co-authored-by: Andrei Rusu <andrei.rusu@beatfactor.net> Co-authored-by: Raluca Simedroni <92971445+simedroniraluca@users.noreply.github.com> * Patch/nasc (#51) * Add conversion factors for NASC computation - Convert backscattering cross-section to scattering cross-section. - Convert meters to nmi. * Fix nasc test --------- Co-authored-by: Raluca Simedroni <92971445+simedroniraluca@users.noreply.github.com> * Merge/add shoal mask (#55) * Mask utils * rename * Mask utils * Weill shoal mask * Decided against implementing the echoview method * Turned out not to need these * Add wrapper function * Refactoring and adding channel selection capability * Merge/add seabed mask (#56) * Initial implementation of the seabed masking. * Code for seabed masking * Changed dilation source package from cv2 to skimage. * some refactoring and add unit test * Update echopype/mask/mask_seabed.py Co-authored-by: Raluca Simedroni <92971445+simedroniraluca@users.noreply.github.com> * Update echopype/tests/mask/test_mask_seabed.py Co-authored-by: Raluca Simedroni <92971445+simedroniraluca@users.noreply.github.com> * Re-added mask_transformation, with only the necessary functions kept * Changed the tests to be able to autodownload its files - must check that I have the correct one though since the test isn't passing * Cleaned up mask_transformation * Copyright notices Co-authored-by: Raluca Simedroni <92971445+simedroniraluca@users.noreply.github.com> * Update echopype/mask/mask_seabed.py Co-authored-by: Raluca Simedroni <92971445+simedroniraluca@users.noreply.github.com> * Update echopype/mask/mask_seabed.py Co-authored-by: Raluca Simedroni <92971445+simedroniraluca@users.noreply.github.com> * Update incorrect file name * Mask transformation consistency * Mask transformation consistency * Removing pytest req * Refactoring * Fix twod test * Refactoring * Test fixes * Fixing merge compatibility * Added dataset generation for datasets enriched with beam angle and location data * Added dataset generation for datasets enriched with beam angle and location data * Added channel selection. * Modified Blackwell methods to use the physical beam angle rather than the raw EK60 beam angle - doing this using a correction on the angle itself (rather than modifying the constants) since the changes are nonlinear so if I want a linear transformation, I need to apply it on the angle data itself - angle * 22 * 128 / 180, which gets it close to how a decently calibrated EK60 instrument as used by Blackwell et al stores its raw angle data. * Indent fix * Merge fixes --------- Co-authored-by: Mihai Boldeanu <mishu_boldy@yahoo.com> Co-authored-by: Andrei Rusu <andrei.rusu@beatfactor.net> Co-authored-by: Andrei Rusu <beatfactor@users.noreply.github.com> Co-authored-by: Raluca Simedroni <92971445+simedroniraluca@users.noreply.github.com> * Merge/add multichannel masks (#57) * Initial multichannel mask * Fixed a has-channels check in apply_mask, added tests * Documentation * Made changes to conftest --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Mihai Boldeanu <mishu_boldy@yahoo.com> Co-authored-by: Mihai Boldeanu <35964395+mihaiboldeanu@users.noreply.github.com> Co-authored-by: Andrei Rusu <andrei.rusu@beatfactor.net> Co-authored-by: Raluca Simedroni <92971445+simedroniraluca@users.noreply.github.com> Co-authored-by: Andrei Rusu <beatfactor@users.noreply.github.com> Co-authored-by: alejandro-ariza <alejandro.ariza@ird.fr>
OceanStreamIO · Sep 21, 2023 · 8433cd7 · 8433cd7
1 parent 6532d62
commit 8433cd7
Show file tree

Hide file tree

Showing 24 changed files with 3,778 additions and 168 deletions.
diff --git a/.github/workflows/windows.yaml b/.github/workflows/windows.yaml
@@ -51,7 +51,7 @@ jobs:
             python-version: ${{ matrix.python-version }}
             architecture: x64
         - name: Cache conda
-          uses: actions/cache@v3.3.1
+          uses: actions/cache@v3.3.2
           env:
             # Increase this value to reset cache if '.ci_helpers/py{0}.yaml' has not changed
             CACHE_NUMBER: 0

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -24,7 +24,7 @@ repos:
         args: ["--profile", "black", "--filter-files"]
 
   - repo: https://github.com/psf/black
-    rev: 23.7.0
+    rev: 23.9.1
     hooks:
       - id: black
 

diff --git a/echopype/clean/__init__.py b/echopype/clean/__init__.py
@@ -1,6 +1,10 @@
 from .api import estimate_noise, remove_noise
+from .api import get_impulse_noise_mask, get_transient_noise_mask, get_attenuation_mask
 
 __all__ = [
     "estimate_noise",
     "remove_noise",
+    "get_impulse_noise_mask",
+    "get_transient_noise_mask",
+    "get_attenuation_mask"
 ]
diff --git a/echopype/clean/api.py b/echopype/clean/api.py
@@ -1,8 +1,19 @@
 """
 Functions for reducing variabilities in backscatter data.
 """
+import pathlib
+from typing import List, Optional, Tuple, Union
+import numpy as np
+import xarray as xr
 
 from ..utils.prov import add_processing_level, echopype_prov_attrs, insert_input_processing_level
+from . import transient_noise
+from .impulse_noise import (
+    _find_impulse_mask_ryan,
+    _find_impulse_mask_ryan_iterable,
+    _find_impulse_mask_wang,
+)
+from . import signal_attenuation
 from .noise_est import NoiseEst
 
 
@@ -74,3 +85,208 @@ def remove_noise(ds_Sv, ping_num, range_sample_num, noise_max=None, SNR_threshol
     ds_Sv = insert_input_processing_level(ds_Sv, input_ds=ds_Sv)
 
     return ds_Sv
+
+
+
+def get_transient_noise_mask(
+    source_Sv: Union[xr.Dataset, str, pathlib.Path],
+    desired_channel: str,
+    mask_type: str = "ryan",
+    **kwargs,
+) -> xr.DataArray:
+    """
+    Create a mask based on the identified signal attenuations of Sv values at 38KHz.
+    This method is based on:
+    Ryan et al. (2015) ‘Reducing bias due to noise and attenuation in
+        open-ocean echo integration data’, ICES Journal of Marine Science,
+        72: 2482–2493.
+
+    Parameters
+    ----------
+    source_Sv: xr.Dataset or str or pathlib.Path
+        If a Dataset this value contains the Sv data to create a mask for,
+        else it specifies the path to a zarr or netcdf file containing
+        a Dataset. This input must correspond to a Dataset that has the
+        coordinate ``channel`` and variables ``frequency_nominal`` and ``Sv``.
+    mask_type: str with either "ryan" or "fielding" based on
+        the preferred method for signal attenuation mask generation
+    Returns
+    -------
+    xr.DataArray
+        A DataArray containing the mask for the Sv data. Regions satisfying the thresholding
+        criteria are filled with ``True``, else the regions are filled with ``False``.
+
+    Raises
+    ------
+    ValueError
+        If neither ``ryan`` or ``fielding`` are given
+
+    """
+    assert mask_type in ["ryan", "fielding"], "mask_type must be either 'ryan' or 'fielding'"
+    selected_channel_Sv = source_Sv.sel(channel=desired_channel)
+    Sv = selected_channel_Sv["Sv"].values
+    r = source_Sv["echo_range"].values[0, 0]
+    if mask_type == "ryan":
+        # Define a list of the keyword arguments your function can handle
+        valid_args = {"m", "n", "thr", "excludeabove", "operation"}
+        # Use dictionary comprehension to filter out any kwargs not in your list
+        filtered_kwargs = {k: v for k, v in kwargs.items() if k in valid_args}
+        mask = transient_noise._ryan(Sv, r, m=5, **filtered_kwargs)
+    elif mask_type == "fielding":
+        # Define a list of the keyword arguments your function can handle
+        valid_args = {"r0", "r1", "roff", "n", "thr"}
+        # Use dictionary comprehension to filter out any kwargs not in your list
+        filtered_kwargs = {k: v for k, v in kwargs.items() if k in valid_args}
+        mask = transient_noise._fielding(Sv, r, **filtered_kwargs)
+    else:
+        raise ValueError("The provided mask_type must be ryan or fielding!")
+
+    mask = np.logical_not(mask)
+    return_mask = xr.DataArray(
+        mask,
+        dims=("ping_time", "range_sample"),
+        coords={"ping_time": source_Sv.ping_time, "range_sample": source_Sv.range_sample},
+    )
+    return return_mask
+
+
+def get_impulse_noise_mask(
+    source_Sv: xr.Dataset,
+    desired_channel: str,
+    thr: Union[Tuple[float, float], int, float],
+    m: Optional[Union[int, float]] = None,
+    n: Optional[Union[int, Tuple[int, int]]] = None,
+    erode: Optional[List[Tuple[int, int]]] = None,
+    dilate: Optional[List[Tuple[int, int]]] = None,
+    median: Optional[List[Tuple[int, int]]] = None,
+    method: str = "ryan",
+) -> xr.DataArray:
+    """
+    Algorithms for masking Impulse noise.
+
+    Parameters
+    ----------
+    source_Sv: xr.Dataset
+        Dataset  containing the Sv data to create a mask
+    desired_channel: str
+        Name of the desired frequency channel.
+    thr: float or tuple
+        User-defined threshold value (dB) (ryan and ryan iterable) o
+        r a 2-element tuple specifying the range of threshold values (wang).
+    m: int or float, optional
+        Vertical binning length (in number of samples or range) (ryan and ryan iterable).
+        Defaults to None.
+    n: int or tuple, optional
+        Number of pings either side for comparisons (ryan),
+        or a 2-element tuple specifying the range (ryan iterable).
+        Defaults to None.
+    erode: List of 2-element tuples, optional
+        List indicating the window's size for each erosion cycle (wang). Defaults to None.
+    dilate: List of 2-element tuples, optional
+        List indicating the window's size for each dilation cycle (wang). Defaults to None.
+    median: List of 2-element tuples, optional
+        List indicating the window's size for each median filter cycle (wang). Defaults to None.
+    method: str, optional
+        The method (ryan, ryan iterable or wang) used to mask impulse noise. Defaults to 'ryan'.
+
+    Returns
+    -------
+    xr.DataArray
+        A DataArray consisting of a mask for the Sv data, wherein True values signify
+        samples that are free of noise.
+    """
+
+    # Our goal is to have a mask where True represents samples that are NOT impulse noise.
+    # So, we negate the obtained mask.
+
+    if method == "ryan":
+        impulse_mask_ryan = _find_impulse_mask_ryan(source_Sv, desired_channel, m, n, thr)
+        noise_free_mask = ~impulse_mask_ryan
+    elif method == "ryan_iterable":
+        impulse_mask_ryan_iterable = _find_impulse_mask_ryan_iterable(
+            source_Sv, desired_channel, m, n, thr
+        )
+        noise_free_mask = ~impulse_mask_ryan_iterable
+    elif method == "wang":
+        impulse_mask_wang = _find_impulse_mask_wang(
+            source_Sv, desired_channel, thr, erode, dilate, median
+        )
+        noise_free_mask = ~impulse_mask_wang
+    else:
+        raise ValueError(f"Unsupported method: {method}")
+
+    return noise_free_mask
+
+
+
+def get_attenuation_mask(
+    source_Sv: Union[xr.Dataset, str, pathlib.Path],
+    desired_channel: str,
+    mask_type: str = "ryan",
+    **kwargs
+) -> xr.DataArray:
+    """
+    Create a mask based on the identified signal attenuations of Sv values at 38KHz.
+    This method is based on:
+    Ryan et al. (2015) ‘Reducing bias due to noise and attenuation in
+        open-ocean echo integration data’, ICES Journal of Marine Science,
+        72: 2482–2493.
+    and,
+    Ariza et al. (2022) 'Acoustic seascape partitioning through functional data analysis',
+    Journal of Biogeography, 00, 1– 15. https://doi.org/10.1111/jbi.14534
+
+
+    Parameters
+    ----------
+    source_Sv: xr.Dataset or str or pathlib.Path
+        If a Dataset this value contains the Sv data to create a mask for,
+        else it specifies the path to a zarr or netcdf file containing
+        a Dataset. This input must correspond to a Dataset that has the
+        coordinate ``channel`` and variables ``frequency_nominal`` and ``Sv``.
+    desired_channel: the Dataset channel to be used for identifying the signal attenuation.
+    mask_type: str with either "ryan" or "ariza" based on the
+                preferred method for signal attenuation mask generation
+    Returns
+    -------
+    xr.DataArray
+        A DataArray containing the mask for the Sv data. Regions satisfying the thresholding
+        criteria are filled with ``True``, else the regions are filled with ``False``.
+
+    Raises
+    ------
+    ValueError
+        If neither ``ryan`` or ``azira`` are given
+
+    Notes
+    -----
+
+
+    Examples
+    --------
+
+    """
+    assert mask_type in ["ryan", "ariza"], "mask_type must be either 'ryan' or 'ariza'"
+    selected_channel_Sv = source_Sv.sel(channel=desired_channel)
+    Sv = selected_channel_Sv["Sv"].values
+    r = source_Sv["echo_range"].values[0, 0]
+    if mask_type == "ryan":
+        # Define a list of the keyword arguments your function can handle
+        valid_args = {"r0", "r1", "n", "thr", "start"}
+        # Use dictionary comprehension to filter out any kwargs not in your list
+        filtered_kwargs = {k: v for k, v in kwargs.items() if k in valid_args}
+        mask = signal_attenuation._ryan(Sv, r, **filtered_kwargs)
+    elif mask_type == "ariza":
+        # Define a list of the keyword arguments your function can handle
+        valid_args = {"offset", "thr", "m", "n"}
+        # Use dictionary comprehension to filter out any kwargs not in your list
+        filtered_kwargs = {k: v for k, v in kwargs.items() if k in valid_args}
+        mask = signal_attenuation._ariza(Sv, r, **filtered_kwargs)
+    else:
+        raise ValueError("The provided mask_type must be ryan or ariza!")
+
+    return_mask = xr.DataArray(
+        mask,
+        dims=("ping_time", "range_sample"),
+        coords={"ping_time": source_Sv.ping_time, "range_sample": source_Sv.range_sample},
+    )
+    return return_mask