Merge pull request #57 from deel-ai/develop

Merge "develop" into "master", preparing version 1.3.0

.gitignore

@@ -7,6 +7,7 @@ __pycache__
 # IDE / Tools files
 .vscode
 .tox
+deel_lip_dev_env
 
 # Files generated:
 logs

CONTRIBUTING.md

@@ -0,0 +1,55 @@
+# Contributing
+
+Thanks for taking the time to contribute!
+
+From opening a bug report to creating a pull request: every contribution is
+appreciated and welcome. If you're planning to implement a new feature or change
+the API, please create an issue first. This way we can ensure that your precious
+work is not in vain.
+
+
+## Setup the development environment
+
+- Clone the repo and go to your local `deel-lip` folder
+- Create a virtual environment and install the development requirements:
+
+  `make prepare-dev && source deel_lip_dev_env/bin/activate`.
+- You are ready to install the library:
+
+  `pip install -e .`
+
+Welcome to the team!
+
+
+## Check your code changes
+
+Before opening a pull request, please make sure you check your code and you run the
+unit tests:
+
+```bash
+# Using make:
+$ make test
+
+# Or using directly tox in your development environment:
+$ tox
+```
+
+This command will:
+- check your code with black PEP-8 formatter and flake8 linter.
+- run `unittest` on the `tests/` folder with Python 3.6, 3.7 and 3.8.
+> Note: If you do not have those 3 interpreters, the tests will only be only performed
+  with your current interpreter.
+
+
+## Submitting your changes
+
+You can submit a pull request. Something that will increase the chance that your pull
+request is accepted:
+
+- Write tests and ensure that the existing ones pass.
+- If `make test` is successful, you have fair chances to pass the CI workflows (linting
+  and tests)
+- Write a [good commit message](https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html) (we follow a lowercase convention).
+- For a major fix/feature make sure your PR has an issue and if it doesn't, please
+  create one. This would help discussion with the community, and polishing ideas in case
+  of a new feature.

Makefile

@@ -0,0 +1,34 @@
+.PHONY: help prepare-dev test test-disable-gpu doc ipynb-to-rst
+.DEFAULT: help
+
+help:
+	@echo "make prepare-dev"
+	@echo "       create and prepare development environment, use only once"
+	@echo "make test"
+	@echo "       run tests and linting on py36, py37, py38"
+	@echo "make test-disable-gpu"
+	@echo "       run test with gpu disabled"
+	@echo "make doc"
+	@echo "       build Sphinx docs documentation"
+	@echo "ipynb-to-rst"
+	@echo "       Transform notebooks to .rst files in documentation and generate the doc"
+
+prepare-dev:
+	python3 -m pip install virtualenv
+	python3 -m venv deel_lip_dev_env
+	. deel_lip_dev_env/bin/activate && pip install --upgrade pip
+	. deel_lip_dev_env/bin/activate && pip install -e .
+	. deel_lip_dev_env/bin/activate && pip install -e .[dev]
+	. deel_lip_dev_env/bin/activate && pip install -e .[docs]
+
+test:
+	. deel_lip_dev_env/bin/activate && tox
+
+test-disable-gpu:
+	. deel_lip_dev_env/bin/activate && CUDA_VISIBLE_DEVICES=-1 tox
+
+doc:
+	. deel_lip_dev_env/bin/activate && cd doc && make html && cd -
+
+ipynb-to-rst:
+	. deel_lip_dev_env/bin/activate && cd doc && ./generate_doc.sh && cd -

README.md

@@ -1,7 +1,9 @@
-# deel-lip
+<img src="doc/source/logo.svg#gh-light-mode-only" alt="logo deel-lip" style="width:500px;"/>
+<img src="doc/source/logo_white.svg#gh-dark-mode-only" alt="logo deel-lip" style="width:500px;"/>
 
 [![Python](https://img.shields.io/pypi/pyversions/deel-lip.svg)](https://pypi.org/project/deel-lip)
 [![PyPI](https://img.shields.io/pypi/v/deel-lip.svg)](https://pypi.org/project/deel-lip)
+[![Downloads](https://pepy.tech/badge/deel-lip)](https://pepy.tech/project/deel-lip)
 [![Documentation](https://img.shields.io/badge/api-reference-blue.svg)](https://deel-lip.readthedocs.io)
 [![deel-lip tests](https://github.com/deel-ai/deel-lip/actions/workflows/python-tests.yml/badge.svg?branch=master)](https://github.com/deel-ai/deel-lip/actions/workflows/python-tests.yml)
 [![deel-lip linters](https://github.com/deel-ai/deel-lip/actions/workflows/python-linters.yml/badge.svg?branch=master)](https://github.com/deel-ai/deel-lip/actions/workflows/python-linters.yml)
@@ -146,8 +148,8 @@ In order to use `deel-lip`, you also need a [valid tensorflow installation](http
 ## Cite this work
 
 This library has been built to support the work presented in the paper
-[*Achieving robustness in classification using optimaltransport with Hinge regularization*](https://arxiv.org/abs/2006.06520)
-which aim provable and efficient robustness by design.
+[*Achieving robustness in classification using optimal transport with Hinge regularization*](https://arxiv.org/abs/2006.06520)
+which aims provable and efficient robustness by design.
 
 This work can be cited as:
 ```latex
@@ -161,20 +163,10 @@ Eprint = {arXiv:2006.06520},
 
 ## Contributing
 
-To contribute, you can open an [issue](https://github.com/deel-ai/deel-lip/issues), or fork this repository and then submit
-changes through a [pull-request](https://github.com/deel-ai/deel-lip/pulls).
-We use [`black`](https://pypi.org/project/black/) to format the code and follow PEP-8 convention. To check
-that your code will pass the lint-checks, you can run:
-
-```bash
-tox -e py36-lint
-```
-
-You need [`tox`](https://tox.readthedocs.io/en/latest/) in order to run this. You can install it via `pip`:
-
-```bash
-pip install tox
-```
+Contributions are welcome! You can open an [issue](https://github.com/deel-ai/deel-lip/issues),
+or fork this repository and then submit changes through a
+[pull-request](https://github.com/deel-ai/deel-lip/pulls).
+Take a look at our [contribution guidelines](CONTRIBUTING.md).
 
 ## License
 

deel/lip/__init__.py

@@ -2,3 +2,14 @@
 # rights reserved. DEEL is a research program operated by IVADO, IRT Saint Exupéry,
 # CRIAQ and ANITI - https://www.deel.ai/
 # =====================================================================================
+from . import activations
+from . import callbacks
+from . import constraints
+from . import initializers
+from . import layers
+from . import unconstrained_layers
+from . import losses
+from . import metrics
+from .model import Sequential, Model, vanillaModel
+from . import normalizers
+from . import utils

deel/lip/activations.py

@@ -120,26 +120,33 @@ def __init__(
         self.data_format = data_format
 
     def build(self, input_shape):
+        input_shape = tf.TensorShape(input_shape)
         super(GroupSort, self).build(input_shape)
         self._init_lip_coef(input_shape)
         if (self.n is None) or (self.n > input_shape[self.channel_axis]):
             self.n = input_shape[self.channel_axis]
         if (input_shape[self.channel_axis] % self.n) != 0:
             raise RuntimeError("self.n has to be a divisor of the number of channels")
+        input_shape = tuple(input_shape.as_list())
+        self.flat_shape = (
+            (-1,) + input_shape[1:-1] + (input_shape[-1] // self.n, self.n)
+        )
+        self.out_shape = (-1,) + input_shape[1:]
 
     def _compute_lip_coef(self, input_shape=None):
         return 1.0
 
+    @tf.function
     def call(self, x, **kwargs):
-        fv = tf.reshape(x, [-1, self.n])
+        fv = tf.reshape(x, self.flat_shape)
         if self.n == 2:
-            b, c = tf.split(fv, 2, 1)
-            newv = tf.concat([tf.minimum(b, c), tf.maximum(b, c)], axis=1)
-            newv = tf.reshape(newv, tf.shape(x))
-            return newv
+            b, c = tf.split(fv, 2, -1)
+            newv = tf.concat([tf.minimum(b, c), tf.maximum(b, c)], axis=-1)
+            newv = tf.reshape(newv, self.out_shape)
+            return newv * self._get_coef()
 
         newv = tf.sort(fv)
-        newv = tf.reshape(newv, tf.shape(x))
+        newv = tf.reshape(newv, self.out_shape)
         return newv * self._get_coef()
 
     def get_config(self):

deel/lip/constraints.py

@@ -9,7 +9,12 @@
 import tensorflow as tf
 from tensorflow.keras import backend as K
 from tensorflow.keras.constraints import Constraint
-from .normalizers import reshaped_kernel_orthogonalization
+from .normalizers import (
+    reshaped_kernel_orthogonalization,
+    DEFAULT_EPS_SPECTRAL,
+    DEFAULT_EPS_BJORCK,
+    DEFAULT_BETA_BJORCK,
+)
 from tensorflow.keras.utils import register_keras_serializable
 
 
@@ -71,7 +76,12 @@ def get_config(self):
 @register_keras_serializable("deel-lip", "SpectralConstraint")
 class SpectralConstraint(Constraint):
     def __init__(
-        self, k_coef_lip=1.0, niter_spectral=3, niter_bjorck=15, u=None
+        self,
+        k_coef_lip=1.0,
+        eps_spectral=DEFAULT_EPS_SPECTRAL,
+        eps_bjorck=DEFAULT_EPS_BJORCK,
+        beta_bjorck=DEFAULT_BETA_BJORCK,
+        u=None,
     ) -> None:
         """
         Ensure that *all* singular values of the weight matrix equals to 1. Computation
@@ -82,13 +92,15 @@ def __init__(
 
         Args:
             k_coef_lip: lipschitz coefficient of the weight matrix
-            niter_spectral: number of iteration to find the maximum singular value.
-            niter_bjorck: number of iteration with Bjorck algorithm..
+            eps_spectral: stopping criterion for the iterative power algorithm.
+            eps_bjorck: stopping criterion Bjorck algorithm.
+            beta_bjorck: beta parameter in bjorck algorithm.
             u: vector used for iterated power method, can be set to None (used for
                 serialization/deserialization purposes).
         """
-        self.niter_spectral = niter_spectral
-        self.niter_bjorck = niter_bjorck
+        self.eps_spectral = eps_spectral
+        self.eps_bjorck = eps_bjorck
+        self.beta_bjorck = beta_bjorck
         self.k_coef_lip = k_coef_lip
         if not (isinstance(u, tf.Tensor) or (u is None)):
             u = tf.convert_to_tensor(u)
@@ -100,16 +112,18 @@ def __call__(self, w):
             w,
             self.u,
             self.k_coef_lip,
-            self.niter_spectral,
-            self.niter_bjorck,
+            self.eps_spectral,
+            self.eps_bjorck,
+            self.beta_bjorck,
         )
         return wbar
 
     def get_config(self):
         config = {
             "k_coef_lip": self.k_coef_lip,
-            "niter_spectral": self.niter_spectral,
-            "niter_bjorck": self.niter_bjorck,
+            "eps_spectral": self.eps_spectral,
+            "eps_bjorck": self.eps_bjorck,
+            "beta_bjorck": self.beta_bjorck,
             "u": None if self.u is None else self.u.numpy(),
         }
         base_config = super(SpectralConstraint, self).get_config()

deel/lip/initializers.py

@@ -4,16 +4,22 @@
 # =====================================================================================
 from tensorflow.keras.initializers import Initializer, Orthogonal
 from tensorflow.keras import initializers
-from .normalizers import reshaped_kernel_orthogonalization
+from .normalizers import (
+    reshaped_kernel_orthogonalization,
+    DEFAULT_EPS_SPECTRAL,
+    DEFAULT_EPS_BJORCK,
+    DEFAULT_BETA_BJORCK,
+)
 from tensorflow.keras.utils import register_keras_serializable
 
 
 @register_keras_serializable("deel-lip", "SpectralInitializer")
 class SpectralInitializer(Initializer):
     def __init__(
         self,
-        niter_spectral=3,
-        niter_bjorck=15,
+        eps_spectral=DEFAULT_EPS_SPECTRAL,
+        eps_bjorck=DEFAULT_EPS_BJORCK,
+        beta_bjorck=DEFAULT_BETA_BJORCK,
         k_coef_lip=1.0,
         base_initializer=Orthogonal(gain=1.0, seed=None),
     ) -> None:
@@ -22,13 +28,16 @@ def __init__(
         normalization.
 
         Args:
-            niter_spectral: number of iteration to do with the iterative power method
-            niter_bjorck: number of iteration to do with the bjorck algorithm
+            eps_spectral: stopping criterion of iterative power method
+            eps_bjorck: float greater than 0, stopping criterion of
+                bjorck algorithm, setting it to None disable orthogonalization
+            beta_bjorck: beta parameter of bjorck algorithm
             base_initializer: method used to generate weights before applying the
                 orthonormalization
         """
-        self.niter_spectral = niter_spectral
-        self.niter_bjorck = niter_bjorck
+        self.eps_spectral = eps_spectral
+        self.eps_bjorck = eps_bjorck
+        self.beta_bjorck = beta_bjorck
         self.k_coef_lip = k_coef_lip
         self.base_initializer = initializers.get(base_initializer)
         super(SpectralInitializer, self).__init__()
@@ -39,15 +48,17 @@ def __call__(self, shape, dtype=None, partition_info=None):
             w,
             None,
             self.k_coef_lip,
-            self.niter_spectral,
-            self.niter_bjorck,
+            self.eps_spectral,
+            self.eps_bjorck,
+            self.beta_bjorck,
         )
         return wbar
 
     def get_config(self):
         return {
-            "niter_spectral": self.niter_spectral,
-            "niter_bjorck": self.niter_bjorck,
+            "eps_spectral": self.eps_spectral,
+            "eps_bjorck": self.eps_bjorck,
+            "beta_bjorck": self.beta_bjorck,
             "k_coef_lip": self.k_coef_lip,
             "base_initializer": initializers.serialize(self.base_initializer),
         }