TensorFlow implementation of the ml-params API and CLI.
The purpose of ml-params is to expose the hooks and levers of ML experiments for external usage, e.g., in GUIs, CLIs, REST & RPC APIs, and parameter and hyperparameter optimisers.
pip install -r requirements.txt
pip install .
After installing as above, follow usage from ml-params
| Other vendors | |
|---|---|
| tensorflow | pytorch |
| keras | skorch |
| flax | sklearn |
| trax | xgboost |
| jax | cntk |
To make the development of ml-params-tensorflow type safer and maintain consistency with the other ml-params implementing projects, the cdd was created.
When TensorFlow itself changes—i.e., a new major version of TensorFlow is releases—then run the sync_properties, as shown in the module-level docstring here ml_params_tensorflow/ml_params/type_generators.py;
To synchronise all the various other APIs, edit one and it'll translate to the others, but make sure you select which one is the gold-standard.
$ pip install python-cdd autoflake black isort
fd (or you can just use find, rewrite to find -type f -name '*.py' -exec)
NOTE: The below steps are available in a script within sync_scripts.
As an example, using the class TensorFlowTrainer methods as truth, this will update the CLI parsers and config classes:
$ python -m cdd sync --class 'ml_params_tensorflow/ml_params/config.py' \
--class-name 'TrainConfig' \
--function 'ml_params_tensorflow/ml_params/trainer.py' \
--function-name 'TensorFlowTrainer.train' \
--argparse-function 'ml_params_tensorflow/ml_params/cli.py' \
--argparse-function-name 'train_parser' \
--truth 'function'
$ python -m cdd sync --class 'ml_params_tensorflow/ml_params/config.py' \
--class-name 'LoadDataConfig' \
--function 'ml_params_tensorflow/ml_params/trainer.py' \
--function-name 'TensorFlowTrainer.load_data' \
--argparse-function 'ml_params_tensorflow/ml_params/cli.py' \
--argparse-function-name 'load_data_parser' \
--truth 'function'
$ python -m cdd sync --class 'ml_params_tensorflow/ml_params/config.py' \
--class-name 'LoadModelConfig' \
--function 'ml_params_tensorflow/ml_params/trainer.py' \
--function-name 'TensorFlowTrainer.load_model' \
--argparse-function 'ml_params_tensorflow/ml_params/cli.py' \
--argparse-function-name 'load_model_parser' \
--truth 'function'
Another example, that you'd run before ^, to generate custom config CLI parsers for members of tf.keras.losses:
$ python -m cdd gen --name-tpl '{name}Config' \
--input-mapping 'ml_params_tensorflow.ml_params.type_generators.exposed_losses' \
--prepend '""" Generated Loss config classes """\nimport tensorflow as tf\n' \
--imports-from-file 'tf.keras.losses.Loss' \
--type 'argparse' \
--output-filename 'ml_params_tensorflow/ml_params/losses.py'
There's a bit of boilerplate here, so let's automate it:
$ for name in 'callbacks' 'losses' 'metrics' 'optimizers'; do
rm 'ml_params_tensorflow/ml_params/'"$name"'.py';
python -m ml_params_tensorflow.ml_params.cdd_cli_gen "$name" 2>/dev/null | xargs python -m cdd gen;
done
Cleanup the code everywhere, removing unused imports and autolinting/autoformatting:
$ fd -HIepy -x sh -c 'autoflake --remove-all-unused-imports -i "$0" && isort --atomic "$0" && python -m black "$0"' {} \;
Licensed under either of
- Apache License, Version 2.0 (LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0)
- MIT license (LICENSE-MIT or https://opensource.org/licenses/MIT)
at your option.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.