SystemDS 0.1.0 (August 31, 2019)

Release Notes

SystemDS 0.1.0 is the initial public release of SystemDS after being forked from Apache SystemML in September 2018 and contains a major refactoring and several experimental features that aim at better support for the end-to-end data science lifecycle.

The major changes (compared to SystemML 1.2) and new features are

• New mechanism for DML-bodied (script-level) builtin functions, and selected new built-in functions for data augmentation, outlier detection, data preprocessing, feature engineering, ML algorithms, and model debugging.
• Various compiler and runtime improvements: new and improved IPA rewrites, new libsvm I/O format, reduced Spark context creation, updated native kernel libraries
• New lineage tracing and reuse (lineage tracing, loop lineage deduplication, full and partial reuse of intermediates, serialization and deserialization of lineage traces) [experimental]
• New tensor data model (basic tensors of different value types, data tensors with schema) [experimental]
• Backported SystemML features on cumulative aggregates, various sparsity estimators, and improved transform.
• Removed baggage: MapReduce compiler and runtime backend, pydml parser and language support, Java-UDF framework, script-level debugger

Acknowledgements

Thanks to Iulian Antonov, Matthias Boehm, Mark Dokter, Kevin Innerebner, Philipp Ortner, Arnab Phani, Benjamin Rath for their contributions to SystemDS 0.1.0 as well as the entire Apache SystemML team for the initial code base, documentation, and other resources.