runtimes too big

[Vaishaal]

Recently our runtime builder started producing runtimes that wouldn't fit in a Lambdas /tmp

So I ran a diff of the shared libraries in a new default runtime and the old runtime we had on s3

And saw the following:

ls -lht --sort=size condaruntime_old/lib/ | head -n 15
total 224M
-rwxrwxr-x 1 ubuntu ubuntu 49M Jan 10 2017 libmkl_avx2.so
-rwxrwxr-x 1 ubuntu ubuntu 28M Jan 10 2017 libmkl_def.so
-rwxrwxr-x 1 ubuntu ubuntu 27M Jan 10 2017 libmkl_intel_thread.so
-rwxrwxr-x 1 ubuntu ubuntu 27M Jan 10 2017 libmkl_core.so
-rwxrwxr-x 1 ubuntu ubuntu 13M Jan 10 2017 libmkl_sequential.so
-rwxrwxr-x 1 ubuntu ubuntu 12M Jan 10 2017 libmkl_vml_avx2.so
-rwxrwxr-x 1 ubuntu ubuntu 9.3M Jan 10 2017 libmkl_intel_lp64.so
-rwxrwxr-x 1 ubuntu ubuntu 8.7M Jan 10 2017 libmkl_intel_ilp64.so
-rwxrwxr-x 1 ubuntu ubuntu 5.4M Jan 10 2017 libmkl_rt.so
-rwxrwxr-x 1 ubuntu ubuntu 5.3M Jan 10 2017 libmkl_vml_def.so
-rwxrwxr-x 1 ubuntu ubuntu 5.2M Jan 10 2017 libmkl_vml_cmpt.so
-rw-rw-r-- 1 ubuntu ubuntu 5.0M Jan 19 2017 libfreetype.a
-rw-r--r-- 1 ubuntu ubuntu 4.7M May 25 2016 libsqlite3.a
-rw-rw-r-- 1 ubuntu ubuntu 4.2M Jan 26 2017 libcrypto.a

ls -lht --sort=size condaruntime_new/lib/ | head -n 15
total 392M
-rwxrwxr-x 1 ubuntu ubuntu 49M Dec 6 21:17 libmkl_ao_worker.so
-rwxrwxr-x 1 ubuntu ubuntu 47M Dec 6 21:17 libmkl_avx2.so
-rwxrwxr-x 1 ubuntu ubuntu 31M Dec 6 21:17 libmkl_core.so
-rwxrwxr-x 1 ubuntu ubuntu 31M Dec 6 21:17 libmkl_intel_thread.so
-rwxrwxr-x 1 ubuntu ubuntu 28M Dec 6 21:17 libmkl_def.so
-rwxrwxr-x 1 ubuntu ubuntu 26M Dec 6 21:17 libmkl_pgi_thread.so
-rwxrwxr-x 1 ubuntu ubuntu 22M Dec 6 21:16 libpython3.6m.a
-rwxrwxr-x 1 ubuntu ubuntu 20M Dec 6 21:17 libmkl_gnu_thread.so
-rwxrwxr-x 1 ubuntu ubuntu 19M Dec 6 21:17 libmkl_tbb_thread.so
-rwxrwxr-x 1 ubuntu ubuntu 17M Dec 6 21:17 libmkl_sequential.so
-rwxrwxr-x 1 ubuntu ubuntu 11M Dec 6 21:17 libmkl_vml_avx2.so
-rwxrwxr-x 1 ubuntu ubuntu 8.6M Dec 6 21:17 libmkl_intel_lp64.so
-rwxrwxr-x 1 ubuntu ubuntu 8.6M Dec 6 21:17 libmkl_gf_lp64.so
-rwxrwxr-x 1 ubuntu ubuntu 7.8M Dec 6 21:17 libmkl_intel_ilp64.so

Basically the new mkl version adds a lot of unnecessary .so files (around 200mb worth), such as support for additional threading libraries, support for Intel® MIC Architecture none of which we need

So I think for now the correct thing to do is to freeze the

[shivaram]

http://hpcwiki.uark.edu/doku.php/mkl_library has a list of library names to what they mean

[Vaishaal]

yeah thats what I used to figure out what the extra .so files were

[rkruegs123]

You can install conda packages with nomkl to avoid this