Download and organise DLRM data automatically #93
Conversation
| os.system(f"unzip {os.path.join(dlrm_data_path, 'criteo', 'day23', 'fp32', 'day_23_sparse_multi_hot.npz')} -d {os.path.join(dlrm_data_path, 'criteo', 'day23', 'fp32', 'day_23_sparse_multi_hot_unpacked')}") | ||
| if os.path.exists(os.path.join(dlrm_data_path, "criteo", "day23", "fp32", "day_23_sparse_multi_hot.npz")): | ||
| file_path = os.path.join(dlrm_data_path, "criteo", "day23", "fp32", "day_23_sparse_multi_hot.npz") | ||
| run_cmd = ("echo {} {} | md5sum -c").format('c46b7e31ec6f2f8768fa60bdfc0f6e40', file_path) |
There was a problem hiding this comment.
we want to do md5sum check even for download right?
There was a problem hiding this comment.
Yes... I tried to handle it in commit 19862dd
| prehook_deps: | ||
| - tags: get,ml-model,dlrm,_pytorch | ||
| update_tags_from_env_with_prefix: | ||
| download_path: CM_DLRM_DATA_PATH |
There was a problem hiding this comment.
_ missing? Also this uses a list right?
| - "on" | ||
| - tags: get,dataset,preprocessed,criteo,_mlc | ||
| update_tags_from_env_with_prefix: | ||
| output_dir: CM_DLRM_DATA_PATH |
| print(f'day_23_labels.npy is missing inside {dlrm_data_path}/criteo/day23/fp32') | ||
| env['CM_DLRM_DATASET_DOWNLOAD'] = True | ||
| # TBD: raw data is not available with mlcommons hosted dataset | ||
| #if not os.path.exists(os.path.join(dlrm_data_path, "criteo", "day23", "raw_data")): |
There was a problem hiding this comment.
We should now turn this check on and ask user to point to this file via an input say criteo_day23_raw_data_path. This is the link to download the rawdata but only manual download is possible.
There was a problem hiding this comment.
would criteo_raw_data_path and CM_CRITEO_RAW_DATA_PATH be a good naming option for input mapping and env variable?
There was a problem hiding this comment.
We need day23 too as we only need to download day23 file - not the entire raw data.
There was a problem hiding this comment.
1dec20a incorporated in this commit
No description provided.