[ENH] Datalad-free alternative for TemplateFlow #7
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This PR provides an alternative for systems without datalad to pull data from TemplateFlow. A new skeleton file containing metadata and empty-files replacing datalad symlinks is now found at ``templateflow/conf/templateflow-skel.zip``. If the repository has not been initiated and datalad is not installed, then this skeleton is unzipped. When the user requests a file via pybids, the file size is checked. If it is zero (i.e., not a symlink and no content), the file is pulled from an S3 bucket. The S3 bucket is easily updated with ``git annex export``. With every build the skeleton file is updated (based off the datalad repository pulled down in a previous step of the build) and added as an artifact to the CircleCI build. If the commit is a tag commit, then the skeleton file is updated before releasing in the ``deploy_pypi`` build. Closes #5.
oesteban
force-pushed
the
enh/datalad-alternative
branch
from
5bbcef7
to
282c652
Compare
effigies
approved these changes
Mar 4, 2019
|
I'll go ahead and check whether the deployment build also works as expected. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR provides an alternative for systems without datalad to pull data from TemplateFlow, using lazy loading.
A new skeleton file containing metadata and empty-files replacing datalad symlinks is now found at
templateflow/conf/templateflow-skel.zip. If the repository has not been initiated and datalad is not installed, then this skeleton is unzipped.When the user requests a file via pybids, the file size is checked. If it is zero (i.e., not a symlink and no content), the file is pulled from an S3 bucket (lazy loading).
The S3 bucket is easily updated with
git annex export.With every build the skeleton file is updated (based off the datalad repository pulled down in a previous step of the build) and added as an artifact to the CircleCI build.
If the commit is a tag commit, then the skeleton file is updated before releasing in the
deploy_pypibuild.