-
Notifications
You must be signed in to change notification settings - Fork 992
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unify url handling with filesources #15497
Unify url handling with filesources #15497
Conversation
9ffb107
to
70ee2cf
Compare
70ee2cf
to
abae3fa
Compare
I was somewhat skeptical but this is very nice - wonderful job! The **kwargs getting passed everywhere is very typical Galaxy and feels at home in this codebase and is a thing I've done with my own plugin frameworks in the past. I think we should perhaps try to get away from that though by adding a bit more structure. It looks like there is a limited number of known options with fixed types for the plugins we've defined - could we capture all of those in a TypedDict and then use typing_extensions.Unpack to annotate all those kwargs.
It looks like TypedDict can specify they don't capture the totality of options (https://mypy.readthedocs.io/en/stable/more_types.html#totality), which would allow downstream plugins to do their own thing as long as they don't conflict with existing plugins. If you're uninterested in working through all the details - would you be willing to at least defined the dictionaries of possible options for me and annotate what interfaces accept which ones with a comment or something? I could probably work through the possibilities but you're much closer to the code now. Even if we don't figure out how to use the experimental mypy features - I think that would serve as good documentation for the module. |
assert_realizes_as(file_sources, test_url, "hello drs world", user_context=user_context) | ||
|
||
|
||
@responses.activate |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So cool!
This is a very good point and I fully agree with the runaway nature of a kwargs dict - it's hard to stop it from devolving into an incomprehensible bag of data or know who's depending or modifying which bits. I added it because I didn't feel I had many options - it was either a matter of polluting the interface with weird properties, introducing another object similar to a UserContext, or tacking it onto the UserContext itself. Ultimately, I just chose the easy way out with kwargs. I didn't know about TypedDicts so I'm happy to look into that and give it a go, and am also ears to other approaches to handling this. |
a3b9a18
to
1f63e03
Compare
@jmchilton I've made the changes that you've proposed. The TypedDict with Unpack works well, and flagged a number of relevant issues, even though it's still experimental and required a special switch to enable. I eventually switched the kwargs over to a FileSourceOptions object. I thought it made for a cleaner design - too many Unpack/TypedDicts are potentially confusing I think. However, I did use TypedDicts with Unpack for all the constructor arguments, and the whole thing seems to work quite nicely together. Thanks for the suggestion and let me know whether this looks better. |
64e1991
to
4bfa44f
Compare
4bfa44f
to
4ccd124
Compare
Hats off to you @nuwang - this is wonderful! |
which was failing with: ``` E galaxy.tool_util.provided_metadata INFO 2023-06-19 12:49:16,823 [pN:main.1,p:15735,tN:LocalRunner.work_thread-0] One or more tool outputs is marked as failed ({'type': 'dataset', 'ext': 'data', 'dataset_id': 1, 'stderr': 'Unable to fetch file:///home/runner/work/planemo/planemo/tests/data/hello.txt\nCould not find handler for URI [file:///home/runner/work/planemo/planemo/tests/data/hello.txt]', 'failed': True}). ``` After galaxyproject/galaxy#15497 and follow-up galaxyproject/galaxy#15794 , data upload using "file://" URLs is allowed only for Galaxy admins. This adds a basic ``file_sources_conf.yml`` configuration file that allows planemo to upload from the test data directory.
This PR unifies URL handling with filesources. Closes: #14658
Features
Backward compatibility
score_url_match
method to indicate its ability to handle a particular url,get_browsable
method to indicate that it supports theSupportsBrowsing
interface and theto_relative_path
method to convert a gxfile url to a relative path. BaseFileSource handles these cases so that most FIleSources do not need any changes.get_scheme
method has been removed from the interface definition as it is no longer essential, but remains in theBaseFileSource
.How to test the changes?
(Select all options that apply)
License