-
Notifications
You must be signed in to change notification settings - Fork 105
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Pydantic V1 IO models for use with Hera Runner #920
Conversation
b6af175
to
43a25f8
Compare
43a25f8
to
6c5b4f1
Compare
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #920 +/- ##
=======================================
+ Coverage 80.7% 81.2% +0.4%
=======================================
Files 50 51 +1
Lines 3908 4045 +137
Branches 793 843 +50
=======================================
+ Hits 3157 3287 +130
- Misses 564 565 +1
- Partials 187 193 +6 ☔ View full report in Codecov by Sentry. |
fe9ff79
to
3a0a39f
Compare
@elliotgunton can you provide a brief examples/details in the PR description of the functionality that is currently enabled? Specifically I am curious how we handle things when a user -
Having that in the PR description would help future readers understand what is happening. |
5bc252c
to
f37fbea
Compare
22fb851
to
d13cb41
Compare
b553cbb
to
66a4443
Compare
66a4443
to
a0dd261
Compare
if len(inspect.signature(source).parameters) != 1: | ||
raise SyntaxError("Only one function parameter can be specified when using a RunnerInput.") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Only one function param that is a RunnerInput
subclass is allowed
if isinstance(annotation, type) and issubclass(annotation, RunnerOutput): | ||
raise ValueError("RunnerOutput cannot be part of a tuple output") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Only one RunnerOutput
is allowed as the function return type. Note that we also haven't dealt with nested Annotated
types above.
@samj1912 ready for re-review! Self-reviewed to double check this PR stands on its own to implement for Pydantic V1 and signposted the single input/output we discussed 🚀 |
Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
* Add annotations test only in this commit Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
* Tidy up kwarg mapping code - may need another refactor * TODO artifact outputs Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
* Add environment variable to script templates using the feature * Use actual error message for ValueError when flag not enabled Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
* Also allow setting defaults of a RunnerInput param via its func param default * Remove runner tests that use multiple inputs * Add script annotations test for using a RunnerInput in a List Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
a0dd261
to
221ecae
Compare
Part of #858, follow up to #920 * Enables users to use Pydantic V2 objects in their scripts while maintaining V1 usage internally for Hera * RunnerInput/Output classes are created in hera.workflows.io depending on the value of _PYDANTIC_VERSION - users will automatically get classes using their (possibly pinned) version of Pydantic * I have not yet managed to get an explicit test that uses the automatic import of V1 classes - we may just have to rely on the Pydantic V1 CI check. --------- Signed-off-by: Elliot Gunton <egunton@bloomberg.net>
Pull Request Checklist
Documentation/examples addedSee Add pydantic IO docs #939Description of PR
Currently hera i/o with annotated params can become extremely verbose. The output syntax is especially error-prone.
This PR introduces custom Input/Output BaseModels for users to subclass, which allow a cleaner arrangement of inputs and outputs for functions. These are available under the
script_pydantic_io
experimental feature flag.With these Pydantic input/output models, the following should be noted:
exit_code
andresult
are reserved attributes for theRunnerOutput
. A user trying to use their own parameters with these names would have to be specified with an annotated parameter e.g.my_exit_code: Annotated[int, Parameter(name="exit_code")]
(TBC with a test)RunnerOutput
to avoid multipleexit_code
s being specified. @samj1912 / @flaviuvadan this is up for debate but I think would encourage better practices to discourage tuples and have a single script template outputting a singleRunnerOutput
subclass, and it keeps the logic clearer from the Hera side. Users can still use inline output parameters alongside theRunnerOutput
return annotationRunnerInput
in the function params is not legalRunnerInput
's__fields__
as defined by pydantic are used to "explode" the input class into constituent parameters for the Argo spec. i.e. using the following class as an input param to a script function:will create the script template
my_func
in yaml with Parametersmy_input_str
andmy_input_int
, NOTmy_input
, see the example