Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remap image_pull_policy field on Container from ImagePullPolicy enum to str #877

Merged
merged 6 commits into from
Nov 23, 2023
Merged

Remap image_pull_policy field on Container from ImagePullPolicy enum to str #877

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
c4b495c
remove image pull policy from container
flaviuvadan Nov 23, 2023
b6e651c
add back image pull policy via spec
flaviuvadan Nov 23, 2023
3c1955d
add back image pull policy
flaviuvadan Nov 23, 2023
2f13b3d
codegen
flaviuvadan Nov 23, 2023
121528c
Merge branch 'main' into fv/784
flaviuvadan Nov 23, 2023
8d18f61
Merge branch 'main' into fv/784
elliotgunton Nov 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 34 additions & 5 deletions scripts/spec.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
import json
import logging
import sys
from typing import Dict, List, Set, Tuple
from typing import Dict, List, Optional, Set, Tuple, Union

import requests

Expand Down Expand Up @@ -56,17 +56,26 @@
# these are specifications of objects with fields that are marked to have a union type of IntOrString. However, K8s
# only accepts one or the other, unfortunately. Here, we remap those fields from their respective `$ref`s, which
# contain a specific type, to another type. The mapping is from the `$ref` to a tuple of the existing type and the
# new type. The dictionary model is:
# new type. While the first piece of the mapping tuple must be defined the second one is optional - if the second
# part of the tuple is None then the field will be removed. The dictionary model is:
# { object name: { field name: ( ( existing field, existing value ) , ( new field, new value ) ) } }
INT_OR_STRING_FIELD_REMAPPING: Dict[str, Dict[str, Tuple[Tuple[str, str], Tuple[str, str]]]] = {
FIELD_REMAPPINGS: Dict[
str, Dict[str, Tuple[Tuple[str, Union[str, List[str]]], Optional[Tuple[str, Union[str, List[str]]]]]]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure the type is helping at this point 😅 The comment is helpful enough, and longer term maybe we should create a helper class within this script?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea... I was very reluctant to add this type, especially as I realized it's growing. Yea, we'll have to refactor this a bit in the future.

] = {
"io.k8s.api.core.v1.HTTPGetAction": {
"port": (
("$ref", "#/definitions/io.k8s.apimachinery.pkg.util.intstr.IntOrString"),
("type", "integer"),
),
},
"io.k8s.api.core.v1.Container": {
"imagePullPolicy": (
("enum", None),
None,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's this None doing here? 🤔

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First part of the tuple says "original field, original field value" second one says "new field, new field value". New field value of None means delete the original field. So, this is removing the enum from Container, allowing codegen to put a string in place of the Container image pull policy enum, and stop generating the ImagePullPolicy enum altogether.

Later in the script you see we have to add a specification for IPP manually for otherwise it's not fully backwards compatible, even if it's not used by Container!

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah of course, I was misreading the outer tuple. Makes sense (almost!), though I would suggest a refactor to helper classes if you have time/want to in a separate PR

)
},
}
for obj_name, field in INT_OR_STRING_FIELD_REMAPPING.items():
for obj_name, field in FIELD_REMAPPINGS.items():
try:
curr_field = spec["definitions"][obj_name]
except KeyError as e:
Expand Down Expand Up @@ -94,7 +103,12 @@

# get the tuple of the existing field and value, and the new field and value
existing_field, existing_value = field[property_to_change][0]
new_field, new_value = field[property_to_change][1]
if field[property_to_change][1] is None:
# if the second one is absent it means we want to delete the existing field
del curr_property[existing_field]
continue
else:
new_field, new_value = field[property_to_change][1]

# check that the existing field and value are the same as the current field and value
assert curr_property[existing_field] == existing_value, (
Expand All @@ -108,6 +122,21 @@
if existing_field != new_field:
del curr_property[existing_field]

# there are also some specifications that have to be introduced manually for backwards compatibility purposes. This
# block allows us to define those specifications and add them to the spec.
MANUAL_SPECIFICATIONS: List[Tuple[str, Dict]] = [
(
"io.k8s.api.core.v1.ImagePullPolicy",
{
"description": "An enum that contains available image pull policy options.",
"type": "string",
"enum": ["Always", "Never", "IfNotPresent"],
},
),
]
for obj_name, obj_spec in MANUAL_SPECIFICATIONS:
spec["definitions"][obj_name] = obj_spec

# finally, we write the spec to the output file that is passed to use assuming the client wants to perform
# something with this file
with open(output_file, "w+") as f:
Expand Down
14 changes: 7 additions & 7 deletions src/hera/events/models/io/k8s/api/core/v1.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,12 +141,6 @@ class ConfigMapKeySelector(BaseModel):
)


class ImagePullPolicy(Enum):
always = "Always"
if_not_present = "IfNotPresent"
never = "Never"


class TerminationMessagePolicy(Enum):
fallback_to_logs_on_error = "FallbackToLogsOnError"
file = "File"
Expand Down Expand Up @@ -1240,6 +1234,12 @@ class WindowsSecurityContextOptions(BaseModel):
)


class ImagePullPolicy(Enum):
always = "Always"
never = "Never"
if_not_present = "IfNotPresent"


class CSIVolumeSource(BaseModel):
driver: str = Field(
...,
Expand Down Expand Up @@ -2828,7 +2828,7 @@ class Container(BaseModel):
" StatefulSets."
),
)
image_pull_policy: Optional[ImagePullPolicy] = Field(
image_pull_policy: Optional[str] = Field(
default=None,
alias="imagePullPolicy",
description=(
Expand Down
12 changes: 6 additions & 6 deletions src/hera/events/models/io/k8s/api/core/v1.pyi
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,11 +39,6 @@ class ConfigMapKeySelector(BaseModel):
name: Optional[str]
optional: Optional[bool]

class ImagePullPolicy(Enum):
always: str
if_not_present: str
never: str

class TerminationMessagePolicy(Enum):
fallback_to_logs_on_error: str
file: str
Expand Down Expand Up @@ -313,6 +308,11 @@ class WindowsSecurityContextOptions(BaseModel):
host_process: Optional[bool]
run_as_user_name: Optional[str]

class ImagePullPolicy(Enum):
always: str
never: str
if_not_present: str

class CSIVolumeSource(BaseModel):
driver: str
fs_type: Optional[str]
Expand Down Expand Up @@ -582,7 +582,7 @@ class Container(BaseModel):
env: Optional[List[EnvVar]]
env_from: Optional[List[EnvFromSource]]
image: str
image_pull_policy: Optional[ImagePullPolicy]
image_pull_policy: Optional[str]
lifecycle: Optional[Lifecycle]
liveness_probe: Optional[Probe]
name: Optional[str]
Expand Down
14 changes: 7 additions & 7 deletions src/hera/workflows/models/io/k8s/api/core/v1.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,12 +141,6 @@ class ConfigMapKeySelector(BaseModel):
)


class ImagePullPolicy(Enum):
always = "Always"
if_not_present = "IfNotPresent"
never = "Never"


class TerminationMessagePolicy(Enum):
fallback_to_logs_on_error = "FallbackToLogsOnError"
file = "File"
Expand Down Expand Up @@ -1240,6 +1234,12 @@ class WindowsSecurityContextOptions(BaseModel):
)


class ImagePullPolicy(Enum):
always = "Always"
never = "Never"
if_not_present = "IfNotPresent"


class CSIVolumeSource(BaseModel):
driver: str = Field(
...,
Expand Down Expand Up @@ -2828,7 +2828,7 @@ class Container(BaseModel):
" StatefulSets."
),
)
image_pull_policy: Optional[ImagePullPolicy] = Field(
image_pull_policy: Optional[str] = Field(
default=None,
alias="imagePullPolicy",
description=(
Expand Down
12 changes: 6 additions & 6 deletions src/hera/workflows/models/io/k8s/api/core/v1.pyi
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,11 +39,6 @@ class ConfigMapKeySelector(BaseModel):
name: Optional[str]
optional: Optional[bool]

class ImagePullPolicy(Enum):
always: str
if_not_present: str
never: str

class TerminationMessagePolicy(Enum):
fallback_to_logs_on_error: str
file: str
Expand Down Expand Up @@ -313,6 +308,11 @@ class WindowsSecurityContextOptions(BaseModel):
host_process: Optional[bool]
run_as_user_name: Optional[str]

class ImagePullPolicy(Enum):
always: str
never: str
if_not_present: str

class CSIVolumeSource(BaseModel):
driver: str
fs_type: Optional[str]
Expand Down Expand Up @@ -582,7 +582,7 @@ class Container(BaseModel):
env: Optional[List[EnvVar]]
env_from: Optional[List[EnvFromSource]]
image: str
image_pull_policy: Optional[ImagePullPolicy]
image_pull_policy: Optional[str]
lifecycle: Optional[Lifecycle]
liveness_probe: Optional[Probe]
name: Optional[str]
Expand Down