Add manager for simple Kubernetes job creation #47

stv0g · 2022-12-02T11:12:31Z

In GitLab by @iripiri on Sep 23, 2021, 13:35

relates to #26

Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

…eter and config file Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

…added to the controller Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

rather set their state to stopped Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

…t and test stages Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de>

Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de>

stv0g · 2025-04-02T14:30:27Z

Hi @iripiri,

Could we somehow break this PR up? Its containing over 80 commits which is almost impossible to review..

Or, I see a lot of small bug fixing commits, could you squash those into the respective commits to which they logically belong? (Not everthing into a single commit, as this would be equally hard to review :/).

iripiri · 2025-04-03T07:05:47Z

Hi @stv0g,
I will do so! Though I'm not completely done here, I am testing & fixing right now, it's been a long time..
Could you review #61 first? That one is done

Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de> pipeline fixes Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de> updatet setuptools version to support license expression Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de> update license definition Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de> read version dynamically Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de> fixes Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de> move version to correct init file Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de>

Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de>

…t to find controller executable Signed-off-by: SystemsPurge <naktiyoussef@proton.me>

stv0g · 2025-05-21T08:16:58Z

Could you review #61 first? That one is done

#61 is merged.

I think we are almost ready to merge this one, after the conflicts have been resolved.

stv0g

First little round of review is done.

Ideally, we should split this PR into two:

Migration to pyproject.toml
New kubernetes-simple component

stv0g · 2025-05-21T08:17:55Z

etc/config_simplekub.yaml

@@ -0,0 +1,17 @@
+---
+broker:
+  url: amqp://admin:vieQuoo2sieDahHee8ohM5aThaibiPei@villas-broker:5672/


Is this a production credential? Looks like your RabbitMQ deployment in K8s?

stv0g · 2025-05-21T08:19:44Z

setup.py

-        'jsonschema>=4.1.0',
-        'psutil',
-        'pyusb'
-    ],
    data_files=[


Cant we add this to pyproject.toml as well?

Glob expressions are supported there as well:

https://setuptools.pypa.io/en/latest/userguide/datafiles.html#package-data

stv0g · 2025-05-21T08:22:13Z

villas/controller/components/simulators/kubernetes.py

+        # job isn't immediately deleted
+        # let the user see that something is happening


Suggested change

# job isn't immediately deleted

# let the user see that something is happening

# Job isn't immediately deleted.

# Let the user see that something is happening.

stv0g · 2025-05-21T08:23:16Z

villas/controller/components/simulators/kubernetes.py

@@ -173,6 +176,9 @@ def _delete_job(self):
        self.job = None
        self.properties['job_name'] = None
        self.properties['pod_names'] = []
+        # job isn't immediately deleted
+        # let the user see that something is happening
+        time.sleep(7)


Sorry, but this is a bad.. Why should we artifically block execution here?
This also stops the controller from processing other actions and stuff..

stv0g · 2025-05-21T08:24:47Z

villas/controller/components/managers/kubernetes_simple.py

+from villas.controller.components.managers.kubernetes import KubernetesManager
+from villas.controller.components.simulators.kubernetes import KubernetesJob
+
+parameters_simple = {


Please turn this into a function which constructs the dictionary and returns it.

The patching of the struct below is not nice, since it modifies the global variable.

stv0g · 2025-05-21T08:25:19Z

villas/controller/components/managers/kubernetes_simple.py

+        name = params.get('name')
+        privileged = params.get('privileged', False)
+        uuid = params.get('uuid')
+        self.logger.info('uuid:')


Please improve the logging message here. This does not need to be spread across two invocations.

stv0g · 2025-05-21T08:25:52Z

villas/controller/components/managers/kubernetes_simple.py

+            super().create(payload)
+            return


Suggested change

super().create(payload)

return

return super().create(payload)

stv0g · 2025-05-21T08:26:22Z

villas/controller/components/managers/kubernetes_simple.py

+        parameters = parameters_simple
+        parameters['name'] = sim_name
+        job = parameters['properties']['job']
+        job['metadata']['name'] = jobname
+        job['spec']['activeDeadlineSeconds'] = adls
+        job_container = job['spec']['template']['spec']['containers'][0]
+        job_container['image'] = image
+        job_container['securityContext']['privileged'] = privileged
+
+        parameters['job'] = job
+
+        if name:
+            parameters['name'] = name
+
+        if uuid:
+            parameters['uuid'] = uuid


As noted above, please construct the parameters within a dedicated function.

iripiri force-pushed the simple-kubernetes-manager branch from 7a3a130 to 0c166d5 Compare August 14, 2023 14:38

iripiri and others added 29 commits August 14, 2023 16:42

add manager for simple kubernetes job creation

de44dea

Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

suppress state changes from event repetitions

8959757

Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

try create of superclass if parameters are missing

d69a26f

Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

delete configmap when job is deleted

298df49

Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

fix api_url for villas-node manager

f13ea00

Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

simplify config handling

df28149

Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

move global status to Controller mixin class

e0d569c

Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

make work dir configurable

a0759cf

Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

pass payload instead of message to action handlers

a9bde8d

Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

fix mixed up state vs status properties in some components

1f30e93

Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

fix mixed up state vs status properties in some components

4e23053

Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

fix mixed up state vs status properties in some components

426cbf2

Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

added first version of an HTTP REST API

22ba436

Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

creating AMQP queue with durable, exclusive and auto_delete flags

4360d29

Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

add first draft of OpenAPI spec

124f55f

Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

fix simple manager

5800d0b

Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

use 'villas' namespace as default

1f695f5

Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

use villas-controller namespace as default

17080eb

Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

move schemas to YAML files and load them from there for all components

d705699

Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

config: show a useful error message when started without broker param…

208f8e8

…eter and config file Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

fix typo

71a4872

Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

config: fix bug when started without config file

075fbfc

Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

validate action parameters against schema

aa6432c

Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

fix linting errors

b79c286

Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

fix some errors in the API spec and schema

bac3246

Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

validate schemas and openapi doc against meta jsonschemas

38c698c

Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

raise a SimulationException if an IC with an existing UUID should be …

a27be13

…added to the controller Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

provide version number within status field of status update

f87608c

Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

relay: do not remove vanished sessions by setting state to gone

e0f7829

rather set their state to stopped Signed-off-by: Iris Koester <iris.m.koester@gmail.com>

stv0g changed the title ~~add manager for simple kubernetes job creation~~ Add manager for simple kubernetes job creation Mar 25, 2025

iripiri added 8 commits March 26, 2025 08:23

changed python docker image due to pip3 command not found in precommi…

67a8761

…t and test stages Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de>

changed back image version and changed CI tag

7f5242d

Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de>

update schema checking

e06754c

Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de>

commented schema checking to run CI, needs to get revised

d0c70d1

Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de>

debug test fail

2ccadc5

Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de>

add current workind directory to PATH

e4dd2b3

Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de>

added init file to villas folder

555fe20

Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de>

remove changes made during pipeline debug/fix

5ac9fa1

Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de>

stv0g changed the title ~~Add manager for simple kubernetes job creation~~ Add manager for simple Kubernetes job creation Mar 31, 2025

iripiri added 9 commits March 31, 2025 10:34

get version (fix ModuleNotFound error), remove tags

52e7338

Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de>

add long description which got lost in last commit

fd0d5c8

Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de>

updated json schema checking

5fdb890

Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de>

added blank lines (flake8)

44e8da4

Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de>

add init.py for modules to be found in tests

05fdaa4

Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de>

merged with pipeline fixes

dd13225

Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de>

watch the resources inside the controller namespace

64f2143

Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de>

start watcher thread AFTER reading environment variables

79a56d6

Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de>

corrected setting the namespace

3df9f3d

Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de>

iripiri and others added 5 commits April 22, 2025 16:04

remove requirements files

d7dd022

Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de>

merged build updates

487a387

Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de>

fix errors, update config

0336da3

Signed-off-by: iripiri <ikoester@eonerc.rwth-aachen.de>

fix: temporary fix for entrypoint of Dockerfile, causing container no…

82bc2ad

…t to find controller executable Signed-off-by: SystemsPurge <naktiyoussef@proton.me>

stv0g commented May 21, 2025

View reviewed changes

stv0g mentioned this pull request Jun 12, 2025

Attempt at making dpsim work consistently #63

Draft

iripiri mentioned this pull request Jul 1, 2025

Cleaned simple kubernetes manager #66

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add manager for simple Kubernetes job creation #47

Add manager for simple Kubernetes job creation #47

Uh oh!

stv0g commented Dec 2, 2022

Uh oh!

stv0g commented Apr 2, 2025

Uh oh!

iripiri commented Apr 3, 2025

Uh oh!

stv0g commented May 21, 2025

Uh oh!

stv0g left a comment

Uh oh!

stv0g May 21, 2025

Uh oh!

stv0g May 21, 2025

Uh oh!

stv0g May 21, 2025

Uh oh!

stv0g May 21, 2025

Uh oh!

stv0g May 21, 2025

Uh oh!

stv0g May 21, 2025

Uh oh!

stv0g May 21, 2025

Uh oh!

stv0g May 21, 2025

Uh oh!

Uh oh!

		# job isn't immediately deleted
		# let the user see that something is happening

Add manager for simple Kubernetes job creation #47

Are you sure you want to change the base?

Add manager for simple Kubernetes job creation #47

Uh oh!

Conversation

stv0g commented Dec 2, 2022

Uh oh!

stv0g commented Apr 2, 2025

Uh oh!

iripiri commented Apr 3, 2025

Uh oh!

stv0g commented May 21, 2025

Uh oh!

stv0g left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!