Use newdeploy executor instead of poolmgr to deploy workflow engine

[erwinvaneyk]

As an intermediate step towards the proxy executor, one step that we already can take is to move the workflow engine to the newdeploy executor. The newdeploy executor aligns much better with the behavior of the workflow engine within the Fission function context.

[ghost]

This can also helpful to add horizontal scalability #184 per flow level without having to implement special autoscaler.

In ideal scenario, whenever new flow deployed, it should have it's own fission function (with it's own k8s deployment, HPA, pods). This way each flow can independently run/scale/fail.

[erwinvaneyk]

Hi @thenamly thanks for your input on this issue 😁

This can also helpful to add horizontal scalability #184 per flow level without having to implement special autoscaler.

This is indeed the main motivation behind this change for me as well.

In ideal scenario, whenever new flow deployed, it should have it's own fission function (with it's own k8s deployment, HPA, pods). This way each flow can independently run/scale/fail.

For Fission Workflows V2 we are moving towards this approach: each flow is executed as a fission function. However I have not yet found a good answer to whether we should represent each flow as an independent function, or have all flows executed under a single "flow" function. Let's discuss this design at some point :)

[erwinvaneyk]

Update: one thing that might be faster to implement is to separate the Fission-Workflows proxy from the workflows engine.