Pack Builder Go Scaffolding Support

[lkingland]

The new Go middleware, which supports instance-based method signatures, lifecycle events, default health endpoints etc (http://github.com/knative-extensions/func-go) has now been integrated into both the Host and S2I builders. The Pack builder, however, is still using the old middleware, which should now be bypassed and then decommissioned.

Currently, the "pack" builder is sent the source code of a Go function directly. However this source code does not include main and thus must be wrapped during buildpack execution. This process should be amended to instead first run the "scaffolding" step (which creates a main file which imports the user's function) in the .func transient directory. Then that directory (containing the main file along with the source code of the function itself) should be sent to the Pack builder for compilation. This is the process of "scaffolding" and will result in the new middleware being used instead of that added by the Buildpack. The legacy buildpack will then be able to be archived.

This update should be very similar to #2203 which implements this process for the Pack builder.

/kind enhancement

[lkingland]

Proposed alternative solution was to perform the scaffolding step within the Buildpack. The pros/cons suggested via slack are below. Captured here because Slack is closed, restricted, nigh unsearchable black hole of information:

Option 1

Inject the scaffolding code and set envvars before build in the func CLI. This is what S2I does for Go does right now.
Con: This will not work with PaC build {well unless we would commit and push the .func dir). Meaning we would need to rework our Tekton Tasks to also do the scaffolding part.
Con: A func project won't be build-able by pack/kpack CLI alone.
Pro: One advantage is that this would work without custom buildpack, we could just us what paketo already offers.

Option 2

Inject the scaffolding code and set envvars during the build inside of the buildpack code itself (using func as a library to do so). This is similar to current approach of pack Go build. The difference would be that func code would be used as a library doing the scaffolding instead of having some hard-coded main() in buildpack source code.
Pro: The PaC build will work just work as it does now without no additional work.
Pro: A func project would be buidable just by pack CLI or any compatible build (e.g. kpack).
Con: We cannot use standard paketo buildpack, however we already maintain our own buildpacks already. So no change there really.

[lkingland]

I would prefer we take option 1 if at all possible, but Option 2 is also good depending on how its implemented. There are in my mind a few more pros/cons to consider.

I’ll try to explain a little of why I lean towards Option 1 (or something similar) here, but lets plan on a longer discussion on our WG call next week to dig a little deeper.

Ideally we would not need a custom buildpack, because the code received by pack for building would already be “scaffolded” by func (have it’s main injected JIT). However, we would need to update the Tekton task to actually use func to perform the build. This is the way func was meant to be used: as a tool to allow systems to perform operations on Functions (such as building and deploying), and in particular CI/CD systems.

Option 2 would work, but I am concerned that we will continue to have more and more problems over time: the Tekton task is supposed to “build and deploy” a Function, but it is not using func logic to do so; it is duplicating the functionality. Can we fix this? If the Tekton task were actually using func directly to build and deploy Function source code, we would be simpler with minimal logic duplication. Now that we have the “Host” builder, there’s no need for a container runtime, so that may make this viable when it was not before.

It’s perhaps worth noting that there is actual logic applied by func core when building and deploying, which are necessary stages of the process converting source to a service. We can not simply deploy a function’s source code without the interpretation of that code through the functions core library into a service. It is an integral part of the process.

While it is possible to skip this step right now by pre-generating instructions for use in Tekton, we are replicating the logic (setting volumes, environment variables, etc. etc), and therefore the amount of things that we need to do in both places will continue to grow. We will inevitably diverge at times and cause bugs or inconsistent feature support.

The architecture of the system has included an expectation that any system/environment wishing to operate on a Function (such as deploying it) will need to use the functions client (either Go library or CLI).

If the implementation of Option 2 is to run a buildpack which delegates to func to create scaffolding before continuing on, then we are quite close to the original intent. We’ll just need to do the same thing for each language runtime. Updating the tekton task to use func directly may be a universal change, and would hit the proverbial nail on the head.

Looking forward to asking some more questions on the WG call this week.