Convert pipelines to obsah

[ekohl]

Currently this doesn't work because the forklift role can't properly find the vagrant/boxes.d directory.

[ekohl]

I've verified this works locally. Obviously we'll need to modify our CI systems, at least to understand the new paths.

Before I proceed with converting all pipelines, I want to know people actually like this.

One question I have is whether it should be bin/forklift or bin/pipeline.

[ehelms]

I would like this to handle more than just pipelines. So I'd be curious what your design for that would look like? Given the design of Obsah allows it run a playbook via command line it would be nice to be have it as the interface to run any of the playbooks herein.

If wanting to start with just pipelines, I'd lean towards some sub-commands to help make it clearer:

forklift pipeline install 
forklift pipeline upgrade

[ekohl]

I agree with that, which I was alluding to with my question is whether it should be bin/forklift or bin/pipeline. I think we can have a subparser for the pipeline command, but I think Obsah can do that. On the other hand, naming it bin/pipeline is easier to start with.

I'll let @evgeni weigh in as well before I make any changes.

[zjhuntin]

I tried to leave some comments about things that I thought might be a little unclear, but overall I think it looks really cool.

I'm unsure where to weigh in on the conversation above, but I agree with Eric that maybe making sub-commands would help. Though getting it out the door sounds good too. Either way this is looking great!

[zjhuntin]

A little more explanation of why the second run is used would make it more understandable for someone starting to use it.

[zjhuntin]

Are these the only operating systems that can be used? Can all the pipelines be run on any of the oses? Is it something more that the user would know when creating the pipeline?

[ekohl]

It maps to the pipeline_os variable which maps back to the base boxes. Of course it doesn't magically work so we will need packages for the OS. So you can use centos6, it will fail during provisioning since there's no packages. We'll also add a centos8 box at some point.

[zjhuntin]

Same here as above, but it does look like you want to eventually have something here for that?

[ekohl]

Currently we only have a set of variables for those 3. That means we can enforce just the 3 options.

[zjhuntin]

Why is dirname listed twice here?

[ekohl]

playbook_dir is pipelines/upgrade so the forklift root is pipelines/upgrade/../..

[evgeni]

obsah, today, can't generate sub-parsers that you'd need to implement sub-commands. I think we can implement these quite easily by adding another directory level in the playbooks folder. so that <playbooks>/name/name.yaml is exposed as binary name and <playbooks>/sub/name/name.yml as binary sub name.

On the other hand, that will require us to merge the pipelines and playbooks folders, or come up with a different solution (multiple playbook folders, each being a sub-parser?).

I think the easiest solution would be to call the binary forklift and the actions pipeline-install and pipeline-upgrade, then we just need to forget the dash in the future.

[ekohl]

I think the easiest solution would be to call the binary forklift and the actions pipeline-install and pipeline-upgrade, then we just need to forget the dash in the future.

Is this easier than naming it pipeline and later add the umbrella forklift to it? This can be done by extending obsah to expose itself as a subparser. This is documented in https://docs.python.org/3/library/argparse.html#sub-commands. You'd end with something like:

parser = argparse.ArgumentParser(prog='forklift')
subparsers = parser.add_subparsers(help='sub-command help')

obsah.add_subparser(subparsers, ApplicationConfig)

[evgeni]

I think the easiest solution would be to call the binary forklift and the actions pipeline-install and pipeline-upgrade, then we just need to forget the dash in the future.

Is this easier than naming it pipeline and later add the umbrella forklift to it? This can be done by extending obsah to expose itself as a subparser. This is documented in https://docs.python.org/3/library/argparse.html#sub-commands. You'd end with something like:

parser = argparse.ArgumentParser(prog='forklift')
subparsers = parser.add_subparsers(help='sub-command help')

obsah.add_subparser(subparsers, ApplicationConfig)

So at the end we have a forklift parser which has two obsah generated/created subparsers for pipelines and playbooks? Yeah, that'd work.

I think I'd still prefer to have the forklift binary now, even if it does only have one subcommand (pipeline) as this will allow to establish knowledge and won't require re-learning (or re-writing CI) when we add it later.

Oh, and this needs rebase please :)

[ekohl]

Currently obsah is rather tied to its own parser since the main function also does the processing. Right now I don't have the time to dive into how that would need to be written so I probably won't get to this until after FOSDEM and friends.

[ekohl]

Rebased to at least resolve the merge conflicts. Depends on theforeman/obsah#8.

[ekohl]

I did notice that this is quite slow because ansible calls the inventory, which calls vagrant status. I think that's a problem we can't really avoid.