Automated firmware image releases

[linuxianer99]

This discussion is about an automated release and upload of firmware images.

There are some challenges:

• Cross-repo-build: the meta-openvario repo can not be build on its own. A release has to be done in the ov-build repo.
• Release in Github: Can be done in the meta-openvario repo, but the storage of the files makes no sense here.

Nice to have:

• List of contained software components (sensord, variod, XCSoar, etc ....) automatically created.

This is just a starting point of the discussion :-)

[MaxKellermann]

I don't understand what ov-build is and why there is a Docker container (ovbuild-container). Yocto builds are already self-contained; they need only a very minimal set of host tools. Adding Docker only adds complexity and overhead, which is error prone and slows down the build.
I also don't understand why you're using repo. Whenever I did Yocto, I had my own git repository with bitbake, oe-core and meta-oe as git submodules. This way, all builds were reproducible, because the definition of the state of one build is completely in my one repository, and it refers to specific commit ids in those submodules. And I can easily create git tags in that repository, and go back to any tag, and get the same build again. Only one git repository, not a complex and hard-to-use combination of several git repositories I have to manage and update for every release.
All of this seems you make your lives harder than necessary, and I don't understand why.

[linuxianer99]

Indeed, i also thought this way.
Think this is also a leftover of the start, where we had angstrom as a basis.

As I build the kas config file (openvario/ov-build), it had the same idea ... Think i will try this way ...

For the docker build container ... We had lots of issues with users, building an image. Therefore we decided to use the CROPS container of Yocto to have a uniform build environment. This approach has proven good, as we have no question any longer ;-)

[linuxianer99]

@MaxKellermann : Do you have an example for a "submodule" based approach ?

I see some problems:

• Generation of local.conf
• Generation of bblayers.conf
• We are based on poky .. so if clone poky layer, it has a gitignore entry for the underlying meta-openembedded layer

[MaxKellermann]

Do you have an example for a "submodule" based approach ?

Not public. My dayjob repository looks like this:

• conf/bblayers.conf
• meta: my own layer (containing conf/layer.conf, conf/distro/..., recipes/...)
• oe/bitbake: git submodule for BitBake
• oe/oe-core: git submodule for OpenEmbedded-Core
• oe/meta-oe: git submodule for meta-oe

For meta-openvario, this could be:

• conf/bblayers.conf
• meta-ov (everything at the root layer moved into this new directory)
• bitbake
• meta-poky
• meta-oe
• meta-sunxi
• ...

[linuxianer99]

Looks good!

So at the moment the poky repository contains the bitbake and all the meta-poky stuff .. Would you suggest to split up the poky repository and add the content separately as submodules ?? Or just keep poky with all the contents it brings ??

[MaxKellermann]

I never understood the point of Poky. It contains copies of stuff from other repositories (BitBake and oe-core). I'd rather integrate those original repositories as individual submodules.

[linuxianer99]

ok!! Thanks for the description!
I will try to do my best to setup this ...

btw. i also never understand this Poky thing, but it is use in every tutorial and startup guide ...

If you don't use Poky, how do you configure your build environment ?? I think the oe-init-build-env is a part of Poky, right ?

Ahh ... found it ... it's in openembedded-core !!

[August2111]

I have always worked with a small update script, not with repo - that was too 'unwieldy' for me.
But working with the suggested submodules seems to me to be much clearer and better!
Maybe this unwieldy juggernaut deters some developers from working here ;-(

[linuxianer99]

@MaxKellermann I tried to do an example ... may you have a look please:

linuxianer99/openvario-distro

[MaxKellermann]

See above, I just posted an example.

[linuxianer99]

@MaxKellermann : May i ask you again, to have a look please ??
Think you have a lot more experience in this topic ...

https://github.com/linuxianer99/meta-openvario/tree/hardknott

Thanks a lot :-)

[MaxKellermann]

Yes, that looks like what I imagined.

[linuxianer99]

@kedder What do you think ??
I would suggest to switch the structure, as it has big benefit on automated build and PR checks ...

[kedder]

I'm totally fine with that structure. I'm not using repo myself and doing git pulls manually anyway.

[linuxianer99]

Great news!!!

Due to the new repository structure, the github action is also quite easy!
I builds like a charm :-)

Thanks to @MaxKellermann for the good ideas and the support doing this !!

So, I suggest to switch to the new repo structure as soon as possible...

@kedder , @August2111 , @Blaubart
What is your opinion ??

[August2111]

I like the new approach very much, that makes life a lot easier :)
I was also thinking about a similar approach, but due to my significantly less experience in this sector, it would never have gotten that far and that consistent!
BTW @linuxianer99 : Is it possible to tag the current repository state?
e.g. the merge with PR #188 (commit eba5eab) with tag v3.3.1 and the merge with PR #191 (commit 73fbee0) with v3.3.2?
I could already prepare accordingly because of the new image naming! Of course, you can also use 'v3.3-rel2' or 'v3.3-rev2' - but in my opinion this will only bloat it unnecessarily...

[linuxianer99]

Of course we can tag, but this has no impact on the naming of the firmware image, which is built. This naming has to be also done in the recipes of yocto.
I suggest, doing one step after the other ... Move to new repo structure, change naming, ...

[linuxianer99]

Btw. this has also big impact on USB recovery stick as well ... So before we change naming, let's check the whole impact of this task ;-)

[August2111]

A tag does not affect the current build process - everything stays the same. But you can already work with it in your fork - and you could also slowly prepare the community for the change from the old to the new version (even if the naming convention is only introduced after the change and has been well tested - I completely agree with you )

[Blaubart]

Thank you for your work!! That makes sense and makes compiling a lot easier!!

[August2111]

A few minor notes:

• I simply called the repository 'OpenVario' - in my opinion, this summarizes everything that is needed to build an image
• This means that the openvario layer can be called the same as before without any confusion: meta-openvario - it also corresponds to the previous layer!
• The 'workdir' in the docker call can be called something else, it doesn't matter, it just has to be the same in both fields of the call line. That's why I prefer the similar path here as with XCSoar: /opt/openvario
• Cloning the repository makes it easier to download the submodules at the same time via --recursive, so you save the two lines at the end:

    git clone --recursive <something/OpenVario.git>
    cd OpenVario

if you have changed the repository name

And one more question: has this ever worked before and was an image built? I still had a few stumbling blocks!

[August2111]

An other point: It is very nice the starting investigations for a much better, easier to handle and faster system - but now the 'official' standard branch 'hardknott' is broken and don't work as expected - and described in the OpenVario wiki!
Please let us switch the hardknott branch back to a functional PR - of course with ov shell and hopefully with XCSoar 7.22 - and do our investigations on another branch, 'hardknott-next', 'master' or 'main' - or what ever!
The current state of our development makes a lot of trouble outside, see issue #202

[MaxKellermann]

So I'm definitely in favor of rethinking our branching

Why? We can accomodate all groups from a single branch, with two different image types: one tiny end-user image with read-only rootfs and no package management, and a larger image with package management and bells and whistles. That's easy to do with Yocto/BitBake.

Talking about branching: don't see why OpenVario has branches called "warrior" and "hardknott". Development should happen in "master". From there, you can branch off "stable" branches. But don't switch the development branch all the time, this is a mess. And it's a mess that we lost git history several times. Our current git history goes back to 2019 only, and other branches go back to 2014. Had we only used one branch, everything would be in there, without those ugly "initial commit" starts which loses all history.

[August2111]

This would be really good - but unfortunately this isn't the current state! Please never starts with a new cleanup branch without any history!

[linuxianer99]

Because nevertheless we have 3 types of user:

• the 'normal' user needs a complete image for sd card - and that is it.
• the 'extended' user is following your description on the Repository page and builds his own image! I know there are quite a few too!
• the 'developer' user is involved in our discussion and knows, what happens and why (From my point of view the smallest group)

So I'm definitely in favor of rethinking our branching

ok .. what is your proposal ??
Please use #213 for the branch discussion ...

[lordfolken]

What is really broken in hardknott? #202 is rather trivial, there's a trivial workaround (Alt-F2), not a showstoppper at all. What's the real breakage?

That XCSoar doesn't launch due to #207 Currently openvario is still referencing 7.22

[MaxKellermann]

If this is urgent enough, somebody should submit a patch to change XCSoar's default DRI device path. That's #207.
I just submitted a fix for #203.
Anything else?