Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(experimental): Build multiple recipes in parallel #182

Merged
merged 42 commits into from
Jun 7, 2024

Conversation

gmpinder
Copy link
Member

@gmpinder gmpinder commented May 13, 2024

The build subcommand can now take in any number of recipe files and will build them all in parallel. Along with this new ability, I've added a way to easily distinguish which part of the build log belongs to which recipe. Check out the docker_build action of this PR for an example.

gif

Tasks

  • Make build log follow same pattern as normal logs to keep things consistent
  • Update color ranges based on @xynydev 's feedback
  • Deal with ANSI control characters in log output
  • Add indicatif to make logs look nicer
  • Add ability to print logs to a file

@gmpinder gmpinder added the type: feature Brand new functionality, features, pages, workflows, endpoints, etc. label May 13, 2024
@gmpinder gmpinder self-assigned this May 13, 2024
Cargo.toml Outdated Show resolved Hide resolved
src/commands/build.rs Outdated Show resolved Hide resolved
src/drivers/docker_driver.rs Outdated Show resolved Hide resolved
utils/src/command_output.rs Outdated Show resolved Hide resolved
@gmpinder gmpinder marked this pull request as draft May 13, 2024 16:05
@gmpinder
Copy link
Member Author

Ok so trying this out, it would seem that although GHAs support using truecolor, Gitlab CI doesn't and makes it look weird. So I'm going to try using standard ANSI colors to keep things as compatible as possible.

Copy link
Contributor

@gerblesh gerblesh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm so far

@gmpinder gmpinder marked this pull request as ready for review May 14, 2024 22:09
@gmpinder gmpinder requested a review from gerblesh May 14, 2024 22:12
@gmpinder gmpinder enabled auto-merge (squash) May 15, 2024 13:22
@gmpinder
Copy link
Member Author

This is probably something that shouldn't be used in a CI with build matrixing capabilities, but could be useful for some in local builds or custom CI systems I guess.

My next goal after this is to make use of buildx bake. There are situations where some stages can be shared amongst builds like compiling stages and bake would help to ensure that they do. Currently this might or might not happen when spawning docker processes during my own testing. I did try it out locally so I guess I could try adding it to this PR now that I know how that works

@gmpinder
Copy link
Member Author

The logs being interwoven like that could get cluttered, though the custom colors help, maybe adding an option to write the logs to some directory instead would make this even better?

Yeah that was the reason for the colors. Earthly has a similar pattern for their matrix builds. I think it would be reasonable/fairly easy to add log redirection too. Helps that I already created a system to handle logs

@gmpinder gmpinder disabled auto-merge May 18, 2024 13:49
@gmpinder gmpinder changed the title feat: Build multiple recipes in parallel feat(experimental): Build multiple recipes in parallel May 19, 2024
@gmpinder
Copy link
Member Author

I'll hold off on the use of buildx bake for this particular PR and add that in later. I will still work on getting logs out

@gmpinder gmpinder marked this pull request as draft May 27, 2024 22:41
@gmpinder gmpinder marked this pull request as ready for review June 7, 2024 03:30
@gmpinder gmpinder merged commit 4ca98c1 into main Jun 7, 2024
8 checks passed
@gmpinder gmpinder deleted the multi-recipe-build branch June 7, 2024 21:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: feature Brand new functionality, features, pages, workflows, endpoints, etc.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants