Add GitHub Actions configuration file

[smorimoto]

This PR adds a configuration file for GitHub Actions.

There are jobs that don't build and/or test for a variety of reasons, but they are all commented out. See the diagram below for what to do on which version and platform.

[VERSION] - [PLATFORM]	Build	Test
4.09.0 - macOS, Ubuntu, Windows	✔️	✔️
4.08.1 - macOS, Ubuntu, Windows	✔️	✔️
4.07.1 - macOS, Ubuntu, Windows	✔️	❌
4.02.3 - macOS, Ubuntu	✔️	❌
4.02.3 - Windows	❌	❌

[ghost]

Thanks for playing with github actions for dune :)

Would this replace Travis and AppVeyor?

[smorimoto]

Yes, that's right. and this works without complex shell scripts.

[ghost]

That's nice.

/cc our CI experts: @avsm @dra27 @craigfe

[smorimoto]

By the way, you can check here that it works correctly.
https://github.com/imbsky/dune/actions/runs/96236696

[dra27]

Nice! No reason for a short period not to run it alongside AppVeyor/Travis and then retire them a few weeks after merging?

[dra27]

Yes, that's right. and this works without complex shell scripts.

I'm not against it at all, but this is because the complex logic is offloaded to another action!

Another important caveat at the moment is run time - part of the complexity (especially on Travis) is managing the caching. The action is presently taking half an hour to run where the AppVeyor takes 90 seconds and Travis around 17 minutes.

[ghost]

Run time is definitely a concern. If there are other benefits to switching to github actions, we can accept a small slowdown, but going from 17 minutes to half an hour is too much.

[rgrinberg]

The run time increase is worrying and bit unexpected. When we switched lsp to github actions from travis, our run times improved significantly.

[ghost]

Maybe it's because of the caching @dra27 described. Do you do anything clever in lsp regarding opam caching?

[rgrinberg]

I don't think we have any special caching (@giltho can confirm). But we also don't install nearly as much dependencies for tests.

[smorimoto]

Yes, that's right. and this works without complex shell scripts.

I'm not against it at all, but this is because the complex logic is offloaded to another action!

Another important caveat at the moment is run time - part of the complexity (especially on Travis) is managing the caching. The action is presently taking half an hour to run where the AppVeyor takes 90 seconds and Travis around 17 minutes.

Yeah, you're absolutely right! My (and our?) goal is to push the many complex tasks that are going on in each repository to a particular Action.

[smorimoto]

For example, I've been working for about 6 months recently to speed up actions/cache, mainly for OCaml projects, and it's about 4~5 times faster as a result. This is probably good for caching dependencies, etc.

I also plan to rewrite the avsm/setup-ocaml to cache the contents of the opam directory so that we can set up our OCaml environment faster.

[giltho]

Yep confirming that we have no caching at all in ocaml-lsp's CI
As far as I know there is no good caching story yet for opam + GH actions workflow yet. But @imbsky is a maintainer of setup-ocaml and is working on caching there as said earlier.

The main advantage of moving to gha is indeed to have all the complexity encapsulated in an action instead of having custom scripts. Dune is a major engine that moves the community forward and it would be very nice for it to use gha so that the action can improve, and therefore improve for the whole community :)

That being said, it would probably be better to wait imo for the caching to work before merging this, or at least not remove the rest of the CI and make the gha checks optional for merging.

[smorimoto]

I'm sorry for committing so much. I was caught in some traps. I think I was able to reflect all　suggestions in code review.

[ghost]

No worries, we'll probably just squash everything before merging. Committing a lot during development is generally good practice :)

Otherwise, I'm personally happy to move to github actions. It's nice to have all the platforms tested in one place. But the run time is definitely a concern, and we should indeed wait until it is close enough to the current run time we get with the combo Travis/AppVeyor.

[smorimoto]

Thank you for your kind words. Yes, it makes sense. I will mark this PR as draft until their speed improves!

[rgrinberg]

I looked at the CI configuration and it doesn't seem like the run times are comparable. Here are some differences:

• We are running the job on 3 versions of OCaml on windows, while the current appveyor runs it only one version of OCaml.

• We are using opam for the windows jobs. In our current setup, we don't use windows.

• We are running the tests only on one version of OCaml and a single OS (Ubuntu). With actions, we are running the tests on 4.07-4.09 and on Mac and Linux.

• We are testing that 4.02, 4.07, 4.08, 4.09 build dune on ALL platforms rather than just one.

In summary, no wonder github actions seems a lot slower -- we are testing quite a bit more configurations. I would suggest the following changes:

• Run the 4.02 build only on a single platform (Linux)
• Run the test suite only on the latest version of OCaml on Mac/Linux.
• Add a CI configuration (build only) that does not use opam. It's important for dune to be usable without opam, so I don't think we should drop this configuration.

[rgrinberg]

Once #3466 is merged, we can drop installing gstat with homebrew by the way.

[smorimoto]

Running jobs on all platforms and versions are done in parallel, so I don't think it will speed up CI.
The most time-consuming part is setting up an OCaml environment and installing dependencies. So, in Windows, for example, only builds and doesn't install any extra dependencies, the job will finish in about 5 minutes.

[smorimoto]

If we want to speed everything up, we need to set up the OCaml environment within seconds to a minute. Because GitHub Actions has a cache limit for users, at least for now.

A repository can have up to 5GB of caches. Once the 5GB limit is reached, older caches will be evicted based on when the cache was last accessed. Caches that are not accessed within the last week will also be evicted.

But, on the Action side is unlimited. So I think we can have the pre-built switch on the setup-ocaml side. And that's what I'm working on now.

[dra27]

@rgrinberg - the important figure, which his comparable is "How long does it all take"... reducing the testing matrix is a fix for that, being able to parallelise is another! 🙂

[smorimoto]

It works for small projects, but the problem is projects like this. The cache limit is only 5 GB and will likely result in cache thrashing which will not get the expected results.
https://github.com/actions/cache#cache-limits

[voodoos]

It works for small projects, but the problem is projects like this. The cache limit is only 5 GB and will likely result in cache thrashing which will not get the expected results.
https://github.com/actions/cache#cache-limits

Ok I see, thanks for you answer !

(Apparently I accidentality deleted my question: I am using action/cache to cache the whole .opam folder on a small project of mine and it works great, I was wondering why it wouldn't scale to Dune CI.)

[rgrinberg]

@imbsky are there any updates on the caching issue? we've discussed this PR at the recent dune meeting and there was renewed interest in it.

[smorimoto]

I started a personal experiment project like this. Hopefully this will reduce the compilation time of the compiler itself. I want to work on this project as soon as I get home after this week's exams and so on.
https://github.com/actions-ml

[smorimoto]

To make it faster, I want to work more closely with the GitHub team rather than doing it in my own way, and while the project above is a bit stalled, things are moving forward.

[ghost]

That's good to hear :) We look forward to a simpler and more centralised CI, the current one based on Travis and AppVeyor is regularly causing us pain.

[rgrinberg]

Replaced by #3712.