Julia definition is out of date

[eitsupi]

The Julia container definitions added in #450 use pre-built images (ghcr.io/julia-vscode/julia-devcontainer), but they have not been updated for almost half a year.
Therefore, the latest version of Julia is 1.6.3, but Julia on containers created using Julia's container definition is still version 1.6.1.

As mentioned in this comment #154 (comment), updating Docker images requires a lot of effort, so I think it is worthwhile to stop using pre-built images and switch to a method where users build locally, similar to many other community container definitions.

Although there is a disadvantage that local builds take more time, I feel that there is also an advantage of being able to switch base images easily.

[Chuxel]

@eitsupi - This is a community definition so I'll loop in the definition owner.

@davidanthoff - Your thoughts on this?

[eitsupi]

@davidanthoff @ThomasHagebols Do you have any future plans for the julia definition?

[eitsupi]

@fonsp Any thoughts?

[Chuxel]

@eitsupi @davidanthoff @ThomasHagebols @fonsp - At this point I suspect we may want to remove this definition from the repository since given we haven't had a response in a few months now. I worry about the impression it leaves current state even though it is marked as community.

[davidanthoff]

The containers are all up-to-date. If we fall behind from time to time, it is extremly simple to add another version, see julia-vscode/julia-devcontainer#10 for an example. PRs are always welcome! I don't think there is a need to change anything.

[eitsupi]

As I wrote at the beginning of this issue, I think that repository should be deprecated and the definition completely rewritten to allow users to easily do local builds.

If we switch from pre-built images to local builds, we will no longer have to worry about update delays, and users can choose any version of Julia (even development versions can be choosed, as requested in this issue julia-vscode/julia-devcontainer#8 ).
(Since it is very easy to install any version of Julia, it may be more convenient to add an installation script to the script library so that any Julia can be installed on base image like mcr.microsoft.com/vscode/devcontainers/base:debian.)

There is also the problem that the build system of that repository only builds amd64 and does not support arm64.

[davidanthoff]

I would suggest we wait with any change until https://github.com/julialang/juliaup is finished and the official way to install Julia. I expect that to be the case in a few weeks (months at most). That will change a lot of things about Julia deployment, and will open up a range of new options for this scenario here as well.

There are also some major benefits of the current structure, in particular for novice users: there is almost no complexity involved, the surface level stuff that they interact with looks extremly simple (essentially just the super simple devcontainer.json file). All the complexity of say the Julia project instantiation etc. is properly packaged away. If we were to put all of that into the template here, then adding the Julia devcontainer definition would add a lot of quite complicated code, which I think would not be desirable.

So, to be honest, I'm not too keen to change the current setup, I think it essentially works well. I can see that it would be nice to have more control for more advanced users, but maybe one way would be to just add another configuration template for those users, say Julia (custom) or something? I don't think it would be a good trade-off to make things more complex for everyone to make life easier for advanced users, they can presumably just figure things out themselves.

[eitsupi]

I think you should take seriously the fact that no contributor has shown up in over six months and you, the owner of the definition, have taken no action.
I really appreciate the great contribution you are making to Julia. However, maintaining Docker images over the long term involves a lot of work, and I don't think the current community is capable of maintaining that repository.

Pre-built images should be eliminated, especially if you care about beginners. (It's much easier for them to edit the Dockerfile in their hands than to send a PR)

[davidanthoff]

However, maintaining Docker images over the long term involves a lot of work, and I don't think the current community is capable of maintaining that repository.

I don't follow at all. Literally one needs to add one number to one line in one file to add a new version, that is it. Everything else is fully automated. If someone doesn't know how to do it, open an issue or ping me. I am thinking about solutions how we can automate even that (as we have a number of places where we need to make that kind of one-line edit when a new Julia version is released, so I might just write a script for that and then that should be handled).

Pre-built images should be eliminated, especially if you care about beginners.

I disagree, I think the current solution is much easier for beginners than a template that drops a large and complicated Dockerfile into their workspace. I think one of the really good things about the current system is that it works really well for folks that don't even know what docker is.

I'm not sure what the right way to resolve this is. From the Julia extension for VS Code team's point of view I would prefer if we just kept things the way they are, but I guess at the end of the day it is MS' call?

[eitsupi]

I have been waiting for 3 months and unfortunately it seems that all I can do is close this issue.
I hope things will get better in the future, thanks.

[eitsupi]

I released a Dev Container Feature to install Julia, ghcr.io/eitsupi/devcontainer-features/julia.
Unfortunately, juliaup does not yet appear to be available on the shell scripts, so I had to use jq to identify the version available for download.
This Dev Container Feature allows users to quickly add any version of Julia on the container.
I hope this helps Julia users.
https://github.com/eitsupi/devcontainer-features/tree/main/src/julia

[Chuxel]

@eitsupi Given no response here, should we remove the Julia definition from this repository in favor of yours?

[davidanthoff]

I don't think anything should be changed. The images over at https://github.com/julia-vscode/julia-devcontainer are up-to-date and work.

I am the person creating Juliaup, which will become the default Julia installer soon (TM). Once that is officially rolled out, I will update the solution over at https://github.com/julia-vscode/julia-devcontainer to use that. But at the moment folks should not be pushed towards Juliaup for something official as it is still in a prerelease mode.

[eitsupi]

@Chuxel Of course I can provide a new Julia template like below, but if @davidanthoff sticks with the current one, all I can do is provide a Feature to install Julia.

{
	"image": "mcr.microsoft.com/devcontainers/base:debian",
	"features": {
		"ghcr.io/eitsupi/devcontainer-features/julia:0": {
			"version": "latest"
		}
	}
}

@davidanthoff

I don't think anything should be changed. The images over at julia-vscode/julia-devcontainer are up-to-date and work.

I think the situation is different now than it was a year ago. What is the reason for sticking with what is here and now outdated?
(ghcr.io/julia-vscode/julia-devcontainer relies on the deprecated script in this repository that are no longer maintained, so that is outdated, I think)
https://github.com/julia-vscode/julia-devcontainer/blob/1dc6fa02bb9fb9e2b856488146a664444039ad24/Dockerfile#L9

[eitsupi]

Given that the definition of Julia has become outdated again, it seems necessary to reopen this.

[davidanthoff]

@eitsupi I just added 1.8.4. It was released during the holidays, so that means there can be a bit of a delay sometimes. One thing you could do that would be super helpful is to just open a PR against https://github.com/julia-vscode/julia-devcontainer/blob/master/.github/workflows/docker-publish.yml#L48 and add the version there, that is really all that is needed.

And as I wrote above, as soon as we have Juliaup more officially I'll do something more automatic.

[eitsupi]

@davidanthoff Thanks. But I am not talking only about the version of Julia that I am describing as outdated.
Could you answer to my question #1114 (comment)?

If someone were to overhaul julia-vscode/julia-devcontainer, it would require a lot of effort and I don't have the bandwidth.
So I recommend giving up julia-vscode/julia-devcontainer and moving to a Dev Container Feature approach.

[davidanthoff]

@eitsupi Alright, I finally caught up with the recent (well, not so recent) devcontainer changes, at least I know now what a feature is :) I think you are right, it seems we should deprecate the container things we have maintained over at the VS Code extension org and move to a feature instead.

I did just add a non-interactive option to the Juliaup installer, I think that was probably the missing piece to make that usable inside a feature, right? I'm going to test it for a few days before I roll it out to everyone.

What do you think about the following plan:

1. We migrate the repo you started for the feature over into the official julialang GitHub org. I think generally this is such an official piece that it would be best to have it there? I can handle that.
2. We move the feature over to use Juliaup. I can also help with that.
3. We deprecate the existing Julia VS Code extension container.

[davidanthoff]

Actually, I'm realizing that the Julia feature right now is in a repo that has quite a number of other features as well. So, I guess my proposal then would be that we only move the Julia feature out of that repo and into a new one in the julialang org.

[eitsupi]

@davidanthoff Thanks for your reply!

I did just add a non-interactive option to the Juliaup installer, I think that was probably the missing piece to make that usable inside a feature, right?

Yes, I forget which issue, but I gave up using Juliaup (or juliainstaller) when I tried it a few months ago because the process of installing it required interactivity.
Things are much simpler if we can write the following process in a shell script.

install_juliaup # a function to install juliaup
su "${USERNAME}" -c "juliaup add ${JULIA_VERSION}"

I assume something like the following code that installs R via rig (R Installation Manager).
https://github.com/rocker-org/devcontainer-features/blob/5b1d9ba0c5af793f582fd161160856b4e48ca1ad/src/r-rig/install.sh#L314-L330

What do you think about the following plan:

I completely agree with your suggestion.
If JuliaLang could maintain Julia Dev Container Feature(s) and Dev Container Template(s), that would be great!
(FYI, I recently did the same thing with mamba-org. After I created a prototype of Dev Container Feature, it has now been transferred and is owned by mamba-org. https://github.com/mamba-org/devcontainer-features)

We move the feature over to use Juliaup. I can also help with that.

Since the Dev Container CLI has testing abilities, it would make sense to refactor after the juliaup has matured.

We deprecate the existing Julia VS Code extension container.

And, just as we would create a repository for Dev Container Features, we must create a repository for Dev Container Templates and migrate to the new templates using Dev Container Features.
Please check the R's case. (#1669, #1673)

In summary, for the new format, we need a repository based on https://github.com/devcontainers/feature-starter and a repository based on https://github.com/devcontainers/template-starter.
You could create these repositories and I could create the PR and port https://github.com/eitsupi/devcontainer-features/tree/main/src/julia to that and create the template prototype, or I could create these repositories and later transfer them to you and you could transfer them to JuliaLang or julia-vscode.
(A recent update added the ability to deprecate the Dev Container Feature so I can deprecate ghcr.io/eitsupi/devcontainer-features/julia after it is migrated.)

[davidanthoff]

@eitsupi cool, this is all very helpful. I started with https://github.com/JuliaLang/devcontainer-features, do you want to take a look at that? Does that look good to you? The tests are nonsense, maybe you could migrate the ones you have over to there? At the moment I can't publish things because I'm waiting for a Julialang admin to allow public packages.

Does it make sense that we test the feature stuff first and the move to templates?

[eitsupi]

@davidanthoff Awsome!

I am wondering if we can specify which version to install via juliaup at the first time.
For example, my Feature can install any version by specifying the following version.

installs the latest 1.6.X

"features": {
    "ghcr.io/eitsupi/devcontainer-features/julia:0": {
        "version": "1.6"
    }
}

installs the latest 1.X.Y

"features": {
    "ghcr.io/eitsupi/devcontainer-features/julia:0": {
        "version": "1"
    }
}

install the lates X.Y.Z

"features": {
    "ghcr.io/eitsupi/devcontainer-features/julia:0": {
        "version": "latest"
    }
}

A non-stable version can also be specified.

"features": {
    "ghcr.io/eitsupi/devcontainer-features/julia:0": {
        "version": "1.9.0-alpha1",
        "allowNonStableVersion": true
    }
}

Does it make sense that we test the feature stuff first and the move to templates?

Definitely.

[davidanthoff]

I am wondering if we can specify which version to install via juliaup at the first time.

Yes, the channel field can be used for that. One can pass any of the named channels like release, lts, beta etc, or any version like 1.8.2, or 1.8 or 1. All the pre-release versions should also work.

On one level the name version is nicer if one is really specifying a full version, but in the end the implementation is now just surfacing Juliaup concepts, and the name for this there is channel, so I thought it would probably make most sense to just stick with that?

[eitsupi]

Yes, the channel field can be used for that. One can pass any of the named channels like release, lts, beta etc, or any version like 1.8.2, or 1.8 or 1. All the pre-release versions should also work.

On one level the name version is nicer if one is really specifying a full version, but in the end the implementation is now just surfacing Juliaup concepts, and the name for this there is channel, so I thought it would probably make most sense to just stick with that?

That's great.

I think channel is fine since it is Juliaup's terminology.
It is possible that naming the Feature Julia (via Juliaup) like Ruby (via rvm) would be better as it would clarify the meaning of the term.
https://github.com/devcontainers/features/tree/main/src/ruby

It would be a good idea to document which versions are valid and to create test cases to verify that they work.

[eitsupi]

@davidanthoff I don't think this Feature is working yet, as evidenced by my additional tests. (JuliaLang/devcontainer-features#3)
install.sh is executed by the root user, but the user in the container (remote user) is not necessarily the root user.
So far Julia seems to be installed only for the root user.

[eitsupi]

Will be closed by devcontainers/devcontainers.github.io#278, thanks!