Isolation of end-to-end tests

[Adirelle]

This is an attempt to isolate the end-to-end tests from each others and from the configuration of the mise project itself.

It takes the following measures:

• Create a temporary working tree for each test in the $TMPDIR / /tmp folder. This folder is removed if the test succeeds.
• Install the dummy plugin from the source tree in the working tree. This allows to do quick and offline tests.
• Get rid of the shared fixtures files. Each test creates the required files in its script.
• Enforce using e2e/run_test to run the tests. They are not executable anymore to prevent running them in the user HOME.
• Use env --ignore-environment ... to tightly control the environment variables. This includes overriding $PATH with a default value.
• Move the test scripts around to mimics the src file tree.
• Rename slow tests using _slow. They are skipped by run_all_tests unless TEST_ALL=1.
• Check that the required commands for a test is available in the (restricted) PATH or skip the test with an explicit message. This should not happen in the dev container.
• Print both successful and failed assertions, with colors if possible. Use Github Action annotations in CI.

Note: this will require tests on MacOS.

[Adirelle]

In these tests, different commands are used to test if a binary is available in PATH:

• which $CMD => which is a POSIX-compliant binary
• type -p $CMD => type is a bash builtin
• command $CMD => command seems to be a fish builtin but I am not sure.

Which one would work on both MacOS and Linux?

[Adirelle]

Some of the plugins (and tests) require some tools to be available in the PATH, e.g. the pipx forge requires pipx to be in the PATH.
According to some testing, it does not support reusing pipx installed by another plugin.

In the e2e tests, what should be done?

1. install the required tool(s) in the test,
2. add the tools in the docker test image and skip the test if the command is not available,
3. or another option?

[Adirelle]

The ubi test is failing because the test expects ubi to be installed in the dev container, which is not the case right yet.

[jdx]

Some of the plugins (and tests) require some tools to be available in the PATH, e.g. the pipx forge requires pipx to be in the PATH.
According to some testing, it does not support reusing pipx installed by another plugin.

maybe try testing on the latest changes since I should have fixed this bug yesterday

[jdx]

I think bootstrapping with mise would be the best approach generally if we can get it working, backends are definitely still a bit wonky though so we'll just do the best we can

[codacy-production]

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ -0.18%	✅ ∅

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (76202de)	16867	13752	81.53%
Head commit (f043512)	16867 (+0)	13722 (-30)	81.35% (-0.18%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#2047)	0	0	∅ (not applicable)

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences

_{Codacy will stop sending the deprecated coverage status from June 5th, 2024. Learn more}

[jdx]

this is such a tiny nit I don't even want you to change the current code, but in the future I prefer sticking to EOF for all here-docs

[Adirelle]

Ah yes. I wrote this at the very beginning then I switched to EOF but forgot this one.