diff --git a/README.md b/README.md index f7ed0418..15a32928 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -
__hook``. For example, the hook that is triggered right before the ``configure`` +step is run is a function named ``pre_configure_hook``. + +Every time these hooks are called, a single argument is provided: an [``EasyBlock``](https://docs.easybuild.io/en/latest/api/easybuild.framework.easyblock.html#easybuild.framework.easyblock.EasyBlock) +instance that represents the easyblock that is being used to perform the installation. +The parsed easyconfig file can be accessed via the ``cfg`` class variable of the ``EasyBlock`` instance. + +These hooks are useful for influencing the installation procedure at a particular stage. + +## Caveats + +There are a couple of important caveats to take into account when implementing hooks. + +### Breaking EasyBuild with hooks + +Since hooks allow you to inject custom code into EasyBuild at runtime, +it is also easy to break EasyBuild by using hooks... + +Make sure to carefully test your hook implementations, and constrain the actions +you take a much as possible, for example by adding conditions to control for which +software names you will actually modify the installation procedure, etc. + +Any errors that are triggered or raised while a hook function is running +will interrupt the EasyBuild session. + +So don't forget: with great power comes great responsibility! + +### Template values + +Depending on the type of hook, you may observe "raw" values of easyconfig parameters where +template values have not been resolved yet, or values in which template values have been resolved already. + +In the ``parse`` hook, you will always see unresolved template values. + +In the pre/post-step hooks you will see resolved template values, +unless you explicitly disable templating. + +To obtain easyconfig parameter values with unresolved template values in step hooks, +you can use the ``disable_templating`` [context manager](https://docs.python.org/3/reference/compound_stmts.html#with). +For example: + +```python +from easybuild.framework.easyconfig.easyconfig import disable_templating +from easybuild.tools.build_log import print_warning + +def pre_source_hook(eb): + """Print warning when software version was found in 'raw' name of source file.""" + with disable_templating(eb.cfg): + for src in eb.cfg['sources']: + if eb.version in src: + msg = "Software version '%s' found in name of source file (%s), " % (eb.version, src) + msg += "please use %(version)s template value instead!" + print_warning(msg) +``` + + +### Manipulating easyconfig parameters + +If you want update a particular easyconfig parameter without overwriting the existing value, +a bit of care has to be taken: you should use the ``update`` method of the ``EasyConfig`` instance +for this, unless you disable template resolution. This is particularly important when +updating easyconfig parameters that have *mutable* value (like a ``list`` or ``dict``). + +Here's a correct example of a pre-install hook: + +```python +def pre_install_hook(eb): + if eb.name == 'pigz': + # always copy the README directory too when installing pigz + eb.cfg.update('files_to_copy', 'README') +``` + +This seemingly equivalent implementation will ***not*** work (the value of the `files_to_copy` +easyconfig parameter will *not* be updated): + +```python +def pre_install_hook(eb): + if eb.name == 'pigz': + # incorrect way of adding 'README' to 'files_to_copy' (DON'T USE THIS!) + eb.cfg['files_to_copy'].append('README') +``` + +To use this coding style successfully, you have to disable the templating mechanism +when updating the easyconfig parameter: + +```python +def pre_install_hook(eb): + if eb.name == 'pigz': + # this works, but it is better to use the 'update' method instead... + with disable_templating(eb.cfg): + eb.cfg['files_to_copy'].append('README') +``` + +--- + +*[[next: Submitting installations as Slurm jobs]](3_03_slurm_jobs.md)* diff --git a/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md b/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md new file mode 100644 index 00000000..d49a7925 --- /dev/null +++ b/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md @@ -0,0 +1,335 @@ +# Submitting installations as Slurm jobs + +*[[back: Using hooks to customise EasyBuild]](3_02_hooks.md)* + +--- + +EasyBuild can submit jobs to different backends including Slurm to install software, +to *distribute* the often time-consuming installation of a set of software applications and +the dependencies they require to a cluster. Each individual package is installed in a separate +job and job dependencies are used to manage the dependencies between package so that no build +is started before the dependencies are in place. + +This is done via the ``--job`` command line option. + +It is important to be aware of some details before you start using this, which we'll cover here. + +!!! Warning "This section is not supported on LUMI, use at your own risk" + + EasyBuild on LUMI is currently not fully configured to support job submission via Slurm. Several + changes would be needed to the configuration of EasyBuild, including the location of the + temporary files and build directory. Those have to be made by hand. + + Due to the setup of the central software stack, this feature is currently useless to install + the central stack. For user installations, there are also limitations as the environment + on the compute nodes is different from the login nodes so, e.g., different locations for + temporary files are being used. These would only be refreshed if the EasyBuild configuration + modules are reloaded on the compute nodes which cannot be done currently in the way Slurm + job submission is set up in EasyBuild. + + Use material in this section with care; it has not been completely tested. + + +## Configuration + +The EasyBuild configuration that is active at the time that ``eb --job`` is used +will be *passed down* into the submitted job automatically, via command line options to the ``eb`` +command that is run in the job script. + +This includes not only command line options used next to ``--job``, but also configuration settings +that are specified via an [EasyBuild configuration file](../../1_Intro/1_07_configuration#configuration-files) or through +[``$EASYBUILD_*`` environment variables](../../1_Intro/1_07_configuration#easybuild_-environment-variables). + +This implies that any EasyBuild configuration files or ``$EASYBUILD_*`` environment variables +that are in place in the job environment are most likely *irrelevant*, since configuration settings +they specify they will most likely be overruled by the corresponding command line options. +It does also imply however that the EasyBuild configuration that is in place when ``eb --job`` is used +should also work on the compute nodes to which the job is submitted as EasyBuild will generate the command +line options used in the job script based on that configuration. + + +## Using ``eb --job`` + +### Job backend + +The default job backend in EasyBuild v4.x is [``GC3Pie``](https://gc3pie.readthedocs.io). +To let EasyBuild submit jobs to Slurm instead, you should set the ``job-backend`` configuration setting +to ``Slurm``, for example by setting the corresponding environment variable: + +```shell +export EASYBUILD_JOB_BACKEND='Slurm' +``` + +On LUMI this is taken care of in the EasyBuild configuration modules such as ``EasyBuild-user``. + + +### Job resources + +To submit an installation as a job, simply use ``eb --job``: + +```shell +eb example.eb --job +``` + +By default, EasyBuild will submit single-core jobs requesting for 24 hours of walltime. +You can tweak the requested resources via the ``job-cores`` and ``job-max-walltime`` configuration options. +For example: + +```shell +# submit job to install example, using 5 cores and 2 hours of max. walltime +eb example.eb --job --job-cores 5 --job-max-walltime 2 +``` + +Note that not all ``job-*`` configuration settings apply to all job backends, +see the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Submitting_jobs.html) for more details. + +### Controlling Slurm submission options + +When using Slurm as a job backend, EasyBuild will automatically generate job scripts which +use the ``eb`` command to perform a single installation. These scripts will be submitted +using the ``sbatch`` command. + +EasyBuild currently doesn't provide away to customize the Slurm submission options, +for example to submit to a particular partition, or to use a particular account, +build you can set the corresponding ``SBATCH_*`` environment variables prior to running ``eb --job``. + +For example, to specify a particular account that should be used for the jobs submitted by EasyBuild +(equivalent with using the ``-A`` or ``--account`` command line option for ``sbatch``): + +```shell +export SBATCH_ACCOUNT='project_XXXXXXXXX' +``` + +Or to submit to a particular Slurm partition (equivalent with the ``-p`` or ``--partition`` option for ``sbatch``): + +```shell +export SBATCH_PARTITION='small' +``` + +For more information about supported ``SBATCH_*`` environment variables, +see the [Slurm documentation](https://slurm.schedmd.com/sbatch.html#lbAJ). + +## Combining ``--job`` and ``--robot`` + +If one or more dependencies are still missing for the software you want to install, +you can combine ``--job`` and ``--robot`` to get EasyBuild to submit a *separate* job +for each of the installations. These jobs will *not* use ``--robot``, they will each only +perform a single installation. + +Dependencies between jobs will be "registered" at submission time, so Slurm will put jobs +on hold until the jobs that install the required (build) dependencies have completed successfully, +and cancel jobs if the job to install a dependency failed for some reason. + +## Attention points + +There are a couple of important things to keep an eye on when submitting installations as jobs... + +### Differences on cluster workernodes + +Sometimes the resources available on the login nodes and cluster workernodes are slightly different, +and you may need to take this into account in your EasyBuild configuration. + +For example, plenty of disk space may be available in the `/tmp` temporary filesystem on a login node, +while the workernodes require you to use a different location for temporary files and directories. +As a result, you may need to slightly change your EasyBuild configuration when submitting installations +as jobs, to avoid that they fail almost instantly due to a lack of disk space. + +Keep in mind that the active EasyBuild configuration is passed down into the submitted jobs, +so any configuration that is present on the workernodes may not have any effect. + +For example, on LUMI it is possible to use ``$XDG_RUNTIME_DIR`` on the login nodes which has +the advantage that any leftovers of failed builds will be cleaned up when the user ends their last +login session on that node, but it is not possible to do so on the compute nodes. + +```shell +# EasByuild is configured to use /tmp/$USER on the login node +uan01 $ eb --show-config | grep buildpath +buildpath (E) = /run/user/XXXXXXXX/easybuild/build + +# use /dev/shm/$USER for build directories when submitting installations as jobs +login01 $ eb --job --buildpath /dev/shm/$USER/easybuild example.eb --robot +``` + + +### Temporary log files and build directories + +The problems for the temporary log files are twofold. First, they may end up in a place +that is not available on the compute nodes. E.g., for the same reasons as for the build +path, the LUMI EasyBuild configuration will place the temporary files in a subdirectory of +``$XDG_RUNTIME_DIR`` on the loginnodes but a subdirectory of ``/dev/shm/$USER`` on the +compute nodes. The second problem however is that if an installation fails, those log files are +not even accessible anymore which may leave you wondering about the actual cause of the failing +installation... + +To remedy this, there are a couple of EasyBuild configuration options you can use: + +* You can use ``--tmp-logdir`` to specify a different location where EasyBuild should store temporary log files, + for example: + ```shell + $ eb --job example.eb --tmp-logdir $HOME/eb_tmplogs + ``` + This will move at least the log file to a suitable place. + +* If you prefer having the entire log file stored in the Slurm job output files, + you can use ``--logtostdout`` when submitting the jobs. This will result in extensive logging + to your terminal window when submitting the jobs, but it will also make EasyBuild + log to ``stdout`` when the installation is running in the job, and hence the log messages will be + captured in the job output files. + +The build directory of course also suffers from the problem of being no longer accessible if the +installation fails, but there it is not so easy to find a solution. Building on a shared file system +is not only much slower, but in particular on parallel file systems like GPFS/SpectrumScale, Lustre +or BeeGFS building sometimes fails in strange ways. One thing you can consider if you cannot do the +build on a login node (e.g., because the code is not suitable for cross-compiling or the configure +system does tests that would fail on the login node), is to rety the installation in an +interactive job, so you can inspect the build directory after the installation fails. + +### Lock files + +EasyBuild creates [locks](https://docs.easybuild.io/en/latest/Locks.html) +to prevent that the same installation is started multiple times on +different system to the same installation directory. + +If an installation fails or gets interrupted, EasyBuild cleans up those locks automatically. + +However, if a Slurm job that is using EasyBuild to install software gets cancelled (because it +ran out of walltime, tried to consume too much memory, through an ``scancel`` command, etc.), +EasyBuild will not get the chance to clean up the lock file. + +If this occurs you will need to either clean up the lock file (which is located in the `software/.locks` +subdirectory of ``installpath``) manually, or re-submit the job with ``eb --job --ignore-locks``. + +## Example + +As an example, we will let EasyBuild submit jobs to install ``AUGUSTUS`` with the ``foss/2020b`` toolchain. + +!!! Warning "This example does not work on LUMI" + + Note that this is an example using the FOSS common toolchain. For this reason it does not work on + LUMI. + +### Configuration + +Before using ``--job``, let's make sure that EasyBuild is properly configured: + +```shell +# Load the EasyBuild-user module (central installations will not work at all +# using job submission) +module load LUMI/21.12 +module load partition/C +module load EasyBuild-user + +# use /tmp for build directories and temporary directories as we have those +# on login and compute nodes. +export EASYBUILD_BUILDPATH=/tmp/$USER/build +export EASYBUILD_TMPDIR=/tmp/$USER/tmp + +# use Slurm as job backend +export EASYBUILD_JOB_BACKEND=Slurm +``` + + +We will also need to inform Slurm that jobs should be submitted into a particular account, and +in a particular partition: + +```shell +export SBATCH_ACCOUNT=project_XXXXXXXXX +export SBATCH_PARTITION='small' +``` + +This will be picked up by the ``sbatch`` commands that EasyBuild will run to submit the software installation jobs. + + +### Submitting jobs to install AUGUSTUS + +Now we can let EasyBuild submit jobs for AUGUSTUS. + +Let's first check what is still missing: + +```shell +$ eb AUGUSTUS-3.4.0-foss-2020b.eb --missing +... +11 out of 61 required modules missing: + +* HTSlib/1.11-GCC-10.2.0 (HTSlib-1.11-GCC-10.2.0.eb) +* lpsolve/5.5.2.11-GCC-10.2.0 (lpsolve-5.5.2.11-GCC-10.2.0.eb) +* Boost/1.74.0-GCC-10.2.0 (Boost-1.74.0-GCC-10.2.0.eb) +* GSL/2.6-GCC-10.2.0 (GSL-2.6-GCC-10.2.0.eb) +* SAMtools/1.11-GCC-10.2.0 (SAMtools-1.11-GCC-10.2.0.eb) +* BCFtools/1.11-GCC-10.2.0 (BCFtools-1.11-GCC-10.2.0.eb) +* METIS/5.1.0-GCCcore-10.2.0 (METIS-5.1.0-GCCcore-10.2.0.eb) +* BamTools/2.5.1-GCC-10.2.0 (BamTools-2.5.1-GCC-10.2.0.eb) +* MPFR/4.1.0-GCCcore-10.2.0 (MPFR-4.1.0-GCCcore-10.2.0.eb) +* SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0 (SuiteSparse-5.8.1-foss-2020b-METIS-5.1.0.eb) +* AUGUSTUS/3.4.0-foss-2020b (AUGUSTUS-3.4.0-foss-2020b.eb) +``` + +Several dependencies are not installed yet, so we will need to use ``--robot`` to ensure that +EasyBuild also submits jobs to install these first. + +To speed up the installations a bit, we will request 8 cores for each submitted job (via ``--job-cores``). +That should be sufficient to let each installation finish in (well) under 1 hour, +so we only request 1 hour of walltime per job (via ``--job-max-walltime``). + +In order to have some meaningful job output files, we also enable trace mode (via ``--trace``). + +``` +$ eb AUGUSTUS-3.4.0-foss-2020b.eb --job --job-cores 8 --job-max-walltime 1 --robot --trace +... +== resolving dependencies ... +... +== List of submitted jobs (11): Boost-1.74.0-GCC-10.2.0 (Boost/1.74.0-GCC-10.2.0): 1000011; GSL-2.6-GCC-10.2.0 (GSL/2.6-GCC-10.2.0): 1000004; SAMtools-1.11-GCC-10.2.0 (SAMtools/1.11-GCC-10.2.0): 1000005; HTSlib-1.11-GCC-10.2.0 (HTSlib/1.11-GCC-10.2.0): 1000006; BCFtools-1.11-GCC-10.2.0 (BCFtools/1.11-GCC-10.2.0): 1000001; lpsolve-5.5.2.11-GCC-10.2.0 (lpsolve/5.5.2.11-GCC-10.2.0): 1000007; BamTools-2.5.1-GCC-10.2.0 (BamTools/2.5.1-GCC-10.2.0): 1000008; METIS-5.1.0-GCCcore-10.2.0 (METIS/5.1.0-GCCcore-10.2.0): 1000009; MPFR-4.1.0-GCCcore-10.2.0 (MPFR/4.1.0-GCCcore-10.2.0): 1000010; SuiteSparse-5.8.1-foss-2020b-METIS-5.1.0 (SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0): 1000002; AUGUSTUS-3.4.0-foss-2020b (AUGUSTUS/3.4.0-foss-2020b): 1000003 +== Submitted parallel build jobs, exiting now +``` + +### Inspecting the submitted jobs + +Once EasyBuild has submitted the jobs, we can inspect them via Slurm's ``squeue`` command: + +``` +$ squeue -u $USER -la + JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON) +1000001 small BCFtools user123 PENDING 0:00 2:00:00 1 (Dependency) +1000002 small SuiteSpa user123 PENDING 0:00 2:00:00 1 (Dependency) +1000003 small AUGUSTUS user123 PENDING 0:00 2:00:00 1 (Dependency) +1000004 small GSL-2.6- user123 RUNNING 0:21 2:00:00 1 node003 +1000005 small SAMtools user123 RUNNING 0:21 2:00:00 1 node007 +1000006 small HTSlib-1 user123 RUNNING 0:21 2:00:00 1 node007 +1000007 small lpsolve- user123 RUNNING 0:21 2:00:00 1 node011 +1000008 small BamTools user123 RUNNING 0:21 2:00:00 1 node011 +1000009 small METIS-5. user123 RUNNING 0:21 2:00:00 1 node013 +1000010 small MPFR-4.1 user123 RUNNING 0:21 2:00:00 1 node029 +1000011 small Boost-1. user123 RUNNING 0:24 2:00:00 1 node029 +``` + +Note that 3 jobs can not be started yet, because those installations require on one or more +missing dependencies. As soon as the jobs for those dependencies (successfully) complete, +these jobs will be able to start. + +### Final result + +After about 20 minutes, AUGUSTUS and all missing dependencies should be installed: + +``` +$ ls -lrt $HOME/EasyBuild/modules/.../*.lua | tail -11 +-rw-rw----. 1 example example 1634 Mar 29 10:13 /users/example/easybuild/modules/all/HTSlib/1.11-GCC-10.2.0.lua +-rw-rw----. 1 example example 1792 Mar 29 10:13 /users/example/easybuild/modules/all/SAMtools/1.11-GCC-10.2.0.lua +-rw-rw----. 1 example example 1147 Mar 29 10:13 /users/example/easybuild/modules/all/BamTools/2.5.1-GCC-10.2.0.lua +-rw-rw----. 1 example example 957 Mar 29 10:13 /users/example/easybuild/modules/all/lpsolve/5.5.2.11-GCC-10.2.0.lua +-rw-rw----. 1 example example 1549 Mar 29 10:13 /users/example/easybuild/modules/all/METIS/5.1.0-GCCcore-10.2.0.lua +-rw-rw----. 1 example example 1525 Mar 29 10:14 /users/example/easybuild/modules/all/GSL/2.6-GCC-10.2.0.lua +-rw-rw----. 1 example example 1221 Mar 29 10:15 /users/example/easybuild/modules/all/MPFR/4.1.0-GCCcore-10.2.0.lua +-rw-rw----. 1 example example 1678 Mar 29 10:15 /users/example/easybuild/modules/all/BCFtools/1.11-GCC-10.2.0.lua +-rw-rw----. 1 example example 1292 Mar 29 10:21 /users/example/easybuild/modules/all/Boost/1.74.0-GCC-10.2.0.lua +-rw-rw----. 1 example example 1365 Mar 29 10:28 /users/example/easybuild/modules/all/SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0.lua +-rw-rw----. 1 example example 2233 Mar 29 10:30 /users/example/easybuild/modules/all/AUGUSTUS/3.4.0-foss-2020b.lua + +$ module avail AUGUSTUS + +-- EasyBuild managed user software for software stack ... -- + AUGUSTUS/3.4.0-foss-2020b +``` + +--- + +*[[next: Module naming schemes]](3_04_module_naming_schemes.md)* diff --git a/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md b/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md new file mode 100644 index 00000000..ce8a6eff --- /dev/null +++ b/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md @@ -0,0 +1,575 @@ +# Module naming schemes + +*[[back: Submitting installations as Slurm jobs]](3_03_slurm_jobs.md)* + +--- + +Up until now we have used the naming scheme in use on LUMI which is for many practical purposes +is indistinguishable from the default EasyBuild naming scheme (`EasyBuildMNS`). +It produces module files with names that closely resemble to the names of the +corresponding easyconfig files. +For example, when installing `zlib-1.2.11-cpeGNU-21.12.eb` the generated module was named +`zlib/1.2.11-cpeGNU-21.12`. + +EasyBuild supports several different module naming schemes: + +```shell +$ eb --avail-module-naming-schemes +List of supported module naming schemes: + EasyBuildMNS + LUMI_FlatMNS + MigrateFromEBToHMNS + HierarchicalMNS + CategorizedHMNS + CategorizedModuleNamingScheme +``` + +In this part of the tutorial we will take a closer look at `HierarchicalMNS`, +which is the standard **hierarchical** module naming scheme included with EasyBuild. + +We will also take a quick look at implementing our own custom module naming scheme. + +## Flat vs hierarchical + +!!! Note + + This text is very much based on the generic EasyBuild tutorials as we cannot demonstrate + all aspects on LUMI as it is configured today. + + On LUMI, the complete module scheme is partly hierarchical, but EasyBuild uses a flat naming + scheme. The two levels in the hierarchy that are present to deal with different versions of + the software stack and the various types of compute nodes, are not managed by EasyBuild. + +The default module naming scheme `EasyBuildMNS` +and the custom `LUMI_FlatMNS` naming scheme are both examples of regular *"flat"* +module naming schemes, which are characterized by: + +- all module files are directly available for loading; +- each module name uniquely identifies a particular installation; + +In contrast, a *hierarchical* module naming scheme +consists of a *hierarchy* of module files. +A fairly typical 3-level scheme (``Core``, ``Compiler`` and ``MPI``) has been +discussed in the [section on Lmod](../../1_Intro/1_02_Lmod#lmod-hierarchy). +This typical Lmod hierarcny would map very well on the EasyBuild common toolchains. + +In fact, for the example + + + +software at the ``Core`` level would be installed with the ``GCCcore`` and ``SYSTEM`` +toolchains. Software at the ``Compiler`` level would be installed with the ``GCC`` +toolchain, while software at the ``MPI`` level after loading the ``OpenMPI`` module +would be installed using the regular ``foss`` toolchain or the ``gompi`` toolchain +(see the diagram in the +["Common toolchains" section of the page on terminology](../../1_Intro/1_05_terminology/#common-toolchains)). + +On LUMI, where software is installed through the Cray Programming Environment with no real choice of +MPI implementation, a two-level arrangement would still make a lot of sense, with at the ``Core`` level +all software compiled with the SYSTEM toolchain while there could be a ``PrgEnv`` level for software +compiled with a particular programming environment aka cpeGNU/cpeCray/cpeAOCC toolchain. Such a scheme +is used on the Cray systems at CSCS. + +To recap, the characteristics of a module hierarchy are: + +* not all module files are directly available for loading; +* some modules serve as a gateway to more modules; +* to access some software installations you will first need to load one or more gateway modules in order + to use them; + +You can probably think of other ways to organize module files in a hierarchical module tree, but here +we will stick to the standard core / compiler / MPI hierarchy. + +### Pros & cons + +So why go through all this trouble of organizing modules hierarchically? + +There are a couple of advantages to this approach: + +* shorter module names (or at least for the version part of the name); +* less overwhelming list of available modules; +* only compatible modules can be loaded together; + +However, the are some minor disadvantages too: + +* not all existing modules are directly visible, so users have to learn how to find + modules using ``module spider`` etc. (but that is not an issue on LUMI as users + also need to use this command anyway to discover software that may not be in all + software stacks); +* gateway modules may have little meaning to end users; + +#### Length of module names + +When using a flat module naming scheme, module names can be fairly long and perhaps confusing. For a `HDF5` installation +with the EasyBuild common toolchains for example, +one might have `HDF5/1.12.1-gompi-2021b` as module name. The `-gompi-2021b` part of the name refers to the toolchain that was +used for this installation, but it may be confusing to some people (what kind of Pokémon is a "gompi"?!). + +In the example module hierarchy shown above, the module for `HDF5` could simply be named `HDF5/1.12.1` which is basically the bare +essentials: software name and version. That's way better, nice and clean! + +#### Amount of available modules + +The output of "`module avail`" can be quite overwhelming if lots of module files +are installed and a flat module naming scheme is used, since *all* modules are +*always* available. +EasyBuild makes it very easy to install lots of software, +so the number of installed modules can easily grow into the hundreds or even thousands... + +This often explosive growth of modules is less of an issue when using a hierarchical module naming scheme, since +initially only a small set of modules are available, and relatively limited +groups of additional modules become available as gateway modules are loaded. + +#### Loading compatible modules + +Since all modules are available at once when using a flat module naming scheme, you can easily load +modules together that are not compatible with each other. + +Imagine loading two modules that were built with a different compiler toolchain (different compiler, +different MPI library). That's likely to end in tears, unless you have the necessary technical expertise +to fully understand what is going on, *and* you are being very careful... + +In a module hierarchy this can be prevented, since modules for software that were installed with a +different compiler and/or a different MPI library are located in a different part of the module hierarchy, +and so these modules can not be loaded together easily. + + +#### Visibility of existing modules + +One downside of a module hierarchy is that not all existing modules are directly available for loading +or are even *visible* to the user, since the output of "`module avail`" only shows a subset of all modules. + +As we have discussed in [the Lmod section](../../1_Intro/1_02_Lmod). +[Lmod](https://lmod.readthedocs.io) provides a powerful solution to search for modules through the +``module spider`` and to some extent also the ``module keyword`` commands. +So as long as the end users are +aware of this additional command, it should not be difficult to discover which software installations exist +and how they can be accessed. The "`module spider`" command will inform the user which of the gateway modules +need to be loaded in order to load a specific module file. + +#### Semantics of gateway modules + +An additional potential problem of a module hierarchy is that the semantics of the gateway modules may not be clear +to end users. They may wonder why they need to pick a specific compiler and MPI library, or which of the +available options is the best one. Maybe they are not even aware what exactly a "compiler" is, or how it is +relevant to the software they need in their bioinformatics pipeline... + +This can be partially resolved by loading a default compiler and MPI module so a particular set of modules +is available right after login, which could be the ones used in the most recent toolchain, or the +recommended versions. More experienced users could then leverage the "`module spider`" command to navigate +the module hierarchy. + +On LUMI this would mean loading a default software stack, but due to the the default modules are currently +loaded on LUMI this was not possible to accomplish without losing other functionality of the module tree. + +## Using a custom module naming scheme + +Next to the module naming schemes that are included with EasyBuild, +you can also define your own module naming scheme (MNS), and configure EasyBuild to use it +(which is exactly what has been done on LUMI to remove a feature of the default ``EasyBuildMNS`` scheme +that we do not use). + +### Implementation + +To use a custom module naming scheme, you need to implement a Python module file, +where you define a Python class that derives from the general [``ModuleNamingScheme``](https://docs.easybuild.io/en/latest/api/easybuild.tools.module_naming_scheme.mns.html#easybuild.tools.module_naming_scheme.mns.ModuleNamingScheme) class. + +For a flat module naming scheme, it is sufficient to implement the ``det_full_module_name`` method, +which should return a string value (the full module name). +You may also need to customize the ``is_short_modname_for`` method, which verifies whether +a given (short) module name is for the software with a particular given name (or not). + +The argument provided to ``det_full_module_name`` can be a small Python dictionary which only specifies +the value of a handful of easyconfig parameters (`name`, `version`, `toolchain`, and `versionsuffix`), +or an ``EasyConfig`` instance which represents a parsed easyconfig file and contains values for *all* +known easyconfig parameters. + +For simple module naming schemes, just have ``name``, ``version``, ``toolchain``, and ``versionsuffix`` +available is sufficient. If it is not, you can list which additional easyconfig parameters are required for +your module naming scheme via a class constant ``REQUIRED_KEYS``. + +A fairly straightforward example of a *flat* module name scheme is the default EasyBuild module naming scheme +[``EasyBuildMNS``](https://github.com/easybuilders/easybuild-framework/blob/main/easybuild/tools/module_naming_scheme/easybuild_mns.py). + +For a *hierarchical* module naming scheme, various additional methods have to be implemented. + +Some of these, like ``det_module_subdir`` and ``det_short_module_name``, determine which part of +the (full) module name determines the location of the module in the module hierarchy, and which part +is the user-facing (or "short") module name. Others, like ``det_modpath_extensions``, +determine which modules are *gateway* modules that open up an additional level of the hierarchy +(by *extending* the ``$MODULEPATH``). + +A typical example of a hierarchical module naming scheme is [``HierarchicalMNS``](https://github.com/easybuilders/easybuild-framework/blob/main/easybuild/tools/module_naming_scheme/hierarchical_mns.py) that comes with EasyBuild (which is quite complex because it includes quite a bit of special handling for particular compiler +toolchains), which implements the traditional ``Core``-``Compiler``-``MPI`` module hierarchy we discussed +above. + +### Configuring EasyBuild + +To let EasyBuild use a custom module naming scheme, you need to: + +* specify the path to the Python module file implementing it via the ``include-module-naming-schemes`` + EasyBuild configuration option; + +* indicate that you also want to *use* this custom module naming scheme via the ``module-naming-scheme`` + EasyBuild configuration option. + +For example: + +```shell +export EASYBUILD_INCLUDE_MODULE_NAMING_SCHEMES=$HOME/easybuild/example_mns.py +export EASYBUILD_MODULE_NAMING_SCHEME=ExampleMNS +``` + +#### Example custom module naming scheme + +Here is an example of a custom module naming scheme, where: + +* the ``versionsuffix`` goes directly after the ``version``; +* all dashes are replaced by underscores; +* all module names are lowercase; + +Note that we also need to customise the ``is_short_modname_for`` method, +to make sure it returns ``True`` when the EasyBuild framework checks whether +``scipy_bundle/2020.11_foss_2020b`` is a module name for ``SciPy-bundle``. + +```python +import os + +from easybuild.tools.module_naming_scheme.mns import ModuleNamingScheme + + +class ExampleMNS(ModuleNamingScheme): + + REQUIRED_KEYS = ['name', 'version', 'versionsuffix', 'toolchain'] + + def det_full_module_name(self, ec): + """ + Determine full module name: + - all lowercase + - replace all dashes with underscores + """ + parts = [ec['version']] + + # versionsuffix directly after version (but only if it's not empty) + if ec['versionsuffix']: + parts.append(ec['versionsuffix']) + + # only add toolchain name/version for non-system toolchain + tc = ec['toolchain'] + if tc['name'].lower() != 'system': + parts.extend([tc['name'], tc['version']]) + + modname = ec['name'] + '/' + '_'.join(parts) + + modname = modname.replace('-', '_').replace('__', '_') + + return modname.lower() + + def is_short_modname_for(self, short_modname, name): + """Determine whether short module name is a module for the software with specified name.""" + return short_modname.startswith(name.lower().replace('-', '_') + '/') +``` + +We can see what the module names with this module naming scheme would like like via ``eb -D``. +E.g., for the common toolchains (the example will not work on LUMI as the default easyconfig +files are not in the robot- and the search path): + +``` +$ eb SciPy-bundle-2020.11-foss-2020b-Python-2.7.18.eb -D + ... + * [ ] $CFGS/g/GCC/GCC-10.2.0.eb (module: gcc/10.2.0) + ... + * [ ] $CFGS/p/Python/Python-2.7.18-GCCcore-10.2.0.eb (module: python/2.7.18_gcccore_10.2.0) + ... + * [ ] $CFGS/o/OpenMPI/OpenMPI-4.0.5-GCC-10.2.0.eb (module: openmpi/4.0.5_gcc_10.2.0) + ... + * [ ] $CFGS/s/SciPy-bundle/SciPy-bundle-2020.03-foss-2020a-Python-2.7.18.eb (module: scipy_bundle/2020.03_python_2.7.18_foss_2020a) +``` + +## Example module hierarchy: HDF5 + +!!! Warning "Example not suitable for LUMI" + **This exercise is meant for a system where the common toolchains can be used and requires an + independent EasyBuild installation in your personal file space**, + because EasyBuild will try to copy the installation log file to each installation directory. + +Now that we know more about hierarchical module naming schemes, +let us see how EasyBuild can help us with generating a hierarchical module tree. + +In this example we will use EasyBuild to generate modules organised in a hierarchy +for some of the software that is already installed in the prepared environment. + +The good news is that the existing installations can be reused. There is absolutely no need +to reinstall the software, we are just creating a different "view" on these software installations. + +### Preparing the environment + +Before running EasyBuild to generate a hierarchical module tree, we have to be a bit careful +with preparing our environment. + +**We must absolutely avoid mixing modules from a flat and hierarchical module naming scheme!** + +Some module files will have the same name in both module trees (like `GCC/10.2.0` for example), +but their contents will be different. +Mixing modules from a flat and hierarchical module tree *will* trigger problems... + +So we have to make sure that the module files we already have in `/easybuild` are *not* visible. +The easiest way to do this is to unload all modules (using "`module purge`") +and resetting the module search path to be empty, which we can do with "`module unuse $MODULEPATH`". + +```shell +module purge +module unuse $MODULEPATH +``` + +In this part of the tutorial, we are assuming you are *not* using an EasyBuild installation provided through +a module. We have just made all modules unavailable, so we would have to first +install EasyBuild again in our hierarchical module tree before we can continue. + +**We strongly recommend using an EasyBuild installation that was [installed via "`pip install`" +or "`pip3 install`"](../../1_Intro/1_06_installation#method-1-using-pip) in this part of the tutorial.** + +An easy way to do this is in the prepared environment used for regular EasyBuild tutorials is to run: + +```shell +unset PIP_PREFIX +pip3 install --user easybuild +export PATH=$HOME/.local/bin:$PATH +export EB_PYTHON=python3 +``` + +### Configuring EasyBuild + +First of all, we need to make sure that EasyBuild is properly configured. +We can do this by defining this set of environment variables: + +```shell +export EASYBUILD_PREFIX=$HOME/easybuild +export EASYBUILD_BUILDPATH=/tmp/$USER +export EASYBUILD_INSTALLPATH_SOFTWARE=/easybuild/software +export EASYBUILD_MODULE_NAMING_SCHEME=HierarchicalMNS +export EASYBUILD_INSTALLPATH_MODULES=$HOME/hmns/modules +``` + +To make sure we didn't make any silly mistakes, we double check using `eb --show-config`: + + +```shell +$ eb --show-config +# +# Current EasyBuild configuration +# (C: command line argument, D: default value, E: environment variable, F: configuration file) +# +buildpath (E) = /tmp/example +containerpath (E) = /home/example/easybuild/containers +installpath (E) = /home/example/easybuild +installpath-modules (E) = /home/example/hmns/modules +installpath-software (E) = /easybuild/software +module-naming-scheme (E) = HierarchicalMNS +packagepath (E) = /home/example/easybuild/packages +prefix (E) = /home/example/easybuild +repositorypath (E) = /home/example/easybuild/ebfiles_repo +robot-paths (D) = /home/example/.local/easybuild/easyconfigs +sourcepath (E) = /home/example/easybuild/sources +``` + +There are a couple of things worth pointing out here: + +* We have defined the `module-naming-scheme` configuration setting to `HierarchicalMNS`, + which makes EasyBuild use the included standard hierarchical module naming scheme (the classic + core / compiler / MPI one we discussed above). +* We have specified *different* locations for the software (via `installpath-software`) + and the module files (via `installpath-modules`). This is important because we want to + reuse the software that is already installed in `/easybuild/software` while we want to + generate an entirely new module tree for it (in `$HOME/hmns/modules`). + +The other configuration settings are the same as before, and mostly irrelevant for this example. + +### Generating modules for HDF5 + +Let us now generate a hierarchical module tree for `HDF5` and all of its dependencies, +including the toolchain. That sounds complicated, and it sort of is since there are +a lot of details you have to get right for the module hierarchy to works as intended, +but EasyBuild can do all the hard work for us. + +The steps we will have to go through are: + +* Tell EasyBuild we want to "install" the `HDF5-1.12.1-gompi-2021b.eb` easyconfig file; +* Enable dependency resolution via `--robot`; +* Assuming the software would have been installed already with the default naming scheme + in a different module directory, instruct EasyBuild to only generate the module files, + not to install the software (since it is + there already in `/easybuild/software`), via the `--module-only` option. + +These steps translate to this single `eb` command: + +``` +$ eb HDF5-1.12.1-gompi-2021b.eb --robot --module-only +... +== building and installing MPI/GCC/11.2.0/OpenMPI/4.1.1/HDF5/1.12.1... +... +== sanity checking... +== cleaning up [skipped] +== creating module... +... +== COMPLETED: Installation ended successfully (took 9 sec) +... +== Build succeeded for 41 out of 41 +``` + +This should take a couple of minutes in total, for generating 41 modules. +Remember that this also includes generating module files for the toolchain and +all of its components. + +In addition, there is a bit more going on one that just generating module files, +since the sanity check step is still being run for each of the installations +when using `--module-only` to ensure the installation is actually functional. +After all, there is no point in generating a module for an obviously broken +installation... + +### Loading the HDF5 module + +After generating the hierarchical module tree for HDF5, how do we access the HDF5 installation through it? + +Here's what the module tree looks like on disk: + +``` +$ ls $HOME/hmns/modules/all +Compiler Core MPI +``` + +Those are basically the 3 levels in the module hierarchy we showed in our example earlier. + +The starting point is the top level of the module hierarchy named `Core`: + +``` +module use $HOME/hmns/modules/all/Core +``` + +Let us see what that gives us in terms of available modules: + +``` +$ module avail + +------------------------ /home/easybuild/hmns/modules/all/Core ------------------------- + binutils/2.37 GCC/11.2.0 gompi/2021b OpenSSL/1.1 + Bison/3.8.2 GCCcore/11.2.0 M4/1.4.19 pkg-config/0.29.2 + flex/2.6.4 gettext/0.21 ncurses/6.2 zlib/1.2.11 +``` + + +Nice and short module names, but only a limited set of them. + +We know a module file exists for `HDF5`, but we can't see it yet (and hence +we can't load it either). + +``` +$ module avail HDF5 +No module(s) or extension(s) found! +Use "module spider" to find all possible modules and extensions. +``` + +Let us see if `module spider` is of any help, as "`module avail`" so kindly suggests: + +``` +$ module spider HDF5 +... + + You will need to load all module(s) on any one of the lines below + before the "HDF5/1.12.1" module is available to load. + + GCC/11.2.0 OpenMPI/4.1.1 +``` + +This tells us we need to load two gateway modules before we can load the module +for HDF5. + +Let us start with loading the `GCC` compiler module: + +``` +module load GCC/11.2.0 +``` + +And then check again which modules are available: + +``` +$ module avail + +--------------------------- /home/easybuild/hmns/modules/all/Compiler/GCC/11.2.0 --------------------------- + OpenMPI/4.1.1 + +------------------------- /home/easybuild/hmns/modules/all/Compiler/GCCcore/11.2.0 ------------------------- + Autoconf/2.71 flex/2.6.4 (D) libreadline/8.1 pkg-config/0.29.2 (D) + Automake/1.16.4 groff/1.22.4 libtool/2.4.6 PMIx/4.1.0 + Autotools/20210726 help2man/1.48.3 libxml2/2.9.10 Szip/2.1.1 + binutils/2.37 (L,D) hwloc/2.5.0 M4/1.4.19 (D) UCX/1.11.2 + Bison/3.7.6 libevent/2.1.12 ncurses/6.2 (D) xorg-macros/1.19.3 + DB/18.1.40 libfabric/1.13.2 numactl/2.0.14 XZ/5.2.5 + expat/2.4.1 libpciaccess/0.16 Perl/5.34.0 zlib/1.2.11 (L,D) + +---------------------------------- /home/easybuild/hmns/modules/all/Core ----------------------------------- + binutils/2.37 GCC/11.2.0 (L) gompi/2021b OpenSSL/1.1 + Bison/3.8.2 (D) GCCcore/11.2.0 (L) M4/1.4.19 pkg-config/0.29.2 + flex/2.6.4 gettext/0.21 ncurses/6.2 zlib/1.2.11 +``` + +Good news, we now have additional modules available! + +The compiler level of our hierarchy actually consists of two directories here: `Compiler/GCCcore/11.2.0` +and `Compiler/GCC/11.2.0`. The modules in the `GCCcore` directory are ones we can use in other compiler +toolchains that use GCC 10.2.0 as a base compiler (the details of that are out of scope here). + +The module we are interested in is `OpenMPI/4.1.1`, which is another gateway module. + +Remember that the "`module spider`" output told us that there does indeed exist a module for `HDF5`, but that +we need to load *both* the `GCC/11.2.0` and `OpenMPI/4.1.1` modules first. + +So, let us do exactly that (remember that `GCC/11.2.0` is already loaded): + +``` +module load OpenMPI/4.1.1 +``` + +If you now check the output of "`module avail`" again, you should see the `HDF5/1.12.1` module: + +``` +$ module avail + +-------- /home/easybuild/hmns/modules/all/MPI/GCC/11.2.0/OpenMPI/4.1.1 ------- + HDF5/1.12.1 + +------------ /home/easybuild/hmns/modules/all/Compiler/GCC/11.2.0 ------------ + OpenMPI/4.1.1 (L) + +... +``` + +To use HDF5, we need to load this `HDF5/1.12.1` module. We can verify that the installation works +using one of the commands provided by HDF5, `h5dump` for example: + +``` +module load HDF5/1.12.1 +``` + +``` +$ h5dump --version +h5dump: Version 1.12.1 +``` + +If you now check which modules are loaded via "`module list`", you will notice that all module names +and nice and short now, which is one of the advantages of using a hierarchical module tree: + +```shell +$ module list + +Currently Loaded Modules: + 1) GCCcore/11.2.0 5) numactl/2.0.14 9) hwloc/2.5.0 13) libfabric/1.13.2 17) HDF5/1.12.1 + 2) zlib/1.2.11 6) XZ/5.2.5 10) OpenSSL/1.1 14) PMIx/4.1.0 + 3) binutils/2.37 7) libxml2/2.9.10 11) libevent/2.1.12 15) OpenMPI/4.1.1 + 4) GCC/11.2.0 8) libpciaccess/0.16 12) UCX/1.11.2 16) Szip/2.1.1 +``` + + +--- + +*[[next: GitHub integration]](3_05_github_integration.md)* diff --git a/docs/2022-CSC_and_LO/3_Advanced/3_05_github_integration.md b/docs/2022-CSC_and_LO/3_Advanced/3_05_github_integration.md new file mode 100644 index 00000000..9900e1b3 --- /dev/null +++ b/docs/2022-CSC_and_LO/3_Advanced/3_05_github_integration.md @@ -0,0 +1,461 @@ +# GitHub integration to facilitate contributing to EasyBuild + +*[[back: Module naming schemes]](3_04_module_naming_schemes.md)* + +--- + +!!! Warning "Preliminary note for LUMI and many other Cray systems" + + Not everything in this section is as useful for EasyBuild installations that build + on the Cray Programming Environment. E.g., the default easyconfig repository does + not contain easyconfigs for the Cray PE. + + Moreover, due to the restricted operating system on the compute nodes of big + Cray systems such as the EX series (COS instead of the full SUSE linux), + the GitHub integration does not work on the compute nodes of a typical setup as + one of the required Python packages fails to work as it needs the DBus daemon. + + Since GitHub integration is not yet really used on LUMI, this section is not fully + tested but left in the tutorial for completeness. + +To contribute changes to the EasyBuild code (framework or easyblocks) or easyconfigs, +you will need to be a bit familiar with Git and GitHub. Or maybe not? + +## Manual contribution procedure + +0) Create and setup a [GitHub account](https://github.com/join) (and register your SSH public key); + +1) Clone and fork the appropriate GitHub repository, for example when contributing an easyconfig file: + +```shell +git clone git@github.com:easybuilders/easybuild-easyconfigs.git +cd easybuild-easyconfigs +git remote add my_fork git@github.com:your_github_account/easybuild-easyconfigs.git +``` + +**You should change '`your_github_account`' in the last line to your own GitHub user name!** + +2) Create and check out a new branch, starting from the (up-to-date) ``develop`` branch: + +``` +git checkout develop +git pull origin develop +git checkout -b example +``` + +3) Stage the changes you want to contribute, after you make sure that your easyconfig file has the +[correct filename](../../1_Intro/1_08_basic_usage/#easyconfig-filenames), and that it's located in the appropriate directory. + +```shell +mkdir -p easybuild/easyconfigs/e/example/ +mv example.eb easybuild/easyconfigs/e/example/example-1.2.3-GCC-9.3.0.eb +git add easybuild/easyconfigs/e/example/example-1.2.3-GCC-9.3.0.eb +``` + +4) Commit those changes with a sensible commit message: + +```shell +git commit -m "This is just an example" +``` + +5) Push your branch to your fork of the repository on GitHub: + +```shell +git push my_fork example +``` + +6) Open the pull request through the GitHub web interface, making sure that: + +* the target branch is correct (should be `develop`); +* an appropriate title is used; +* a short description of the changes is provided; +* the changes are indeed the ones you want to propose; +* clicking the (correct) green button; + ++ +That didn't exactly motivate you to contribute, did it... + +## Github integration features + +Over the years we noticed that some people were keen on contributing to EasyBuild, +but they were not very familiar with Git or GitHub. That meant they had to overcome a +relatively steep learning curve before they could contribute... + ++ +In addition, the contribution workflow can be a bit daunting and time consuming, +even if you're already familiar with the procedure. You will have dozens of +branches flying around in no time, and if you get stuck in a weird corner +with `git` you may quickly end up demotivated. + +This is frustrating not only for the people who wanted to contribute but +also for the EasyBuild maintainers, and it doesn't agree with the philosophy of +a project that aims to *automate* tedious software installation procedures. + +At the end of 2015 efforts were made to tackle this issue by implementing +GitHub integration features in EasyBuild, which automate the contribution +workflow by running `git` commands and interacting with the [GitHub API](https://docs.github.com/en/rest). + +We will briefly go over some of these features here, but they are also covered in detail [in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Integration_with_GitHub.html). + +### Requirements & configuration + +First of all, the GitHub integration features impose a couple of additional [requirements](https://docs.easybuild.io/en/latest/Integration_with_GitHub.html) +and configuration. + + +**Additional dependencies** + +Both the `GitPython` and `keyring` Python packages as well as the `keyrings.cryptfile` add-on package must be installed. + +!!! Note + You may experiences problems installing the ``cryptography`` Python packages, + which is a dependency of keyring. The underlying cause is that you need to have + the [``Rust``](https://www.rust-lang.org/) compiler installed to install the latest version + of ``cryptography`` (see [here](https://github.com/pyca/cryptography/issues/5771)). + + You can work around this issue using: + + ```shell + pip3 install --user 'cryptography<3.4' + ``` + +**SSH public key in GitHub account** + +You need to have a GitHub account that has your SSH public key registered in it +(via [https://github.com/settings/keys](https://github.com/settings/keys)). + +If you need to generate an SSH key pair, you can run the following command: + +```shell +ssh-keygen -t rsa -b 4096 +``` + +You can copy the SSH public key from the output of this command: + +```shell +cat .ssh/id_rsa.pub +``` + + +**Forked repository in GitHub** + +In addition, you must have *forked* the EasyBuild repository you want to contribute to +(for example [https://github.com/easybuilders/easybuild-easyconfigs](https://github.com/easybuilders/easybuild-easyconfigs)). + +**EasyBuild configuration, incl. GitHub token** + +You also have to configure EasyBuild a bit more, so it knows about your +GitHub user name *and* has a GitHub token available in order to perform actions +in GitHub with your credentials. + +To do this, you should define the `github-user` configuration option and +run the "`eb --install-github-token`" command: + +```shell +# replace 'ebtutorial' with your own GitHub username! +$ export EASYBUILD_GITHUB_USER=ebtutorial +$ eb --install-github-token +``` + +To create a GitHub token: + +* Visit [https://github.com/settings/tokens](https://github.com/settings/tokens). +* Click *"Personal access tokens"*. +* Click followed by *"Generate new token"*. +* Give the token a name (for example *"Token for EasyBuild"*). +* Select both the '`repo`' and '`gist`' scopes. +* Click the green *"Generate token"* button. +* Copy the generated token. +* Paste the token when asked by `--install-github-token` (and hit *Enter*). +* Enter a password to encrypt your GitHub token. + +The output should look something like this: + +```shell +$ eb --install-github-token +== temporary log file in case of crash /tmp/eb-9z0bdve9/easybuild-hfpti62w.log +Token: +Validating token... +Token seems to be valid, installing it. +Please set a password for your new keyring: +Please confirm the password: +Token 'fed..987' installed! +``` + + +**Checking status of GitHub integration** + +You can check the status of the GitHub integration using "`eb --check-github`": + +```shell +$ eb --check-github +== temporary log file in case of crash /tmp/eb-4ckdlyfy/easybuild-gp69ev2w.log + +Checking status of GitHub integration... + +Making sure we're online...OK + +* GitHub user...ebtutorial => OK +Please enter password for encrypted keyring: +* GitHub token...fed..987 (len: 40) => OK (validated) +* git command...OK ("git version 1.8.3.1; ") +* GitPython module...OK (GitPython version 3.1.3) +* push access to ebtutorial/easybuild-easyconfigs repo @ GitHub...OK +* creating gists...OK +* location to Git working dirs... not found (suboptimal) + +All checks PASSed! + +Status of GitHub integration: +* --from-pr: OK +* --new-pr: OK +* --review-pr: OK +* --update-pr: OK +* --upload-test-report: OK +``` + +If you see '`OK`' for each of the status checks, you're all set +to try out the GitHub integration features! + +!!! Note + If your SSH private key is protected with a password, you may need + to enter your password a couple of times when running "`eb --check-github`". + + You can avoid this by + [using an SSH agent](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent). + +!!! Note + If you see the ``push access`` check fail with ``Failed to fetch branch 'main'``, + you will need to rename the ``master`` branch in your fork of the ``easybuild-easyconfigs`` + repository from ``master`` to ``main`` (this is required since EasyBuild v4.3.3). + + This can be done via the *pencil* icon at + [https://github.com/YOUR_GITHUB_ACCOUNT/easybuild-easyconfigs/branches](https://github.com/YOUR_GITHUB_ACCOUNT/easybuild-easyconfigs/branches) + (replace ``YOUR_GITHUB_ACCOUNT`` with the name of your GitHub account in this URL). + +### Creating pull requests + +The GitHub integration in EasyBuild allows you to **create pull requests +using the `eb` command**, without even leaving your shell environment. +How cool is that‽ + +To create a pull request to the `easybuild-easyconfigs` repository, +you can either do it in a single go by +running "`eb --new-pr`" and passing it one or more easyconfig files to add +into the pull request. + +The more detailed option is to first create a branch in your repository fork +in GitHub via "`eb --new-branch-github`" and then later open the pull request +via "`eb --new-pr-from-branch`". This method can be useful when preparing multiple +interdependent pull requests, or to check whether your changes pass the unit tests +(which are run automatically in the GitHub Actions CI environment for +all branches pushed to your fork). + +The `--new-pr` option can also be used to open pull requests to the easyblocks +and framework repositories, and it will even automatically determine the target +repository based on the contents of the files you provide. Whoa! + +You can control the target repository for your pull request using +`--pr-target-account` (default is `easybuilders`) and `--pr-target-repo`. + +If you want you can carefully double check your work before actually +opening the pull request by doing a dry run via "`eb --dry-run --new-pr`" +or "`eb -D --new-pr`". + +Finally, you can use "`eb --preview-pr`" to see how the easyconfig files +for which you plan to create a pull request differ from existing easyconfig +files. + +### Updating pull requests + +To update an existing pull request with additional changes +you can use "`eb --update-pr`" and pass the pull request ID, +alongside the paths to the updated files. + +If you have only created a branch (for example via `eb --new-branch-github`) +you can update it via `--update-branch-github` in the same way, +passing the branch name instead of a pull request ID. + +### Using a pull request + +Next to creating and updating branches and pull requests +you can also *use* easyconfig files and easyblocks from a pull request, +regardless of its status (open, merged, or closed). This is particularly +useful when testing contributions, or to install software for which +support is not yet included in the latest EasyBuild release. + +Using the `--from-pr` option you can install easyconfig files from the +pull request with specified ID. By default all easyconfig files that are +touched by the pull request will be installed, but you can specify +particular ones to use as well. It is generally advised to also use the +`--robot` option to ensure that the easyconfig files are installed in the +correct order with respect to dependencies. + +Similarly, using a new or updated easyblock from a pull request is as simple +as using the `--include-easyblocks-from-pr` option. And of course you can +combine it with `--from-pr`! + +Via `--upload-test-report` you can let EasyBuild submit a comment into the +easyconfig pull request to show that the installation worked on your system. This is +useful for others to know, in particular EasyBuild maintainers, since the comment +will include information about your system (OS, processor, etc.) and your EasyBuild configuration. + +## Demo + +!!! Warning "Not currently suited for LUMI" + + This is text from a previous (non-Cray) version of the tutorial. The demo does require + a number of files not installed on LUMI after going through this tutorial, + and someone who can master the repository used to clean up again. + However, having a look at the output (which was generated in the spring of 2021 on + the CSC system puhti) still offers some information, so the demo was left in this + version of the tutorial. + +That is a lot to digest, so let us make this a bit more concrete with an example: +we will open a pull request for the [`eb-tutorial` example software](../../2_Using/2_02_creating_easyconfig_files/#example) to *a fork* of the [`easybuild-easyconfigs` repository](https://github.com/easybuilders/easybuild-easyconfigs) using the `eb` command, +and submit a test report in it. + +!!! Note + Make sure that you have correctly configured the GitHub integration, + [see above](#requirements-configuration). + +### Creating pull request + +We first configure EasyBuild to target the `ebtutorial` GitHub account rather +than the default `easybuilders` GitHub organisation, +by defining the `pr-target-account` configuration setting: + +```shell +export EASYBUILD_PR_TARGET_ACCOUNT=ebtutorial +``` + +In the output of "`eb --show-config`" you should see a line like this: + +``` +pr-target-account (E) = ebtutorial +``` + +We only do this to avoid that lots of pull requests for the `eb-tutorial` +example software are opened in the [central easyconfigs repository](https://github.com/easybuilders/easybuild-easyconfigs). + +Opening a pull request is as simple as running "`eb --new-pr`" and passing +the easyconfig file: + +```shell +$ eb --new-pr example.eb +== temporary log file in case of crash /tmp/eb-ggr6scbq/easybuild-hnk271xj.log +== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... +== fetching branch 'develop' from https://github.com/ebtutorial/easybuild-easyconfigs.git... +== copying files to /tmp/eb-ggr6scbq/git-working-dirxwk1fzaw/easybuild-easyconfigs... +== pushing branch '20200622095415_new_pr_eb-tutorial100' to remote 'github_ebtutorial_qgtfU' (git@github.com:ebtutorial/easybuild-easyconfigs.git) +Enter passphrase for key '/home/example/.ssh/id_rsa': +Please enter password for encrypted keyring: + +Opening pull request +* target: ebtutorial/easybuild-easyconfigs:develop +* from: ebtutorial/easybuild-easyconfigs:20200622095415_new_pr_eb-tutorial100 +* title: "{tools}[GCC/10.2.0] eb-tutorial v1.0.1" +* labels: new +* description: +""" +(created using `eb --new-pr`) + +""" +* overview of changes: + easybuild/easyconfigs/e/eb-tutorial/eb-tutorial-1.0.1-GCC-10.2.0.eb | 26 ++++++++++++++++++++++++++ + 1 file changed, 26 insertions(+) + +Opened pull request: https://github.com/ebtutorial/easybuild-easyconfigs/pull/ +== Temporary log file(s) /tmp/eb-ggr6scbq/easybuild-hnk271xj.log* have been removed. +== Temporary directory /tmp/eb-ggr6scbq has been removed. +``` + +Take a moment to grasp what we did here: we ran **a single `eb` command** which +took care of the **[whole contribution procedure](#contribution-procedure)** for us, including: + +* Cloning the `easybuilders/easybuild-easyconfigs` repository and checking out the `develop` branch (in a temporary + directory); +* Picking a sensible name for a branch and creating it; +* Adding the `eb-tutorial` easyconfig file to the branch, in the correct location + (`easybuild/easyconfigs/e/eb-tutorial/`) and with the correct filename (`eb-tutorial-1.0.1-GCC-10.2.0.eb`); +* Pushing the branch to our fork (`example/easybuild-easyconfigs`); +* Actually opening the pull request, using an informative title. + +That is so... easy! + +This feature not only *significantly* lowers the bar for contributing, +it also saves quite a bit of time since you don't need to double check +various details (like targeting the `develop` branch) or spend time on +coming up with a nice looking title or funny branch name (although you +still can if you really want to). + +There are a couple of nice side effects too, like not having any local branches +to tidy up on once the pull request gets merged (since `--new-pr` created the +branch only in a temporary directory). + +If many contributions are made via `--new-pr` it also simplifies the task +of EasyBuild maintainers, since pull requests opened this way have a particular +structure to them and thus are easier to digest because they look familiar. + +### Uploading test report + +After opening the pull request, we should also upload a test report to show that the installation is working. +This is just as easy as creating the pull request. + +First make sure that the pre-installed software in the prepared environment +is available, since the required dependencies for `eb-tutorial` are already +installed there: + +```shell +module use /easybuild/modules/all +``` + +You can verify which dependencies are still missing using `--from-pr` combined with `--missing`: + +```shell +# change '1' to the ID of your own pull request (see output of --new-pr) +$ eb --from-pr 1 --missing +== temporary log file in case of crash /tmp/eb-ioi9ywm1/easybuild-e3v0xa1b.log +Please enter password for encrypted keyring: +== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... + +1 out of 20 required modules missing: + +* eb-tutorial/1.0.1-GCC-10.2.0 (eb-tutorial-1.0.1-GCC-10.2.0.eb) +``` + +Uploading a test report boils down to combining `--from-pr` with `--upload-test-report`: + +```shell +# change '1' to the ID of your own pull request (see output of --new-pr) +$ eb --rebuild --from-pr 1 --upload-test-report +Please enter password for encrypted keyring: +... +== processing EasyBuild easyconfig /tmp/eb-bnb1pv3n/files_pr65/e/eb-tutorial/eb-tutorial-1.0.1-GCC-10.2.0.eb +== building and installing eb-tutorial/1.0.1-GCC-10.2.0... +... +== COMPLETED: Installation ended successfully (took 2 sec) +... +Adding comment to easybuild-easyconfigs issue #65: 'Test report by @ebtutorial +**SUCCESS** +Build succeeded for 1 out of 1 (1 easyconfigs in this PR) +example - Linux centos linux 7.8.2003, x86_64, Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz (haswell), Python 3.6.8 +See https://gist.github.com/f7c74159c809029afd99e30e4d994ef1 for a full test report.' +== Test report uploaded to https://gist.github.com/f7c74159c809029afd99e30e4d994ef1 and mentioned in a comment in easyconfigs PR#1 +``` + +Note that we may need to use `--rebuild` here since `eb-tutorial` may already be installed. + +This results in a comment being added to the pull request: + ++ +The gist linked from this comment provides more detailed information: + ++ +--- + +*[[next: Additional reading]](../4_00_additional_reading.md)* diff --git a/docs/2022-CSC_and_LO/3_Advanced/index.md b/docs/2022-CSC_and_LO/3_Advanced/index.md new file mode 100644 index 00000000..d7831506 --- /dev/null +++ b/docs/2022-CSC_and_LO/3_Advanced/index.md @@ -0,0 +1,15 @@ +# Part III: Advanced topics + +*[[back to start page]](index.md)* + +In this section we mostly cover "good to know that they exist" features as they are not used +on LUMI, or not really accessible to regular user installations that are performed with the +LUMI ``EasyBuild-user`` module. Hooks are used on LUMI, but it is not really advised to +overwrite the centrally defined hooks with a local file. And the whole structure of the +EasyBuild integration is also set up to make use of the GitHub integration in the future. + +* [Using EasyBuild as a library](3_01_easybuild_library.md) +* [Using hooks to customise EasyBuild](3_02_hooks.md) +* [Submitting installations as Slurm jobs](3_03_slurm_jobs.md) +* [Module naming schemes (incl. hierarchical)](3_04_module_naming_schemes.md) +* [GitHub integration to facilitate contributing to EasyBuild](3_05_github_integration.md) diff --git a/docs/2022-CSC_and_LO/4_00_additional_reading.md b/docs/2022-CSC_and_LO/4_00_additional_reading.md new file mode 100644 index 00000000..14f8454c --- /dev/null +++ b/docs/2022-CSC_and_LO/4_00_additional_reading.md @@ -0,0 +1,54 @@ +# Additional reading + +*[[back: GitHub integration]](3_Advanced/3_05_github_integration.md)* + +--- + +## EasyBuild + +- [EasyBuild documentation](https://docs.easybuild.io/en/latest/#) +- [EasyBuild web site](https://easybuild.io/) containing links to much of the recent information on + EasyBuild. +- [EasyBuild YouTube channel](https://www.youtube.com/c/EasyBuilders) +- This tutorial is an evolution of the + [EasyBuild tutorial prepared for the LUMI User Support Team, spring '21](https://easybuilders.github.io/easybuild-tutorial/2021-lust/) + given by Kenneth Hoste (UGent, EasyBuild lead developer) and Luca Marsella (CSCS) + - [Recordings are available on YouTube](https://www.youtube.com/watch?v=JTRw8hqi6x0&list=PLhnGtSmEGEQh573bk3BeOj_KCRBBiA5OT) +- The EasyBuild setup on LUMI is partly insprired on the setup used at CSCS on their Cray systems + - [Section on the CSCS setup](https://easybuilders.github.io/easybuild-tutorial/2021-lust/cray/easybuild_at_cscs/) + from the LUST EasyBuild tutorial spring '21. + - [CSCS site presentation](https://easybuild.io/eum21/#eb-site-talk-cscs) + during the [6th EasyBuild User Meeting in 2021](https://easybuild.io/eum21/) + - CSCS site presentation ([video](https://www.youtube.com/watch?v=rhoHmUhzSGw&list=PLhnGtSmEGEQidEM8MZKkOaVutgt9WmqI0)/[slides](https://users.ugent.be/~kehoste/eum20/eum20_11_luca_cscs.pdf)) + during the [5h EasyBuild User Meeting in 2020](https://github.com/easybuilders/easybuild/wiki/5th-EasyBuild-User-Meeting) + - CSCS site presentation ([video](https://www.youtube.com/watch?list=PLhnGtSmEGEQhEjG5hD70tRwL3n58aWNHc&t=1451&v=qLuHeaFmydM&feature=youtu.be)) + during the [4h EasyBuild User Meeting in 2020](https://github.com/easybuilders/easybuild/wiki/4th-EasyBuild-User-Meeting) + +## HPE Cray Programming Environment + +- [LUMI documentation: "Developing" section](https://docs.lumi-supercomputer.eu/development/) +- The Cray PE is mostly documented through man pages. There used to be some documentation on the + Cray web site also but the documentation system got reworked after the merger with HPE. + The documentation is now in the [HPE Support Centre](https://support.hpe.com/) where it + is very difficult to find the right version of the documents. +- The [PE-Cray](https://github.com/PE-Cray) GitHub project also provides some additional + documentation, including + - [some whitepapers](https://github.com/PE-Cray/whitepapers), + - [documentation for Cray OpenSMHEMX](https://github.com/PE-Cray/cray-openshmemx) and + - [information about Cray DSMML](https://github.com/PE-Cray/cray-dsmml) + though much of that information ia also not updated regularly anymore. + + +## LUMI + +- [LUMI web site](https://lumi-supercomputer.eu/) +- [LUMI User Documentation](https://docs.lumi-supercomputer.eu/) + - [Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/) + - [Organisation of the software stack](https://docs.lumi-supercomputer.eu/computing/softwarestacks/) + - [EasyBuild on LUMI](https://docs.lumi-supercomputer.eu/software/installing/easybuild/) +- [EasyBuild on LUMI presentation](https://easybuild.io/eum22/#lumi) + ([video](https://www.youtube.com/watch?v=hZezVG6lJNk&list=PLhnGtSmEGEQgCneeSQvYoIZrbv7wIKlo2)/[slides](https://easybuild.io/eum22/001_eum22_EasyBuild_on_LUMI.pdf)) + during the [7th EasyBuild User Meeting in 2022](https://easybuild.io/eum22/) + + +*[[next: Overview]](index.md)* diff --git a/docs/2022-CSC_and_LO/index-lust.md b/docs/2022-CSC_and_LO/index-lust.md new file mode 100644 index 00000000..b8cee2ab --- /dev/null +++ b/docs/2022-CSC_and_LO/index-lust.md @@ -0,0 +1,123 @@ +<<<<<<< HEAD +# EasyBuild tutorial for CSC and the LUMI consortium +======= +# EasyBuild tutorial for LUST + +Overview page of the introductory tutorial on [EasyBuild](https://easybuild.io) for the CSC +and Local Organizations who want to contribute to the LUMI EasyBuild setup. +>>>>>>> 5573462b (Initial commit, already contains the structure to work on the first tutorial.) + +This tutorial is based extensively on the more generic EasyBuild tutorials build by +the EasyBuild community and maintained on +[the EasyBuild tutorial site](https://easybuilders.github.io/easybuild-tutorial/). +See that site for [credits to the contributors of those tutorials](https://easybuilders.github.io/easybuild-tutorial/#contributors). + +<<<<<<< HEAD +## Scope + +This is an introductory tutorial to [EasyBuild](https://easybuild.io), +a command line tool for installing (scientific) software on High Performance Computing (HPC) systems. +This tutorial is specifically for EasyBuild as implemented on [LUMI](https://lumi-supercomputer.eu) +and has been developed for CSC and the Local Organisations within the LUMI consortium. Yet +much of the material is useful to a broader community of EasyBuild users on Cray systems +or even EasyBuild users in general. +For more generic EasyBuild tutorials, see the [EasyBuild tutorial site](https://easybuilders.github.io/easybuild-tutorial/). + +This tutorial aims to explain the core concepts of EasyBuild, +get you started with using it, make you familiar with some of the features it provides, +and show how it is used on LUMI to maintain the central software stacks and offer the users +an easy environment to install packages on top of the central stack and thus create their own +customised environment. + +Through hands-on exercises and demos, you will learn how EasyBuild can help you +to get scientific software installed in an efficient way. + + +## Intended audience + +This tutorial is primarily intended for people new to EasyBuild, but even if you're already familiar +with the project it could be interesting to step through it. + +Our main target audience includes: + +- application experts in LUST and the local organizations who want to contribute to the + software stack on LUMI or support their users; +- developers who want to make their developments available to LUMI users; +- advanced users who want to customize available build recipes or develop their own recipes. + + +## Prerequisites + +We expect you to be (a little bit) familiar with: + +- using a Linux command line interface; +- the (absolute) basics of compiling software from source. + +EasyBuild requires: + +- GNU/Linux (any distribution), +- Python 2.7 or 3.5+, though a fairly recent version of Python 3 is highly recommended, +- an environment modules tool (see the ``module`` command). On LUMI we use [Lmod](https://lmod.readthedocs.io), + a modern environment modules tool implemented in Lua. + +However, the LUMI version of the tutorial is currently specifically for the Cray Programming Environment which is not +freely available, so unless you have access to a system with this environment you cannot really do local development. + + +## Contents + +- [Part I: **Introduction to EasyBuild on Cray systems**](1_Intro/index.md) + - [What is EasyBuild?](1_Intro/1_01_what_is_easybuild.md) + - [The Lmod module system](1_Intro/1_02_Lmod.md) + - [The HPE Cray Programming Environment](1_Intro/1_03_CPE.md) + - [LUMI software stacks](1_Intro/1_04_LUMI_software_stack.md) + - [Terminology](1_Intro/1_05_terminology.md) + - [Installation](1_Intro/1_06_installation.md) + - [Configuration](1_Intro/1_07_configuration.md) + - [Basic usage](1_Intro/1_08_basic_usage.md) *(hands-on)* +- [Part II: **Using EasyBuild**](2_Using/index.md) + - [Troubleshooting](2_Using/2_01_troubleshooting.md) *(hands-on)* + - [Creating easyconfig files](2_Using/2_02_creating_easyconfig_files.md) *(hands-on)* + - [Using external modules from the Cray PE](2_Using/2_03_external_modules.md) + - [Implementing easyblocks](2_Using/2_04_implementing_easyblocks.md) *(hands-on)* +- [Part III: **Advanced topics**](3_Advanced/index.md) + - [Using EasyBuild as a library](3_Advanced/3_01_easybuild_library.md) + - [Using hooks to customise EasyBuild](3_Advanced/3_02_hooks.md) + - [Submitting installations as Slurm jobs](3_Advanced/3_03_slurm_jobs.md) + - [Module naming schemes (incl. hierarchical)](3_Advanced/3_04_module_naming_schemes.md) + - [GitHub integration to facilitate contributing to EasyBuild](3_Advanced/3_05_github_integration.md) +- [**Additional reading**](4_00_additional_reading.md) +======= +## Contents + +TODO + + + + + + +## OLD TEXT + +- [Part I: **Introduction to EasyBuild**](part1_intro.md) *(Tue March 9th 2021, 9am-12 CET)* + * [What is EasyBuild?](what_is_easybuild.md) + * [Terminology](terminology.md) + * [Installation](installation.md) *(hands-on)* + * [Configuration](configuration.md) *(hands-on)* + * [Basic usage](basic_usage.md) *(hands-on)* +- [Part II: **Using EasyBuild**](part2_using.md) *(Tue March 23rd 2021, 9am-12 CET)* + * [Troubleshooting](troubleshooting.md) *(hands-on)* + * [Creating easyconfig files](creating_easyconfig_files.md) *(hands-on)* + * [Implementing easyblocks](implementing_easyblocks.md) *(hands-on)* +- [Part III: **Advanced topics**](part3_advanced.md) *(Tue March 30th 2021, 9am-12 CEST)* + * [Using EasyBuild as a library](easybuild_library.md) *(hands-on)* + * [Using hooks to customise EasyBuild](hooks.md) *(hands-on)* + * [Submitting installations as Slurm jobs](slurm_jobs.md) *(hands-on)* + * [Module naming schemes (incl. hierarchical)](module_naming_schemes.md) *(hands-on)* + * [GitHub integration to facilitate contributing to EasyBuild](github_integration.md) *(hands-on)* +- [Part IV: **EasyBuild on Cray systems**](part4_cray.md) *(Friday June 18th 2021, 09-12 CEST)* + * [Introduction to Cray Programming Environment](cray/introduction.md) *(hands-on)* + * [Cray External Modules](cray/external_modules.md) *(hands-on)* + * [Cray Custom Toolchains](cray/custom_toolchains.md) *(hands-on)* + * [EasyBuild at CSCS](cray/easybuild_at_cscs.md) *(hands-on)* +>>>>>>> 5573462b (Initial commit, already contains the structure to work on the first tutorial.) diff --git a/docs/2022-CSC_and_LO/index.md b/docs/2022-CSC_and_LO/index.md new file mode 100644 index 00000000..1de6a052 --- /dev/null +++ b/docs/2022-CSC_and_LO/index.md @@ -0,0 +1,81 @@ +# EasyBuild tutorial for CSC and the LUMI consortium + +This tutorial is based extensively on the more generic EasyBuild tutorials build by +the EasyBuild community and maintained on +[this site](https://easybuilders.github.io/easybuild-tutorial/). +See that site for [credits to the contributors of those tutorials](https://easybuilders.github.io/easybuild-tutorial/#contributors). + +## Scope + +This is an introductory tutorial to [EasyBuild](https://easybuild.io), +a command line tool for installing (scientific) software on High Performance Computing (HPC) systems. +This tutorial is specifically for EasyBuild as implemented on [LUMI](https://lumi-supercomputer.eu) +and has been developed for CSC and the Local Organisations within the LUMI consortium. Yet +much of the material is useful to a broader community of EasyBuild users on Cray systems +or even EasyBuild users in general. +For more generic EasyBuild tutorials, see the [EasyBuild tutorial site](https://easybuilders.github.io/easybuild-tutorial/). + +This tutorial aims to explain the core concepts of EasyBuild, +get you started with using it, make you familiar with some of the features it provides, +and show how it is used on LUMI to maintain the central software stacks and offer the users +an easy environment to install packages on top of the central stack and thus create their own +customised environment. + +Through hands-on exercises and demos, you will learn how EasyBuild can help you +to get scientific software installed in an efficient way. + + +## Intended audience + +This tutorial is primarily intended for people new to EasyBuild, but even if you're already familiar +with the project it could be interesting to step through it. + +Our main target audience includes: + +- application experts in LUST and the local organizations who want to contribute to the + software stack on LUMI or support their users; +- developers who want to make their developments available to LUMI users; +- advanced users who want to customize available build recipes or develop their own recipes. + + +## Prerequisites + +We expect you to be (a little bit) familiar with: + +- using a Linux command line interface; +- the (absolute) basics of compiling software from source. + +EasyBuild requires: + +- GNU/Linux (any distribution), +- Python 2.7 or 3.5+, though a fairly recent version of Python 3 is highly recommended, +- an environment modules tool (see the ``module`` command). On LUMI we use [Lmod](https://lmod.readthedocs.io), + a modern environment modules tool implemented in Lua. + +However, the LUMI version of the tutorial is currently specifically for the Cray Programming Environment which is not +freely available, so unless you have access to a system with this environment you cannot really do local development. + + +## Contents + +- [Part I: **Introduction to EasyBuild on Cray systems**](1_Intro/index.md) + - [What is EasyBuild?](1_Intro/1_01_what_is_easybuild.md) + - [The Lmod module system](1_Intro/1_02_Lmod.md) + - [The HPE Cray Programming Environment](1_Intro/1_03_CPE.md) + - [LUMI software stacks](1_Intro/1_04_LUMI_software_stack.md) + - [Terminology](1_Intro/1_05_terminology.md) + - [Installation](1_Intro/1_06_installation.md) + - [Configuration](1_Intro/1_07_configuration.md) + - [Basic usage](1_Intro/1_08_basic_usage.md) *(hands-on)* +- [Part II: **Using EasyBuild**](2_Using/index.md) + - [Troubleshooting](2_Using/2_01_troubleshooting.md) *(hands-on)* + - [Creating easyconfig files](2_Using/2_02_creating_easyconfig_files.md) *(hands-on)* + - [Using external modules from the Cray PE](2_Using/2_03_external_modules.md) + - [Implementing easyblocks](2_Using/2_04_implementing_easyblocks.md) *(hands-on)* +- [Part III: **Advanced topics**](3_Advanced/index.md) + - [Using EasyBuild as a library](3_Advanced/3_01_easybuild_library.md) + - [Using hooks to customise EasyBuild](3_Advanced/3_02_hooks.md) + - [Submitting installations as Slurm jobs](3_Advanced/3_03_slurm_jobs.md) + - [Module naming schemes (incl. hierarchical)](3_Advanced/3_04_module_naming_schemes.md) + - [GitHub integration to facilitate contributing to EasyBuild](3_Advanced/3_05_github_integration.md) +- [**Additional reading**](4_00_additional_reading.md) diff --git a/mkdocs.yml b/mkdocs.yml index 1e3ef2c9..c7c97216 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -33,6 +33,32 @@ nav: - Contributing to EasyBuild: 2022-isc22/contributing.md - Comparison with other tools: 2022-isc22/comparison_other_tools.md - Archive: + - "CSC (spring '22)": + - (overview): 2022-CSC_and_LO/index.md + - Introduction to EasyBuild: + - (overview): 2022-CSC_and_LO/1_Intro/index.md + - What is EasyBuild?: 2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md + - The Lmod module system: 2022-CSC_and_LO/1_Intro/1_02_Lmod.md + - The HPE Cray Programming Environment: 2022-CSC_and_LO/1_Intro/1_03_CPE.md + - LUMI software stacks: 2022-CSC_and_LO/1_Intro/1_04_LUMI_software_stack.md + - Terminology: 2022-CSC_and_LO/1_Intro/1_05_terminology.md + - Installation: 2022-CSC_and_LO/1_Intro/1_06_installation.md + - Configuration: 2022-CSC_and_LO/1_Intro/1_07_configuration.md + - Basic usage: 2022-CSC_and_LO/1_Intro/1_08_basic_usage.md + - Using EasyBuild: + - (overview): 2022-CSC_and_LO/2_Using/index.md + - Troubleshooting: 2022-CSC_and_LO/2_Using/2_01_troubleshooting.md + - Creating easyconfig files: 2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md + - Using external modules from the Cray PE: 2022-CSC_and_LO/2_Using/2_03_external_modules.md + - Implementing easyblocks: 2022-CSC_and_LO/2_Using/2_04_implementing_easyblocks.md + - Advanced topics: + - (overview): 2022-CSC_and_LO/3_Advanced/index.md + - Using EasyBuild as a library: 2022-CSC_and_LO/3_Advanced/3_01_easybuild_library.md + - Using hooks to customise EasyBuild: 2022-CSC_and_LO/3_Advanced/3_02_hooks.md + - Submitting installations as Slurm jobs: 2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md + - Module naming schemes (incl. hierarchical): 2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md + - GitHub integration to facilitate contributing to EasyBuild: 2022-CSC_and_LO/3_Advanced/3_05_github_integration.md + - Additional reading: 2022-CSC_and_LO/4_00_additional_reading.md - "ISC'21 (25 June 2021)": - (overview): 2021-isc21/index.md - Practical info: 2021-isc21/practical_info.md @@ -92,6 +118,33 @@ nav: - Contributing to EasyBuild: 2020-06-isc20/contributing.md - Comparison with other tools: 2020-06-isc20/comparison_other_tools.md - Getting help: 2020-06-isc20/getting_help.md + - "CSC (spring '22)": + - (overview): 2022-CSC_and_LO/index.md + - Introduction to EasyBuild: + - (overview): 2022-CSC_and_LO/1_Intro/index.md + - What is EasyBuild?: 2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md + - The Lmod module system: 2022-CSC_and_LO/1_Intro/1_02_Lmod.md + - The HPE Cray Programming Environment: 2022-CSC_and_LO/1_Intro/1_03_CPE.md + - LUMI software stacks: 2022-CSC_and_LO/1_Intro/1_04_LUMI_software_stack.md + - Terminology: 2022-CSC_and_LO/1_Intro/1_05_terminology.md + - Installation: 2022-CSC_and_LO/1_Intro/1_06_installation.md + - Configuration: 2022-CSC_and_LO/1_Intro/1_07_configuration.md + - Basic usage: 2022-CSC_and_LO/1_Intro/1_08_basic_usage.md + - Using EasyBuild: + - (overview): 2022-CSC_and_LO/2_Using/index.md + - Troubleshooting: 2022-CSC_and_LO/2_Using/2_01_troubleshooting.md + - Creating easyconfig files: 2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md + - Using external modules from the Cray PE: 2022-CSC_and_LO/2_Using/2_03_external_modules.md + - Implementing easyblocks: 2022-CSC_and_LO/2_Using/2_04_implementing_easyblocks.md + - Advanced topics: + - (overview): 2022-CSC_and_LO/3_Advanced/index.md + - Using EasyBuild as a library: 2022-CSC_and_LO/3_Advanced/3_01_easybuild_library.md + - Using hooks to customise EasyBuild: 2022-CSC_and_LO/3_Advanced/3_02_hooks.md + - Submitting installations as Slurm jobs: 2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md + - Module naming schemes (incl. hierarchical): 2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md + - GitHub integration to facilitate contributing to EasyBuild: 2022-CSC_and_LO/3_Advanced/3_05_github_integration.md + - Additional reading: 2022-CSC_and_LO/4_00_additional_reading.md + plugins: # show revision date at bottom of each page - git-revision-date-localized @@ -119,7 +172,11 @@ markdown_extensions: # notes, warnings, hints, ... - admonition # code blocks with syntax highlighting, graphs - - pymdownx.superfences + - pymdownx.superfences: + custom_fences: + - name: mermaid + class: mermaid + format: !!python/name:pymdownx.superfences.fence_code_format # clickable details - pymdownx.details # tabbed contents