diff --git a/README.md b/README.md index f7ed0418..15a32928 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -
+å Welcome to the repository that hosts the sources of the official **[EasyBuild](https://easybuild.io) tutorial**, see https://easybuilders.github.io/easybuild-tutorial. @@ -20,13 +20,17 @@ which makes it very easy to preview the result of the changes you make locally. * Start the MkDocs built-in dev-server to preview the tutorial as you work on it: - make preview + ```bash + make preview + ``` - or + or - mkdocs serve + ```bash + mkdocs serve + ``` - Visit http://127.0.0.1:8000 to see the local live preview of the changes you make. + Visit http://127.0.0.1:8000 to see the local live preview of the changes you make. * If you prefer building a static preview you can use ``make`` or ``mkdocs build``, which should result in a ``site/`` subdirectory that contains the rendered documentation. diff --git a/docs/2021-LUST/index.md b/docs/2021-LUST/index.md new file mode 100644 index 00000000..c2adbc65 --- /dev/null +++ b/docs/2021-LUST/index.md @@ -0,0 +1,5 @@ +# EasyBuild tutorial for LUST + +Overview page of the introductory tutorial on [EasyBuild](https://easybuild.io) for the *[LUMI](https://www.lumi-supercomputer.eu) User Support team (LUST)*. + +The tutorial is available on [the EasyBuilders tutoral web site](https://easybuilders.github.io/easybuild-tutorial/2021-lust/). \ No newline at end of file diff --git a/docs/2021-lust/index.md b/docs/2021-lust/index.md index 53f3b9ac..c2adbc65 100644 --- a/docs/2021-lust/index.md +++ b/docs/2021-lust/index.md @@ -2,24 +2,4 @@ Overview page of the introductory tutorial on [EasyBuild](https://easybuild.io) for the *[LUMI](https://www.lumi-supercomputer.eu) User Support team (LUST)*. -- [Part I: **Introduction to EasyBuild**](part1_intro.md) *(Tue March 9th 2021, 9am-12 CET)* - * [What is EasyBuild?](what_is_easybuild.md) - * [Terminology](terminology.md) - * [Installation](installation.md) *(hands-on)* - * [Configuration](configuration.md) *(hands-on)* - * [Basic usage](basic_usage.md) *(hands-on)* -- [Part II: **Using EasyBuild**](part2_using.md) *(Tue March 23rd 2021, 9am-12 CET)* - * [Troubleshooting](troubleshooting.md) *(hands-on)* - * [Creating easyconfig files](creating_easyconfig_files.md) *(hands-on)* - * [Implementing easyblocks](implementing_easyblocks.md) *(hands-on)* -- [Part III: **Advanced topics**](part3_advanced.md) *(Tue March 30th 2021, 9am-12 CEST)* - * [Using EasyBuild as a library](easybuild_library.md) *(hands-on)* - * [Using hooks to customise EasyBuild](hooks.md) *(hands-on)* - * [Submitting installations as Slurm jobs](slurm_jobs.md) *(hands-on)* - * [Module naming schemes (incl. hierarchical)](module_naming_schemes.md) *(hands-on)* - * [GitHub integration to facilitate contributing to EasyBuild](github_integration.md) *(hands-on)* -- [Part IV: **EasyBuild on Cray systems**](part4_cray.md) *(Friday June 18th 2021, 09-12 CEST)* - * [Introduction to Cray Programming Environment](cray/introduction.md) *(hands-on)* - * [Cray External Modules](cray/external_modules.md) *(hands-on)* - * [Cray Custom Toolchains](cray/custom_toolchains.md) *(hands-on)* - * [EasyBuild at CSCS](cray/easybuild_at_cscs.md) *(hands-on)* +The tutorial is available on [the EasyBuilders tutoral web site](https://easybuilders.github.io/easybuild-tutorial/2021-lust/). \ No newline at end of file diff --git a/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md b/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md new file mode 100644 index 00000000..fa5075f5 --- /dev/null +++ b/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md @@ -0,0 +1,320 @@ +# What is EasyBuild? + +*[[back: Course overview]](../index.md)* + + + +**[EasyBuild](https://easybuild.io)** is a **software build and installation framework** +that manages (scientific) software in an efficient way, without compromising on performance. +Easybuild is especially tailored towards [**High Performance Computing (HPC)**](https://en.wikipedia.org/wiki/Supercomputer) systems, +and provides a **command-line interface** to facilitate the installation and management of a scientific software stack. + +--- + + + + + +## Elevator pitch + +EasyBuild intends to relieve HPC support teams from manually managing +software installations while at the same time **providing a consistent and well performing +scientific software stack** to end users of HPC systems. + +It serves as a **uniform interface for installing scientific software** +and saves valuable time (and frustration) by the automation of tedious, boring and repetitive tasks. + +In addition, EasyBuild can **empower scientific researchers to self-manage their software stack**, +and it can serve as a tool that can be leveraged for **building optimized container images**. + +The project has grown to become a **platform for collaboration** among HPC sites worldwide. + +--- + +On LUMI we pay a lot of attention to empowering users to create their custom software stack which +is best suited for their project without being confronted with hundreds of modules that are not +useful to them. It also enables them to easily customize the build instructions to create a +project-specific build or even to install software with custom patches to adapt to their needs. +For this we offer both [EasyBuild](https://easybuild.io/) and [Spack](https://spack.io/) +as tools but obviously this tutorial concentrates on EasyBuild alone. + +EasyBuild can also be used to [generate container recipes](https://docs.easybuild.io/en/latest/Containers.html) +that will then use EasyBuild when building +the container to install platform-optimised software in the container. However, as that functionality +is still under development on LUMI it is not yet covered in this LUMI-specific version of the tutorial. + + +--- + +## Key features + +EasyBuild is capable of **fully autonomously installing (scientific) software**, +including making sure that all necessary dependencies are installed, +and automatically generating environment module files. + +***No*** **admin privileges are required**: it is sufficient to have write permissions +to the preferred software installation prefix. + +EasyBuild is [**highly configurable**](https://docs.easybuild.io/en/latest/Configuration.html) via configuration files, +environment variables, and command line options. Various elements of the functionality can be +[**dynamically extended**](https://docs.easybuild.io/en/latest/Including_additional_Python_modules.html) via plugins, +and [**hooks**](https://docs.easybuild.io/en/latest/Hooks.html) are available for further site-specific customizations. + +The installation procedure executed by EasyBuild is thoroughly +[**logged**](https://docs.easybuild.io/en/latest/Logfiles.html), and is fully transparent via support for +[**dry runs**](https://docs.easybuild.io/en/latest/Extended_dry_run.html) and +[**tracing**](https://docs.easybuild.io/en/latest/Tracing_progress.html) the software installation procedure +as it is performed. + +EasyBuild supports using a **custom module naming scheme**, allows for +*hierarchical* module naming schemes, and **integrates with various other tools** ranging from resource managers ([Slurm](https://slurm.schedmd.com) and [GC3Pie](https://github.com/gc3pie/gc3pie)), +container tools ([Singularity](https://github.com/apptainer/singularity) and [Docker](https://www.docker.com)), +packaging tools ([FPM](https://fpm.readthedocs.io)), and so on. + +--- + +## Focus points + +EasyBuild was created specifically for **installing scientific software on HPC systems**, +which is reflected in some of the design choices that were made. + + +### Performance + +EasyBuild strongly prefers to **build software from source code**, whenever possible. + +This is important to ensure that the binaries that are installed can maximally exploit +the capabilities of the system architecture on which the software will be run. + +For that same reason, EasyBuild out-of-the-box **optimizes software for the processor architecture of the build host** +by default, via compiler options like ``-march=native`` (GCC), ``-xHost`` (Intel compilers), etc. +This behaviour [can be changed via the ``--optarch`` configuration setting](https://docs.easybuild.io/en/latest/Controlling_compiler_optimization_flags.html). +When using the HPE Cray Programming Environment, the target modules are used instead. +In principle it is possible to cross-compile with EasyBuild, but this comes with some pitfalls. +By default EasyBuild tries to run tests if the build process of a package provides that +functionality, or can run simple tests of its own, and of course these tests will fail if +the host CPU does not support the full instruction set of the target CPU. Secondly, more +and more software has broken build processes where developers thought that it was a good +idea to hard-code options such as ``-march=native`` or ``-xHost`` in a way that EasyBuild +cannot easily overwrite those. + + +### Reproducibility + +In addition to performance, **reproducibility of installations** is a core aspect of EasyBuild. + +Most software installations performed with EasyBuild use a **particular compiler toolchain**, +with which we aim to be in control over the build environment and avoid relying on tools and libraries +provided by the operating system. For similar reasons, we try to **provide all required dependencies through EasyBuild** as well, +with a few notable exceptions, like ``OpenSSL`` for security reasons, and Infiniband and GPU drivers which +are too closely intertwined with the operating system. + +For both toolchains and dependencies, **fixed software versions** are specified in the +easyconfig files. That way, easyconfig files can easily be shared with others: +if they worked for you it is very likely that they will work for others too, because the vast majority of the +software stack is controlled by EasyBuild. + +As we shall see when discussing the HPE Cray Programming Environment, the reproducibility is a bit more limited +in that environment due to the way the programming environment selects shared libraries at runtime. + + +### Community effort + +In a number of different ways, we try to encourage EasyBuild users to **collaborate** and help each other out. + +We actively recommend people to report problems and bugs, to submit ideas for additional features and improvements, +and to [**contribute back**](https://docs.easybuild.io/en/latest/Contributing.html) when possible, be it +by opening pull requests to the [GitHub repositories](https://github.com/easybuilders) or the [documentation](https://docs.easybuild.io). + +The EasyBuild community also provides packages for two standard toolchains, called the +[**common toolchains**](https://docs.easybuild.io/en/latest/Common-toolchains.html). +The so-called ``foss`` is based on free and open software (GNU compilers, FlexiBlas with OpenBLAS backend, LaPACK, FFTW and Open MPI) +while the ``intel`` toolchain uses the full stack of Intel compilers, libraries and MPI. Currently these toolchains +cannot be supported on LUMI though as getting Open MPI to run on LUMI is challenging and as the Intel libraries and +some MPI versions also produce slow or incorrect results on AMD CPUs (and it is not entirely clear if Intel MPI is compatible +with SlingShot 11, though in principle it should). +Focusing the efforts on those two popular common toolchains makes it possible to support a lot of build recipes that are +collected in the [central repository](https://github.com/easybuilders/easybuild-easyconfigs). +They can still serve as a basis to develop build recipes for the HPE Cray Programming Environment. + +Last but not least, EasyBuild provides various [**GitHub integration features**](https://docs.easybuild.io/en/latest/Integration_with_GitHub.html) +that greatly facilitate the contribution process: opening, updating, and testing pull requests, +reviewing incoming contributions, and much more can all be done directly from the EasyBuild +command line. This not only saves time, effort, brain cycles, and mouse clicks for contributors, +but it also makes the review process for *maintainers* significantly easier. +All together this leads to improved stability and consistency. + +--- + +## What EasyBuild is ***not*** + +EasyBuild is ***not*** **YABT (Yet Another Build Tool)**: it does *not* replace established build +tools like CMake or ``make``, it wraps around them. +If the installation procedure of a software package involves running some unholy trinity of tools while whispering +the correct magic incantations, EasyBuild automates this process for you. + +It is ***not*** **a replacement for traditional Linux package managers** like ``yum``, ``dnf`` or ``apt``. +EasyBuild relies on certain tools and libraries provided by the operating system. This includes glibc, OpenSSL, +drivers for Infiniband and GPUs, and so on. It is required that these tools are installed and managed by other means. +This is typically done via the package management tool that comes with your Linux distribution. + +Finally, EasyBuild is ***not*** **a magic solution to all your (software installation) problems**. +You may, and probably still will occasionally, run into compiler errors unless somebody has already taken care of the problem for you. +This is even more likely to happen in the Cray environment as builds are not often tested with the Cray compilers +or even with the AMD compilers. + +--- + +## Requirements + +Since EasyBuild is primarily intended for installing (scientific) software on HPC systems, the main target is +systems running some flavor of **GNU/Linux**. EasyBuild also works on *macOS*, +but significant effort would be required to use it extensively there. + + + +EasyBuild is implemented in [**Python**](https://www.python.org), and is compatible with Python 2.7 and 3.5+. +Since [Python 2 is no longer maintained](https://blog.python.org/2020/04/python-2718-last-release-of-python-2.html), +it is recommended to use a recent version of Python 3 for running EasyBuild. + +An [**environment modules**](https://en.wikipedia.org/wiki/Environment_Modules_(software)) tool is required for running EasyBuild. +The modern Lua-based implementation **[Lmod](https://lmod.readthedocs.io)** is strongly recommended, +but other (Tcl-based) implementations are also supported. +Both Tcl and Lua are supported as syntax for environment module files. +On LUMI Lmod is used as the module tool. + +--- + +## Development + + +EasyBuild is **actively developed** by a worldwide community under the +[**GPLv2 open source license**](https://www.gnu.org/licenses/old-licenses/gpl-2.0.html) +via the [**``easybuilders``** organisation on GitHub](https://github.com/easybuilders), in the following repositories: + +- [easybuild-framework](https://github.com/easybuilders/easybuild-framework) +- [easybuild-easyblocks](https://github.com/easybuilders/easybuild-easyblocks) +- [easybuild-easyconfigs](https://github.com/easybuilders/easybuild-easyconfigs) +- [easybuild](https://github.com/easybuilders/easybuild) (website, documentation) +- [easybuild-tutorial](https://github.com/easybuilders/easybuild-tutorial) (the generic EasyBuild tutorials) + +On LUMI the generic EasyBuild installation is used but extended and adapted using LUMI-specifics +plugins and hooks maintained in the [LUMI software stack repository](https://github.com/Lumi-supercomputer/LUMI-SoftwareStack). + +Stable versions of EasyBuild have been released roughly every 6-8 weeks since 2012, +and are published via the *Python Package Index (PyPI)*, +see [https://pypi.org/project/easybuild](https://pypi.org/project/easybuild/). + +**Comprehensive testing** practices are applied throughout the +development cycle, with extensive suites of unit and integration tests being run in a CI environment (GitHub Actions), +consistent testing of incoming contributions (pull requests), and thorough regression testing before every release. + +--- + +## A worldwide community + +EasyBuild has grown into a **community project** since it was first released publicy in 2012, +and is used and developed by dozens (hundreds?) of HPC centres and consortia worldwide. + +Since 2016 the EasyBuild community has gathered for a yearly [**EasyBuild User Meeting**](https://easybuild.io/eum), +which is an open and highly interactive event that provides a great opportunity to meet fellow EasyBuild users, +developers, and maintainers, and to discuss related topics, learn about new aspects of the tool, and how other HPC +sites leverage it, and integrate it with other tools like [Jenkins](https://www.youtube.com/watch?v=bzW45cfRUMU), +[Ansible](https://www.youtube.com/watch?v=Qr2udsAjkG0), [CernVM-FS](https://www.youtube.com/watch?v=_0j5Shuf2uE), +[ReFrame](https://www.youtube.com/watch?v=KN9QwQIrso4) etc. + +HPC sites, consortia, and companies that use EasyBuild include (but are not limited to): + + + + + + + + + + + + + + + + + + + + + + + +--- + +## Getting help + +### *Documentation* + +If you have any questions about EasyBuild, make sure to check the EasyBuild documentation first, +which is available at [**https://docs.easybuild.io**](https://docs.easybuild.io). + +It is not unlikely that an answer to your question is available there already. +The [search box](https://docs.easybuild.io/en/latest/search.html) on the top left may be helpful. + +### GitHub + + + +You can open an issue in one of the EasyBuild repositories in [the `easybuilders` GitHub organisation](https://github.com/easybuilders/) to +ask questions, report problems or share suggestions and ideas for additional +improvements. + +Please try to post the issue in the most suitable repository. If you are +not sure which repository is most suited just go ahead and open the issue anyway, +the EasyBuild maintainers can move your issue if deemed appropriate. + +Also here you can use the search box on the top left, to see if somebody else +has already asked a similar question or reported a similar problem. + +### Mailing list + +Feel free to ask any questions or provide feedback via the EasyBuild mailing list +`easybuild@lists.ugent.be`. + +Keep in mind that **you must be subscribed to the mailing list in order to +post messages there!** You can subscribe via [this link](https://lists.ugent.be/wws/subscribe/easybuild). + +In 2016-2017 over 1,000 messages were posted to the EasyBuild mailing list. +More recently this has decreased to about 700 messages per year, mostly due +to the EasyBuild Slack channel. + +### Slack + + + +The [EasyBuild Slack](https://easybuild.slack.com/) is a quick and easy way +to get in touch with the EasyBuild community. +Over 400 people have joined it so far, and there is usually somebody awake in there +who may be able to help you out. + +**To join the EasyBuild Slack request an invitation via +[https://easybuild-slack.herokuapp.com](https://easybuild-slack.herokuapp.com/).** + +If you are not a big fan of Slack, you can also consider joining the `#easybuild` +IRC channel (available on the [Freenode IRC network](https://freenode.net/)). +A bot relays messages between this IRC channel and the `#general` channel in the EasyBuild Slack. + + +### Conference calls + +Every other week on Wednesday an EasyBuild conference call is set up to discuss recent +developments, provide an outlook to the next EasyBuild release, and +provide answers to questions that can be raised by the people attending the call. + +These conference calls are open to anyone interested in the EasyBuild project, +and have been organised since November 2013. + +More information, including the dates and times of the next calls, is available +[on the EasyBuild Wiki](https://github.com/easybuilders/easybuild/wiki/Conference-calls). + +--- + +*[[next: The Lmod module system]](1_02_Lmod.md)* diff --git a/docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md new file mode 100644 index 00000000..ee0e5856 --- /dev/null +++ b/docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md @@ -0,0 +1,825 @@ +# The Lmod module system + +*[[back: What is EasyBuild?]](1_01_what_is_easybuild.md)* + +--- + +## Modules + +*Module* is a massively overloaded term in (scientific) software and IT in general +(kernel modules, Python modules, and so on). +In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. + +[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept +on HPC systems: it is a way to specify changes that should be made to one or more +[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a +[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file +is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or +[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, +and specifies which environment variables should be updated, and how (append, +prepend, (re)define, undefine, etc.) upon loading the environment module. +Unloading the environment module will restore the shell environment to its previous state. + +Environment module files are processed via a **modules tool**, of which there +are several conceptually similar yet slightly different implementations. +The oldest module tool still in use today is Environment Modules 3.2, implemented in C and +supporting module files written in Tcl. After a gap in development, Xavier Delaruelle of CEA +developed [Environment Modules 4 and 5](https://sourceforge.net/projects/modules/) which is +fully implemented on Tcl. An alternative module tool is [Lmod](https://lmod.readthedocs.io), +developed by Robert McLay at TACC and implemented in LUA. This tool supports natively LUA +module files but also offers a high degree of compatibility with Tcl-based module files +developed for Environment Modules fia a translation layer and some API translation. + +The Cray PE offers a choice between the old-style Environment Modules 3.2 and Lmod, but no +packages or official support for Environment Modules 4 or 5. At the user level, +Environment Modules 3.2 and Lmod have many commands in common, but with different options. +Lmod also has some powerful features that are lacking in Environment Modules 3.2. + +!!! Note "The Cray PE on LUMI" + On LUMI, Lmod was selected as the module tool. One area where there are significant + differences between Environment Modules 3.2 (and also the newer versions) and Lmod is + in the commands for discovering modules on the system. If you are not familiar with Lmod + and its commands for users, it is worthwhile to read the + [LUMI documentation page on Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/). + Some of those commands are also discussed on this page. + +--- + +## Lmod hierarchy + +### User view + +Lmod supports a module hierarchy. In a hierarchy, there is a distinction between the *installed +modules* and the *available modules*. Available modules are those that can be loaded directly +without first loading any other module, while the installed modules is the complete set of +modules that one could load one way or another. A typical use case +is a hierarchy to deal with different compilers on a system and different MPI implementations. +After all, it is a common practice to only link libraries and application code compiled with the +same compiler to avoid compatibility problems between compilers (and to be able to use advanced +features such as link time optimization). This is even more important for MPI, as Open MPI and +MPCIH-derived MPI implementations have incompatible Application Binary Interfaces. This would lead +to a hierarchy with 3 levels: + +1. The ``Core`` level containing the modules for the compilers themselves, e.g., one or more versions + of the GNU compiler suite and one or more versions of LLVM-based compilers. + + Loading a compiler module would then make the next level available: + +2. The ``Compiler`` level, containing modules for libraries and packages that only rely on the compilers + but do not use MPI, as well as the MPI modules, e.g., a version of Open MPI and a version of MPICH. + + Loading one of the MPI modules would then make the next level available: + +3. The ``MPI`` level, containing libraries and applications that depend on the compiler used and the MPI + implementation. + +??? Example "A simple Lmod hierarchy with a single compiler" + + Here is a simple example of such a 3-level module hierarchy + (that almost could have been generated by EasyBuild): + + + + In this example the ``Core`` level only includes a single module `GCC/9.3.0`, + while the ``Compiler`` level includes two modules: `OpenMPI/4.0.3` and `MPICH/3.3.2`. + In the ``MPI `` level, three modules are available: one for `FFTW`, one for `ScaLAPACK`, + and one for `HDF5`. + + Initially only the modules on the top level of a module hierarchy are available for loading. + If you run "`module avail`", the command that is used to view all modules that are available + for loading, with this example module hierarchy, you will only see the `GCC/9.3.0` module. + + Some modules in the top level of the hierarchy act as a "gateway" to modules in the + next level below. + To make additional modules available for loading one of these gateway modules has to be loaded. + In our example, loading the `GCC/9.3.0` module results in two additional modules coming into + view from the ``Compiler`` level, as indicated by the arrows: the modules for `OpenMPI` and `MPICH`. + These correspond to installations of `OpenMPI` + and `MPICH` that were built using `GCC/9.3.0`. + + Similarly, the `OpenMPI/4.0.3` module serves as a gateway to the three modules in the ``MPI`` level. + Only by loading the `OpenMPI` module will these additional three modules become + available for loading. They correspond to software installations built using the ``GCC/9.3.0`` + compiler with ``OpenMPI/4.0.3``. + +Now assume that we have two compilers in the hierarchy, Compiler_A and Compiler_B. Their modules would reside +at the ``Core`` level. Both compilers provide the same MPI implementation, MPI_C. So there would be two modules +for ``MPI_C`` in two different subdirectories at the ``Compiler`` level. And further assume that we have an +application, Appl_E, compiled with both Compiler_A and Compiler_B and using MPI_C. For that application there would +also be two module files at the ``MPI`` level, one in a subdirectory corresponding ao Compiler_A and MPI_C and one +in a subdirectory corresponding to Compiler_B and MPI_C. + +```mermaid +graph TD; +A[Compiler_A] --> AC[MPI_C]; +A --> AD[MPI_D] +B[Compiler_B] --> BC[MPI_C]; +AC --> ACE[Appl_E]; +AD --> ADE[Appl_E] +BC --> BCE[Appl_E]; +``` + + +To be able to load the module for Appl_E, a user should +first load Compiler_A, then load MPI_C and only then is it possible to load the module for Appl_E: + +```bash +module load Compiler_A MPI_C Appl_E +``` + +What is interesting is what happens if the user now loads Compiler_B: + +```bash +module load Compiler_B +``` + +In a properly designed and implemented hierarchy, Lmod will unload Compiler_A which will also trigger the unloading/deactivation +of MPI_C and Appl_E. It will then load the module for Compiler_B and proceed with looking if it can find another module for +MPI_C. That will then be loaded which now makes a different module for Appl_E available, which Lmod will proceed to load. If it +cannot find an exact match for the version, Lmod will even try to locate a different version. Hence the situation after loading +Compiler_B is that now modules are loaded for Compiler_B, MPI_C for Compiler_B and Appl_E for Compiler_A with MPI_C. +All this requires very little effort from the module file programmer and very little logic in the module files. E.g., rather +then implementing a single module file for Appl_E that would require logic to see which compiler and MPI implementation is loaded +and depending on those adapt the path to the binaries, several very simple modules need to be written with very little +logic, and one could add an Appl_E module for a different compiler or MPI implementation without touching any of the already +existing module files for that application. + +Similarly, if after + +```bash +module load Compiler_A MPI_C Appl_E +``` + +one does + +```bash +module load MPI_D +``` + +then MPI_C gets unloaded, Lmod notices that it also has to unload/deactivate Appl_E, then will load MPI_D for Compiler_A and +finally will notice that there is an equivalent Appl_E module available again, and Lmod will load that one also. However, +now loading Compiler_B will cause a warning that MPI_D and Appl_E have been deactivated as there is no module name MPI_D in +any version for Compiler_B. + + +### Building blocks + +Some mechanisms in Lmod make implementing a hierarchy fairly easy (though there are a lot of hidden pitfalls) + +- The *MODULEPATH* environment variable determines which modules are available. MODULEPATH is different from any other + path-style variable in Lmod in that any change will immediately trigger a re-evaluation of which modules are available + and trigger deactivating modules that are no longer available when a directory is removed from the MODULEPATH or + looking for alternatives for deactivated modules when a directory is added to the MODULEPATH. + +- The *"one name rule"*: Lmod cannot have two modules loaded with the same name (but a different version). By default, when loading + a module with the name of an already loaded module, Lmod will automatically swap the old one with the new one, i.e., unload the + already loaded module and load the new one. + +- The *family* concept: It is possible to declare a module to be part of a family using a command in the module file. No two modules + of the same family can be loaded at the same time, and Lmod will again by default auto-swap the already loaded one with the one + being loaded. The procedure is different though as Lmod now first has to read the new module file to discover the family, and this + may lead to more side effects. But that discussion is outside the scope of this tutorial. + + The family concept was for a long time a unique feature of Lmod, but it has been added now also to Environment Modules version 5.1. + + +### Implementation details + +The above example could be implemented using 8 module files: One for each compiler, three for the MPI modules +(two for MPI_C and one for MPI_D) and +three for the application modules. + +``` +moduleroot +├── Core +│ ├── Compiler_A +│ │ └── version_A.lua +│ └── Compiler_B +│ └── version_B.lua +├── Compiler +│ ├── Compiler_A +│ │ └── version_A +│ │ ├── MPI_C +│ │ │ └── version_C.lua +│ │ └── MPI_D +│ │ └── version_D.lua +│ └── Compiler_B +│ └── version_B +│ └── MPI_C +│ └── version_C.lua +└── MPI + ├── Compiler_A + │ └── version_A + │ ├── MPI_C + │ │ └── version_C + │ │ └── Appl_E + │ │ └── version_E.lua + │ └── MPI_D + │ └── version_D + │ └── Appl_E + │ └── version_E.lua + └── Compiler_B + └── version_B + └── MPI_C + └── version_C + └── Appl_E + └── version_E.lua +``` + +Besides the module functions needed to create the environment needed to run the compiler, the module file for +Compiler_A would need only two lines to implement the hierarchy: + +```Lua +family('Compiler') +prepend_path('MODULEPATH', 'moduleroot/Compiler/Compiler_A/version_A') +``` + +There are now two different ``version_C.lua`` files. One contains the necessary calls to module functions to +initialise the environment to use the version compiled with Compiler_A/version_A while the other contains the +necessary functions to do that for Compiler_B/version_B. Again, two more lines are needed to implement the hierarchy. +E.g., for ``moduleroot/Compiler/Compiler_A/version_A/MPI_C/version_C.lua``: + +```Lua +family('MPI') +prepend_path('MODULEPATH', 'moduleroot/MPI/Compiler_A/version_A/MPI_C/version_C') +``` + +Finally two versions of the ``version_E.lua`` file are needed, one to prepare the environment for using the +package with Compiler_A anmd MPI_C and one for using the package with Compiler_B and MPI_C. However, these +are just regular modules and no additions are needed to work for the hierarchy. + +Both EasyBuild and Spack support Lmod hierarchies and with these tools it is also fairly automatic to create +different versions of the module files for each compiler and MPI library used to build the application. When +hand-writing modules it may be more interesting to have a generic module which would work for all those cases +and that is also possible with Lmod. Lmod does have a range of *introspection functions* that a module can use +to figure out its name, version and place in the module tree. All that would be needed is that the various +instances of the module file are at the correct location in the module tree and link to the generic file which +can be outside the module tree. In fact, this feature is used on LUMI to implement the modules that load a +particular version of the hardware for a particular section of LUMI. + +--- + +## Finding modules + +In a hierarchical setup, not all modules are available at login. This implies that a user cannot use +``module avail`` to discover which software is available on the system. To this end Lmod has powerful +search commands. It is important to understand how these commands work to ensure that the proper information +is included in the module files to improve discoverability of software. + +!!! Note "Documentation in the LUMI documentation" + Extensive information on search commands with examples of how to use them on LUMI can be found + in the [LUMI documentation](https://docs.lumi-supercomputer.eu/), in + [the computing section, "Module environment page", "Finding modules" section](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/#finding-modules). + + +### module spider command + +The available modules at any point in time are often only a subset of all installed modules on a +system. However, Lmod provides the ``module spider`` command to search for a module with a given name +among all installed modules and to tell you how this module can be loaded (i.e., which other modules +need to be loaded to make the module available). + +The ``module spider`` command has three levels, producing different outputs: + + 1. ``module spider`` without further arguments will produce a list of all + installed software and show some basic information about those packages. + Some packages may have an ``(E)`` behind their name and will appear in blue + (in the default colour scheme) which means that they are part of a different + package. These are called *extensions* of packages or modules. + This is explained a little further in this page. + + Note that ``module spider`` will also search in packages that are hidden from + being displayed. These packages can be loaded and used. However administrators + may have decided to hide them + either because they are not useful to regular users or because they think that + they will rarely or never be directly loaded by a user and want to avoid + overloading the module display. + + 2. ``module spider__hook``. For example, the hook that is triggered right before the ``configure`` +step is run is a function named ``pre_configure_hook``. + +Every time these hooks are called, a single argument is provided: an [``EasyBlock``](https://docs.easybuild.io/en/latest/api/easybuild.framework.easyblock.html#easybuild.framework.easyblock.EasyBlock) +instance that represents the easyblock that is being used to perform the installation. +The parsed easyconfig file can be accessed via the ``cfg`` class variable of the ``EasyBlock`` instance. + +These hooks are useful for influencing the installation procedure at a particular stage. + +## Caveats + +There are a couple of important caveats to take into account when implementing hooks. + +### Breaking EasyBuild with hooks + +Since hooks allow you to inject custom code into EasyBuild at runtime, +it is also easy to break EasyBuild by using hooks... + +Make sure to carefully test your hook implementations, and constrain the actions +you take a much as possible, for example by adding conditions to control for which +software names you will actually modify the installation procedure, etc. + +Any errors that are triggered or raised while a hook function is running +will interrupt the EasyBuild session. + +So don't forget: with great power comes great responsibility! + +### Template values + +Depending on the type of hook, you may observe "raw" values of easyconfig parameters where +template values have not been resolved yet, or values in which template values have been resolved already. + +In the ``parse`` hook, you will always see unresolved template values. + +In the pre/post-step hooks you will see resolved template values, +unless you explicitly disable templating. + +To obtain easyconfig parameter values with unresolved template values in step hooks, +you can use the ``disable_templating`` [context manager](https://docs.python.org/3/reference/compound_stmts.html#with). +For example: + +```python +from easybuild.framework.easyconfig.easyconfig import disable_templating +from easybuild.tools.build_log import print_warning + +def pre_source_hook(eb): + """Print warning when software version was found in 'raw' name of source file.""" + with disable_templating(eb.cfg): + for src in eb.cfg['sources']: + if eb.version in src: + msg = "Software version '%s' found in name of source file (%s), " % (eb.version, src) + msg += "please use %(version)s template value instead!" + print_warning(msg) +``` + + +### Manipulating easyconfig parameters + +If you want update a particular easyconfig parameter without overwriting the existing value, +a bit of care has to be taken: you should use the ``update`` method of the ``EasyConfig`` instance +for this, unless you disable template resolution. This is particularly important when +updating easyconfig parameters that have *mutable* value (like a ``list`` or ``dict``). + +Here's a correct example of a pre-install hook: + +```python +def pre_install_hook(eb): + if eb.name == 'pigz': + # always copy the README directory too when installing pigz + eb.cfg.update('files_to_copy', 'README') +``` + +This seemingly equivalent implementation will ***not*** work (the value of the `files_to_copy` +easyconfig parameter will *not* be updated): + +```python +def pre_install_hook(eb): + if eb.name == 'pigz': + # incorrect way of adding 'README' to 'files_to_copy' (DON'T USE THIS!) + eb.cfg['files_to_copy'].append('README') +``` + +To use this coding style successfully, you have to disable the templating mechanism +when updating the easyconfig parameter: + +```python +def pre_install_hook(eb): + if eb.name == 'pigz': + # this works, but it is better to use the 'update' method instead... + with disable_templating(eb.cfg): + eb.cfg['files_to_copy'].append('README') +``` + +--- + +*[[next: Submitting installations as Slurm jobs]](3_03_slurm_jobs.md)* diff --git a/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md b/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md new file mode 100644 index 00000000..d49a7925 --- /dev/null +++ b/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md @@ -0,0 +1,335 @@ +# Submitting installations as Slurm jobs + +*[[back: Using hooks to customise EasyBuild]](3_02_hooks.md)* + +--- + +EasyBuild can submit jobs to different backends including Slurm to install software, +to *distribute* the often time-consuming installation of a set of software applications and +the dependencies they require to a cluster. Each individual package is installed in a separate +job and job dependencies are used to manage the dependencies between package so that no build +is started before the dependencies are in place. + +This is done via the ``--job`` command line option. + +It is important to be aware of some details before you start using this, which we'll cover here. + +!!! Warning "This section is not supported on LUMI, use at your own risk" + + EasyBuild on LUMI is currently not fully configured to support job submission via Slurm. Several + changes would be needed to the configuration of EasyBuild, including the location of the + temporary files and build directory. Those have to be made by hand. + + Due to the setup of the central software stack, this feature is currently useless to install + the central stack. For user installations, there are also limitations as the environment + on the compute nodes is different from the login nodes so, e.g., different locations for + temporary files are being used. These would only be refreshed if the EasyBuild configuration + modules are reloaded on the compute nodes which cannot be done currently in the way Slurm + job submission is set up in EasyBuild. + + Use material in this section with care; it has not been completely tested. + + +## Configuration + +The EasyBuild configuration that is active at the time that ``eb --job`` is used +will be *passed down* into the submitted job automatically, via command line options to the ``eb`` +command that is run in the job script. + +This includes not only command line options used next to ``--job``, but also configuration settings +that are specified via an [EasyBuild configuration file](../../1_Intro/1_07_configuration#configuration-files) or through +[``$EASYBUILD_*`` environment variables](../../1_Intro/1_07_configuration#easybuild_-environment-variables). + +This implies that any EasyBuild configuration files or ``$EASYBUILD_*`` environment variables +that are in place in the job environment are most likely *irrelevant*, since configuration settings +they specify they will most likely be overruled by the corresponding command line options. +It does also imply however that the EasyBuild configuration that is in place when ``eb --job`` is used +should also work on the compute nodes to which the job is submitted as EasyBuild will generate the command +line options used in the job script based on that configuration. + + +## Using ``eb --job`` + +### Job backend + +The default job backend in EasyBuild v4.x is [``GC3Pie``](https://gc3pie.readthedocs.io). +To let EasyBuild submit jobs to Slurm instead, you should set the ``job-backend`` configuration setting +to ``Slurm``, for example by setting the corresponding environment variable: + +```shell +export EASYBUILD_JOB_BACKEND='Slurm' +``` + +On LUMI this is taken care of in the EasyBuild configuration modules such as ``EasyBuild-user``. + + +### Job resources + +To submit an installation as a job, simply use ``eb --job``: + +```shell +eb example.eb --job +``` + +By default, EasyBuild will submit single-core jobs requesting for 24 hours of walltime. +You can tweak the requested resources via the ``job-cores`` and ``job-max-walltime`` configuration options. +For example: + +```shell +# submit job to install example, using 5 cores and 2 hours of max. walltime +eb example.eb --job --job-cores 5 --job-max-walltime 2 +``` + +Note that not all ``job-*`` configuration settings apply to all job backends, +see the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Submitting_jobs.html) for more details. + +### Controlling Slurm submission options + +When using Slurm as a job backend, EasyBuild will automatically generate job scripts which +use the ``eb`` command to perform a single installation. These scripts will be submitted +using the ``sbatch`` command. + +EasyBuild currently doesn't provide away to customize the Slurm submission options, +for example to submit to a particular partition, or to use a particular account, +build you can set the corresponding ``SBATCH_*`` environment variables prior to running ``eb --job``. + +For example, to specify a particular account that should be used for the jobs submitted by EasyBuild +(equivalent with using the ``-A`` or ``--account`` command line option for ``sbatch``): + +```shell +export SBATCH_ACCOUNT='project_XXXXXXXXX' +``` + +Or to submit to a particular Slurm partition (equivalent with the ``-p`` or ``--partition`` option for ``sbatch``): + +```shell +export SBATCH_PARTITION='small' +``` + +For more information about supported ``SBATCH_*`` environment variables, +see the [Slurm documentation](https://slurm.schedmd.com/sbatch.html#lbAJ). + +## Combining ``--job`` and ``--robot`` + +If one or more dependencies are still missing for the software you want to install, +you can combine ``--job`` and ``--robot`` to get EasyBuild to submit a *separate* job +for each of the installations. These jobs will *not* use ``--robot``, they will each only +perform a single installation. + +Dependencies between jobs will be "registered" at submission time, so Slurm will put jobs +on hold until the jobs that install the required (build) dependencies have completed successfully, +and cancel jobs if the job to install a dependency failed for some reason. + +## Attention points + +There are a couple of important things to keep an eye on when submitting installations as jobs... + +### Differences on cluster workernodes + +Sometimes the resources available on the login nodes and cluster workernodes are slightly different, +and you may need to take this into account in your EasyBuild configuration. + +For example, plenty of disk space may be available in the `/tmp` temporary filesystem on a login node, +while the workernodes require you to use a different location for temporary files and directories. +As a result, you may need to slightly change your EasyBuild configuration when submitting installations +as jobs, to avoid that they fail almost instantly due to a lack of disk space. + +Keep in mind that the active EasyBuild configuration is passed down into the submitted jobs, +so any configuration that is present on the workernodes may not have any effect. + +For example, on LUMI it is possible to use ``$XDG_RUNTIME_DIR`` on the login nodes which has +the advantage that any leftovers of failed builds will be cleaned up when the user ends their last +login session on that node, but it is not possible to do so on the compute nodes. + +```shell +# EasByuild is configured to use /tmp/$USER on the login node +uan01 $ eb --show-config | grep buildpath +buildpath (E) = /run/user/XXXXXXXX/easybuild/build + +# use /dev/shm/$USER for build directories when submitting installations as jobs +login01 $ eb --job --buildpath /dev/shm/$USER/easybuild example.eb --robot +``` + + +### Temporary log files and build directories + +The problems for the temporary log files are twofold. First, they may end up in a place +that is not available on the compute nodes. E.g., for the same reasons as for the build +path, the LUMI EasyBuild configuration will place the temporary files in a subdirectory of +``$XDG_RUNTIME_DIR`` on the loginnodes but a subdirectory of ``/dev/shm/$USER`` on the +compute nodes. The second problem however is that if an installation fails, those log files are +not even accessible anymore which may leave you wondering about the actual cause of the failing +installation... + +To remedy this, there are a couple of EasyBuild configuration options you can use: + +* You can use ``--tmp-logdir`` to specify a different location where EasyBuild should store temporary log files, + for example: + ```shell + $ eb --job example.eb --tmp-logdir $HOME/eb_tmplogs + ``` + This will move at least the log file to a suitable place. + +* If you prefer having the entire log file stored in the Slurm job output files, + you can use ``--logtostdout`` when submitting the jobs. This will result in extensive logging + to your terminal window when submitting the jobs, but it will also make EasyBuild + log to ``stdout`` when the installation is running in the job, and hence the log messages will be + captured in the job output files. + +The build directory of course also suffers from the problem of being no longer accessible if the +installation fails, but there it is not so easy to find a solution. Building on a shared file system +is not only much slower, but in particular on parallel file systems like GPFS/SpectrumScale, Lustre +or BeeGFS building sometimes fails in strange ways. One thing you can consider if you cannot do the +build on a login node (e.g., because the code is not suitable for cross-compiling or the configure +system does tests that would fail on the login node), is to rety the installation in an +interactive job, so you can inspect the build directory after the installation fails. + +### Lock files + +EasyBuild creates [locks](https://docs.easybuild.io/en/latest/Locks.html) +to prevent that the same installation is started multiple times on +different system to the same installation directory. + +If an installation fails or gets interrupted, EasyBuild cleans up those locks automatically. + +However, if a Slurm job that is using EasyBuild to install software gets cancelled (because it +ran out of walltime, tried to consume too much memory, through an ``scancel`` command, etc.), +EasyBuild will not get the chance to clean up the lock file. + +If this occurs you will need to either clean up the lock file (which is located in the `software/.locks` +subdirectory of ``installpath``) manually, or re-submit the job with ``eb --job --ignore-locks``. + +## Example + +As an example, we will let EasyBuild submit jobs to install ``AUGUSTUS`` with the ``foss/2020b`` toolchain. + +!!! Warning "This example does not work on LUMI" + + Note that this is an example using the FOSS common toolchain. For this reason it does not work on + LUMI. + +### Configuration + +Before using ``--job``, let's make sure that EasyBuild is properly configured: + +```shell +# Load the EasyBuild-user module (central installations will not work at all +# using job submission) +module load LUMI/21.12 +module load partition/C +module load EasyBuild-user + +# use /tmp for build directories and temporary directories as we have those +# on login and compute nodes. +export EASYBUILD_BUILDPATH=/tmp/$USER/build +export EASYBUILD_TMPDIR=/tmp/$USER/tmp + +# use Slurm as job backend +export EASYBUILD_JOB_BACKEND=Slurm +``` + + +We will also need to inform Slurm that jobs should be submitted into a particular account, and +in a particular partition: + +```shell +export SBATCH_ACCOUNT=project_XXXXXXXXX +export SBATCH_PARTITION='small' +``` + +This will be picked up by the ``sbatch`` commands that EasyBuild will run to submit the software installation jobs. + + +### Submitting jobs to install AUGUSTUS + +Now we can let EasyBuild submit jobs for AUGUSTUS. + +Let's first check what is still missing: + +```shell +$ eb AUGUSTUS-3.4.0-foss-2020b.eb --missing +... +11 out of 61 required modules missing: + +* HTSlib/1.11-GCC-10.2.0 (HTSlib-1.11-GCC-10.2.0.eb) +* lpsolve/5.5.2.11-GCC-10.2.0 (lpsolve-5.5.2.11-GCC-10.2.0.eb) +* Boost/1.74.0-GCC-10.2.0 (Boost-1.74.0-GCC-10.2.0.eb) +* GSL/2.6-GCC-10.2.0 (GSL-2.6-GCC-10.2.0.eb) +* SAMtools/1.11-GCC-10.2.0 (SAMtools-1.11-GCC-10.2.0.eb) +* BCFtools/1.11-GCC-10.2.0 (BCFtools-1.11-GCC-10.2.0.eb) +* METIS/5.1.0-GCCcore-10.2.0 (METIS-5.1.0-GCCcore-10.2.0.eb) +* BamTools/2.5.1-GCC-10.2.0 (BamTools-2.5.1-GCC-10.2.0.eb) +* MPFR/4.1.0-GCCcore-10.2.0 (MPFR-4.1.0-GCCcore-10.2.0.eb) +* SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0 (SuiteSparse-5.8.1-foss-2020b-METIS-5.1.0.eb) +* AUGUSTUS/3.4.0-foss-2020b (AUGUSTUS-3.4.0-foss-2020b.eb) +``` + +Several dependencies are not installed yet, so we will need to use ``--robot`` to ensure that +EasyBuild also submits jobs to install these first. + +To speed up the installations a bit, we will request 8 cores for each submitted job (via ``--job-cores``). +That should be sufficient to let each installation finish in (well) under 1 hour, +so we only request 1 hour of walltime per job (via ``--job-max-walltime``). + +In order to have some meaningful job output files, we also enable trace mode (via ``--trace``). + +``` +$ eb AUGUSTUS-3.4.0-foss-2020b.eb --job --job-cores 8 --job-max-walltime 1 --robot --trace +... +== resolving dependencies ... +... +== List of submitted jobs (11): Boost-1.74.0-GCC-10.2.0 (Boost/1.74.0-GCC-10.2.0): 1000011; GSL-2.6-GCC-10.2.0 (GSL/2.6-GCC-10.2.0): 1000004; SAMtools-1.11-GCC-10.2.0 (SAMtools/1.11-GCC-10.2.0): 1000005; HTSlib-1.11-GCC-10.2.0 (HTSlib/1.11-GCC-10.2.0): 1000006; BCFtools-1.11-GCC-10.2.0 (BCFtools/1.11-GCC-10.2.0): 1000001; lpsolve-5.5.2.11-GCC-10.2.0 (lpsolve/5.5.2.11-GCC-10.2.0): 1000007; BamTools-2.5.1-GCC-10.2.0 (BamTools/2.5.1-GCC-10.2.0): 1000008; METIS-5.1.0-GCCcore-10.2.0 (METIS/5.1.0-GCCcore-10.2.0): 1000009; MPFR-4.1.0-GCCcore-10.2.0 (MPFR/4.1.0-GCCcore-10.2.0): 1000010; SuiteSparse-5.8.1-foss-2020b-METIS-5.1.0 (SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0): 1000002; AUGUSTUS-3.4.0-foss-2020b (AUGUSTUS/3.4.0-foss-2020b): 1000003 +== Submitted parallel build jobs, exiting now +``` + +### Inspecting the submitted jobs + +Once EasyBuild has submitted the jobs, we can inspect them via Slurm's ``squeue`` command: + +``` +$ squeue -u $USER -la + JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON) +1000001 small BCFtools user123 PENDING 0:00 2:00:00 1 (Dependency) +1000002 small SuiteSpa user123 PENDING 0:00 2:00:00 1 (Dependency) +1000003 small AUGUSTUS user123 PENDING 0:00 2:00:00 1 (Dependency) +1000004 small GSL-2.6- user123 RUNNING 0:21 2:00:00 1 node003 +1000005 small SAMtools user123 RUNNING 0:21 2:00:00 1 node007 +1000006 small HTSlib-1 user123 RUNNING 0:21 2:00:00 1 node007 +1000007 small lpsolve- user123 RUNNING 0:21 2:00:00 1 node011 +1000008 small BamTools user123 RUNNING 0:21 2:00:00 1 node011 +1000009 small METIS-5. user123 RUNNING 0:21 2:00:00 1 node013 +1000010 small MPFR-4.1 user123 RUNNING 0:21 2:00:00 1 node029 +1000011 small Boost-1. user123 RUNNING 0:24 2:00:00 1 node029 +``` + +Note that 3 jobs can not be started yet, because those installations require on one or more +missing dependencies. As soon as the jobs for those dependencies (successfully) complete, +these jobs will be able to start. + +### Final result + +After about 20 minutes, AUGUSTUS and all missing dependencies should be installed: + +``` +$ ls -lrt $HOME/EasyBuild/modules/.../*.lua | tail -11 +-rw-rw----. 1 example example 1634 Mar 29 10:13 /users/example/easybuild/modules/all/HTSlib/1.11-GCC-10.2.0.lua +-rw-rw----. 1 example example 1792 Mar 29 10:13 /users/example/easybuild/modules/all/SAMtools/1.11-GCC-10.2.0.lua +-rw-rw----. 1 example example 1147 Mar 29 10:13 /users/example/easybuild/modules/all/BamTools/2.5.1-GCC-10.2.0.lua +-rw-rw----. 1 example example 957 Mar 29 10:13 /users/example/easybuild/modules/all/lpsolve/5.5.2.11-GCC-10.2.0.lua +-rw-rw----. 1 example example 1549 Mar 29 10:13 /users/example/easybuild/modules/all/METIS/5.1.0-GCCcore-10.2.0.lua +-rw-rw----. 1 example example 1525 Mar 29 10:14 /users/example/easybuild/modules/all/GSL/2.6-GCC-10.2.0.lua +-rw-rw----. 1 example example 1221 Mar 29 10:15 /users/example/easybuild/modules/all/MPFR/4.1.0-GCCcore-10.2.0.lua +-rw-rw----. 1 example example 1678 Mar 29 10:15 /users/example/easybuild/modules/all/BCFtools/1.11-GCC-10.2.0.lua +-rw-rw----. 1 example example 1292 Mar 29 10:21 /users/example/easybuild/modules/all/Boost/1.74.0-GCC-10.2.0.lua +-rw-rw----. 1 example example 1365 Mar 29 10:28 /users/example/easybuild/modules/all/SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0.lua +-rw-rw----. 1 example example 2233 Mar 29 10:30 /users/example/easybuild/modules/all/AUGUSTUS/3.4.0-foss-2020b.lua + +$ module avail AUGUSTUS + +-- EasyBuild managed user software for software stack ... -- + AUGUSTUS/3.4.0-foss-2020b +``` + +--- + +*[[next: Module naming schemes]](3_04_module_naming_schemes.md)* diff --git a/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md b/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md new file mode 100644 index 00000000..ce8a6eff --- /dev/null +++ b/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md @@ -0,0 +1,575 @@ +# Module naming schemes + +*[[back: Submitting installations as Slurm jobs]](3_03_slurm_jobs.md)* + +--- + +Up until now we have used the naming scheme in use on LUMI which is for many practical purposes +is indistinguishable from the default EasyBuild naming scheme (`EasyBuildMNS`). +It produces module files with names that closely resemble to the names of the +corresponding easyconfig files. +For example, when installing `zlib-1.2.11-cpeGNU-21.12.eb` the generated module was named +`zlib/1.2.11-cpeGNU-21.12`. + +EasyBuild supports several different module naming schemes: + +```shell +$ eb --avail-module-naming-schemes +List of supported module naming schemes: + EasyBuildMNS + LUMI_FlatMNS + MigrateFromEBToHMNS + HierarchicalMNS + CategorizedHMNS + CategorizedModuleNamingScheme +``` + +In this part of the tutorial we will take a closer look at `HierarchicalMNS`, +which is the standard **hierarchical** module naming scheme included with EasyBuild. + +We will also take a quick look at implementing our own custom module naming scheme. + +## Flat vs hierarchical + +!!! Note + + This text is very much based on the generic EasyBuild tutorials as we cannot demonstrate + all aspects on LUMI as it is configured today. + + On LUMI, the complete module scheme is partly hierarchical, but EasyBuild uses a flat naming + scheme. The two levels in the hierarchy that are present to deal with different versions of + the software stack and the various types of compute nodes, are not managed by EasyBuild. + +The default module naming scheme `EasyBuildMNS` +and the custom `LUMI_FlatMNS` naming scheme are both examples of regular *"flat"* +module naming schemes, which are characterized by: + +- all module files are directly available for loading; +- each module name uniquely identifies a particular installation; + +In contrast, a *hierarchical* module naming scheme +consists of a *hierarchy* of module files. +A fairly typical 3-level scheme (``Core``, ``Compiler`` and ``MPI``) has been +discussed in the [section on Lmod](../../1_Intro/1_02_Lmod#lmod-hierarchy). +This typical Lmod hierarcny would map very well on the EasyBuild common toolchains. + +In fact, for the example + + + +software at the ``Core`` level would be installed with the ``GCCcore`` and ``SYSTEM`` +toolchains. Software at the ``Compiler`` level would be installed with the ``GCC`` +toolchain, while software at the ``MPI`` level after loading the ``OpenMPI`` module +would be installed using the regular ``foss`` toolchain or the ``gompi`` toolchain +(see the diagram in the +["Common toolchains" section of the page on terminology](../../1_Intro/1_05_terminology/#common-toolchains)). + +On LUMI, where software is installed through the Cray Programming Environment with no real choice of +MPI implementation, a two-level arrangement would still make a lot of sense, with at the ``Core`` level +all software compiled with the SYSTEM toolchain while there could be a ``PrgEnv`` level for software +compiled with a particular programming environment aka cpeGNU/cpeCray/cpeAOCC toolchain. Such a scheme +is used on the Cray systems at CSCS. + +To recap, the characteristics of a module hierarchy are: + +* not all module files are directly available for loading; +* some modules serve as a gateway to more modules; +* to access some software installations you will first need to load one or more gateway modules in order + to use them; + +You can probably think of other ways to organize module files in a hierarchical module tree, but here +we will stick to the standard core / compiler / MPI hierarchy. + +### Pros & cons + +So why go through all this trouble of organizing modules hierarchically? + +There are a couple of advantages to this approach: + +* shorter module names (or at least for the version part of the name); +* less overwhelming list of available modules; +* only compatible modules can be loaded together; + +However, the are some minor disadvantages too: + +* not all existing modules are directly visible, so users have to learn how to find + modules using ``module spider`` etc. (but that is not an issue on LUMI as users + also need to use this command anyway to discover software that may not be in all + software stacks); +* gateway modules may have little meaning to end users; + +#### Length of module names + +When using a flat module naming scheme, module names can be fairly long and perhaps confusing. For a `HDF5` installation +with the EasyBuild common toolchains for example, +one might have `HDF5/1.12.1-gompi-2021b` as module name. The `-gompi-2021b` part of the name refers to the toolchain that was +used for this installation, but it may be confusing to some people (what kind of Pokémon is a "gompi"?!). + +In the example module hierarchy shown above, the module for `HDF5` could simply be named `HDF5/1.12.1` which is basically the bare +essentials: software name and version. That's way better, nice and clean! + +#### Amount of available modules + +The output of "`module avail`" can be quite overwhelming if lots of module files +are installed and a flat module naming scheme is used, since *all* modules are +*always* available. +EasyBuild makes it very easy to install lots of software, +so the number of installed modules can easily grow into the hundreds or even thousands... + +This often explosive growth of modules is less of an issue when using a hierarchical module naming scheme, since +initially only a small set of modules are available, and relatively limited +groups of additional modules become available as gateway modules are loaded. + +#### Loading compatible modules + +Since all modules are available at once when using a flat module naming scheme, you can easily load +modules together that are not compatible with each other. + +Imagine loading two modules that were built with a different compiler toolchain (different compiler, +different MPI library). That's likely to end in tears, unless you have the necessary technical expertise +to fully understand what is going on, *and* you are being very careful... + +In a module hierarchy this can be prevented, since modules for software that were installed with a +different compiler and/or a different MPI library are located in a different part of the module hierarchy, +and so these modules can not be loaded together easily. + + +#### Visibility of existing modules + +One downside of a module hierarchy is that not all existing modules are directly available for loading +or are even *visible* to the user, since the output of "`module avail`" only shows a subset of all modules. + +As we have discussed in [the Lmod section](../../1_Intro/1_02_Lmod). +[Lmod](https://lmod.readthedocs.io) provides a powerful solution to search for modules through the +``module spider`` and to some extent also the ``module keyword`` commands. +So as long as the end users are +aware of this additional command, it should not be difficult to discover which software installations exist +and how they can be accessed. The "`module spider`" command will inform the user which of the gateway modules +need to be loaded in order to load a specific module file. + +#### Semantics of gateway modules + +An additional potential problem of a module hierarchy is that the semantics of the gateway modules may not be clear +to end users. They may wonder why they need to pick a specific compiler and MPI library, or which of the +available options is the best one. Maybe they are not even aware what exactly a "compiler" is, or how it is +relevant to the software they need in their bioinformatics pipeline... + +This can be partially resolved by loading a default compiler and MPI module so a particular set of modules +is available right after login, which could be the ones used in the most recent toolchain, or the +recommended versions. More experienced users could then leverage the "`module spider`" command to navigate +the module hierarchy. + +On LUMI this would mean loading a default software stack, but due to the the default modules are currently +loaded on LUMI this was not possible to accomplish without losing other functionality of the module tree. + +## Using a custom module naming scheme + +Next to the module naming schemes that are included with EasyBuild, +you can also define your own module naming scheme (MNS), and configure EasyBuild to use it +(which is exactly what has been done on LUMI to remove a feature of the default ``EasyBuildMNS`` scheme +that we do not use). + +### Implementation + +To use a custom module naming scheme, you need to implement a Python module file, +where you define a Python class that derives from the general [``ModuleNamingScheme``](https://docs.easybuild.io/en/latest/api/easybuild.tools.module_naming_scheme.mns.html#easybuild.tools.module_naming_scheme.mns.ModuleNamingScheme) class. + +For a flat module naming scheme, it is sufficient to implement the ``det_full_module_name`` method, +which should return a string value (the full module name). +You may also need to customize the ``is_short_modname_for`` method, which verifies whether +a given (short) module name is for the software with a particular given name (or not). + +The argument provided to ``det_full_module_name`` can be a small Python dictionary which only specifies +the value of a handful of easyconfig parameters (`name`, `version`, `toolchain`, and `versionsuffix`), +or an ``EasyConfig`` instance which represents a parsed easyconfig file and contains values for *all* +known easyconfig parameters. + +For simple module naming schemes, just have ``name``, ``version``, ``toolchain``, and ``versionsuffix`` +available is sufficient. If it is not, you can list which additional easyconfig parameters are required for +your module naming scheme via a class constant ``REQUIRED_KEYS``. + +A fairly straightforward example of a *flat* module name scheme is the default EasyBuild module naming scheme +[``EasyBuildMNS``](https://github.com/easybuilders/easybuild-framework/blob/main/easybuild/tools/module_naming_scheme/easybuild_mns.py). + +For a *hierarchical* module naming scheme, various additional methods have to be implemented. + +Some of these, like ``det_module_subdir`` and ``det_short_module_name``, determine which part of +the (full) module name determines the location of the module in the module hierarchy, and which part +is the user-facing (or "short") module name. Others, like ``det_modpath_extensions``, +determine which modules are *gateway* modules that open up an additional level of the hierarchy +(by *extending* the ``$MODULEPATH``). + +A typical example of a hierarchical module naming scheme is [``HierarchicalMNS``](https://github.com/easybuilders/easybuild-framework/blob/main/easybuild/tools/module_naming_scheme/hierarchical_mns.py) that comes with EasyBuild (which is quite complex because it includes quite a bit of special handling for particular compiler +toolchains), which implements the traditional ``Core``-``Compiler``-``MPI`` module hierarchy we discussed +above. + +### Configuring EasyBuild + +To let EasyBuild use a custom module naming scheme, you need to: + +* specify the path to the Python module file implementing it via the ``include-module-naming-schemes`` + EasyBuild configuration option; + +* indicate that you also want to *use* this custom module naming scheme via the ``module-naming-scheme`` + EasyBuild configuration option. + +For example: + +```shell +export EASYBUILD_INCLUDE_MODULE_NAMING_SCHEMES=$HOME/easybuild/example_mns.py +export EASYBUILD_MODULE_NAMING_SCHEME=ExampleMNS +``` + +#### Example custom module naming scheme + +Here is an example of a custom module naming scheme, where: + +* the ``versionsuffix`` goes directly after the ``version``; +* all dashes are replaced by underscores; +* all module names are lowercase; + +Note that we also need to customise the ``is_short_modname_for`` method, +to make sure it returns ``True`` when the EasyBuild framework checks whether +``scipy_bundle/2020.11_foss_2020b`` is a module name for ``SciPy-bundle``. + +```python +import os + +from easybuild.tools.module_naming_scheme.mns import ModuleNamingScheme + + +class ExampleMNS(ModuleNamingScheme): + + REQUIRED_KEYS = ['name', 'version', 'versionsuffix', 'toolchain'] + + def det_full_module_name(self, ec): + """ + Determine full module name: + - all lowercase + - replace all dashes with underscores + """ + parts = [ec['version']] + + # versionsuffix directly after version (but only if it's not empty) + if ec['versionsuffix']: + parts.append(ec['versionsuffix']) + + # only add toolchain name/version for non-system toolchain + tc = ec['toolchain'] + if tc['name'].lower() != 'system': + parts.extend([tc['name'], tc['version']]) + + modname = ec['name'] + '/' + '_'.join(parts) + + modname = modname.replace('-', '_').replace('__', '_') + + return modname.lower() + + def is_short_modname_for(self, short_modname, name): + """Determine whether short module name is a module for the software with specified name.""" + return short_modname.startswith(name.lower().replace('-', '_') + '/') +``` + +We can see what the module names with this module naming scheme would like like via ``eb -D``. +E.g., for the common toolchains (the example will not work on LUMI as the default easyconfig +files are not in the robot- and the search path): + +``` +$ eb SciPy-bundle-2020.11-foss-2020b-Python-2.7.18.eb -D + ... + * [ ] $CFGS/g/GCC/GCC-10.2.0.eb (module: gcc/10.2.0) + ... + * [ ] $CFGS/p/Python/Python-2.7.18-GCCcore-10.2.0.eb (module: python/2.7.18_gcccore_10.2.0) + ... + * [ ] $CFGS/o/OpenMPI/OpenMPI-4.0.5-GCC-10.2.0.eb (module: openmpi/4.0.5_gcc_10.2.0) + ... + * [ ] $CFGS/s/SciPy-bundle/SciPy-bundle-2020.03-foss-2020a-Python-2.7.18.eb (module: scipy_bundle/2020.03_python_2.7.18_foss_2020a) +``` + +## Example module hierarchy: HDF5 + +!!! Warning "Example not suitable for LUMI" + **This exercise is meant for a system where the common toolchains can be used and requires an + independent EasyBuild installation in your personal file space**, + because EasyBuild will try to copy the installation log file to each installation directory. + +Now that we know more about hierarchical module naming schemes, +let us see how EasyBuild can help us with generating a hierarchical module tree. + +In this example we will use EasyBuild to generate modules organised in a hierarchy +for some of the software that is already installed in the prepared environment. + +The good news is that the existing installations can be reused. There is absolutely no need +to reinstall the software, we are just creating a different "view" on these software installations. + +### Preparing the environment + +Before running EasyBuild to generate a hierarchical module tree, we have to be a bit careful +with preparing our environment. + +**We must absolutely avoid mixing modules from a flat and hierarchical module naming scheme!** + +Some module files will have the same name in both module trees (like `GCC/10.2.0` for example), +but their contents will be different. +Mixing modules from a flat and hierarchical module tree *will* trigger problems... + +So we have to make sure that the module files we already have in `/easybuild` are *not* visible. +The easiest way to do this is to unload all modules (using "`module purge`") +and resetting the module search path to be empty, which we can do with "`module unuse $MODULEPATH`". + +```shell +module purge +module unuse $MODULEPATH +``` + +In this part of the tutorial, we are assuming you are *not* using an EasyBuild installation provided through +a module. We have just made all modules unavailable, so we would have to first +install EasyBuild again in our hierarchical module tree before we can continue. + +**We strongly recommend using an EasyBuild installation that was [installed via "`pip install`" +or "`pip3 install`"](../../1_Intro/1_06_installation#method-1-using-pip) in this part of the tutorial.** + +An easy way to do this is in the prepared environment used for regular EasyBuild tutorials is to run: + +```shell +unset PIP_PREFIX +pip3 install --user easybuild +export PATH=$HOME/.local/bin:$PATH +export EB_PYTHON=python3 +``` + +### Configuring EasyBuild + +First of all, we need to make sure that EasyBuild is properly configured. +We can do this by defining this set of environment variables: + +```shell +export EASYBUILD_PREFIX=$HOME/easybuild +export EASYBUILD_BUILDPATH=/tmp/$USER +export EASYBUILD_INSTALLPATH_SOFTWARE=/easybuild/software +export EASYBUILD_MODULE_NAMING_SCHEME=HierarchicalMNS +export EASYBUILD_INSTALLPATH_MODULES=$HOME/hmns/modules +``` + +To make sure we didn't make any silly mistakes, we double check using `eb --show-config`: + + +```shell +$ eb --show-config +# +# Current EasyBuild configuration +# (C: command line argument, D: default value, E: environment variable, F: configuration file) +# +buildpath (E) = /tmp/example +containerpath (E) = /home/example/easybuild/containers +installpath (E) = /home/example/easybuild +installpath-modules (E) = /home/example/hmns/modules +installpath-software (E) = /easybuild/software +module-naming-scheme (E) = HierarchicalMNS +packagepath (E) = /home/example/easybuild/packages +prefix (E) = /home/example/easybuild +repositorypath (E) = /home/example/easybuild/ebfiles_repo +robot-paths (D) = /home/example/.local/easybuild/easyconfigs +sourcepath (E) = /home/example/easybuild/sources +``` + +There are a couple of things worth pointing out here: + +* We have defined the `module-naming-scheme` configuration setting to `HierarchicalMNS`, + which makes EasyBuild use the included standard hierarchical module naming scheme (the classic + core / compiler / MPI one we discussed above). +* We have specified *different* locations for the software (via `installpath-software`) + and the module files (via `installpath-modules`). This is important because we want to + reuse the software that is already installed in `/easybuild/software` while we want to + generate an entirely new module tree for it (in `$HOME/hmns/modules`). + +The other configuration settings are the same as before, and mostly irrelevant for this example. + +### Generating modules for HDF5 + +Let us now generate a hierarchical module tree for `HDF5` and all of its dependencies, +including the toolchain. That sounds complicated, and it sort of is since there are +a lot of details you have to get right for the module hierarchy to works as intended, +but EasyBuild can do all the hard work for us. + +The steps we will have to go through are: + +* Tell EasyBuild we want to "install" the `HDF5-1.12.1-gompi-2021b.eb` easyconfig file; +* Enable dependency resolution via `--robot`; +* Assuming the software would have been installed already with the default naming scheme + in a different module directory, instruct EasyBuild to only generate the module files, + not to install the software (since it is + there already in `/easybuild/software`), via the `--module-only` option. + +These steps translate to this single `eb` command: + +``` +$ eb HDF5-1.12.1-gompi-2021b.eb --robot --module-only +... +== building and installing MPI/GCC/11.2.0/OpenMPI/4.1.1/HDF5/1.12.1... +... +== sanity checking... +== cleaning up [skipped] +== creating module... +... +== COMPLETED: Installation ended successfully (took 9 sec) +... +== Build succeeded for 41 out of 41 +``` + +This should take a couple of minutes in total, for generating 41 modules. +Remember that this also includes generating module files for the toolchain and +all of its components. + +In addition, there is a bit more going on one that just generating module files, +since the sanity check step is still being run for each of the installations +when using `--module-only` to ensure the installation is actually functional. +After all, there is no point in generating a module for an obviously broken +installation... + +### Loading the HDF5 module + +After generating the hierarchical module tree for HDF5, how do we access the HDF5 installation through it? + +Here's what the module tree looks like on disk: + +``` +$ ls $HOME/hmns/modules/all +Compiler Core MPI +``` + +Those are basically the 3 levels in the module hierarchy we showed in our example earlier. + +The starting point is the top level of the module hierarchy named `Core`: + +``` +module use $HOME/hmns/modules/all/Core +``` + +Let us see what that gives us in terms of available modules: + +``` +$ module avail + +------------------------ /home/easybuild/hmns/modules/all/Core ------------------------- + binutils/2.37 GCC/11.2.0 gompi/2021b OpenSSL/1.1 + Bison/3.8.2 GCCcore/11.2.0 M4/1.4.19 pkg-config/0.29.2 + flex/2.6.4 gettext/0.21 ncurses/6.2 zlib/1.2.11 +``` + + +Nice and short module names, but only a limited set of them. + +We know a module file exists for `HDF5`, but we can't see it yet (and hence +we can't load it either). + +``` +$ module avail HDF5 +No module(s) or extension(s) found! +Use "module spider" to find all possible modules and extensions. +``` + +Let us see if `module spider` is of any help, as "`module avail`" so kindly suggests: + +``` +$ module spider HDF5 +... + + You will need to load all module(s) on any one of the lines below + before the "HDF5/1.12.1" module is available to load. + + GCC/11.2.0 OpenMPI/4.1.1 +``` + +This tells us we need to load two gateway modules before we can load the module +for HDF5. + +Let us start with loading the `GCC` compiler module: + +``` +module load GCC/11.2.0 +``` + +And then check again which modules are available: + +``` +$ module avail + +--------------------------- /home/easybuild/hmns/modules/all/Compiler/GCC/11.2.0 --------------------------- + OpenMPI/4.1.1 + +------------------------- /home/easybuild/hmns/modules/all/Compiler/GCCcore/11.2.0 ------------------------- + Autoconf/2.71 flex/2.6.4 (D) libreadline/8.1 pkg-config/0.29.2 (D) + Automake/1.16.4 groff/1.22.4 libtool/2.4.6 PMIx/4.1.0 + Autotools/20210726 help2man/1.48.3 libxml2/2.9.10 Szip/2.1.1 + binutils/2.37 (L,D) hwloc/2.5.0 M4/1.4.19 (D) UCX/1.11.2 + Bison/3.7.6 libevent/2.1.12 ncurses/6.2 (D) xorg-macros/1.19.3 + DB/18.1.40 libfabric/1.13.2 numactl/2.0.14 XZ/5.2.5 + expat/2.4.1 libpciaccess/0.16 Perl/5.34.0 zlib/1.2.11 (L,D) + +---------------------------------- /home/easybuild/hmns/modules/all/Core ----------------------------------- + binutils/2.37 GCC/11.2.0 (L) gompi/2021b OpenSSL/1.1 + Bison/3.8.2 (D) GCCcore/11.2.0 (L) M4/1.4.19 pkg-config/0.29.2 + flex/2.6.4 gettext/0.21 ncurses/6.2 zlib/1.2.11 +``` + +Good news, we now have additional modules available! + +The compiler level of our hierarchy actually consists of two directories here: `Compiler/GCCcore/11.2.0` +and `Compiler/GCC/11.2.0`. The modules in the `GCCcore` directory are ones we can use in other compiler +toolchains that use GCC 10.2.0 as a base compiler (the details of that are out of scope here). + +The module we are interested in is `OpenMPI/4.1.1`, which is another gateway module. + +Remember that the "`module spider`" output told us that there does indeed exist a module for `HDF5`, but that +we need to load *both* the `GCC/11.2.0` and `OpenMPI/4.1.1` modules first. + +So, let us do exactly that (remember that `GCC/11.2.0` is already loaded): + +``` +module load OpenMPI/4.1.1 +``` + +If you now check the output of "`module avail`" again, you should see the `HDF5/1.12.1` module: + +``` +$ module avail + +-------- /home/easybuild/hmns/modules/all/MPI/GCC/11.2.0/OpenMPI/4.1.1 ------- + HDF5/1.12.1 + +------------ /home/easybuild/hmns/modules/all/Compiler/GCC/11.2.0 ------------ + OpenMPI/4.1.1 (L) + +... +``` + +To use HDF5, we need to load this `HDF5/1.12.1` module. We can verify that the installation works +using one of the commands provided by HDF5, `h5dump` for example: + +``` +module load HDF5/1.12.1 +``` + +``` +$ h5dump --version +h5dump: Version 1.12.1 +``` + +If you now check which modules are loaded via "`module list`", you will notice that all module names +and nice and short now, which is one of the advantages of using a hierarchical module tree: + +```shell +$ module list + +Currently Loaded Modules: + 1) GCCcore/11.2.0 5) numactl/2.0.14 9) hwloc/2.5.0 13) libfabric/1.13.2 17) HDF5/1.12.1 + 2) zlib/1.2.11 6) XZ/5.2.5 10) OpenSSL/1.1 14) PMIx/4.1.0 + 3) binutils/2.37 7) libxml2/2.9.10 11) libevent/2.1.12 15) OpenMPI/4.1.1 + 4) GCC/11.2.0 8) libpciaccess/0.16 12) UCX/1.11.2 16) Szip/2.1.1 +``` + + +--- + +*[[next: GitHub integration]](3_05_github_integration.md)* diff --git a/docs/2022-CSC_and_LO/3_Advanced/3_05_github_integration.md b/docs/2022-CSC_and_LO/3_Advanced/3_05_github_integration.md new file mode 100644 index 00000000..9900e1b3 --- /dev/null +++ b/docs/2022-CSC_and_LO/3_Advanced/3_05_github_integration.md @@ -0,0 +1,461 @@ +# GitHub integration to facilitate contributing to EasyBuild + +*[[back: Module naming schemes]](3_04_module_naming_schemes.md)* + +--- + +!!! Warning "Preliminary note for LUMI and many other Cray systems" + + Not everything in this section is as useful for EasyBuild installations that build + on the Cray Programming Environment. E.g., the default easyconfig repository does + not contain easyconfigs for the Cray PE. + + Moreover, due to the restricted operating system on the compute nodes of big + Cray systems such as the EX series (COS instead of the full SUSE linux), + the GitHub integration does not work on the compute nodes of a typical setup as + one of the required Python packages fails to work as it needs the DBus daemon. + + Since GitHub integration is not yet really used on LUMI, this section is not fully + tested but left in the tutorial for completeness. + +To contribute changes to the EasyBuild code (framework or easyblocks) or easyconfigs, +you will need to be a bit familiar with Git and GitHub. Or maybe not? + +## Manual contribution procedure + +0) Create and setup a [GitHub account](https://github.com/join) (and register your SSH public key); + +1) Clone and fork the appropriate GitHub repository, for example when contributing an easyconfig file: + +```shell +git clone git@github.com:easybuilders/easybuild-easyconfigs.git +cd easybuild-easyconfigs +git remote add my_fork git@github.com:your_github_account/easybuild-easyconfigs.git +``` + +**You should change '`your_github_account`' in the last line to your own GitHub user name!** + +2) Create and check out a new branch, starting from the (up-to-date) ``develop`` branch: + +``` +git checkout develop +git pull origin develop +git checkout -b example +``` + +3) Stage the changes you want to contribute, after you make sure that your easyconfig file has the +[correct filename](../../1_Intro/1_08_basic_usage/#easyconfig-filenames), and that it's located in the appropriate directory. + +```shell +mkdir -p easybuild/easyconfigs/e/example/ +mv example.eb easybuild/easyconfigs/e/example/example-1.2.3-GCC-9.3.0.eb +git add easybuild/easyconfigs/e/example/example-1.2.3-GCC-9.3.0.eb +``` + +4) Commit those changes with a sensible commit message: + +```shell +git commit -m "This is just an example" +``` + +5) Push your branch to your fork of the repository on GitHub: + +```shell +git push my_fork example +``` + +6) Open the pull request through the GitHub web interface, making sure that: + +* the target branch is correct (should be `develop`); +* an appropriate title is used; +* a short description of the changes is provided; +* the changes are indeed the ones you want to propose; +* clicking the (correct) green button; + + + +That didn't exactly motivate you to contribute, did it... + +## Github integration features + +Over the years we noticed that some people were keen on contributing to EasyBuild, +but they were not very familiar with Git or GitHub. That meant they had to overcome a +relatively steep learning curve before they could contribute... + + + +In addition, the contribution workflow can be a bit daunting and time consuming, +even if you're already familiar with the procedure. You will have dozens of +branches flying around in no time, and if you get stuck in a weird corner +with `git` you may quickly end up demotivated. + +This is frustrating not only for the people who wanted to contribute but +also for the EasyBuild maintainers, and it doesn't agree with the philosophy of +a project that aims to *automate* tedious software installation procedures. + +At the end of 2015 efforts were made to tackle this issue by implementing +GitHub integration features in EasyBuild, which automate the contribution +workflow by running `git` commands and interacting with the [GitHub API](https://docs.github.com/en/rest). + +We will briefly go over some of these features here, but they are also covered in detail [in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Integration_with_GitHub.html). + +### Requirements & configuration + +First of all, the GitHub integration features impose a couple of additional [requirements](https://docs.easybuild.io/en/latest/Integration_with_GitHub.html) +and configuration. + + +**Additional dependencies** + +Both the `GitPython` and `keyring` Python packages as well as the `keyrings.cryptfile` add-on package must be installed. + +!!! Note + You may experiences problems installing the ``cryptography`` Python packages, + which is a dependency of keyring. The underlying cause is that you need to have + the [``Rust``](https://www.rust-lang.org/) compiler installed to install the latest version + of ``cryptography`` (see [here](https://github.com/pyca/cryptography/issues/5771)). + + You can work around this issue using: + + ```shell + pip3 install --user 'cryptography<3.4' + ``` + +**SSH public key in GitHub account** + +You need to have a GitHub account that has your SSH public key registered in it +(via [https://github.com/settings/keys](https://github.com/settings/keys)). + +If you need to generate an SSH key pair, you can run the following command: + +```shell +ssh-keygen -t rsa -b 4096 +``` + +You can copy the SSH public key from the output of this command: + +```shell +cat .ssh/id_rsa.pub +``` + + +**Forked repository in GitHub** + +In addition, you must have *forked* the EasyBuild repository you want to contribute to +(for example [https://github.com/easybuilders/easybuild-easyconfigs](https://github.com/easybuilders/easybuild-easyconfigs)). + +**EasyBuild configuration, incl. GitHub token** + +You also have to configure EasyBuild a bit more, so it knows about your +GitHub user name *and* has a GitHub token available in order to perform actions +in GitHub with your credentials. + +To do this, you should define the `github-user` configuration option and +run the "`eb --install-github-token`" command: + +```shell +# replace 'ebtutorial' with your own GitHub username! +$ export EASYBUILD_GITHUB_USER=ebtutorial +$ eb --install-github-token +``` + +To create a GitHub token: + +* Visit [https://github.com/settings/tokens](https://github.com/settings/tokens). +* Click *"Personal access tokens"*. +* Click followed by *"Generate new token"*. +* Give the token a name (for example *"Token for EasyBuild"*). +* Select both the '`repo`' and '`gist`' scopes. +* Click the green *"Generate token"* button. +* Copy the generated token. +* Paste the token when asked by `--install-github-token` (and hit *Enter*). +* Enter a password to encrypt your GitHub token. + +The output should look something like this: + +```shell +$ eb --install-github-token +== temporary log file in case of crash /tmp/eb-9z0bdve9/easybuild-hfpti62w.log +Token: +Validating token... +Token seems to be valid, installing it. +Please set a password for your new keyring: +Please confirm the password: +Token 'fed..987' installed! +``` + + +**Checking status of GitHub integration** + +You can check the status of the GitHub integration using "`eb --check-github`": + +```shell +$ eb --check-github +== temporary log file in case of crash /tmp/eb-4ckdlyfy/easybuild-gp69ev2w.log + +Checking status of GitHub integration... + +Making sure we're online...OK + +* GitHub user...ebtutorial => OK +Please enter password for encrypted keyring: +* GitHub token...fed..987 (len: 40) => OK (validated) +* git command...OK ("git version 1.8.3.1; ") +* GitPython module...OK (GitPython version 3.1.3) +* push access to ebtutorial/easybuild-easyconfigs repo @ GitHub...OK +* creating gists...OK +* location to Git working dirs... not found (suboptimal) + +All checks PASSed! + +Status of GitHub integration: +* --from-pr: OK +* --new-pr: OK +* --review-pr: OK +* --update-pr: OK +* --upload-test-report: OK +``` + +If you see '`OK`' for each of the status checks, you're all set +to try out the GitHub integration features! + +!!! Note + If your SSH private key is protected with a password, you may need + to enter your password a couple of times when running "`eb --check-github`". + + You can avoid this by + [using an SSH agent](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent). + +!!! Note + If you see the ``push access`` check fail with ``Failed to fetch branch 'main'``, + you will need to rename the ``master`` branch in your fork of the ``easybuild-easyconfigs`` + repository from ``master`` to ``main`` (this is required since EasyBuild v4.3.3). + + This can be done via the *pencil* icon at + [https://github.com/YOUR_GITHUB_ACCOUNT/easybuild-easyconfigs/branches](https://github.com/YOUR_GITHUB_ACCOUNT/easybuild-easyconfigs/branches) + (replace ``YOUR_GITHUB_ACCOUNT`` with the name of your GitHub account in this URL). + +### Creating pull requests + +The GitHub integration in EasyBuild allows you to **create pull requests +using the `eb` command**, without even leaving your shell environment. +How cool is that‽ + +To create a pull request to the `easybuild-easyconfigs` repository, +you can either do it in a single go by +running "`eb --new-pr`" and passing it one or more easyconfig files to add +into the pull request. + +The more detailed option is to first create a branch in your repository fork +in GitHub via "`eb --new-branch-github`" and then later open the pull request +via "`eb --new-pr-from-branch`". This method can be useful when preparing multiple +interdependent pull requests, or to check whether your changes pass the unit tests +(which are run automatically in the GitHub Actions CI environment for +all branches pushed to your fork). + +The `--new-pr` option can also be used to open pull requests to the easyblocks +and framework repositories, and it will even automatically determine the target +repository based on the contents of the files you provide. Whoa! + +You can control the target repository for your pull request using +`--pr-target-account` (default is `easybuilders`) and `--pr-target-repo`. + +If you want you can carefully double check your work before actually +opening the pull request by doing a dry run via "`eb --dry-run --new-pr`" +or "`eb -D --new-pr`". + +Finally, you can use "`eb --preview-pr`" to see how the easyconfig files +for which you plan to create a pull request differ from existing easyconfig +files. + +### Updating pull requests + +To update an existing pull request with additional changes +you can use "`eb --update-pr`" and pass the pull request ID, +alongside the paths to the updated files. + +If you have only created a branch (for example via `eb --new-branch-github`) +you can update it via `--update-branch-github` in the same way, +passing the branch name instead of a pull request ID. + +### Using a pull request + +Next to creating and updating branches and pull requests +you can also *use* easyconfig files and easyblocks from a pull request, +regardless of its status (open, merged, or closed). This is particularly +useful when testing contributions, or to install software for which +support is not yet included in the latest EasyBuild release. + +Using the `--from-pr` option you can install easyconfig files from the +pull request with specified ID. By default all easyconfig files that are +touched by the pull request will be installed, but you can specify +particular ones to use as well. It is generally advised to also use the +`--robot` option to ensure that the easyconfig files are installed in the +correct order with respect to dependencies. + +Similarly, using a new or updated easyblock from a pull request is as simple +as using the `--include-easyblocks-from-pr` option. And of course you can +combine it with `--from-pr`! + +Via `--upload-test-report` you can let EasyBuild submit a comment into the +easyconfig pull request to show that the installation worked on your system. This is +useful for others to know, in particular EasyBuild maintainers, since the comment +will include information about your system (OS, processor, etc.) and your EasyBuild configuration. + +## Demo + +!!! Warning "Not currently suited for LUMI" + + This is text from a previous (non-Cray) version of the tutorial. The demo does require + a number of files not installed on LUMI after going through this tutorial, + and someone who can master the repository used to clean up again. + However, having a look at the output (which was generated in the spring of 2021 on + the CSC system puhti) still offers some information, so the demo was left in this + version of the tutorial. + +That is a lot to digest, so let us make this a bit more concrete with an example: +we will open a pull request for the [`eb-tutorial` example software](../../2_Using/2_02_creating_easyconfig_files/#example) to *a fork* of the [`easybuild-easyconfigs` repository](https://github.com/easybuilders/easybuild-easyconfigs) using the `eb` command, +and submit a test report in it. + +!!! Note + Make sure that you have correctly configured the GitHub integration, + [see above](#requirements-configuration). + +### Creating pull request + +We first configure EasyBuild to target the `ebtutorial` GitHub account rather +than the default `easybuilders` GitHub organisation, +by defining the `pr-target-account` configuration setting: + +```shell +export EASYBUILD_PR_TARGET_ACCOUNT=ebtutorial +``` + +In the output of "`eb --show-config`" you should see a line like this: + +``` +pr-target-account (E) = ebtutorial +``` + +We only do this to avoid that lots of pull requests for the `eb-tutorial` +example software are opened in the [central easyconfigs repository](https://github.com/easybuilders/easybuild-easyconfigs). + +Opening a pull request is as simple as running "`eb --new-pr`" and passing +the easyconfig file: + +```shell +$ eb --new-pr example.eb +== temporary log file in case of crash /tmp/eb-ggr6scbq/easybuild-hnk271xj.log +== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... +== fetching branch 'develop' from https://github.com/ebtutorial/easybuild-easyconfigs.git... +== copying files to /tmp/eb-ggr6scbq/git-working-dirxwk1fzaw/easybuild-easyconfigs... +== pushing branch '20200622095415_new_pr_eb-tutorial100' to remote 'github_ebtutorial_qgtfU' (git@github.com:ebtutorial/easybuild-easyconfigs.git) +Enter passphrase for key '/home/example/.ssh/id_rsa': +Please enter password for encrypted keyring: + +Opening pull request +* target: ebtutorial/easybuild-easyconfigs:develop +* from: ebtutorial/easybuild-easyconfigs:20200622095415_new_pr_eb-tutorial100 +* title: "{tools}[GCC/10.2.0] eb-tutorial v1.0.1" +* labels: new +* description: +""" +(created using `eb --new-pr`) + +""" +* overview of changes: + easybuild/easyconfigs/e/eb-tutorial/eb-tutorial-1.0.1-GCC-10.2.0.eb | 26 ++++++++++++++++++++++++++ + 1 file changed, 26 insertions(+) + +Opened pull request: https://github.com/ebtutorial/easybuild-easyconfigs/pull/ +== Temporary log file(s) /tmp/eb-ggr6scbq/easybuild-hnk271xj.log* have been removed. +== Temporary directory /tmp/eb-ggr6scbq has been removed. +``` + +Take a moment to grasp what we did here: we ran **a single `eb` command** which +took care of the **[whole contribution procedure](#contribution-procedure)** for us, including: + +* Cloning the `easybuilders/easybuild-easyconfigs` repository and checking out the `develop` branch (in a temporary + directory); +* Picking a sensible name for a branch and creating it; +* Adding the `eb-tutorial` easyconfig file to the branch, in the correct location + (`easybuild/easyconfigs/e/eb-tutorial/`) and with the correct filename (`eb-tutorial-1.0.1-GCC-10.2.0.eb`); +* Pushing the branch to our fork (`example/easybuild-easyconfigs`); +* Actually opening the pull request, using an informative title. + +That is so... easy! + +This feature not only *significantly* lowers the bar for contributing, +it also saves quite a bit of time since you don't need to double check +various details (like targeting the `develop` branch) or spend time on +coming up with a nice looking title or funny branch name (although you +still can if you really want to). + +There are a couple of nice side effects too, like not having any local branches +to tidy up on once the pull request gets merged (since `--new-pr` created the +branch only in a temporary directory). + +If many contributions are made via `--new-pr` it also simplifies the task +of EasyBuild maintainers, since pull requests opened this way have a particular +structure to them and thus are easier to digest because they look familiar. + +### Uploading test report + +After opening the pull request, we should also upload a test report to show that the installation is working. +This is just as easy as creating the pull request. + +First make sure that the pre-installed software in the prepared environment +is available, since the required dependencies for `eb-tutorial` are already +installed there: + +```shell +module use /easybuild/modules/all +``` + +You can verify which dependencies are still missing using `--from-pr` combined with `--missing`: + +```shell +# change '1' to the ID of your own pull request (see output of --new-pr) +$ eb --from-pr 1 --missing +== temporary log file in case of crash /tmp/eb-ioi9ywm1/easybuild-e3v0xa1b.log +Please enter password for encrypted keyring: +== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... + +1 out of 20 required modules missing: + +* eb-tutorial/1.0.1-GCC-10.2.0 (eb-tutorial-1.0.1-GCC-10.2.0.eb) +``` + +Uploading a test report boils down to combining `--from-pr` with `--upload-test-report`: + +```shell +# change '1' to the ID of your own pull request (see output of --new-pr) +$ eb --rebuild --from-pr 1 --upload-test-report +Please enter password for encrypted keyring: +... +== processing EasyBuild easyconfig /tmp/eb-bnb1pv3n/files_pr65/e/eb-tutorial/eb-tutorial-1.0.1-GCC-10.2.0.eb +== building and installing eb-tutorial/1.0.1-GCC-10.2.0... +... +== COMPLETED: Installation ended successfully (took 2 sec) +... +Adding comment to easybuild-easyconfigs issue #65: 'Test report by @ebtutorial +**SUCCESS** +Build succeeded for 1 out of 1 (1 easyconfigs in this PR) +example - Linux centos linux 7.8.2003, x86_64, Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz (haswell), Python 3.6.8 +See https://gist.github.com/f7c74159c809029afd99e30e4d994ef1 for a full test report.' +== Test report uploaded to https://gist.github.com/f7c74159c809029afd99e30e4d994ef1 and mentioned in a comment in easyconfigs PR#1 +``` + +Note that we may need to use `--rebuild` here since `eb-tutorial` may already be installed. + +This results in a comment being added to the pull request: + + + +The gist linked from this comment provides more detailed information: + + + +--- + +*[[next: Additional reading]](../4_00_additional_reading.md)* diff --git a/docs/2022-CSC_and_LO/3_Advanced/index.md b/docs/2022-CSC_and_LO/3_Advanced/index.md new file mode 100644 index 00000000..d7831506 --- /dev/null +++ b/docs/2022-CSC_and_LO/3_Advanced/index.md @@ -0,0 +1,15 @@ +# Part III: Advanced topics + +*[[back to start page]](index.md)* + +In this section we mostly cover "good to know that they exist" features as they are not used +on LUMI, or not really accessible to regular user installations that are performed with the +LUMI ``EasyBuild-user`` module. Hooks are used on LUMI, but it is not really advised to +overwrite the centrally defined hooks with a local file. And the whole structure of the +EasyBuild integration is also set up to make use of the GitHub integration in the future. + +* [Using EasyBuild as a library](3_01_easybuild_library.md) +* [Using hooks to customise EasyBuild](3_02_hooks.md) +* [Submitting installations as Slurm jobs](3_03_slurm_jobs.md) +* [Module naming schemes (incl. hierarchical)](3_04_module_naming_schemes.md) +* [GitHub integration to facilitate contributing to EasyBuild](3_05_github_integration.md) diff --git a/docs/2022-CSC_and_LO/4_00_additional_reading.md b/docs/2022-CSC_and_LO/4_00_additional_reading.md new file mode 100644 index 00000000..14f8454c --- /dev/null +++ b/docs/2022-CSC_and_LO/4_00_additional_reading.md @@ -0,0 +1,54 @@ +# Additional reading + +*[[back: GitHub integration]](3_Advanced/3_05_github_integration.md)* + +--- + +## EasyBuild + +- [EasyBuild documentation](https://docs.easybuild.io/en/latest/#) +- [EasyBuild web site](https://easybuild.io/) containing links to much of the recent information on + EasyBuild. +- [EasyBuild YouTube channel](https://www.youtube.com/c/EasyBuilders) +- This tutorial is an evolution of the + [EasyBuild tutorial prepared for the LUMI User Support Team, spring '21](https://easybuilders.github.io/easybuild-tutorial/2021-lust/) + given by Kenneth Hoste (UGent, EasyBuild lead developer) and Luca Marsella (CSCS) + - [Recordings are available on YouTube](https://www.youtube.com/watch?v=JTRw8hqi6x0&list=PLhnGtSmEGEQh573bk3BeOj_KCRBBiA5OT) +- The EasyBuild setup on LUMI is partly insprired on the setup used at CSCS on their Cray systems + - [Section on the CSCS setup](https://easybuilders.github.io/easybuild-tutorial/2021-lust/cray/easybuild_at_cscs/) + from the LUST EasyBuild tutorial spring '21. + - [CSCS site presentation](https://easybuild.io/eum21/#eb-site-talk-cscs) + during the [6th EasyBuild User Meeting in 2021](https://easybuild.io/eum21/) + - CSCS site presentation ([video](https://www.youtube.com/watch?v=rhoHmUhzSGw&list=PLhnGtSmEGEQidEM8MZKkOaVutgt9WmqI0)/[slides](https://users.ugent.be/~kehoste/eum20/eum20_11_luca_cscs.pdf)) + during the [5h EasyBuild User Meeting in 2020](https://github.com/easybuilders/easybuild/wiki/5th-EasyBuild-User-Meeting) + - CSCS site presentation ([video](https://www.youtube.com/watch?list=PLhnGtSmEGEQhEjG5hD70tRwL3n58aWNHc&t=1451&v=qLuHeaFmydM&feature=youtu.be)) + during the [4h EasyBuild User Meeting in 2020](https://github.com/easybuilders/easybuild/wiki/4th-EasyBuild-User-Meeting) + +## HPE Cray Programming Environment + +- [LUMI documentation: "Developing" section](https://docs.lumi-supercomputer.eu/development/) +- The Cray PE is mostly documented through man pages. There used to be some documentation on the + Cray web site also but the documentation system got reworked after the merger with HPE. + The documentation is now in the [HPE Support Centre](https://support.hpe.com/) where it + is very difficult to find the right version of the documents. +- The [PE-Cray](https://github.com/PE-Cray) GitHub project also provides some additional + documentation, including + - [some whitepapers](https://github.com/PE-Cray/whitepapers), + - [documentation for Cray OpenSMHEMX](https://github.com/PE-Cray/cray-openshmemx) and + - [information about Cray DSMML](https://github.com/PE-Cray/cray-dsmml) + though much of that information ia also not updated regularly anymore. + + +## LUMI + +- [LUMI web site](https://lumi-supercomputer.eu/) +- [LUMI User Documentation](https://docs.lumi-supercomputer.eu/) + - [Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/) + - [Organisation of the software stack](https://docs.lumi-supercomputer.eu/computing/softwarestacks/) + - [EasyBuild on LUMI](https://docs.lumi-supercomputer.eu/software/installing/easybuild/) +- [EasyBuild on LUMI presentation](https://easybuild.io/eum22/#lumi) + ([video](https://www.youtube.com/watch?v=hZezVG6lJNk&list=PLhnGtSmEGEQgCneeSQvYoIZrbv7wIKlo2)/[slides](https://easybuild.io/eum22/001_eum22_EasyBuild_on_LUMI.pdf)) + during the [7th EasyBuild User Meeting in 2022](https://easybuild.io/eum22/) + + +*[[next: Overview]](index.md)* diff --git a/docs/2022-CSC_and_LO/index-lust.md b/docs/2022-CSC_and_LO/index-lust.md new file mode 100644 index 00000000..b8cee2ab --- /dev/null +++ b/docs/2022-CSC_and_LO/index-lust.md @@ -0,0 +1,123 @@ +<<<<<<< HEAD +# EasyBuild tutorial for CSC and the LUMI consortium +======= +# EasyBuild tutorial for LUST + +Overview page of the introductory tutorial on [EasyBuild](https://easybuild.io) for the CSC +and Local Organizations who want to contribute to the LUMI EasyBuild setup. +>>>>>>> 5573462b (Initial commit, already contains the structure to work on the first tutorial.) + +This tutorial is based extensively on the more generic EasyBuild tutorials build by +the EasyBuild community and maintained on +[the EasyBuild tutorial site](https://easybuilders.github.io/easybuild-tutorial/). +See that site for [credits to the contributors of those tutorials](https://easybuilders.github.io/easybuild-tutorial/#contributors). + +<<<<<<< HEAD +## Scope + +This is an introductory tutorial to [EasyBuild](https://easybuild.io), +a command line tool for installing (scientific) software on High Performance Computing (HPC) systems. +This tutorial is specifically for EasyBuild as implemented on [LUMI](https://lumi-supercomputer.eu) +and has been developed for CSC and the Local Organisations within the LUMI consortium. Yet +much of the material is useful to a broader community of EasyBuild users on Cray systems +or even EasyBuild users in general. +For more generic EasyBuild tutorials, see the [EasyBuild tutorial site](https://easybuilders.github.io/easybuild-tutorial/). + +This tutorial aims to explain the core concepts of EasyBuild, +get you started with using it, make you familiar with some of the features it provides, +and show how it is used on LUMI to maintain the central software stacks and offer the users +an easy environment to install packages on top of the central stack and thus create their own +customised environment. + +Through hands-on exercises and demos, you will learn how EasyBuild can help you +to get scientific software installed in an efficient way. + + +## Intended audience + +This tutorial is primarily intended for people new to EasyBuild, but even if you're already familiar +with the project it could be interesting to step through it. + +Our main target audience includes: + +- application experts in LUST and the local organizations who want to contribute to the + software stack on LUMI or support their users; +- developers who want to make their developments available to LUMI users; +- advanced users who want to customize available build recipes or develop their own recipes. + + +## Prerequisites + +We expect you to be (a little bit) familiar with: + +- using a Linux command line interface; +- the (absolute) basics of compiling software from source. + +EasyBuild requires: + +- GNU/Linux (any distribution), +- Python 2.7 or 3.5+, though a fairly recent version of Python 3 is highly recommended, +- an environment modules tool (see the ``module`` command). On LUMI we use [Lmod](https://lmod.readthedocs.io), + a modern environment modules tool implemented in Lua. + +However, the LUMI version of the tutorial is currently specifically for the Cray Programming Environment which is not +freely available, so unless you have access to a system with this environment you cannot really do local development. + + +## Contents + +- [Part I: **Introduction to EasyBuild on Cray systems**](1_Intro/index.md) + - [What is EasyBuild?](1_Intro/1_01_what_is_easybuild.md) + - [The Lmod module system](1_Intro/1_02_Lmod.md) + - [The HPE Cray Programming Environment](1_Intro/1_03_CPE.md) + - [LUMI software stacks](1_Intro/1_04_LUMI_software_stack.md) + - [Terminology](1_Intro/1_05_terminology.md) + - [Installation](1_Intro/1_06_installation.md) + - [Configuration](1_Intro/1_07_configuration.md) + - [Basic usage](1_Intro/1_08_basic_usage.md) *(hands-on)* +- [Part II: **Using EasyBuild**](2_Using/index.md) + - [Troubleshooting](2_Using/2_01_troubleshooting.md) *(hands-on)* + - [Creating easyconfig files](2_Using/2_02_creating_easyconfig_files.md) *(hands-on)* + - [Using external modules from the Cray PE](2_Using/2_03_external_modules.md) + - [Implementing easyblocks](2_Using/2_04_implementing_easyblocks.md) *(hands-on)* +- [Part III: **Advanced topics**](3_Advanced/index.md) + - [Using EasyBuild as a library](3_Advanced/3_01_easybuild_library.md) + - [Using hooks to customise EasyBuild](3_Advanced/3_02_hooks.md) + - [Submitting installations as Slurm jobs](3_Advanced/3_03_slurm_jobs.md) + - [Module naming schemes (incl. hierarchical)](3_Advanced/3_04_module_naming_schemes.md) + - [GitHub integration to facilitate contributing to EasyBuild](3_Advanced/3_05_github_integration.md) +- [**Additional reading**](4_00_additional_reading.md) +======= +## Contents + +TODO + + + + + + +## OLD TEXT + +- [Part I: **Introduction to EasyBuild**](part1_intro.md) *(Tue March 9th 2021, 9am-12 CET)* + * [What is EasyBuild?](what_is_easybuild.md) + * [Terminology](terminology.md) + * [Installation](installation.md) *(hands-on)* + * [Configuration](configuration.md) *(hands-on)* + * [Basic usage](basic_usage.md) *(hands-on)* +- [Part II: **Using EasyBuild**](part2_using.md) *(Tue March 23rd 2021, 9am-12 CET)* + * [Troubleshooting](troubleshooting.md) *(hands-on)* + * [Creating easyconfig files](creating_easyconfig_files.md) *(hands-on)* + * [Implementing easyblocks](implementing_easyblocks.md) *(hands-on)* +- [Part III: **Advanced topics**](part3_advanced.md) *(Tue March 30th 2021, 9am-12 CEST)* + * [Using EasyBuild as a library](easybuild_library.md) *(hands-on)* + * [Using hooks to customise EasyBuild](hooks.md) *(hands-on)* + * [Submitting installations as Slurm jobs](slurm_jobs.md) *(hands-on)* + * [Module naming schemes (incl. hierarchical)](module_naming_schemes.md) *(hands-on)* + * [GitHub integration to facilitate contributing to EasyBuild](github_integration.md) *(hands-on)* +- [Part IV: **EasyBuild on Cray systems**](part4_cray.md) *(Friday June 18th 2021, 09-12 CEST)* + * [Introduction to Cray Programming Environment](cray/introduction.md) *(hands-on)* + * [Cray External Modules](cray/external_modules.md) *(hands-on)* + * [Cray Custom Toolchains](cray/custom_toolchains.md) *(hands-on)* + * [EasyBuild at CSCS](cray/easybuild_at_cscs.md) *(hands-on)* +>>>>>>> 5573462b (Initial commit, already contains the structure to work on the first tutorial.) diff --git a/docs/2022-CSC_and_LO/index.md b/docs/2022-CSC_and_LO/index.md new file mode 100644 index 00000000..1de6a052 --- /dev/null +++ b/docs/2022-CSC_and_LO/index.md @@ -0,0 +1,81 @@ +# EasyBuild tutorial for CSC and the LUMI consortium + +This tutorial is based extensively on the more generic EasyBuild tutorials build by +the EasyBuild community and maintained on +[this site](https://easybuilders.github.io/easybuild-tutorial/). +See that site for [credits to the contributors of those tutorials](https://easybuilders.github.io/easybuild-tutorial/#contributors). + +## Scope + +This is an introductory tutorial to [EasyBuild](https://easybuild.io), +a command line tool for installing (scientific) software on High Performance Computing (HPC) systems. +This tutorial is specifically for EasyBuild as implemented on [LUMI](https://lumi-supercomputer.eu) +and has been developed for CSC and the Local Organisations within the LUMI consortium. Yet +much of the material is useful to a broader community of EasyBuild users on Cray systems +or even EasyBuild users in general. +For more generic EasyBuild tutorials, see the [EasyBuild tutorial site](https://easybuilders.github.io/easybuild-tutorial/). + +This tutorial aims to explain the core concepts of EasyBuild, +get you started with using it, make you familiar with some of the features it provides, +and show how it is used on LUMI to maintain the central software stacks and offer the users +an easy environment to install packages on top of the central stack and thus create their own +customised environment. + +Through hands-on exercises and demos, you will learn how EasyBuild can help you +to get scientific software installed in an efficient way. + + +## Intended audience + +This tutorial is primarily intended for people new to EasyBuild, but even if you're already familiar +with the project it could be interesting to step through it. + +Our main target audience includes: + +- application experts in LUST and the local organizations who want to contribute to the + software stack on LUMI or support their users; +- developers who want to make their developments available to LUMI users; +- advanced users who want to customize available build recipes or develop their own recipes. + + +## Prerequisites + +We expect you to be (a little bit) familiar with: + +- using a Linux command line interface; +- the (absolute) basics of compiling software from source. + +EasyBuild requires: + +- GNU/Linux (any distribution), +- Python 2.7 or 3.5+, though a fairly recent version of Python 3 is highly recommended, +- an environment modules tool (see the ``module`` command). On LUMI we use [Lmod](https://lmod.readthedocs.io), + a modern environment modules tool implemented in Lua. + +However, the LUMI version of the tutorial is currently specifically for the Cray Programming Environment which is not +freely available, so unless you have access to a system with this environment you cannot really do local development. + + +## Contents + +- [Part I: **Introduction to EasyBuild on Cray systems**](1_Intro/index.md) + - [What is EasyBuild?](1_Intro/1_01_what_is_easybuild.md) + - [The Lmod module system](1_Intro/1_02_Lmod.md) + - [The HPE Cray Programming Environment](1_Intro/1_03_CPE.md) + - [LUMI software stacks](1_Intro/1_04_LUMI_software_stack.md) + - [Terminology](1_Intro/1_05_terminology.md) + - [Installation](1_Intro/1_06_installation.md) + - [Configuration](1_Intro/1_07_configuration.md) + - [Basic usage](1_Intro/1_08_basic_usage.md) *(hands-on)* +- [Part II: **Using EasyBuild**](2_Using/index.md) + - [Troubleshooting](2_Using/2_01_troubleshooting.md) *(hands-on)* + - [Creating easyconfig files](2_Using/2_02_creating_easyconfig_files.md) *(hands-on)* + - [Using external modules from the Cray PE](2_Using/2_03_external_modules.md) + - [Implementing easyblocks](2_Using/2_04_implementing_easyblocks.md) *(hands-on)* +- [Part III: **Advanced topics**](3_Advanced/index.md) + - [Using EasyBuild as a library](3_Advanced/3_01_easybuild_library.md) + - [Using hooks to customise EasyBuild](3_Advanced/3_02_hooks.md) + - [Submitting installations as Slurm jobs](3_Advanced/3_03_slurm_jobs.md) + - [Module naming schemes (incl. hierarchical)](3_Advanced/3_04_module_naming_schemes.md) + - [GitHub integration to facilitate contributing to EasyBuild](3_Advanced/3_05_github_integration.md) +- [**Additional reading**](4_00_additional_reading.md) diff --git a/mkdocs.yml b/mkdocs.yml index 1e3ef2c9..c7c97216 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -33,6 +33,32 @@ nav: - Contributing to EasyBuild: 2022-isc22/contributing.md - Comparison with other tools: 2022-isc22/comparison_other_tools.md - Archive: + - "CSC (spring '22)": + - (overview): 2022-CSC_and_LO/index.md + - Introduction to EasyBuild: + - (overview): 2022-CSC_and_LO/1_Intro/index.md + - What is EasyBuild?: 2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md + - The Lmod module system: 2022-CSC_and_LO/1_Intro/1_02_Lmod.md + - The HPE Cray Programming Environment: 2022-CSC_and_LO/1_Intro/1_03_CPE.md + - LUMI software stacks: 2022-CSC_and_LO/1_Intro/1_04_LUMI_software_stack.md + - Terminology: 2022-CSC_and_LO/1_Intro/1_05_terminology.md + - Installation: 2022-CSC_and_LO/1_Intro/1_06_installation.md + - Configuration: 2022-CSC_and_LO/1_Intro/1_07_configuration.md + - Basic usage: 2022-CSC_and_LO/1_Intro/1_08_basic_usage.md + - Using EasyBuild: + - (overview): 2022-CSC_and_LO/2_Using/index.md + - Troubleshooting: 2022-CSC_and_LO/2_Using/2_01_troubleshooting.md + - Creating easyconfig files: 2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md + - Using external modules from the Cray PE: 2022-CSC_and_LO/2_Using/2_03_external_modules.md + - Implementing easyblocks: 2022-CSC_and_LO/2_Using/2_04_implementing_easyblocks.md + - Advanced topics: + - (overview): 2022-CSC_and_LO/3_Advanced/index.md + - Using EasyBuild as a library: 2022-CSC_and_LO/3_Advanced/3_01_easybuild_library.md + - Using hooks to customise EasyBuild: 2022-CSC_and_LO/3_Advanced/3_02_hooks.md + - Submitting installations as Slurm jobs: 2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md + - Module naming schemes (incl. hierarchical): 2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md + - GitHub integration to facilitate contributing to EasyBuild: 2022-CSC_and_LO/3_Advanced/3_05_github_integration.md + - Additional reading: 2022-CSC_and_LO/4_00_additional_reading.md - "ISC'21 (25 June 2021)": - (overview): 2021-isc21/index.md - Practical info: 2021-isc21/practical_info.md @@ -92,6 +118,33 @@ nav: - Contributing to EasyBuild: 2020-06-isc20/contributing.md - Comparison with other tools: 2020-06-isc20/comparison_other_tools.md - Getting help: 2020-06-isc20/getting_help.md + - "CSC (spring '22)": + - (overview): 2022-CSC_and_LO/index.md + - Introduction to EasyBuild: + - (overview): 2022-CSC_and_LO/1_Intro/index.md + - What is EasyBuild?: 2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md + - The Lmod module system: 2022-CSC_and_LO/1_Intro/1_02_Lmod.md + - The HPE Cray Programming Environment: 2022-CSC_and_LO/1_Intro/1_03_CPE.md + - LUMI software stacks: 2022-CSC_and_LO/1_Intro/1_04_LUMI_software_stack.md + - Terminology: 2022-CSC_and_LO/1_Intro/1_05_terminology.md + - Installation: 2022-CSC_and_LO/1_Intro/1_06_installation.md + - Configuration: 2022-CSC_and_LO/1_Intro/1_07_configuration.md + - Basic usage: 2022-CSC_and_LO/1_Intro/1_08_basic_usage.md + - Using EasyBuild: + - (overview): 2022-CSC_and_LO/2_Using/index.md + - Troubleshooting: 2022-CSC_and_LO/2_Using/2_01_troubleshooting.md + - Creating easyconfig files: 2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md + - Using external modules from the Cray PE: 2022-CSC_and_LO/2_Using/2_03_external_modules.md + - Implementing easyblocks: 2022-CSC_and_LO/2_Using/2_04_implementing_easyblocks.md + - Advanced topics: + - (overview): 2022-CSC_and_LO/3_Advanced/index.md + - Using EasyBuild as a library: 2022-CSC_and_LO/3_Advanced/3_01_easybuild_library.md + - Using hooks to customise EasyBuild: 2022-CSC_and_LO/3_Advanced/3_02_hooks.md + - Submitting installations as Slurm jobs: 2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md + - Module naming schemes (incl. hierarchical): 2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md + - GitHub integration to facilitate contributing to EasyBuild: 2022-CSC_and_LO/3_Advanced/3_05_github_integration.md + - Additional reading: 2022-CSC_and_LO/4_00_additional_reading.md + plugins: # show revision date at bottom of each page - git-revision-date-localized @@ -119,7 +172,11 @@ markdown_extensions: # notes, warnings, hints, ... - admonition # code blocks with syntax highlighting, graphs - - pymdownx.superfences + - pymdownx.superfences: + custom_fences: + - name: mermaid + class: mermaid + format: !!python/name:pymdownx.superfences.fence_code_format # clickable details - pymdownx.details # tabbed contents