How to Use TDAQ Software in Containers
While the ATLAS Trigger/DAQ software is typically running on bare metal or virtual machines, it can also be used inside a container. Use case are
- Building software - e.g. all TDAQ nightly and release builds are done in a container
- Developing software - e.g. if the host system is different from Alma/RedHat like Ubuntu, Debian, ...
- Running software - e.g. to learn about the TDAQ Software by running partitions and using the IGUI
We discuss the following container technologies: apptainer
, podman
and docker
, where the latter two
are almost identical for the user. We use podman
in the examples, and point out the few scenarios where
docker
is different.
TDAQ Images
TDAQ provides images for the use of the software in various formats. The source of all these images is always an OCI image, if necessary converted to another format. This guarantees a single source of truth on what will be part of the container.
Build Containers
Build containers are simply OS images where all the necessary dependencies to build and run the full TDAQ software are installed, but not the TDAQ software itself. They are created in the tdaq_ci project and are available in the associated container registry.
The images contain the hardware tag as used in the TDAQ and LCG software:
gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:x86_64-el9
gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:aarch64-el9
gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:x86_64-centos7
gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:aarch64-centos7
From EL 9 onward users should simply use multi-arch image that supports both x86 and arm64:
gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:el9
These images are used to bootstrap the TDAQ software from scratch, or they are used together with CVMFS which provides the pre-built TDAQ software (both nightlies and releases).
Full Stack Containers
These are inclusive containers that contain the full TDAQ and LCG stack. They are built on top of the previously described containers in this project. They can be used independently from CVMFS or where CVMFS is not available. Their disadvantage is their large size (about 12 GByte).
At the moment a multi-arch image of release tdaq-11-02-00
is available:
registry.cern.ch/atlas-tdaq/tdaq:11.2.0
Note that only the default configuration of a given release (here $ARCH-el9-gcc13-opt
) is made available
like this. Private images for a different configuration can be created easily following the example of the
gitlab project referenced above.
Podman
podman is available in most modern Linux distributions, as well as on lxplus
.
Since it requires no special privileges it might be easier to get it installed on a shared system than docker
.
In the following the podman
command can be replaced with docker
if only the latter is available.
See that later section on lxplus
about special options for that cluster.
Use with CVMFS
If CVMFS is available, the TDAQ software can be used in straightforward way:
mkdir work
podman run -it --rm -v $HOME:$HOME -w $HOME --userns=keep-id -v /cvmfs:/cvmfs:ro,shared gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:el9
% cm_setup tdaq-11-02-00
We bind mount the /cvmfs
directory into the container, so all the setup scripts will find the TDAQ software at
the expected place. Note that cm_setup
is an alias that works out of the box inside the container.
Here we also make the $HOME
directory from the host available to the container. The --userns=keep-id
makes sure that ownership on the host and inside the container
are the same. We also make the $HOME
directory the default work directory. This is most useful for developing software, otherwise you can omit these options.
Customize the Container
If you are developing new software and need additional system packages, they can be easily installed by not using the --userns=keep-id
option.
In this case you will have root access inside the container and can e.g. run dnf
:
podman run -it --rm gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:el9
# dnf install ...
Of course, you would have to repeat this every time you start a new container. In this case it is better to build your
own customized container with a Dockerfile
:
FROM gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:el9
RUN dnf install -y ... && dnf clean all
podman build -t my/tdaq .
podman run ... my/tdaq
Use without CVMFS
If no CVMFS is available you can use the full stack container. Usage is otherwise almost the same:
mkdir work
podman run -it --rm --userns=keep-id -v $HOME:$HOME -w $HOME registry.cern.ch/atlas-tdaq/tdaq:11.2.0
% cm_setup tdaq-11-02-00
After the initial download operations should be faster than with CVMFS, since the latter downloads and caches things on-demand as they are used.
Use of X11 Based Applications
We need a few more argument to have the container be able to run X11 based applications, like the IGUI:
podman run -it --rm --userns=keep-id -v /tmp/.X11-unix/X0:/tmp/X11-unix/X0 -v $HOME:$HOME -w $HOME -v /cvmfs:/cvmfs:ro,shared gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:el9
% cm_setup tdaq-11-02-00
% mkdir /tmp/work && cd /tmp/work
% export TDAQ_IPC_INIT_REF=file:$(pwd)/init.ref
% pm_part_hlt.py -p test
% setup_daq test.data.xml initial
% setup_daq test.data.xml test
You should now get an IGUI
display on your screen and be able to interact with the test
partition.
lxplus
On lxplus make sure to have the following in the file $HOME/.config/containers/storage.conf
:
[storage]
driver = "overlay"
[storage.options.overlay]
ignore_chown_errors = "true"
mount_program = "/usr/bin/fuse-overlayfs"
Advanced Topics: Installation into a Volume
When no CVMFS is available (or its use not desired) there is another option on how to get the TDAQ software. The idea is to install the software from RPMs into
a podman
/docker
volume and then attach the volume the container you want to run. This keeps the OS image and the TDAQ software separated in a way, and both
can be updated independently.
To install the TDAQ software into a volume:
curl -L https://gitlab.cern.ch/api/v4/projects/141622/packages/generic/atlas-dnf5/v5.1.9.3/atlas-dnf5.podman
chmod a+x atlas-dnf5.podman
export ATLAS_DNF5_INSTALL_DIR=/sw/atlas
export ATLAS_DNF5_HOST_INSTALL_DIR=atlas-sw
export ATLAS_DNF5_STATE_DIR=atlas-sw-state
./atlas-dnf5.podman install -y tdaq-11-02-00_x86_64-el9-gcc13-opt
And then run it:
podman run -it --rm -v atlas-sw:/sw/atlas gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:el9
# cm_setup tdaq-11-02-00
As new TDAQ patches become available:
export ATLAS_DNF5_INSTALL_DIR=/sw/atlas
export ATLAS_DNF5_HOST_INSTALL_DIR=atlas-sw
export ATLAS_DNF5_STATE_DIR=atlas-sw-state
./atlas-dnf5.podman upgrade
docker
If you replace podman
above with docker
you will realize that files created inside the container in the work
area are not owned by you but by root
. You should add the
following additional option to each run
command instead of the --userns=keep-id
option:
docker run --user $(id -u):$(id -g) ...
Even if your user name is not known inside the container, you will have the same user and group ID and files created inside the container can be modified by you on the host side.
apptainer
apptainer is requires no special privileges beyond the kernel supporting user namespaces.
The optional suid
features of apptainer
are not needed for most use cases. It can be installed from rpm
or deb
packages available on their release page on github. apptainer
is available on lxplus.cern.ch
. It provides the easiest integration with the user's host environment, as it mounts the home directory and the current working directory by default
and makes user/group ids identical.
If CVMFS is available and the host system is Fedora or RedHat (or clone) based, the apptainer
installation of ATLAS can be used directly. Putting the following in
your .bashrc
selects either a local installation (preferred if available) or the CVMFS version:
command -v apptainer > /dev/null || alias apptainer=/cvmfs/atlas.cern.ch/repo/containers/sw/apptainer/$(arch)-el9/current/bin/apptainer
For other Linux distributions you need a local installation.
Use with CVMFS
OCI images have to converted for use by apptainer
. This has already be done for you and the images are available on CVMFS:
apptainer shell -p -B /cvmfs /cvmfs/unpacked.cern.ch/gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:x86_64-el9
apptainer> cm_setup tdaq-11-02-00
The -p
option enables a private PID namespace, so inside the container you see only the processes started there, not the host processes.
Note that you have to use the full hardware tag here, as apptainer
images are not multi arch capable.
Note also that the full stack image is not available on CVMFS, as using the image and the software from the existing places is equivalent to it.
Use without CVMFS
If don't want to use the CVMFS image, you can use the following syntax
apptainer shell -p -B /cvmfs docker://gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:x86_64-el9
You can also get your own apptainer
image in SIF format in the following way:
apptainer build tdaq-x86_64-el9.sif docker://gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:x86_64-el9
apptainer shell -p -B /cvmfs tdaq-x86_64-el9.sif
Apptainer> cm_setup tdaq-11-02-00
If you have no CVMFS available at all, you need again the full stack container:
apptainer shell -p tdaq-11-02-00.sif docker://registry.cern.ch/tdaq:11.2.0
or
apptainer build tdaq-11-02-00.sif docker://registry.cern.ch/tdaq:11.2.0
apptainer shell -p -B /cvmfs tdaq-11-02-00.sif
Apptainer> cm_setup tdaq-11-02-00
Use of X11 Based Applications
Unlike podman
and docker
the X11 based applications should work out of the box with apptainer
.
Modifications to the Image
Unlike podman
and docker
, apptainer
does not allow easy modifications of the image as it by default
runs as unprivileged user inside the container. Depending on your use case, please check the documentation
about overlays in apptainer
, as well as the --fakeroot
and --writable
or --writable-tmpfs
options.
Advanced Topic: Running a different architecture
Both podman
and docker
allow you to run an architecture different from the host if the relevant qemu
binaries are
installed. E.g. on Ubuntu
sudo apt install qemu-user-static
On Alma 9 (on a system where you have root access):
sudo podman run --rm --privileged docker.io/alekitto/qemu-user-static --reset -p yes
and then
podman run -it --rm --platform linux/arm64 -v /cvmfs:/cvmfs:ro,shared gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:el9
# arch
will allow you to do basic development for aarch64 architecture on an Intel/AMD host. Performance will be vastly inferior to a native system, though.
For apptainer
you have to specify the correct image as it has no equivalent to the --platform
or --arch
options
of podman
and docker
:
apptainer shell -B /cvmfs gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:aarch64-el9
# arch