How to Use TDAQ Software in Containers

While the ATLAS Trigger/DAQ software is typically running on bare metal or virtual machines, it can also be used inside a container. Use case are

  • Building software - e.g. all TDAQ nightly and release builds are done in a container
  • Developing software - e.g. if the host system is different from Alma/RedHat like Ubuntu, Debian, ...
  • Running software - e.g. to learn about the TDAQ Software by running partitions and using the IGUI

We discuss the following container technologies: apptainer, podman and docker, where the latter two are almost identical for the user. We use podman in the examples, and point out the few scenarios where docker is different.

TDAQ Images

TDAQ provides images for the use of the software in various formats. The source of all these images is always an OCI image, if necessary converted to another format. This guarantees a single source of truth on what will be part of the container.

Build Containers

Build containers are simply OS images where all the necessary dependencies to build and run the full TDAQ software are installed, but not the TDAQ software itself. They are created in the tdaq_ci project and are available in the associated container registry.

The images contain the hardware tag as used in the TDAQ and LCG software:

  • gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:x86_64-el9
  • gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:aarch64-el9
  • gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:x86_64-centos7
  • gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:aarch64-centos7

From EL 9 onward users should simply use multi-arch image that supports both x86 and arm64:

  • gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:el9

These images are used to bootstrap the TDAQ software from scratch, or they are used together with CVMFS which provides the pre-built TDAQ software (both nightlies and releases).

Full Stack Containers

These are inclusive containers that contain the full TDAQ and LCG stack. They are built on top of the previously described containers in this project. They can be used independently from CVMFS or where CVMFS is not available. Their disadvantage is their large size (about 12 GByte).

At the moment a multi-arch image of release tdaq-11-02-00 is available:

  • registry.cern.ch/atlas-tdaq/tdaq:11.2.0

Note that only the default configuration of a given release (here $ARCH-el9-gcc13-opt) is made available like this. Private images for a different configuration can be created easily following the example of the gitlab project referenced above.

Podman

podman is available in most modern Linux distributions, as well as on lxplus. Since it requires no special privileges it might be easier to get it installed on a shared system than docker. In the following the podman command can be replaced with docker if only the latter is available.

See that later section on lxplus about special options for that cluster.

Use with CVMFS

If CVMFS is available, the TDAQ software can be used in straightforward way:

mkdir work
podman run -it --rm -v $HOME:$HOME -w $HOME --userns=keep-id -v /cvmfs:/cvmfs:ro,shared gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:el9
% cm_setup tdaq-11-02-00

We bind mount the /cvmfs directory into the container, so all the setup scripts will find the TDAQ software at the expected place. Note that cm_setup is an alias that works out of the box inside the container.

Here we also make the $HOME directory from the host available to the container. The --userns=keep-id makes sure that ownership on the host and inside the container are the same. We also make the $HOME directory the default work directory. This is most useful for developing software, otherwise you can omit these options.

Customize the Container

If you are developing new software and need additional system packages, they can be easily installed by not using the --userns=keep-id option. In this case you will have root access inside the container and can e.g. run dnf:

podman run -it --rm gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:el9
# dnf install ...

Of course, you would have to repeat this every time you start a new container. In this case it is better to build your own customized container with a Dockerfile:

FROM gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:el9
RUN dnf install -y ... && dnf clean all

podman build -t my/tdaq .
podman run ... my/tdaq

Use without CVMFS

If no CVMFS is available you can use the full stack container. Usage is otherwise almost the same:

mkdir work
podman run -it --rm --userns=keep-id -v $HOME:$HOME -w $HOME registry.cern.ch/atlas-tdaq/tdaq:11.2.0
% cm_setup tdaq-11-02-00

After the initial download operations should be faster than with CVMFS, since the latter downloads and caches things on-demand as they are used.

Use of X11 Based Applications

We need a few more argument to have the container be able to run X11 based applications, like the IGUI:

podman run -it --rm --userns=keep-id -v /tmp/.X11-unix/X0:/tmp/X11-unix/X0  -v $HOME:$HOME -w $HOME -v /cvmfs:/cvmfs:ro,shared gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:el9
% cm_setup tdaq-11-02-00
% mkdir /tmp/work && cd /tmp/work
% export TDAQ_IPC_INIT_REF=file:$(pwd)/init.ref
% pm_part_hlt.py -p test
% setup_daq test.data.xml initial
% setup_daq test.data.xml test

You should now get an IGUI display on your screen and be able to interact with the test partition.

lxplus

On lxplus make sure to have the following in the file $HOME/.config/containers/storage.conf:

[storage]
driver = "overlay"

[storage.options.overlay]
ignore_chown_errors = "true"
mount_program = "/usr/bin/fuse-overlayfs"

Advanced Topics: Installation into a Volume

When no CVMFS is available (or its use not desired) there is another option on how to get the TDAQ software. The idea is to install the software from RPMs into a podman/docker volume and then attach the volume the container you want to run. This keeps the OS image and the TDAQ software separated in a way, and both can be updated independently.

To install the TDAQ software into a volume:

curl -L https://gitlab.cern.ch/api/v4/projects/141622/packages/generic/atlas-dnf5/v5.1.9.3/atlas-dnf5.podman
chmod a+x atlas-dnf5.podman
export ATLAS_DNF5_INSTALL_DIR=/sw/atlas
export ATLAS_DNF5_HOST_INSTALL_DIR=atlas-sw
export ATLAS_DNF5_STATE_DIR=atlas-sw-state
./atlas-dnf5.podman install -y tdaq-11-02-00_x86_64-el9-gcc13-opt

And then run it:

podman run -it --rm -v atlas-sw:/sw/atlas gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:el9
# cm_setup tdaq-11-02-00

As new TDAQ patches become available:

export ATLAS_DNF5_INSTALL_DIR=/sw/atlas
export ATLAS_DNF5_HOST_INSTALL_DIR=atlas-sw
export ATLAS_DNF5_STATE_DIR=atlas-sw-state
./atlas-dnf5.podman upgrade

docker

If you replace podman above with docker you will realize that files created inside the container in the work area are not owned by you but by root. You should add the following additional option to each run command instead of the --userns=keep-id option:

docker run --user $(id -u):$(id -g) ...

Even if your user name is not known inside the container, you will have the same user and group ID and files created inside the container can be modified by you on the host side.

apptainer

apptainer is requires no special privileges beyond the kernel supporting user namespaces. The optional suid features of apptainer are not needed for most use cases. It can be installed from rpm or deb packages available on their release page on github. apptainer is available on lxplus.cern.ch. It provides the easiest integration with the user's host environment, as it mounts the home directory and the current working directory by default and makes user/group ids identical.

If CVMFS is available and the host system is Fedora or RedHat (or clone) based, the apptainer installation of ATLAS can be used directly. Putting the following in your .bashrc selects either a local installation (preferred if available) or the CVMFS version:

command -v apptainer > /dev/null || alias apptainer=/cvmfs/atlas.cern.ch/repo/containers/sw/apptainer/$(arch)-el9/current/bin/apptainer

For other Linux distributions you need a local installation.

Use with CVMFS

OCI images have to converted for use by apptainer. This has already be done for you and the images are available on CVMFS:

apptainer shell -p -B /cvmfs /cvmfs/unpacked.cern.ch/gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:x86_64-el9
apptainer> cm_setup tdaq-11-02-00

The -p option enables a private PID namespace, so inside the container you see only the processes started there, not the host processes.

Note that you have to use the full hardware tag here, as apptainer images are not multi arch capable.

Note also that the full stack image is not available on CVMFS, as using the image and the software from the existing places is equivalent to it.

Use without CVMFS

If don't want to use the CVMFS image, you can use the following syntax

apptainer shell -p -B /cvmfs docker://gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:x86_64-el9

You can also get your own apptainer image in SIF format in the following way:

apptainer build tdaq-x86_64-el9.sif docker://gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:x86_64-el9
apptainer shell -p -B /cvmfs tdaq-x86_64-el9.sif
Apptainer> cm_setup tdaq-11-02-00

If you have no CVMFS available at all, you need again the full stack container:

apptainer shell -p tdaq-11-02-00.sif docker://registry.cern.ch/tdaq:11.2.0

or

apptainer build tdaq-11-02-00.sif docker://registry.cern.ch/tdaq:11.2.0
apptainer shell -p -B /cvmfs tdaq-11-02-00.sif
Apptainer> cm_setup tdaq-11-02-00

Use of X11 Based Applications

Unlike podman and docker the X11 based applications should work out of the box with apptainer.

Modifications to the Image

Unlike podman and docker, apptainer does not allow easy modifications of the image as it by default runs as unprivileged user inside the container. Depending on your use case, please check the documentation about overlays in apptainer, as well as the --fakeroot and --writable or --writable-tmpfs options.

Advanced Topic: Running a different architecture

Both podman and docker allow you to run an architecture different from the host if the relevant qemu binaries are installed. E.g. on Ubuntu

sudo apt install qemu-user-static

On Alma 9 (on a system where you have root access):

sudo podman run --rm --privileged docker.io/alekitto/qemu-user-static --reset -p yes

and then

podman run -it --rm --platform linux/arm64 -v /cvmfs:/cvmfs:ro,shared gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:el9
# arch

will allow you to do basic development for aarch64 architecture on an Intel/AMD host. Performance will be vastly inferior to a native system, though.

For apptainer you have to specify the correct image as it has no equivalent to the --platform or --arch options of podman and docker:

apptainer shell -B /cvmfs gitlab-registry.cern.ch/atlas-tdaq-software/tdaq_ci:aarch64-el9
# arch