tdaq-09-02-01

Introduction

The ATLAS TDAQ software version tdaq-09-02-01 has been released on 16th September 2020.

Availability and Installation

Outside of Point 1 the software should be used via CVMFS. It's official location is

/cvmfs/atlas.cern.ch/repo/sw/tdaq/tdaq/tdaq-09-02-01/

At Point 1 the software is as usual available at

/sw/atlas/tdaq/tdaq-09-02-01/

The software can also be installed locally via ayum.

git clone https://gitlab.cern.ch/atlas-sit/ayum.git
source ayum/setup.sh

Modify the prefix entries in the yum repository files in ayum/etc/yum.repos.d/*.repo to point to the desired destination.

ayum install tdaq-09-02-01_x86_64-centos7-gcc8-opt

In case the LCG RPMs are not found, add this to etc/yum.repos.d/lcg.repo:

[lcg-repo-98]
name=LCG 98 Repository
baseurl=http://lcgpackages.web.cern.ch/lcgpackages/lcg/repo/7/LCG_98/
enabled=1
prefix=[...your prefix...]

Configurations

The release is available for the following configurations:

• x86_64-centos7-gcc8-opt
• x86_64-centos7-gcc8-dbg

External Software

LCG_98python3

The version of the external LCG software is LCG_98python3.

TDAQ Specific External Software

Package	Version
cmzq	3.0.2
zyre	1.1.0
libfabric	1.6.2
nlohmann/json	2.1.1
pugixml	1.9
ipbus-software	2.7.8
microhttpd	0.9.59
mailinglogger	5.1.0
netifaces	0.10.9
Twisted	20.3.0
paramiko[gssapi]	2.7.1

CES

Package: CES
Jira: ATDAQCCCES

Live documentation about recoveries and procedure can be found here.

Changes in recoveries and/or procedures:

• HLT recoveries: sending FORK_AFTER_CRASH only if RUNNING (ADHI-4780).

Internal changes:

• The ESPER CEP engine has been updated to version 8.5;
• Fixed memory leak in the Metaspace (ATDAQCCCES-154);
• Added sTGC as a possible source of trigger on hold;
• Avoid concurrent executions of the ESPER compiler.

Igui

Package: Igui

• Commit & Reload functionality adapted to the new OKS GIT back-end (ADTCC-227):

  A new reload panel has been developed in order to allow proper configuration reload when the new OKS GIT back-end is used. In this new version of the panel, the user is asked to select the configuration version to reload on the basis of the last commits available in the GIT repository. The selection of the configuration to be reloaded is no more file-based, but the user is asked to select a GIT commit that, in principle, may include changes to several separate files. In such a way, there is not any risk no more to select a set of incompatible files. The list of modified files per commit is available un-hiding a specific column in the table (that can be done changing the table properties using the little button located on the right upper corner of the table). Of course, the selection of a specific commit implies to import all the changes applied in the previous commits.

MonInfoGatherer

Removed copy of lz4 libray from sources.

ProcessManager

Package: ProcessManager

The ProcessManager twiki can be found here.

Internal changes:

• pmg_start_app: fixed issue with parsing complex environment in command line options.

RCUtils

Package: RCUtils

• Adapting to the new OKS GIT back-end (ADTCC-227):
  The proper OKS version to use is set by the setup_daq script;
  The RunParams IS server is started early by setup_daq in order to allow storing information about the OKS version to use;
• Fixed several issues in setup_daq building the RootController environment in the case of complex environment variables;
• Several fixes to the DAQ efficiency tools in the area of the Hold Trigger accounting.

RunControl

Package: RunControl
Jira: ATDAQCCRC

This is the link to the main RunControl twiki.

Internal changes:

• Python library: fixed initialization of the python sub-system;
• When OKS GIT is used, the RootController sets the currently used configuration version (ADTCC-227).

SFOng

• replaced deprecated tbb::tbb_hasher with std::hash
• replaced deprecated tbb::atomic with std::atomic
• fixed: starting a new run without going to shutdown (stop/start) could produce Late events at the beginning of the new run for a transition period

TDAQExtJars

Package: TDAQExtJars

• The following packages have been updated to a newer version:
  JFreeChart (from 1.0.16 to 1.5.0)
  jide-oss/common (from 3.5.8 to 3.7.7)
  com.jhlabs.filters (2.0.235-1)
  mysql-connector-java (from 3.0.16 to 8.0.19)
  oracle.ojdbc (from 11.2.0.3.0 to 19.3.0.0)
  jldap (from 2007.10.12 to 2009-10-07)
  esper (from 8.4.0 to 8.5.0)

TriggerCommander

Package: TriggerCommander

Clients

A backward compatible change (for clients) has been introduced in the TriggerCommander/MasterTrigger.h interface. The hold() method has been extended with a boolean flag extended_lvl1id which defaults to false.

The method will return the last ECR as it did in the past. When the new flag is set to true it will instead return the full Level 1 ID (from which the ECR can be extracted as the uppermost 8 bits).

Implementations of the MasterTrigger interface

An implementation of the MasterTrigger interface has to modify its overloaded method:

class X : public MasterTrigger {
  ...

  uint32_t hold(const std::string& dm, bool extended_lvl1id = false) override;
  ...
};

coldpie

• Added few missing features that were not implemented before
• Record indexing operator now returns field value instead of Field instance

dbe

Package: dbe
Jira: ATLASDBE

• Schema editor: the schema file to be loaded can be passed as a command line argument using the -f option;

• Support for new archival mechanism based on the OKS-GIT back-end (ADTCC-241):

  Configurations archived in the GIT repository can be loaded using the -v command line option and specifying the GIT hash corresponding the desired configuration.

DVS GUI (graphical UI for DVS)

See also related DVS and TestManager packages.

dynlibs - Load shared libraries

This package is deprecated from tdaq-09-00-00 onwards.

Please use plain dlopen() or boost::dll instead. Note that unlike in this package, the boost::dll::shared_library object has to stay in scope as long as the shared library is used !

Example of possible replacement

#include <boost/dll.hpp>

double example()
{
   boost::dll::shared_library lib("libmyplugin.so",
                                  boost::dll::load_mode::type::search_system_folders |
                                  boost::dll::load_mode::type::rtld_now);
   // Get pointer to function with given signature
   auto f = lib.get<double(double, double)>("my_function")
   return f(10.2, 3.141);
}

emon

• Connections between event samplers and monitors have been optimized. Existing configurations should be adjusted to benefit form that. Previously the NumberOfSamplers parameter has been used to define a number of samplers to be connected by all monitors of a group that uses the same selection criteria. In the new implementation this number defines a number of samplers that each individual monitor has to connect. That makes no difference for monitors that used to connect to a single sampler and do't form a group. For the monitors that share the same selection criteria, like for example the Global Monitoring tasks, this number should be changed to the old number divided to the number of the monitors in the group. For Athena monitoring the corresponding parameter of a JO file is called KeyCount.

jeformat

Package: jeformat

Changes in public APIs:

• Added sTGC to the detector mask.

ls

Package: ls

• Fixed issue with sqlite file permissions (ADTCC-245):

  The name of the user running the logger application can now be set in the connection string with a place-holder. As an example, here is a possible set of command line options:

  -e mts -S app!=mts-sender* -t ${TDAQ_VERSION} -p ${TDAQ_PARTITION} -c ${LOGGER_CONNECT} -n RunParams -s 10000 -b sqlite_file:${TDAQ_LOGS_PATH}/Logger_BackupDb_$(USER).db

  The $(USER) placeholder will be then replaced with the real user name.

mda

Implemented support for future upgrades of MDA database schema. Original intent was to enable storing histograms in ROOT file in their original form without replacing colons with underscores but that did not work out (see ADHI-4177 for details). Schema version support ay still be useful for future updates so I keep it.

OKS

tdaq-09-02-01

Postponed changes

The postponed changes are stored into a git branch and should not be applied immediately in the ongoing data taking session. If needed, they can be shared with other experts and tested using DAQ control and configuration tools. At an appropriate moment the branch will be merged with master branch and deleted.

When branch is created, the gitea pull request (also known as merge request in gitlab) can be created. All gitea pull requests can be accessed from a single point and applied using gitea web interface. In future, the pull requests will be integrated with DAQ Shifter Assistant.

A git branch can be created and updated using git, oks and gitea web interfaces.

Create and update branch

Create or checkout a branch into temporary private area:

$ cd `oks_clone_repository -b <branch-name>`

A branch can also be created using git command line and gitea web interfaces.

Modify oks files

Edit changes using config tools, a text editor or gitea web interface, for example:

$ export TDAQ_DB_USER_REPOSITORY=`pwd`
$ oks_data_editor x/y/z
$ vim x/y/z

Commit and push modifications

Use oks commit:

$ oks-commit.sh -m 'put here your commit message' -u `pwd`

or git command line interface:

$ git commit -m 'put here your commit message' x/y/z
$ git push origin <branch-name>

Create gitea pull request (recommended)

Create a new pull request for your branch using gitea Web interface (you need to be authorised first) using URL:

<gitea-release-url>/compare/master...<branch-name>

Merge postponed changes

When the changes have to be merged with the master branch, use git command line interface:

$ git checkout master
$ git merge <branch-name>
$ git push origin master
$ git push origin --delete <branch-name>
$ git branch -D <branch-name>

or, if the gitea pull request was created, use gitea web interface. Browse all pull requests:

<gitea-release-url>/pulls

Select your pull request, merge changes and delete branch.

How to run partition or use config-based tools

In case, if the postponed changes need to be validated running DAQ partition, a shared file system has to be used to checkout a branch. For example create it in NFS area:

$ export SHARED=<some-path> # e.g. "/tbed/scratch/`whoami`/$CMTRELEASE" on TestBed or "$HOME/oks/$CMTRELEASE" on Point-1
$ mkdir -p $SHARED
$ cd $SHARED
$ oks_clone_repository -b <branch-name> -o .
$ chgrp -R zp .
$ chmod -R g+w .
$ export TDAQ_DB_USER_REPOSITORY=`pwd`

Then edit and commit changes to branch and run setup:

$ setup_daq <path-to-partition-file> <partition-name>

rn (Run Number)

tdaq-09-01-00

Jira: ADTCC-242

Store start of run timestamp and run duration with nanoseconds resolution. Store TDAQ release name. If oks repository is used, store oks config version into run number database and tag oks repository by run number / partition name.

swrod

This release introduces an OKS configuration schema change to facilitate splitting of the TDAQ and detector specific portions of SW ROD configuration. The new schema has a new class called SwRodDataChannel that should be used by detector experts to configure how data for a given data channel has to be collected and processed. This class emerges from the legacy SwRodRob one and steals two relationships from it: * Contains relationship should contain a set of e-links to be used to receive data for the data channel SwRodDataChannel * CustomLib relationship shall point to the custom plugin that will be used for data assembling and post-processing of the fragments produced for the given data channel

Finally an instance of the SwRodDataChannel class has to be linked to an instance of the SwRodRob class via the Contains relationship of the latter. Note that despite the fact that this relationship allows multiple objects to be referenced an instance of the SwRodRob must point to exactly one instance of the SwRodDataChannel class. If that is not the case SW ROD configuration will fail and an error message will be produced.

More details about the new way of configuring SW ROD can be found in the updated Reference Manual as well as in the README file of the package.