tdaq-09-04-00

Doxygen Javadoc

The ATLAS TDAQ software version tdaq-09-04-00 has been released on 16th September 2021.

Availability and Installation

Outside of Point 1 the software should be used via CVMFS. It's official location is 

/cvmfs/atlas.cern.ch/repo/sw/tdaq/tdaq/tdaq-09-04-00/

At Point 1 the software is as usual available at

/sw/atlas/tdaq/tdaq-09-04-00/

The software can also be installed locally via ayum.

git clone https://gitlab.cern.ch/atlas-sit/ayum.git
source ayum/setup.sh

Modify the prefix entries in the yum repository files in ayum/etc/yum.repos.d/*.repo to point to the desired destination.

ayum install tdaq-09-04-00_x86_64-centos7-gcc11-opt

In case the LCG RPMs are not found, add this to etc/yum.repos.d/lcg.repo:

[lcg-repo-101]
name=LCG 101 Repository
baseurl=http://lcgpackages.web.cern.ch/lcgpackages/lcg/repo/7/LCG_101/
enabled=1
prefix=[...your prefix...]

Configurations

The release is available for the following configurations:

  • x86_64-centos7-gcc8-opt
  • x86_64-centos7-gcc8-dbg
  • x86_64-centos7-gcc11-opt (default at Point 1)
  • x86_64-centos7-gcc11-dbg
  • x86_64-centos8-gcc11-opt (experimental)
  • x86_64-centos8-gcc11-dbg (experimental)

External Software

LCG_101

The version of the external LCG software is LCG_101.

CORAL and COOL - 3.3.9

These two packages are no longer part of LCG. They are included in tdaq-common and their use should be transparent for users.

There are several CORAL libraries which are really loadable plugins, and user should not link against them. In the past this was possible, but in this release only the offical targets are available.

If you get a link error, remove the CORAL library if is not in this list and see if things work.

The official libraries are:

  • CORAL::CoralBase
  • CORAL::CoralServerBase
  • CORAL::CoralMonitor
  • CORAL::CoralStubs
  • CORAL::CoralCommon
  • CORAL::CoralServerProxy
  • CORAL::CoralKernel
  • CORAL::CoralSockets

TDAQ Specific External Software

Package Version Requested by
beautifulsoup4 4.10.0 L1Calo
cmzq 3.0.2 FELIX
zyre 1.1.0 FELIX
libfabric 1.11.0 FELIX
colorama 0.4.4 DCS
opcua 0.98.13 DCS
parallel-ssh 2.6.0.post1 TDAQ/PartitionMaker
pugixml 1.9 L1Calo, L1CTP
ipbus-software 2.8.1 L1Calo, L1CTP
microhttpd 0.9.73 TDAQ/pbeast
mailinglogger 5.1.0 TDAQ/SFO
netifaces 0.11.0 TDAQ
Twisted 21.7.0 TDAQ/web services
paramiko[gssapi] 2.7.2 was: TDAQ/PartitionMaker
urwid 2.1.2 TDAQ

Use of Python 2 based system commands

With the move to Python 3 inside the TDAQ software there is now a fundamental incompability between the normal system setup and the TDAQ environment. If one wants to call a system command that is implemented in Python (2 for CentOS 7), like yum or auth-get-sso-cookie the environment has manipulated.

In most cases something like the following will be enough:

env -u PYTHONHOME -u PYTHONPATH auth-get-sso-cookie ...

If the command loads compiled libraries as well, it may be necessary to add a -u LD_LIBRARY_PATH as well.

BeamSpotUtils

  • Updates to support new HLT histogram naming convention for Run3.
  • Add support for new track-based method.
  • Improvements and refactoring to support easier testing.

CES

Package: CES
Jira: ATDAQCCCES

Live documentation about recoveries and procedure can be found here.

coca

  • Tag coca-03-15-00
  • Saving exact file size and checksum in database, some operations do checks for consistency. The code still supports all known versions of schema.
  • Tag coca-03-14-00
  • Improve cache space management across multiple instances of coca_server, now they can cleanup files left from other instances (older releases).
  • Tag coca-03-13-00
  • Some fixes for handling of removed archives (we never removed archives yet).
  • Add unit tests for database classes (running on SQLite schema).
  • Add code to support schema versions up to 3.
  • Tag coca-03-12-10
  • Add schema migration scripts for versions up to 3 (for Oracle only).

coldpie

  • tag coldpie-00-03-06
  • add few missing methods to Database class wrapper

config

Java exceptions become checked

Now, the java config exceptions are derived from ers.Issue. This makes them checked. Most methods of config and generated DAL classes might throw an exception, that either must be caught in try / catch block or forwarded. The config.ConfigException base class can be used to simplify exceptions handling.

try
  {
    config.Configuration db = new config.Configuration("");
    dal.Partition p = dal.Algorithms.get_partition(db, "test");
  }
catch(final config.ConfigException ex)
  {
    ers.Logger.error(ex);
  }

Any jar using config or generated DAL must include Jers/ers.jar in CMakeLists.txt.

Export configuration to Boost ptree

There are two new methods in the Configuration class to export configuration data and schema to the Boost.PropertyTree:

void export_schema(boost::property_tree::ptree& tree, ...);
void export_data(boost::property_tree::ptree& tree, ...);

The classes, objects and data files can be selected using regular expressions.

The schema ptree follows this structure:

{
    "class-name" {
        "abstract" : <value>
        "description" : <value>
        "superclasses" [
            <value>,
            ...
        ]
        "attributes" {
          "attribute-name" {
            "type" : <value>
            "range" : <value>
            "format" : <value>
            "is-not-null" : <value>
            "is-multi-value" : <value>
            "default-value" : <value>
            "description" : <value>
          },
          ...
        }
        "relationships" {
          "relationship-name" {
            "type" : <value>
            "cardinality" : <value>
            "is-aggregation" : <value>
            "description" : <value>
          },
          ...
        }
    }
}

The "superclasses", "attributes" and "relationships" nodes are only added, when non-empty. Similarly, the "description", "range", "format", "is-not-null", "is-multi-value", "default-value" and "is-aggregation" elements are only added, when meaningful.

The data ptree follows this structure:

{
    "class-name" {
        "object-id" {
            "attribute-name" : <value>
            "multivalue-attribute-name" [
              <value>,
              ...
            ]
            "relationship-name" : <value>
            "multivalue-relationship-name" [
              <value>,
              ...
            ]
        },
        ...
    },
    ...
}

A relationship value is stored in oks format: "object-id@class-name". The multivalue attributes and relationships are stored as arrays (can be empty).

There are two new utilities to export schema and data into json, xml and ptree-info formats using above methods: * config_export_schema * config_export_data

Rename object in python binding

The python binding supports now renaming an OKS object.

import config

db = config.Configuration('oksconfig:myfile.data.xml')
hltsv = db.get_obj('HLTSVApplication', 'HLTSV')
hltsv.rename('HLTSV-new-name')
db.commit('Changed name of HLT supervisor application')

On the DAL level:

import config

db = config.Configuration('oksconfig:myfile.data.xml')
hltsv = db.get_dal('HLTSVApplication', 'HLTSV')
hltsv.rename('HLTSV-new-name')
db.update_dal(hltsv)
db.commit('Changed name of the HLT supervisor application')

dbe

Package: dbe
Jira: ATLASDBE

DVS GUI (graphical UI for DVS)

See also related DVS and TestManager packages.

dynlibs - Load shared libraries

This package is deprecated from tdaq-09-00-00 onwards.

Please use plain dlopen() or boost::dll instead. Note that unlike in this package, the boost::dll::shared_library object has to stay in scope as long as the shared library is used !

Example of possible replacement

#include <boost/dll.hpp>

double example()
{
   boost::dll::shared_library lib("libmyplugin.so",
                                  boost::dll::load_mode::type::search_system_folders |
                                  boost::dll::load_mode::type::rtld_now);
   // Get pointer to function with given signature
   auto f = lib.get<double(double, double)>("my_function")
   return f(10.2, 3.141);
}

emon

  • Connections between event samplers and monitors have been optimized. Existing configurations should be adjusted to benefit form that. Previously the NumberOfSamplers parameter has been used to define a number of samplers to be connected by all monitors of a group that uses the same selection criteria. In the new implementation this number defines a number of samplers that each individual monitor has to connect. That makes no difference for monitors that used to connect to a single sampler and do't form a group. For the monitors that share the same selection criteria, like for example the Global Monitoring tasks, this number should be changed to the old number divided to the number of the monitors in the group. For Athena monitoring the corresponding parameter of a JO file is called KeyCount.

Igui

Package: Igui

jeformat

Package: jeformat

ls

Package: ls

mda

Small fix in mda_register script, updates for Python3.

MonInfoGatherer

Alternative implementation for merging non-histogram data (ADHI-4842). This should resolve most of the timing issues we have seen with DCM data in the past. It is enabled by default but can be disabled with ISDynAnyNG configuration parameter (see ConfigParameters.md).

OKS

Show hidden oks data xml attributes for text editors

Use new oks_refresh utility with -x option to enforce all attributes to be shown.

Deprecated OWL date/time format

The OWL date/time formats are deprecated and will be removed in next public release. ERS warning is reported by oks library, when a file containing data stored in deprecated format is loaded. Such file can be refreshed using oks editors or new oks_refresh utility. For example:

$ git clone <url> .
$ TDAQ_DB_PATH=`pwd`:$TDAQ_DB_PATH oks_refresh -f ./<file>
$ git commit -m "refresh <file> to update date/time format" ./<file>
$ git push origin master

Ordering of multi-value attributes and relationships

Use new ordered attribute for multi-value attribute or relationship to sort its data on save. The default value is no. Implement sort of multi-value attributes and relationships in OKS data editor.

owl

The owl package consisted for historical reasons of at least three different functional areas, and has been split after release tdaq-09-02-01.

The process management related functionality is now in the new pmgsync package. If you include any of these headers check carefully if you really need them, as most of this is handled internally by the various run control libraries.

The threadpool package has been separated as well. As it is not used in TDAQ itself, it is no longer part of the release. The git repository contains a functioning package that can be dropped into a release if needed. There is also a thread pool implementation in the ipc package.

The following classes in the owl package have been deprecated and will be removed in a future release:

  • OWLThread (use std::thread instead)
  • OWLMutex and related mutex and lock classes (use std::mutex etc. instead)
  • OWLCondition (use std::condition_var instead)

There is currently no replacement for OWLSemaphore in the C++17 standard, however, there will be one in the upcoming C++20. If you only use the class to stop your main thread when a signal is received in another thread, consider using the new helper functions in ipc/signal.h instead.

P-BEAST

  • introduce v4 data format with more efficient compaction (next data series timestamps stores difference from previous one instead of base file timestamp)
  • rename pbeast_zip to pbeast_copy and add more compress/uncompress options
  • add pbeast_compare to compare two pbeast files
  • add pbeast_refresh_repo.sh utility to refresh repository using latest file format (v4)

ProcessManager

Package: ProcessManager

The ProcessManager twiki can be found here.

RCUtils

Package: RCUtils

rn (Run Number)

tdaq-09-01-00

Jira: ADTCC-242

Store start of run timestamp and run duration with nanoseconds resolution. Store TDAQ release name. If oks repository is used, store oks config version into run number database and tag oks repository by run number / partition name.

RunControl

Package: RunControl
Jira: ATDAQCCRC

This is the link to the main RunControl twiki.

SFOng

  • replaced deprecated tbb::tbb_hasher with std::hash
  • replaced deprecated tbb::atomic with std::atomic
  • fixed: starting a new run without going to shutdown (stop/start) could produce Late events at the beginning of the new run for a transition period

swrod

GBT Fragment Building Timeout

This version implements fragment building timeout for GBT fragment building algorithm. One can specify the timeout value via the new attribute of the SwRodGBTModeBuilder class called DataReceivingTimeout. This attribute contains a number of milliseconds to wait after receiving the first data chunk for a particular ROB fragment to consider this fragment as built even if not all data chunks have been received. By default this attribute is set to zero, which disables the timeout.

Configuration schema changes

  • A new DataReceivingTimeout attribute has been added to the SwRodGBTModeBuilder class. It defines a timeout in milliseconds for ROB fragments building. If the given number of milliseconds is passed after receiving the first data chunk for a particular ROB fragment this fragment will be considered as built and will be passed to the fragment consumers even if it does not contain data chunks from all the E-links associated with the given ROB.

  • A new DeferProcessing attribute has been added to the SwRodCustomProcessor class. If it is set to true the processing will be applied only when serialization of the ROB fragment is requested. This happens for example when the fragment is about to be written to a file or been sent over the network. This may be used to reduce computing resources for the fragments that require heavy processing but are rarely requested by the HLT. Default value of this attribute is false.

  • A new ProfileExecution attribute has been added to the SwRodCustomProcessor class. If it is set to true the processor will keep a record of total time of the custom processing execution and will print it to the standard output when SW ROD is terminated. Default value is false.

Custom Plugin Test application

A new application that can be used for validation and profiling of a custom plugin has been added. The application is called swrod_custom_plugin_test and can be used in the following way: * For the first time it has to be given five parameters: a name of the data file to be used as data source, a name of the shared library that implements the plugin to be tested and the names of the three custom functions which this plugin implements. Optionally one can use -o option to save the new configuration to the given Json file. * If the Json configuration file has been produced the test application can be started with this file as a sole input parameter using -i command line option.

TDAQExtJars

Package: TDAQExtJars

transport

Clients using classes from the transport package should look into more modern network libraries like boost::asio until the C++ standard contains an official network library.

TriggerCommander

Package: TriggerCommander