How to install Python packages

The LCG release corresponding to a give TDAQ software release contains a lot of Python packages that are compatible with the Python version used, and usually much newer than the system version.

If you have need for a generally useful package that should be in LCG, you can contact Andrei.Kazarov@cern.ch or Reiner.Hauser@cern.ch and we can request it in the next LIM meeting. Alternatively ask the offline software coordinators.

Avoid to make the request directly to LCG, or at least let any of us know, as we may be asked about it in the next meeting.

Alternative, if the package is more special purpose, you can ask us to add it to the TDAQ release.

Python Packages on /cvmfs but not in Point 1

Only a subset of the cvmfs Python packages is installed at Point 1, those that are explicitly referenced by either TDAQ or offline software. You can ask us to add an explicit dependency in TDAQ on another package, so it will show up automatically we install the next release.

If you need the package for the current release which is missing the dependency (the usual case...), simply ask us to install manually. This is technically not a big deal, but ideally things should be setup for the next release to avoid this step.

Python Runtime setup

While pure Python packages are typically found with no problem, those which have C extensions, shared libraries etc. require manual setup. The TDAQ software does this only for those package it needs itself. This is to avoid an ever growing environment.

In addition the environment in a running partition is usually not setup to execute arbitrary Python scripts. It is the responsibility to the rsp. subdetector to do this.

The TDAQ release contains a helper script to deal with this. Say you want access to the pip command from the LCG release. The following shows you the necessary entries for PATH, LD_LIBRARY_PATH, PYTHONPATH:

get_lcg_env --all pip

If you just want to use it interactively, you typically want to merge it with your current environment:

eval $(get_lcg_env --all --prepend pip)

and you are good to go and use pip.

Private Python Packages

Say you need a Python package for a special purpose that doesn't really justify adding it to LCG or TDAQ. Installing it yourself at Point 1 is not trivial, as e.g. pip does not work there.

The following procedure allows you to install such packages in a private area.

Outside of Point 1 setup the desired TDAQ release and a local working directory.

cm_setup tdaq-09-04-00
mkdir local
cd local 
export PYTHONUSERBASE=$(pwd)
eval $(get_lcg_env --all --prepend pip)

Now install the packages you want with pip install --user ...

pip install --user pygit2

You will see that it resolves existing packages from LCG and TDAQ. Your local directory contains just what is needed on top of that. Take note of them, in case some of them are not installed at Point 1.

That's it. Now simply copy the whole local directory to Point 1. If you look at it, it's structure is just like the /usr area. Add the relevant bin, lib, lib/python3.9/site-packages to your PATH, LD_LIBRARY_PATH and PYTHONPATH and you are done.

Check if any of the dependencies coming from LCG are missing and request them (see above).

This method is basically the same as used in TDAQ to install additional external packages. Look at the cmake code if you want to integrate this e.g. in your subdetector release.