How to submit a package tag
As Developer
The procedure for submitting a new version of a package to either the nightly or a production release is the same:
tagpkg <package-name> <release-name> <version>
where <package-name> is the package you want to submit.
Typically, this should be all for the nightly release. Read on for more details, and what to expect for a production release.
Package Versions
The <release-name> should be either of the form tdaq-XX-YY-ZZ for
a numbered release, or just nightly (which is a symbolic name for
tdaq-99-00-00).
For submitting a new package version to tdaq-common you use the
obvious tdaq-common-XX-YY-ZZ or tdaq-common-nightly.
The <version> can be anything that resolves to a valid
git revision. E.g. you
can explicitly tag your package:
git tag -a PKG-01-02-00
git push --tags
tagpkg PKG nightly PKG-01-02-00
Or you can just reference a branch:
git push origin master
tagpkg PKG nightly master
Note that we made sure to push the latest version to gitlab
before running the tagpkg command. This variant will use
the commit of the current HEAD of the master
branch on the gitlab.cern.ch server.
It does not care at all about your local version (which might be
ahead of the server version).
The first variant using a tag on the other hand will check that this tag exists on the server and fail otherwise.
Listing Package Versions
You can use
tagpkg -l PKG nightly
tagpkg --list PKG nightly
to list the current version in a release. It will print a symbolic name like a tag (if possible), but also the full SHA1 commit.
What happens now ?
The tagpkg command will create a merge request for your new version
to be incorporated in the specified release. Your proposed changes (the new
version of your package) are on a branch called <user>/<release-name>/<package-name>
where <user> is your login name.
In the following you can replace the relevant names or links for tdaq-common in the obvious way.
Tag for a nightly release
TL;DR
% tagpkg PKG nightly VERSION
Ideally this should be all you need.
You will typically receive several e-mails after you ran this command with a new tag for the nightly release:
- One to tell you that a merge request has been created.
- One to tell you that a merge request has been assigned (to you).
- One to tell you that the pipeline has succeeded (hopefully) and the branch has been merged (and deleted). This may take a while depending how much of the release had to be re-compiled due to your changes. Expect minimum around 30 minutes for this, or up to an hour and a half maximum.
At this point your change is part of the nightly release.
If the pipeline did not succeed, your changes are not merged, however. You should go to the dashboard and see what didn't work.
If the problem is in your package, fix it, and simply submit a new version. It will show up in the existing merge request as a new commit. Rinse and repeat until you succeed. If you want to give up for any reason, you close the merge request (maybe add a comment to explain why).
If the problem is in someones else's package, it is up to you to follow it up with the relevant developer. You cannot merge a package that leads to a failure in your's or another person's package. See below how to handle this.
Access to merged release
The CI pipeline will actually build a whole new release after including your new package version, run all tests etc. It will also make this build available on CVMFS. These are special "releases" which you can find with the
cm_setup -l
cm_setup --list
command. The name of the release will be a date/timestamp followed by your login name. This should be enough to allow you to find the correct build. It should take about 15 minutes after you received the e-mail about the pipeline success or failure before it will be visible on cvmfs.
You can setup such a release the same way as a normal release.
cm_setup 2022-12-28T1240-rhauser
and build against it from a work area etc. Such a release will stay around for about 1-2 days, a cron job at 9am will delete older versions. You can ask us to keep it for longer if necessary.
If you submit new commits to your merge request (because the first one did not build e.g.) you will get a new build with a new timestamp.
Tags for a production release
TL;DR
% tagpkg PKG tdaq-10-00-00 VERSION
- Wait until 'Testing Required' label is set and MR assigned to you
- Test new version on cvmfs.
- Change to 'Testing OK' label (if tests were successfull).
The difference between a submission to the nightly release and a production release is that the latter will not automatically merge the changes, even if the pipeline succeeds. At least one manual approval is required. Usually this is someone in charge of the relevant working group. It is meant as a safeguard against random submissions to a production release.
Every merge request is given one (or more) labels depending on the package which is changed. The labels roughly correspond to the TDAQ working groups (Control, Configuration, Monitoring, Readout, Dataflow, RCD, Level1, Trigger, CMake).
Note that anybody can sign up for notifications on these labels. It is expected that people with approval rights do this for the areas they are responsible for.
The association between packages and labels is part of the main repository and can be changed as needed.
Note that the assignee for the merge request will somebody with TDAQ librarian privileges.
The merging is done by the TDAQ librarian before the RPMs are build and installed
on cvmfs. A Testing Required label will be added and the merge request will
be assigned back to the developer. At this point you should test the new version
on cvmfs and change the label to 'Testing OK' (if it is working).
No other action should be required. You also should not assign other labels like P1 as
they are meant to indicate when the package is installed at Point 1.
If your merge request does not make any progress, leave a comment on the merge request which will get our attention.
Options to tagpkg
Running tagpkg without arguments or with the -h or --help option will
show you additional features.
Submitting Multiple Packages/Tags
If you have a change that requires simultanuous modifications to more than one package, you have to submit them all at once. You can do this in various ways.
The easiest is to run tagpkg and add a list of additional packages after the --:
tagpkg pkg1 nightly tag1 -- pkg2 tag2 pkg3 tag3...
You can also put the packages and tags in a text file, one per line:
pkg1 tag1
pkg2 tag2
pkg3 tag3
and submit via
tagpkg --tags nightly ./packages.txt
You can also create your final merge request piecemeal by making it a draft and adding changes:
tagpkg --draft --branch=feature/awesome pkg1 nightly tag1
tagpkg --branch=feature/awesome pkg2 nightly tag2
tagpkg --branch=feature/awesome pkg3 nightly tag3
Here we used --draft option to avoid the automatic merging of the MR.
We also defined our own branch name instead of relying on the default one.
This feature is best used if you discover unexpected build errors, and then either you or another developer incrementally submits fixes to the same branch until everything compiles.
Do not use this method to submit N tags one after another from the command line - each will trigger a full build. Use one of the previous methods mentioned above instead.
Setting Commit Message or Merge Request Title
You can set an explicit commit message with the --msg=... option. You
can also use --edit option which will open your preferred editor on the
commit message before submission.
You can set the title of the merge request with the --title=... option. This
can make it more useful for other people to find it.
Adding Labels
You can add additional labels with the --labels=L1,L2,.. option. This
should be rarely necessary as it is done automatically now. You can use this
if you want to alert experts from an unrelated working area to your
changes.
Create a Draft Merge Request
You can also a create draft merge request by making
use of the --draft option.
tagpkg --draft pkg1 nightly tag1
Such a MR which will not be automatically merged until you
remove the Draft status in the web interface.
Approval
If you a working group coordinator or have approval rights, please sign up for label notications as described above.
If a notification for a production release alerts you, go to the merge request, check that the build succeeded and the changes are ok, then approve the MR.
You will also get notifications for nightly MRs, for which you have to do nothing in most cases.
New Major Developments
It may be tempting to use the above facilities to do some longer term developments by using a draft MR. However, it is cumbersome if breaking changes are introduced and multiple developers have to fix things over a period of time (finding the right build to work against etc.)
Instead developers can create their own private nightly branch. The
branch name has to be of the form tdaq-99-01-NN where the NN should
be an unused number (use cm_setup --list to check).
You can do this via the web interface, or by checking out the top level package itself:
getpkg tdaq-cmake
cd tdaq-cmake
git branch tdaq-99-01-05
git push origin tdaq-99-01-05
From now on you can submit tags to this release as usual.
Working Model
Development branches are treated differently from nightly branches (tdaq-99-00-NN) or production branches (anything else).
There is no approval necessary, just as for the nightly branch. The pipeline build is made available on cvmfs about 15 minutes after it finished and replaces the previous version of the same name. So at any given time there is only one version of an development branch available. You set it up as usual
cm_setup tdaq-99-01-05
Each new tag will lead to a new version. Note that the build may break and will be installed anyway. This is done on purpose since it is meant to be used for disruptive developments. E.g. you can change an API, build the release and point other developers to it. They can in turn set the release up, fix their dependent package and submit it as well. Ideally the builds will all be successful after a while.
One other distinct feature is that a force push is allowed into these branches. If you have a longer on-going development, you can rebase your branch from time to time onto the main nightly (conflicts you have to fix manually, as usual).
getpkg tdaq-cmake tdaq-99-01-05
cd tdaq-cmake
git rebase origin/tdaq-99-00-00
git push -f origin tdaq-99-01-05
If you've made only changes to the various package versions, it should be safe to merge the final result back into nightly. To test it locally:
getpkg tdaq-cmake tdaq-99-00-00
cd tdaq-cmake
git merge tdaq-99-01-05
Then create a merge request from the web interface. It will create a CI pipeline and make sure everything still compiles before doing the merge.
tdaq-common
By default these development releases will build tdaq-common from the nightly branch (tdaq-common-99-00-00) but call it e.g. tdaq-common-99-01-05.
If you want to make breaking changes to tdaq-common, you will need some additional manual modifications.
First you will have to create a real branch for tdaq-common:
getpkg tdaq-common-cmake
cd tdaq-common-cmake
git branch tdaq-common-99-01-05
git push origin tdaq-common-99-01-05
Now you can make modifications to this branch until tdaq-common compiles without errors.
Then modify the CI instructions in your TDAQ branch:
getpkg tdaq-cmake tdaq-99-01-05
cd tdaq-cmake
vi .gitlab-ci.yml
Find the following:
dev/x86_64-centos7-gcc11-opt:
extends: build/x86_64-centos7-gcc11-opt
tags:
- tdaq-ci
variables:
CCACHE_OPTION: "-D USE_CCACHE=TRUE"
TDAQ_VERSION: $CI_MERGE_REQUEST_TARGET_BRANCH_NAME
TDAQ_COMMON_VERSION: "tdaq-common-99-00-00"
and change the last line to:
TDAQ_COMMON_VERSION: "tdaq-common-99-01-05"
It is important that you don't merge this branch into tdaq-99-00-00, as
it will also modify .gitlab-ci.yml. Instead compile the list of
packages and submit them in one step to nightly. Note that you also
have to coordinate this with the corresponding changes to tdaq-common.
Check with librarians to make sure.