Staging Plugin Documentation ============================ :author: openSUSE Team at SUSE :toc: Introduction ------------ [id="intro"] The staging plugin is used to enable the new factory workflow to work with staging projects. This plugin implements some functionality that some day will be directly implemented in OBS itself. Meanwhile this plugin is used as a playground field to explore these new concepts and operations. This plugin is used by the release manager to move requests into staging projects to evaluate the behavior and inter-compatibility of different changes working together. An staging project is a mere project linked to a subset of Factory (MinimalX) , but with some packages that are different as a result of one or several requests. When the release manager assigns a request in a staging project, this request is review inside the project. There is a precondition here, the user that has the role of staging manager needs to have permission in OBS, and they need to be: * Maintainer of openSUSE:Factory:Staging (needs current staging manager) * Group member of factory-staging (needs OBS admin) Installation ------------ [id="install"] Clone this repository: -------------------------------------------------------------------------------- git clone https://github.com/openSUSE/openSUSE-release-tools.git -------------------------------------------------------------------------------- Link +osc-staging.py+ and the library either to +~/.osc-plugins+ or +/var/lib/osc-plugins+ -------------------------------------------------------------------------------- cd openSUSE-release-tools ln -sr ./osc-staging.py ./osclib ~/.osc-plugins -------------------------------------------------------------------------------- Optionally, one can create directories for each project of interest in order to alleviate the need to use the +--project option+ for each command. For example: -------------------------------------------------------------------------------- # Or use a directory already initialized by osc. mkdir -p "openSUSE:Leap:42.3/.osc" cd "openSUSE:Leap:42.3" echo openSUSE:Leap:42.3 > .osc/_project # runs against openSUSE:Leap:42.3 osc staging list # runs against openSUSE:Factory osc staging -p Factory list -------------------------------------------------------------------------------- The following snippet may be useful for addition to a bash PS1. -------------------------------------------------------------------------------- if [ -f .osc/_project ] ; then echo " <<$(head -n 1 .osc/_project)>> \$ "; fi -------------------------------------------------------------------------------- This provides a clear indication of the project against which the commands will run and can be merged with an existing complex PS1. Command line ------------ [id="cli"] This plugin is expected to be used interactively. The basic commands are: * *accept* Accept all requests assigned to a staging project. * *check* Check the status of the different request inside the staging project, including the status in openQA. * *cleanup_rings* Check the status of the different rings in Factory. * *rebase* Freeze the packages that are not affected in a staging project. In the past, this command used to be called 'freeze', which is still a valid alias. * *list* Accept all the requests from packages that are not in any ring, detect request that supersede the current ones that are actually tracked in staging projects, and list the remaining requests. * *select* Move a request (or a list of requests) to a staging project, or move requests between staging projects. * *unselect* Remove a request from a staging project, and publish it again into the backlog. List ~~~~ This action will list all the requests that are candidates to be assigned to a different staging project. To decide this list, this command will perform two actions with side effects into the requests status. The first action is to accept requests that do not affect packages that are not inside one of the rings that are defined in Factory. This acceptance will move the request to the next status of revision (maybe waiting to the manual review or the legal review). The second action is to detect if a request supersedes a different one that is currently in a staging project. If this is the case, this command will replace the old request by the new one, declining the superseded one after this. Because of the different side effects, this command needs to be executed with the proper permissions in OBS. Select ~~~~~~ The release manager or the staging project manager can take a request and assign it into a staging project using the +select+ command. Actually we have https://build.opensuse.org/project/subprojects/openSUSE:Factory:Staging[ten staging projects], that are managed according to some internal rules. Every time that the manager decides to move a request to one of those projects, a metadata attached to these projects is updated, and a new comment is generated. The metadata associated to a staging project contains the list of requests that are assigned to the project, and is used by the plugin to track and manage the different requests. For example, the command: -------------------------------------------------------------------------------- osc staging select A 12345 -------------------------------------------------------------------------------- will add the request #12345 into the staging project openSUSE:Factory:Staging:A. This command will take care that this request was not previously in the staging project A or in a different one. If this is the case, it will show an error message and exit without selecting the request. If we want to move a request from one staging project to another one, we need to use the parameter +--move+: -------------------------------------------------------------------------------- osc staging select B --move 12345 -------------------------------------------------------------------------------- This command will move #12345 from openSUSE:Factory:Staging:A to openSUSE:Factory:Staging:B You can also merge staging projects by giving other staging projects as arguments: -------------------------------------------------------------------------------- osc staging select B --move A C -------------------------------------------------------------------------------- This moves all requests currently staged in A and C into B Unselect ~~~~~~~~ The +unselect+ command will untrack a request from a staging project, returning it back to the original backlog (without approving or declining the request) So for example, if #12345 is being tracked in A, the command: -------------------------------------------------------------------------------- osc staging unselect 12345 -------------------------------------------------------------------------------- will find the correct staging project and remove the request from it. Check ~~~~~ Before accepting the requests inside a staging project, the user can check the state of those requests. The +check+ command will check the project status, taking care of superseded requests or already accepted requests. This command will also check the status in openQA of the project. This command can be called without special permissions. Accept ~~~~~~ If the current status of the staging project is good, this command will change the review status of the different requests assigned to the project, accepting them. Internally, the +accept+ command contains a call to the +check+ command to make sure that the request can be accepted. After this command, the staging project status will be disabled, to avoid the overload of OBS. Cleanup Rings ~~~~~~~~~~~~~ https://build.opensuse.org/project/subprojects/openSUSE:Factory:Rings[Rings] are collections of packages that are deeply interconnected, and that are building basis for a different ring or for the rest of the distribution. A ring is a way to organize Factory into inter-dependent packages that are again used to build a different layer of Factory itself. Actually we have identified three rings: * https://build.opensuse.org/project/show/openSUSE:Factory:Rings:0-Bootstrap[openSUSE:Factory:Rings:0-Bootstrap] * https://build.opensuse.org/project/show/openSUSE:Factory:Rings:1-MinimalX[openSUSE:Factory:Rings:1-MinimalX] And you can find a better description in this https://www.youtube.com/watch?v=K-wTVGqKFR8[very old presentation] from a conference celebrated in 2014 in Dubrovnik, Croatia. This command is used to check the current status of the rings and to find undesirable dependencies of the packages that conform the rings. Rebase ~~~~~~ Factory (or the subset MinimalX) is always a moving target, even with the staging projects. If we want to check the status of the request assigned to a staging project, sometimes it is desirable to have a frozen status of the source packages that are part of Factory but not of the staging project. This command is used to build frozenlink-kind-of links for a staging project. Use cases --------- [id="usecases"] Use Case 1: Checking Factory ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The staging manager wants to see the status of Factory every morning. -------------------------------------------------------------------------------- osc staging check -------------------------------------------------------------------------------- With this status they will contact the author of the request, or will rebuild the packages if there is a suspect or a random fail. After that, the staging manager can check the important packages that are pending in the queue. -------------------------------------------------------------------------------- osc staging list -------------------------------------------------------------------------------- This command can update the request attached to a staging project, replacing the superseded one. The list shows the name of the ring where this package is found. Use Case 2: Selecting packages from Ring:0-Bootstrap ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The +list+ command shows the name of the ring where we can find the package involved in the request. This information is important, because actually only the staging project A supports these kinds of packages. -------------------------------------------------------------------------------- osc staging select A 12345 -------------------------------------------------------------------------------- This command will put the request #12345 into the staging project A. If A is full, the user can wait until A is empty again before putting new packages from Ring:0. Use Case 3: Dispatching packages ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The staging manager wants to move some packages into different staging projects. The complex part is to decide how to distribute the packages here. The staging manager needs to make sure that packages that have related changes (e.g. new +rpmlint+ check and the packages having fixes for it) are tested in one letter. -------------------------------------------------------------------------------- osc staging select B 22221 22222 osc staging select C 22223 osc staging select B 22224 -------------------------------------------------------------------------------- The +select+ also has a --move to correct mistakes done on first run. -------------------------------------------------------------------------------- osc staging select --move C 22224 -------------------------------------------------------------------------------- Also the staging manager can return some request into the original queue. -------------------------------------------------------------------------------- osc staging unselect 22224 -------------------------------------------------------------------------------- Staging projects should not be too small, but not too big either - and staging projects that are almost done testing shouldn't get a re-triggered build. So in practice adding them in large batches has proven useful, i.e. adding to B for half a day and then open up C and add to it from then on and only look back at B if there is a problem. Use Case 4: Rebasing staging projects ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From time to time the staging projects need a rebase to make sure that they are still working with the current status of Factory. For this the staging manager can use the +freeze+ command to update the links of the packages. -------------------------------------------------------------------------------- osc staging freeze -------------------------------------------------------------------------------- Of course, this will be done only when the project is in green status and Factory, the base, is also green in https://openqa.opensuse.org/tests/?sort=-mtime&hours=18&match=staging&ob=[openQA]. In other case we can see errors in the staging project that comes from Factory. A pro-tip: the ring projects should be basically built and tested - quite challenging to find the right moment. Use Case 5: A full day in Factory ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Checking the current status -------------------------------------------------------------------------------- osc staging check -------------------------------------------------------------------------------- In the list we found a request that is independent, we moved it to a isolated staging project. -------------------------------------------------------------------------------- osc staging select B 12345 -------------------------------------------------------------------------------- There is also a Ring:0 package, that needs to be in A -------------------------------------------------------------------------------- osc staging select A 12300 -------------------------------------------------------------------------------- Also we found three YaST packages that are related. -------------------------------------------------------------------------------- osc staging select C 22201 22202 22203 -------------------------------------------------------------------------------- We wait a bit and we check the result in openQA. We see that the packages work properly in OBS (compile correctly), but there is something wrong in openQA: some of the tests are failing. In this situation we can: * Rebuild the image in openQA to see if this is a random problem. * If openQA is red again, check the packages that can be problematic in the staging project, maybe reading the changelog. * With this information, remove one of the requests from the staging project, putting it back to the queue. -------------------------------------------------------------------------------- osc staging unselect C 22202 -------------------------------------------------------------------------------- After a while we see some packages that are failing in OBS, we need to discard that is a random fail, we re-trigger the build: -------------------------------------------------------------------------------- osc rebuildpac $PROJ $PKG $REPO $ARCH -------------------------------------------------------------------------------- From time to time, we see that there is a missing dependency for one of the packages that is in the staging project, and this dependency is not in the subset of Factory (MinimalX) that is linked in the staging project (for example: ImageMagick needs libqr to build properly). In this case we need to +linpack+ this package into the staging project: -------------------------------------------------------------------------------- osc linkpac openSUSE:Factory liblqr openSUSE:Factory:Staging:F --------------------------------------------------------------------------------