411 lines
15 KiB
Plaintext
411 lines
15 KiB
Plaintext
Staging Plugin Documentation
|
|
============================
|
|
:author: openSUSE Team at SUSE
|
|
:toc:
|
|
|
|
|
|
Introduction
|
|
------------
|
|
[id="intro"]
|
|
|
|
The staging plugin is used to enable the new
|
|
https://progress.opensuse.org/workflow/factory-proposal.html[factory
|
|
workflow] to work with staging projects. This plugin implements some
|
|
functionality that some day will be directly implemented in OBS
|
|
itself. Meanwhile this plugin is used as a playground field to
|
|
explore these new concepts and operations.
|
|
|
|
This plugin is used by the release manager to move requests into
|
|
staging projects to evaluate the behavior and inter-compatibility of
|
|
different changes working together.
|
|
|
|
An staging project is a mere project linked to a subset of Factory
|
|
(MinimalX) , but with some packages that are different as a result of
|
|
one or several requests. When the release manager assigns a request
|
|
in a staging project, this request is review inside th project.
|
|
|
|
There is a precondition here, the user that has the role of staging
|
|
manager needs to have permission in OBS, and they need to be:
|
|
|
|
* Maintainer of openSUSE:Factory:Staging (needs current staging manager)
|
|
|
|
* Group member of factory-staging (needs OBS admin)
|
|
|
|
|
|
Installation
|
|
------------
|
|
[id="install"]
|
|
|
|
Clone this repository:
|
|
|
|
--------------------------------------------------------------------------------
|
|
git clone https://github.com/openSUSE/openSUSE-release-tools.git
|
|
--------------------------------------------------------------------------------
|
|
|
|
Link +osc-staging.py+ and the library either to +~/.osc-plugins+ or
|
|
+/var/lib/osc-plugins+
|
|
|
|
--------------------------------------------------------------------------------
|
|
cd openSUSE-release-tools
|
|
ln -sr ./osc-staging.py ./osclib ~/.osc-plugins
|
|
--------------------------------------------------------------------------------
|
|
|
|
Optionally, one can create directories for each project of interest in order to
|
|
alleviate the need to use the +--project option+ for each command. For example:
|
|
|
|
--------------------------------------------------------------------------------
|
|
# Or use a directory already initialized by osc.
|
|
mkdir -p "openSUSE:Leap:42.3/.osc"
|
|
cd "openSUSE:Leap:42.3"
|
|
echo openSUSE:Leap:42.3 > .osc/_project
|
|
|
|
# runs against openSUSE:Leap:42.3
|
|
osc staging list
|
|
|
|
# runs against openSUSE:Factory
|
|
osc staging -p Factory list
|
|
--------------------------------------------------------------------------------
|
|
|
|
The following snippet may be useful for addition to a bash PS1.
|
|
|
|
--------------------------------------------------------------------------------
|
|
if [ -f .osc/_project ] ; then
|
|
echo " <<$(head -n 1 .osc/_project)>> \$ ";
|
|
fi
|
|
--------------------------------------------------------------------------------
|
|
|
|
This provides a clear indication of the project against which the commands will
|
|
run and can be merged with an existing complex PS1.
|
|
|
|
|
|
Command line
|
|
------------
|
|
[id="cli"]
|
|
|
|
This plugin is expected to be used interactively. The basic commands
|
|
are:
|
|
|
|
* *accept* Accept all requests assigned to a staging project.
|
|
|
|
* *check* Check the status of the different request inside the staging
|
|
project, including the status in openQA.
|
|
|
|
* *cleanup_rings* Check the status of the different rings in Factory.
|
|
|
|
* *freeze* Freeze the packages that are not affected in a staging project.
|
|
|
|
* *list* Accept all the requests from packages that are not in any
|
|
ring, detect request that supersede the current ones that are
|
|
actually tracked in staging projects, and list the remaining
|
|
requests.
|
|
|
|
* *select* Move a request (or a list of requests) to a staging
|
|
project, or move requests between staging projects.
|
|
|
|
* *unselect* Remove a request from a staging project, and publish it
|
|
again into the backlog.
|
|
|
|
|
|
List
|
|
~~~~
|
|
|
|
This action will list all the requests that are candidates to be
|
|
assigned to a different staging project. To decide this list, this
|
|
command will perform two actions with side effects into the requests
|
|
status.
|
|
|
|
The first action is to accept requests that do not affect packages
|
|
that are not inside one of the rings that are defined in Factory.
|
|
This acceptance will move the request to the next status of revision
|
|
(maybe waiting to the manual review or the legal review).
|
|
|
|
The second action is to detect if a request supersedes a different one
|
|
that is currently in a staging project. If this is the case, this
|
|
command will replace the old request by the new one, declining the
|
|
superseded one after this.
|
|
|
|
Because of the different side effects, this command needs to be
|
|
executed with the proper permissions in OBS.
|
|
|
|
|
|
Select
|
|
~~~~~~
|
|
|
|
The release manager or the staging project manager can take a request
|
|
and assign it into a staging project using the +select+ command.
|
|
|
|
Actually we have
|
|
https://build.opensuse.org/project/subprojects/openSUSE:Factory:Staging[ten
|
|
staging projects], that are managed according to some internal rules.
|
|
Every time that the manager decides to move a request to one of those
|
|
projects, a metadata attached to these projects is updated, and a new
|
|
comment is generated.
|
|
|
|
The metadata associated to a staging project contains the list of
|
|
requests that are assigned to the project, and is used by the plugin
|
|
to track and manage the different requests. For example, the command:
|
|
|
|
--------------------------------------------------------------------------------
|
|
osc staging select A 12345
|
|
--------------------------------------------------------------------------------
|
|
|
|
will add the request #12345 into the staging project
|
|
openSUSE:Factory:Staging:A. This command will take care that this
|
|
request was not previously in the staging project A or in a different
|
|
one. If this is the case, it will show an error message and exit without
|
|
selecting the request. If we want to move a request from one staging
|
|
project to another one, we need to use the parameter +--move+:
|
|
|
|
--------------------------------------------------------------------------------
|
|
osc staging select --move B 12345
|
|
--------------------------------------------------------------------------------
|
|
|
|
This command will move #12345 from openSUSE:Factory:Staging:A to
|
|
openSUSE:Factory:Staging:B
|
|
|
|
|
|
Unselect
|
|
~~~~~~~~
|
|
|
|
The +unselect+ command will untrack a request from a staging project,
|
|
returning it back to the original backlog (without approving or
|
|
declining the request) So for example, if #12345 is being tracked in
|
|
A, the command:
|
|
|
|
--------------------------------------------------------------------------------
|
|
osc staging unselect 12345
|
|
--------------------------------------------------------------------------------
|
|
|
|
will find the correct staging project and remove the request from it.
|
|
|
|
|
|
Check
|
|
~~~~~
|
|
|
|
Before accepting the requests inside a staging project, the user can
|
|
check the state of those requests. The +check+ command will check the
|
|
project status, taking care of superseded requests or already accepted
|
|
requests.
|
|
|
|
This command will also check the status in openQA of the project.
|
|
|
|
This command can be called without special permissions.
|
|
|
|
|
|
Accept
|
|
~~~~~~
|
|
|
|
If the current status of the staging project is good, this command
|
|
will change the review status of the different requests assigned to
|
|
the project, accepting them.
|
|
|
|
Internally, the +accept+ command contains a call to the +check+
|
|
command to make sure that the request can be accepted.
|
|
|
|
After this command, the staging project status will be disabled, to
|
|
avoid the overload of OBS.
|
|
|
|
|
|
Cleanup Rings
|
|
~~~~~~~~~~~~~
|
|
|
|
https://build.opensuse.org/project/subprojects/openSUSE:Factory:Rings[Rings]
|
|
are collections of packages that are deeply interconnected, and that
|
|
are building basis for a different ring or for the rest of the
|
|
distribution. A ring is a way to organize Factory into
|
|
inter-dependent packages that are again used to build a different
|
|
layer of Factory itself.
|
|
|
|
Actually we have identified three rings:
|
|
|
|
* https://build.opensuse.org/project/show/openSUSE:Factory:Rings:0-Bootstrap[openSUSE:Factory:Rings:0-Bootstrap]
|
|
* https://build.opensuse.org/project/show/openSUSE:Factory:Rings:1-MinimalX[openSUSE:Factory:Rings:1-MinimalX]
|
|
* https://build.opensuse.org/project/show/openSUSE:Factory:Rings:2-TestDVD[openSUSE:Factory:Rings:2-TestDVD]
|
|
|
|
And you can find a better description in the
|
|
https://www.youtube.com/watch?v=K-wTVGqKFR8[talk conference]
|
|
celebrated in 2014 in Dubrovnik, Croatia.
|
|
|
|
This command is used to check the current status of the rings and to
|
|
find undesirable dependencies of the packages that conform the rings.
|
|
|
|
|
|
Freeze
|
|
~~~~~~
|
|
|
|
Factory (or the subset MinimalX) is always a moving target, even with
|
|
the staging projects. If we want to check the status of the request
|
|
assigned to a staging project, sometimes it is desirable to have a frozen
|
|
status of the source packages that are part of Factory but not of the
|
|
staging project.
|
|
|
|
This command is used to build frozenlink-kind-of links for a staging
|
|
project.
|
|
|
|
|
|
Use cases
|
|
---------
|
|
[id="usecases"]
|
|
|
|
|
|
Use Case 1: Checking Factory
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
The staging manager wants to see the status of Factory every morning.
|
|
|
|
--------------------------------------------------------------------------------
|
|
osc staging check
|
|
--------------------------------------------------------------------------------
|
|
|
|
With this status they will contact the author of the request,
|
|
or will rebuild the packages if there is a suspect or a random fail.
|
|
|
|
After that, the staging manager can check the important packages that
|
|
are pending in the queue.
|
|
|
|
--------------------------------------------------------------------------------
|
|
osc staging list
|
|
--------------------------------------------------------------------------------
|
|
|
|
This command can update the request attached to a staging project,
|
|
replacing the superseded one. The list shows the name of the ring
|
|
where this package is found.
|
|
|
|
|
|
Use Case 2: Selecting packages from Ring:0-Bootstrap
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
The +list+ command shows the name of the ring where we can find the
|
|
package involved in the request. This information is important,
|
|
because actually only the staging project A supports these kinds of
|
|
packages.
|
|
|
|
--------------------------------------------------------------------------------
|
|
osc staging select A 12345
|
|
--------------------------------------------------------------------------------
|
|
|
|
This command will put the request #12345 into the staging project A.
|
|
If A is full, the user can wait until A is empty again before putting
|
|
new packages from Ring:0.
|
|
|
|
|
|
Use Case 3: Dispatching packages
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
The staging manager wants to move some packages into different staging
|
|
projects. The complex part is to decide how to distribute the
|
|
packages here. The staging manager needs to make sure that packages
|
|
that have related changes (e.g. new +rpmlint+ check and the packages
|
|
having fixes for it) are tested in one letter.
|
|
|
|
--------------------------------------------------------------------------------
|
|
osc staging select B 22221 22222
|
|
osc staging select C 22223
|
|
osc staging select B 22224
|
|
--------------------------------------------------------------------------------
|
|
|
|
The +select+ also has a --move to correct mistakes done on first run.
|
|
|
|
--------------------------------------------------------------------------------
|
|
osc staging select --move C 22224
|
|
--------------------------------------------------------------------------------
|
|
|
|
Also the staging manager can return some request into the original
|
|
queue.
|
|
|
|
--------------------------------------------------------------------------------
|
|
osc staging unselect 22224
|
|
--------------------------------------------------------------------------------
|
|
|
|
Staging projects should not be too small, but not too big either - and
|
|
staging projects that are almost done testing shouldn't get a
|
|
re-triggered build. So in practice adding them in large batches has
|
|
proven useful, i.e. adding to B for half a day and then open up C and
|
|
add to it from then on and only look back at B if there is a problem.
|
|
|
|
|
|
Use Case 4: Rebasing staging projects
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
From time to time the staging projects need a rebase to make sure that
|
|
they are still working with the current status of Factory. For this
|
|
the staging manager can use the +freeze+ command to update the links
|
|
of the packages.
|
|
|
|
--------------------------------------------------------------------------------
|
|
osc staging freeze
|
|
--------------------------------------------------------------------------------
|
|
|
|
Of course, this will be done only when the project is in green status
|
|
and Factory, the base, is also green in
|
|
https://openqa.opensuse.org/tests/?sort=-mtime&hours=18&match=staging&ob=[openQA].
|
|
In other case we can see errors in the staging project that comes from
|
|
Factory.
|
|
|
|
A pro-tip: the ring projects should be basically built and tested -
|
|
quite challenging to find the right moment.
|
|
|
|
|
|
Use Case 5: A full day in Factory
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Checking the current status
|
|
|
|
--------------------------------------------------------------------------------
|
|
osc staging check
|
|
--------------------------------------------------------------------------------
|
|
|
|
In the list we found a request that is independent, we moved it to a
|
|
isolated staging project.
|
|
|
|
--------------------------------------------------------------------------------
|
|
osc staging select B 12345
|
|
--------------------------------------------------------------------------------
|
|
|
|
There is also a Ring:0 package, that needs to be in A
|
|
|
|
--------------------------------------------------------------------------------
|
|
osc staging select A 12300
|
|
--------------------------------------------------------------------------------
|
|
|
|
Also we found three YaST packages that are related.
|
|
|
|
--------------------------------------------------------------------------------
|
|
osc staging select C 22201 22202 22203
|
|
--------------------------------------------------------------------------------
|
|
|
|
We wait a bit and we check the result in openQA. We see that the
|
|
packages work properly in OBS (compile correctly), but there is
|
|
something wrong in openQA: some of the tests are failing.
|
|
|
|
In this situation we can:
|
|
|
|
* Rebuild the image in openQA to see if this is a random problem.
|
|
|
|
* If openQA is red again, check the packages that can be problematic
|
|
in the staging project, maybe reading the changelog.
|
|
|
|
* With this information, remove one of the requests from the staging
|
|
project, putting it back to the queue.
|
|
|
|
--------------------------------------------------------------------------------
|
|
osc staging unselect C 22202
|
|
--------------------------------------------------------------------------------
|
|
|
|
After a while we see some packages that are failing in OBS, we need to
|
|
discard that is a random fail, we re-trigger the build:
|
|
|
|
--------------------------------------------------------------------------------
|
|
osc rebuildpac $PROJ $PKG $REPO $ARCH
|
|
--------------------------------------------------------------------------------
|
|
|
|
From time to time, we see that there is a missing dependency for one
|
|
of the packages that is in the staging project, and this dependency is
|
|
not in the subset of Factory (MinimalX) that is linked in the staging
|
|
project (for example: ImageMagick needs libqr to build properly). In
|
|
this case we need to +linpack+ this package into the staging project:
|
|
|
|
--------------------------------------------------------------------------------
|
|
osc linkpac openSUSE:Factory liblqr openSUSE:Factory:Staging:F
|
|
--------------------------------------------------------------------------------
|