Staging Plugin Documentation
============================
:author: openSUSE Team at SUSE
:toc:


Introduction
------------
[id="intro"]

The staging plugin is used to enable the new factory
workflow to work with staging projects.  This plugin implements some
functionality that some day will be directly implemented in OBS
itself.  Meanwhile this plugin is used as a playground field to
explore these new concepts and operations.

This plugin is used by the release manager to move requests into
staging projects to evaluate the behavior and inter-compatibility of
different changes working together.

An staging project is a mere project linked to a subset of Factory
(MinimalX) , but with some packages that are different as a result of
one or several requests.  When the release manager assigns a request
in a staging project, this request is review inside the project.

There is a precondition here, the user that has the role of staging
manager needs to have permission in OBS, and they need to be:

* Maintainer of openSUSE:Factory:Staging (needs current staging manager)

* Group member of factory-staging (needs OBS admin)


Installation
------------
[id="install"]

Clone this repository:

--------------------------------------------------------------------------------
git clone https://github.com/openSUSE/openSUSE-release-tools.git
--------------------------------------------------------------------------------

Link +osc-staging.py+ and the library either to +~/.osc-plugins+ or
+/var/lib/osc-plugins+

--------------------------------------------------------------------------------
cd openSUSE-release-tools
ln -sr ./osc-staging.py ./osclib ~/.osc-plugins
--------------------------------------------------------------------------------

Optionally, one can create directories for each project of interest in order to
alleviate the need to use the +--project option+ for each command. For example:

--------------------------------------------------------------------------------
# Or use a directory already initialized by osc.
mkdir -p "openSUSE:Leap:42.3/.osc"
cd "openSUSE:Leap:42.3"
echo openSUSE:Leap:42.3 > .osc/_project

# runs against openSUSE:Leap:42.3
osc staging list

# runs against openSUSE:Factory
osc staging -p Factory list
--------------------------------------------------------------------------------

The following snippet may be useful for addition to a bash PS1.

--------------------------------------------------------------------------------
if [ -f .osc/_project ] ; then
  echo " <<$(head -n 1 .osc/_project)>> \$ ";
fi
--------------------------------------------------------------------------------

This provides a clear indication of the project against which the commands will
run and can be merged with an existing complex PS1.


Command line
------------
[id="cli"]

This plugin is expected to be used interactively.  The basic commands
are:

* *accept* Accept all requests assigned to a staging project.

* *check* Check the status of the different request inside the staging
  project, including the status in openQA.

* *cleanup_rings* Check the status of the different rings in Factory.

* *rebase* Freeze the packages that are not affected in a staging project.
  In the past, this command used to be called 'freeze', which is still
  a valid alias.

* *list* Accept all the requests from packages that are not in any
  ring, detect request that supersede the current ones that are
  actually tracked in staging projects, and list the remaining
  requests.

* *select* Move a request (or a list of requests) to a staging
  project, or move requests between staging projects.

* *unselect* Remove a request from a staging project, and publish it
  again into the backlog.


List
~~~~

This action will list all the requests that are candidates to be
assigned to a different staging project.  To decide this list, this
command will perform two actions with side effects into the requests
status.

The first action is to accept requests that do not affect packages
that are not inside one of the rings that are defined in Factory.
This acceptance will move the request to the next status of revision
(maybe waiting to the manual review or the legal review).

The second action is to detect if a request supersedes a different one
that is currently in a staging project.  If this is the case, this
command will replace the old request by the new one, declining the
superseded one after this.

Because of the different side effects, this command needs to be
executed with the proper permissions in OBS.


Select
~~~~~~

The release manager or the staging project manager can take a request
and assign it into a staging project using the +select+ command.

Actually we have
https://build.opensuse.org/project/subprojects/openSUSE:Factory:Staging[ten
staging projects], that are managed according to some internal rules.
Every time that the manager decides to move a request to one of those
projects, a metadata attached to these projects is updated, and a new
comment is generated.

The metadata associated to a staging project contains the list of
requests that are assigned to the project, and is used by the plugin
to track and manage the different requests.  For example, the command:

--------------------------------------------------------------------------------
osc staging select A 12345
--------------------------------------------------------------------------------

will add the request #12345 into the staging project
openSUSE:Factory:Staging:A.  This command will take care that this
request was not previously in the staging project A or in a different
one.  If this is the case, it will show an error message and exit without
selecting the request.  If we want to move a request from one staging
project to another one, we need to use the parameter +--move+:

--------------------------------------------------------------------------------
osc staging select B --move 12345
--------------------------------------------------------------------------------

This command will move #12345 from openSUSE:Factory:Staging:A to
openSUSE:Factory:Staging:B

You can also merge staging projects by giving other staging projects as arguments:

--------------------------------------------------------------------------------
osc staging select B --move A C
--------------------------------------------------------------------------------

This moves all requests currently staged in A and C into B

Unselect
~~~~~~~~

The +unselect+ command will untrack a request from a staging project,
returning it back to the original backlog (without approving or
declining the request) So for example, if #12345 is being tracked in
A, the command:

--------------------------------------------------------------------------------
osc staging unselect 12345
--------------------------------------------------------------------------------

will find the correct staging project and remove the request from it.


Check
~~~~~

Before accepting the requests inside a staging project, the user can
check the state of those requests.  The +check+ command will check the
project status, taking care of superseded requests or already accepted
requests.

This command will also check the status in openQA of the project.

This command can be called without special permissions.


Accept
~~~~~~

If the current status of the staging project is good, this command
will change the review status of the different requests assigned to
the project, accepting them.

Internally, the +accept+ command contains a call to the +check+
command to make sure that the request can be accepted.

After this command, the staging project status will be disabled, to
avoid the overload of OBS.


Cleanup Rings
~~~~~~~~~~~~~

https://build.opensuse.org/project/subprojects/openSUSE:Factory:Rings[Rings]
are collections of packages that are deeply interconnected, and that
are building basis for a different ring or for the rest of the
distribution.  A ring is a way to organize Factory into
inter-dependent packages that are again used to build a different
layer of Factory itself.

Actually we have identified three rings:

* https://build.opensuse.org/project/show/openSUSE:Factory:Rings:0-Bootstrap[openSUSE:Factory:Rings:0-Bootstrap]
* https://build.opensuse.org/project/show/openSUSE:Factory:Rings:1-MinimalX[openSUSE:Factory:Rings:1-MinimalX]

And you can find a better description in this
https://www.youtube.com/watch?v=K-wTVGqKFR8[very old presentation] from a conference
celebrated in 2014 in Dubrovnik, Croatia.

This command is used to check the current status of the rings and to
find undesirable dependencies of the packages that conform the rings.


Rebase
~~~~~~

Factory (or the subset MinimalX) is always a moving target, even with
the staging projects.  If we want to check the status of the request
assigned to a staging project, sometimes it is desirable to have a frozen
status of the source packages that are part of Factory but not of the
staging project.

This command is used to build frozenlink-kind-of links for a staging
project.


Use cases
---------
[id="usecases"]


Use Case 1: Checking Factory
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The staging manager wants to see the status of Factory every morning.

--------------------------------------------------------------------------------
osc staging check
--------------------------------------------------------------------------------

With this status they will contact the author of the request,
or will rebuild the packages if there is a suspect or a random fail.

After that, the staging manager can check the important packages that
are pending in the queue.

--------------------------------------------------------------------------------
osc staging list
--------------------------------------------------------------------------------

This command can update the request attached to a staging project,
replacing the superseded one.  The list shows the name of the ring
where this package is found.


Use Case 2: Selecting packages from Ring:0-Bootstrap
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The +list+ command shows the name of the ring where we can find the
package involved in the request.  This information is important,
because actually only the staging project A supports these kinds of
packages.

--------------------------------------------------------------------------------
osc staging select A 12345
--------------------------------------------------------------------------------

This command will put the request #12345 into the staging project A.
If A is full, the user can wait until A is empty again before putting
new packages from Ring:0.


Use Case 3: Dispatching packages
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The staging manager wants to move some packages into different staging
projects.  The complex part is to decide how to distribute the
packages here.  The staging manager needs to make sure that packages
that have related changes (e.g. new +rpmlint+ check and the packages
having fixes for it) are tested in one letter.

--------------------------------------------------------------------------------
osc staging select B 22221 22222
osc staging select C 22223
osc staging select B 22224
--------------------------------------------------------------------------------

The +select+ also has a --move to correct mistakes done on first run.

--------------------------------------------------------------------------------
osc staging select --move C 22224
--------------------------------------------------------------------------------

Also the staging manager can return some request into the original
queue.

--------------------------------------------------------------------------------
osc staging unselect 22224
--------------------------------------------------------------------------------

Staging projects should not be too small, but not too big either - and
staging projects that are almost done testing shouldn't get a
re-triggered build.  So in practice adding them in large batches has
proven useful, i.e. adding to B for half a day and then open up C and
add to it from then on and only look back at B if there is a problem.


Use Case 4: Rebasing staging projects
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

From time to time the staging projects need a rebase to make sure that
they are still working with the current status of Factory.  For this
the staging manager can use the +freeze+ command to update the links
of the packages.

--------------------------------------------------------------------------------
osc staging freeze
--------------------------------------------------------------------------------

Of course, this will be done only when the project is in green status
and Factory, the base, is also green in
https://openqa.opensuse.org/tests/?sort=-mtime&hours=18&match=staging&ob=[openQA].
In other case we can see errors in the staging project that comes from
Factory.

A pro-tip: the ring projects should be basically built and tested -
quite challenging to find the right moment.


Use Case 5: A full day in Factory
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Checking the current status

--------------------------------------------------------------------------------
osc staging check
--------------------------------------------------------------------------------

In the list we found a request that is independent, we moved it to a
isolated staging project.

--------------------------------------------------------------------------------
osc staging select B 12345
--------------------------------------------------------------------------------

There is also a Ring:0 package, that needs to be in A

--------------------------------------------------------------------------------
osc staging select A 12300
--------------------------------------------------------------------------------

Also we found three YaST packages that are related.

--------------------------------------------------------------------------------
osc staging select C 22201 22202 22203
--------------------------------------------------------------------------------

We wait a bit and we check the result in openQA.  We see that the
packages work properly in OBS (compile correctly), but there is
something wrong in openQA: some of the tests are failing.

In this situation we can:

* Rebuild the image in openQA to see if this is a random problem.

* If openQA is red again, check the packages that can be problematic
  in the staging project, maybe reading the changelog.

* With this information, remove one of the requests from the staging
  project, putting it back to the queue.

--------------------------------------------------------------------------------
osc staging unselect C 22202
--------------------------------------------------------------------------------

After a while we see some packages that are failing in OBS, we need to
discard that is a random fail, we re-trigger the build:

--------------------------------------------------------------------------------
osc rebuildpac $PROJ $PKG $REPO $ARCH
--------------------------------------------------------------------------------

From time to time, we see that there is a missing dependency for one
of the packages that is in the staging project, and this dependency is
not in the subset of Factory (MinimalX) that is linked in the staging
project (for example: ImageMagick needs libqr to build properly).  In
this case we need to +linpack+ this package into the staging project:

--------------------------------------------------------------------------------
osc linkpac openSUSE:Factory liblqr openSUSE:Factory:Staging:F
--------------------------------------------------------------------------------
