openSUSE-release-tools/README.md

[![Build Status](https://github.com/openSUSE/openSUSE-release-tools/workflows/CI/badge.svg?branch=master)](https://github.com/openSUSE/openSUSE-release-tools/actions?query=branch%3Amaster)
[![codecov](https://codecov.io/gh/openSUSE/openSUSE-release-tools/branch/master/graph/badge.svg?token=MqVygxmguE)](https://codecov.io/gh/openSUSE/openSUSE-release-tools)
[![openSUSE Tumbleweed package](https://repology.org/badge/version-for-repo/opensuse_tumbleweed/opensuse-release-tools.svg)](https://repology.org/metapackage/opensuse-release-tools)

# openSUSE-release-tools

This repository contains a set of tools to aid in the process of building, testing and releasing
(open)SUSE based distributions and their corresponding maintenance updates. You can find more
information in [docs/processes.md](docs/processes.md). The [CONTENTS.md](CONTENTS.md) file contains
a list of the tools that are included in the repository.

![Rethink release tooling presentation overview](docs/res/workflow-overview.svg)

Everything denoted with a cloud is largely in this repository while the rest is the [open-build-service (OBS)](https://github.com/openSUSE/open-build-service).

## Installation

For non-development usage just install the package.

    zypper in openSUSE-release-tools

Many sub-packages are provided which can be found either by searching or [looking on the build service](https://build.opensuse.org/package/binaries/openSUSE:Tools/openSUSE-release-tools/openSUSE_Factory).

    zypper se openSUSE-release-tools osc-plugin

If CI builds are needed add the [appropriate `openSUSE:Tools` repository](https://software.opensuse.org//download.html?project=openSUSE%3ATools&package=openSUSE-release-tools).

## Usage

All tools provide help documentation accessible via `--help`.

For `osc` plugins include the plugin name after `osc` like the following.

    osc staging --help

For other tools execute the tool directly.

    osrt-repo-checker --help

See the [docs](/docs) directory or a specific tool directory for specific tool documentation outside of `--help`. The [wiki](/wiki) also contains some additional documentation.

## Development

    git clone https://github.com/openSUSE/openSUSE-release-tools.git

If working on an `osc` plugin create symlinks for the plugin and `osclib` in either `~/.osc-plugins` or `/usr/lib/osc-plugins`. For example to install the _staging_ plugin do the following.

    mkdir -p ~/.osc-plugins
    ln -sr ./osc-staging.py ./osclib ~/.osc-plugins

It can also be useful to work against a development copy of `osc` either to utilize new features or to debug/fix functionality. To do so one must place the development copy in the path to be loaded and utilize the wrapper script if working on `osc` plugins. One method to accomplish this is shown below.

    # outside of openSUSE-release-tools checkout
    git clone https://github.com/openSUSE/osc.git

    # inside openSUSE-release-tools checkout
    # note the ending /osc which points to the osc directory within the checkout
    ln -s /path/to/osc/osc ./

    # to utilize the wrapper for working on osc plugins from osrt checkout
    $(realpath ./osc)/../osc-wrapper.py --version

Using [Docker Compose](https://docs.docker.com/compose/), a containerized OBS can be started via one command. The default credentials are `Admin` and `opensuse` on [0.0.0.0:3000](http://0.0.0.0:3000). You can change the port by setting the environment variable `OSRT_EXPOSED_OBS_PORT`.

    docker-compose -f dist/ci/docker-compose.yml up api

To make things easier, you can add an alias to refer to this instance (do not forget to adjust the port if you are using a different one):

    cat <<EOF >> ~/.config/osc/oscrc

    [http://0.0.0.0:3000]
    user = Admin
    pass = opensuse
    aliases = local
    EOF

Then you can use the new `local` alias to access this new instance.

    osc -A local api /about

Some tests will attempt to run against the local OBS, but not all. It's still recommended to run this through docker-compose (see below)

    pytest tests/*.py

## Running Continuous Integration

This repository includes all the needed files to set up and run the Continuous Integration test suite. The idea is to use Docker Compose to orchestrate a set of containers, including an OBS instance, and run [the tests](tests/) on top of them. Although they automatically run [on GitHub Actions](https://github.com/features/actions) (more on that later), it is easy to run them locally. The following commands must be executed from the root of the repository.


    # Mount the current path at the /code directory on the container
    sed -i -e "s,../..:,$PWD:," dist/ci/docker-compose.yml

    # Run the linter
    docker-compose -f dist/ci/docker-compose.yml run flaker

    # Run the test full suite (it may take some time)...
    docker-compose -f dist/ci/docker-compose.yml run test

    # .. or just run a single test (i.e., the 'tests/util_tests.py')
    docker-compose -f dist/ci/docker-compose.yml run test run_as_tester pytest tests/util_tests.py

    # We are finished. Now you can shut the containers down.
    docker-compose -f dist/ci/docker-compose.yml down

The [docker-compose.yml](dist/ci/docker-compose.yml) mentions two container images that are built in the [openSUSE:Tools:Images](https://build.opensuse.org/project/show/openSUSE:Tools:Images) project:

* `osrt-miniobs-for-ci` is the base of OBS-related services (API, caches, SMTP, and so on).
* `osrt-testenv-tumbleweed` used to run the tests. The code and the tests are mounted in the `/code` directory of this container.

As mentioned before, the main repository uses GitHub Actions to automatically run the tests when a pull request is opened or the code is pushed to the master branch. You can find the details in the
[workflow definition](.github/workflows/ci-test.yml). Note that, in addition to the steps listed before, code coverage data is submitted to [Codecov](https://app.codecov.io/gh/openSUSE/openSUSE-release-tools).

### Debugging Failures in CI

This section lists a few tricks to debug problems in the CI. You will use your local setup so, as a first step, you need to be able to run the tests as described in the previous section.
To see the logs from all the containers, the following command can be executed:

  docker-compose -f dist/ci/docker-compose.yml logs -f --tail=10

You can run commands in any container by using the docker-compose `exec` command. For instance, you can connect to a container through a shell with the following command (in this case, it will connect to the container behind the `api` service):

  docker-compose -f dist/ci/docker-compose.yml exec api sh

Or you could check the API logs by issuing the following command:

  docker-compose -f dist/ci/docker-compose.yml exec api sh -c 'tail -f /srv/www/obs/api/log/*.log'

To debug problems in the test suite or in the code, place a `breakpoint()` call and you will get access to Python's debugger.

You can access your testing OBS instance at `http://0.0.0.0:3000` and log in using "Admin" as username and "opensuse" as password. To prevent the data being removed while you are inspecting the OBS instance, you can put a call to the `breakpoint()` function.

Finally, if you miss anything for debugging, you can use `zypper` to install it.

### Adding tests

Testing the release tools isn't quite trivial as a lot of these tools rely on running openSUSE infrastructure. Some of the workflows we replay (not mock) in above described docker-compose setup. So each test will setup the required projects and e.g. staging workflows in a local containerized OBS installation and then do its assertions. If you want to add coverage, best check existing unit tests in tests/*.py. A generic test case looks similiar to this:

``` {.python title="Basic Test Example" }
class TestExample(unittest.TestCase):

def test_basic(self):
    # Keep the workflow in local scope so that ending the test case will destroy it.
    # Destroying the workflow will also delete all created projects and packages. The
    # created workflow has a target project, but most of the test assets need to be created
    # as needed
    wf = OBSLocal.FactoryWorkflow()
    staging = wf.create_staging('A', freeze=True)
    wf.create_submit_request('devel:wine', 'wine')

    ret = SelectCommand(wf.api, staging.name).perform(['wine'])
    self.assertEqual(True, ret)
```

To ease having many such tests, we also have the `OBSLocal` class, which moves the creation of the workflow into `setUp` and the destruction in `tearDwon` functions of pytest. The principle stays the same though.

``` {.python title="OBSLocal Usage"}
class TestExampleWithOBS(OBSLocal.TestCase):
    """
    Tests for various api calls to ensure we return expected content
    """

    def setUp(self):
        super(TestExampleWithOBS, self).setUp()
        self.wf = OBSLocal.FactoryWorkflow()
        self.wf.setup_rings()
        self.staging_b = self.wf.create_staging('B')

    def tearDown(self):
        del self.wf
        super(TestExampleWithOBS, self).tearDown()

    def test_list_projects(self):
        """
        List projects and their content
        """
        staging_a = self.wf.create_staging('A')

        # Prepare expected results
        data = [staging_a.name, self.staging_b.name]

        # Compare the results
        self.assertEqual(data, self.wf.api.get_staging_projects())
```

Note that we have some (older) test cases using httpretty, but those are very special cases and require you a lot of extra mocking as you can't mix httpretty and testing against the minimal OBS. So every extra
call that osc libraries or our code do, will require changes in your test case. It can still be a viable option, especially if more than OBS is involved.

The method that you can combine with `OBSLocal` though is using MagicMock. This class is used to mock individual functions. So splitting the code to use helper functions to retrieve information and then
mocking this inside the test case can be a good alternative to mocking the complete HTTP traffic.