Working with the SUSE Kernel Sources
====================================

This document gives an overview of how SUSE Linux kernels are created, and
describes tasks like building individual kernels and creating external kernel
modules.


Overview
--------

The SUSE kernels are generated from the upstream Linux kernel sources found at
<https://kernel.org/>, on top of which a number of patches are applied. The
expanded kernel source tree is configured and built, resulting in a binary
kernel.

The add-on patches and configuration files are maintained in a Git repository at
<https://github.com/SUSE/kernel-source>, with an alternative mirror at
<https://github.com/openSUSE/kernel-source>.

A script named `scripts/tar-up.sh` packs up the files in the repository in a
form suitable for rpmbuild. When building the RPM packages, the following
binary packages get created:

* kernel-source

  The kernel source tree, generated by unpacking the vanilla kernel sources and
  applying the patches. The kernel sources are used by a number of other
  packages. They can also be used for compiling additional kernel modules.

* kernel-devel

  A complement of the kernel-source package that is needed to build external
  modules.

* kernel-`$FLAVOR`

  A number of binary kernels, for example, kernel-default for standard use,
  kernel-debug with extra debugging features, etc. These packages are all
  generated from the same kernel sources, and differ in the kernel
  configurations used.

* kernel-`$FLAVOR`-base

  A subset of kernel-`$FLAVOR`, for use in virtualized guests that only require
  a few device and filesystem drivers.

* kernel-`$FLAVOR`-devel

  The files used for generating kernel module packages for use with
  kernel-`$FLAVOR`.

* kernel-syms

  A meta package that pulls in the relevant kernel-`$FLAVOR`-devel packages for
  a given architecture.

The repository contains configuration files (`.config`) for all SUSE kernel
flavors. All configuration files are included in the dependencies of the
kernel-syms package, see [Where to find configuration
files](#where-to-find-configuration-files) below.

In the installed system, the kernel-source package installs files in the
following directories:

* `/usr/src/linux-$VERSION-$RELEASE/`

  The kernel sources.

* `/usr/src/linux`

  A symbolic link to `/usr/src/linux-$VERSION-$RELEASE/`.

* `/usr/share/doc/packages/kernel-source-$VERSION-$RELEASE/`

  This document.

The kernel-`$FLAVOR`-devel packages install the following files:

* `/usr/src/linux-$VERSION-$RELEASE-obj/$ARCH/$FLAVOR/`

  Kernel build object files for one kernel flavor. These files are used for
  compiling additional kernel modules.

* `/usr/src/linux-obj/$ARCH/$FLAVOR`

  A symbolic link to `/usr/src/linux-$VERSION-$RELEASE-obj/$ARCH/$FLAVOR/`.


Compiling your own kernel
-------------------------

The kernel sources are found in the kernel-source package. A binary kernel can
be built from these sources as follows:

1. Install the kernel-source package.

2. Create a build directory for use in configuring and building the kernel.
   Using `/usr/src/linux/` directly requires root privileges and will cause
   problems if you need to build kernel modules for other installed kernels.

3. Configure the kernel, see [How to configure the kernel
   sources](#how-to-configure-the-kernel-sources). For example,
   `make -C /usr/src/linux O=$PWD oldconfig`.

4. Build the kernel and all its modules by invoking `make`.

5. Make sure that `/etc/modprobe.d/10-unsupported-modules.conf` contains

       allow_unsupported_modules 1

   otherwise modprobe will refuse to load any modules.

6. Install the modules and kernel: `make modules_install`, followed by
   `make install`. This automatically creates an initrd for the new kernel and
   adds an entry to the GRUB boot menu.

Instead of building binary kernels by hand, you can also build one of the
kernel-`$FLAVOR` packages using RPM.


Building additional (external) modules
--------------------------------------

A single binary kernel module generally only works for a specific version of the
kernel source tree, for a specific architecture and configuration. This means
that for each binary kernel that SUSE ships, a custom module must be built. This
requirement is to some extent relaxed by the modversion mechanism: modversions
attach a checksum to each symbol (function or variable) exported to modules by
the kernel. This allows to use kernel modules that have been built for a kernel
with a different version or release number in many cases, as long as none of the
symbols the module uses have changed between the two kernel versions.

When releasing maintenance or security update kernels for a specific product, we
carefully try to keep the kernel ABI stable. Despite this, we sometimes have no
choice but to break binary compatibility. In this case, those kernel modules
must be rebuilt.

Additional kernel modules for one of the SUSE kernel flavors can be built in the
following ways:

1. by doing an ad-hoc module build, using one of the standard configurations in
   `/usr/src/linux-obj/$ARCH/$FLAVOR`, or

2. by creating a Kernel Module Package (KMP) as described in the Kernel Module
   Packages Manual,
   <https://documentation.suse.com/sbp/all/html/SBP-KMP-Manual-SLE12SP2/index.html>.

The first method involves the following steps:

1. Install the kernel-devel package.

2. Install the kernel-`$FLAVOR`-devel package. This is necessary for symbol
   version information (`CONFIG_MODVERSIONS`).

3. Compile the module(s) by changing into the module source directory and typing
   `make -C /usr/src/linux-obj/$ARCH/$FLAVOR M=$PWD`. Substitute `$ARCH` and
   `$FLAVOR` with the architecture and flavor for which to build the module(s).

   If the installed kernel sources match the running kernel, you can build
   modules for the running kernel by using the path
   `/lib/modules/$(uname -r)/build` as the `-C` option in the above command.
   (`build` is a symlink to `/usr/src/linux-obj/$ARCH/$FLAVOR`.)

4. Install the module(s) with
   `make -C /usr/src/linux-obj/$ARCH/$FLAVOR M=$PWD modules_install`.

Whenever building modules, please use the kernel build infrastructure as much as
possible, and do not try to circumvent it. The `Documentation/kbuild` directory
in the kernel sources documents kbuild makefiles.


Supported vs. unsupported modules
---------------------------------

As an extension to the mainline kernel, modules can be tagged as supported
(directly by SUSE, or indirectly by a third party) or unsupported. Modules which
are known to be flakey or for which SUSE does not have the necessary expertise
are marked as unsupported. Modules for which SUSE has third-party support
agreements are marked as externally supported. Modules for which SUSE provides
direct support are marked as supported.

The support status of a module can be queried with the modinfo tool. Modinfo
will report one of the following:

* direct support by SUSE: "supported: yes",
* third-party support: "supported: external",
* unsupported modules: no supported tag.

At runtime, the support status of a module can be obtained by reading
`/sys/module/$MODULE/supported`.

Note that this information is available only if the module was not built
directly into the kernel. Builtin modules are implicitly supported.

The aggregated support status for the entire kernel can be inspected by reading
`/sys/kernel/supported`. The value is also included in Oopses.

The setting of the "unsupported" kernel command line parameter and
`/proc/sys/kernel/unsupported` controls whether unsupported modules can be
loaded or not, and whether or not loading an unsupported module causes a warning
in the system log:

* 0 = only allow supported modules,
* 1 = warn when loading unsupported modules,
* 2 = don't warn.

Irrespective of this setting, loading an unsupported module sets a kernel taint
flag. The taint status of the kernel can be inspected in
`/proc/sys/kernel/tainted`. The taint flags are also included in Oopses.
Relevant bits have the following meaning:

| Bit | Log |     Number | Reason that got the kernel tainted                 |
|----:|----:|-----------:|----------------------------------------------------|
|   0 | G/P |          1 | proprietary module was loaded                      |
|  12 | ␣/O |       4096 | externally-built ("out-of-tree") module was loaded |
|  13 | ␣/E |       8192 | unsigned module was loaded                         |
|  16 | ␣/X |      65536 | module with third-party support was loaded         |
|  31 | ␣/N | 2147483648 | unsupported module was loaded                      |

Bits 16 and 31 are specific to the SUSE kernels. Since SLE15-SP6, loading an
externally supported module does not taint the kernel, but bit 16 (X) is still
tracked per module and can be read in `/sys/module/$MODULE/taint`.

Out-of-tree modules do not have the supported flag set by default; that
is, they are marked as unsupported. For building externally supported
modules, please get in touch with the Solid Driver Program team, led by
Scott Bahling, at <mailto:soliddriver@suse.com>.


Patch selection mechanism
-------------------------

The SUSE kernels consist of the upstream kernel sources on top of which a number
of patches is applied. The file `series.conf` determines which patches are
applied and which are excluded. A script named `guards` converts `series.conf`
into a plain list of patch files to be applied. Guards decides which patches to
include and exclude based on a list of symbols. From the kernel-source.src.rpm
package, a fully patched kernel source tree can be generated from vanilla
sources + patches like this:

1. Install the package:

       # rpm -i kernel-source-$RPMVERSION-$RPMRELEASE.src.rpm

2. Unpack the patches and the kernel sources:

       # cd /usr/src/packages/SOURCES/
       # for f in patches.*.tar.bz2; do \
           tar -xjf "$f" || break; \
         done
       # tar -xJf linux-$SRCVERSION.tar.xz

3. Apply the patches

       # for p in $(./guards < series.conf); do \
           patch -d linux-$SRCVERSION -p1 < $p || break; \
         done

The configuration script `config.conf` which is similar to `series.conf` is
used for configuration file selection, see the section [Where to find
configuration files](#where-to-find-configuration-files).

The file format of `series.conf` and `config.conf` should be apparent from the
comments in `series.conf` and from the guards(1) manual page. You can generate
this page by running `pod2man` on the `guards` script.


Where to find configuration files
---------------------------------

Kernel configuration files are stored in the kernel Git repository. When packing
up the repository, they end up in `config.tar.bz2`.

The kernel-`$FLAVOR` packages are based on `config/$ARCH/$FLAVOR`, for instance,
kernel-default is based on `config/$ARCH/default`. The kernel-`$FLAVOR` packages
install their configuration files as `/boot/config-$VERSION-$RELEASE-$FLAVOR`.
The config is also packaged in the kernel-`$FLAVOR`-devel package as
`/usr/src/linux-obj/$ARCH/$FLAVOR/.config`.

In addition, the running kernel exposes a gzip compressed version of its
configuration file as `/proc/config.gz`.


How to configure the kernel sources
-----------------------------------

Before a binary kernel is built or an additional loadable module for an existing
kernel is created, the kernel must be configured.

In order for a loadable module to work with an existing kernel, it must be
created with a configuration that is identical to the kernel's configuration, or
at least very close to that. Each configuration is contained in a single file.
The kernel-syms package installs configurations for all standard SUSE kernel
variants, so for building only external kernel modules it is not necessary to
configure the kernel sources.

Configuring the kernel sources for a specific configuration is straightforward:

* Locate the configuration file you want to use, see [Where to find
  configuration files](#where-to-find-configuration-files) above.

* Copy the configuration to the file `.config` in your build directory.

* Run the following commands in sequence to apply the configuration, generate
  version information files, etc.:

      $ make -C /usr/src/linux O=$PWD clean
      $ make -C /usr/src/linux O=$PWD oldconfig

  If the kernel sources do not match the configuration file exactly,
  `make oldconfig` will prompt for settings that are undefined. When no
  `.config` file is initially copied to the target build directory, the command
  automatically uses `/boot/config-$(uname -r)` as the starting configuration.
  Alternatively to `make oldconfig`, you can also use `make menuconfig` for
  a text menu oriented user interface.

  Once this step is completed, a `Makefile` will have been created that
  eliminates the need to specify the locations of the kernel source and the
  build directory.

* Update the configuration appropriately for the target use.

  Configuration files for SUSE kernels include settings to integrate with
  signing support provided by the Open Build Service. When using such a
  configuration file directly, the build might fail due to missing files needed
  for signing kernel modules.

  The minimal steps to enable module signing and have the kernel build
  automatically generate a new key pair are as follows:

      $ ./source/scripts/config --enable CONFIG_MODULE_SIG \
          --enable CONFIG_MODULE_SIG_ALL --undefine CONFIG_MODULE_SIG_KEY
      $ make olddefconfig

  Please refer to the upstream documentation located at
  `/usr/src/linux/Documentation/admin-guide/module-signing.rst` for complete
  information on how to configure module signing.


How to add custom patches
-------------------------

Patches are typically added to the `patches.suse/` directory and an appropriate
place in `series.conf`. When the kernel-source package is exported from the Git
repository, the patch will be automatically added to the matching patch tarball.

If your goal is to create a kernel with only a few additional patches and you
don't want to be bothered with using the Git repository, there is an easier way.

The kernel-source SRPM ships with two empty archives that can be filled and
automatically expanded when building the kernel. You can use these to add your
own patches and config options without disturbing the rest of the kernel
package. This is useful if you are using the openSUSE Build Service and link to
the main kernel-source project instead of creating your own branch. The
advantage to this is that your project will automatically receive all the
changes that go into the main project without any further effort.

To add a patch using this mechanism, just add it to the `patches.addon.tar.bz2`
archive and add an entry to a `series` file inside the archive. The archive will
be expanded automatically after the other kernel patches when the source tree is
constructed.

Some patches may add new Kconfig options. The `config.addon.tar.bz2` archive
contains the same hierarchy as `config.tar.bz2`, but is under `config.addon/`.
You can add your new config options to files named after their `config/`
counterparts.

For example, the file used to configure the x86_64 default kernel is named
`config/x86_64/default`. To add config options to that kernel, you would create
a new file called `config.addon/x86_64/default` with the options as formatted in
a normal Linux kernel `.config` file. This is important because the kernel build
is non-interactive and will fail if it encounters new config options without
entries in the config file.


Module load paths
-----------------

Modules that belong to a specific kernel release are installed in
`/lib/modules/$VERSION-$RELEASE-$FLAVOR/`. Modules from KMPs must be installed
below `/lib/modules/$VERSION-$RELEASE-$FLAVOR/updates/` and similar: modules
below `updates/` have priority over other modules.

When KMPs contain modules that are compatible between multiple installed
kernels, symlinks are used to make those modules available to those compatible
kernels like this:

`/lib/modules/$VERSION-$OTHER_RELEASE-$FLAVOR/weak-updates/foo.ko` →
  `/lib/modules/$VERSION-$RELEASE-$FLAVOR/updates/foo.ko`

Modules in the `weak-updates/` directory have lower priority than modules in
`/lib/modules/$VERSION-$OTHER_RELEASE-$FLAVOR/updates/`, and higher priority
than other modules in `/lib/modules/$VERSION-$OTHER_RELEASE-$FLAVOR/`.


Driver update disks
-------------------

A Driver Update Disk (DUD) is an update archive which makes it possible to use
new device drivers to run installation of an (open)SUSE distribution. It allows
to install the distribution on devices that were not supported at the time the
distribution was created and be able to boot the installed system afterwards
without having to manually install the new device drivers after the
installation.

For information how to create such an update, refer to the mkdud and mksusecd
tools, and their documentation:

* <https://github.com/openSUSE/mkdud>,
* <https://github.com/openSUSE/mksusecd>.


References
----------

General:

* Documentation in the kernel source tree,
* LWN.net (Linux Weekly News), <https://lwn.net/>,
* Kernel newbies, <https://kernelnewbies.org/>.

Loadable kernel modules:

* Peter Jay Salzman, Michael Burian, Ori Pomerantz: The Linux Kernel Module
  Programming Guide, Version 2.6,
  <https://tldp.org/LDP/lkmpg/2.6/html/index.html>.

Kernel module packages:

* Kernel Module Packages Manual, SUSE Linux Enterprise 12 SP2 or later and SUSE
  Linux Enterprise 15,
  <https://documentation.suse.com/sbp/all/html/SBP-KMP-Manual-SLE12SP2/index.html>,
* SUSE SolidDriver Program, <https://drivers.suse.com/doc/SolidDriver/>.