Add documentation for osc-batch-submit

Improve the wording of a message
Add mkdir command in the generated shell script to create the logs directory
2025-05-21 17:47:27 +02:00 · 2025-05-21 17:46:36 +02:00 · 2025-05-21 17:46:36 +02:00 · 2025-05-21 17:46:36 +02:00 · 2025-05-21 17:46:36 +02:00 · 2025-05-21 17:46:36 +02:00
9 changed files with 259 additions and 37 deletions
--- a/README.md
+++ b/README.md
@@ -1,3 +1,197 @@
 # osc-batch-submit

-Tool to batch submit requests with osc
+Tool to batch submit requests with osc. This helps when submitting large software stacks from one OBS project to another.
+For example, KDE Frameworks + Plasma + Applications consist on ~500 packages and Qt6 has ~30 packages.
+
+# Installation
+
+The easiest is to install the package from OBS in https://build.opensuse.org/package/show/home:alarrosa:packages/python-osc-batch-submit .
+
+# Basic Usage
+
+The basic usage consists in running osc-batch-submit with a command and configuration file as parameters. Something like:
+
+```
+osc-batch-submit submitrequest gstreamer-from-Factory-to-SLFO.conf
+```
+
+or
+
+```
+osc-batch-submit copypac qt6-from-Factory-to-home-prj.conf
+```
+
+That creates a shell script file with all the osc commands you need.
+
+You should then review the commands (specially if osc-batch-submit tells you to check some message logs, more on this below) and once they're ok, just run this script.
+
+# Reviewing the shell script
+
+The shell script will contain a list of osc commands with appropriate parameters and automatically generated message logs . This is easy when, for example, a SR just updates
+a package with a newer version, or adds a patch. But there are situations in which osc-batch-submit will warn you that it's better for someone to review a message log if,
+for example, it detects a patch being removed. In this case, osc-batch-submit will tell you to check the message log files to better explain what's being submitted (and
+more importantly, to check if something may be being submitted by mistake).
+
+In some circumstances, while reviewing the shell script you might notice a package will be submitted that you don't really want to submit, in this case, there's no need to rerun osc-batch-submit. Just comment (or remove) the related osc command from the script.
+
+Note that before every osc command, the shell script contains a command to print some internal information. This is done this way because when you run the script, all the script output is also written to a logfile and these "headers" are used to identify which osc command output refer to each package. This allows to later use the `list-srs` command to check the status of the SRs generated.
+
+# General Usage
+
+# The configuration file
+
+The configuration file sets a series of variables that define how osc-batch-submit works: the packages that will be submitted (or copied), from which project(s) and to which project among other data. This configuration file is written in python format as it's loaded as if it was a python file.
+
+### ibs
+
+The first variable that needs to be defined is an `ibs` boolean that defines if the osc commands will be run on build.opensuse.org (if False) or build.suse.de (if True).
+
+### packages
+
+The `packages` variable is a string that defines a set expression that, when evaluated, returns the set of the packages that osc-batch-submit will operate on. The set expression can use unions (`∪`), interesections (`∩`) and substractions (`∖`) on ibs/obs projects or lists of packages (package names written between curly brackets). It's also possible to filter packages with a regular expression by using ` * {pattern}`. Let's see some examples:
+
+```
+packages = '(KDE:Applications ∪ KDE:Extra) ∩ openSUSE:Factory'
+```
+Selects all packages in KDE:Applications or KDE:Extra that are already in openSUSE:Factory
+
+```
+packages = '(openSUSE:Backports:SLE-15-SP4 ∩ KDE:Frameworks5 ∩ openSUSE:Factory) ∖ SUSE:SLE-15-SP5:GA'
+```
+  Selects all packages that are in openSUSE:Backports:SLE-15-SP4 that also are in KDE:Frameworks5 and openSUSE:Factory, except if they are in SUSE:SLE-15-SP5:GA
+```
+packages = '{libqt5-qt3d, libqt5-qtcharts , libqt5-qtconnectivity, libqt5-qtdatavis3d, libqt5-qtdoc}'
+```
+Selects that list of packages
+
+```
+packages = '((openSUSE.org:multimedia:libs * {.*gst.*}) ∩ SUSE:SLFO:Main) ∪ {pipewire, wireplumber}'
+```
+ Selects all packages in multimedia:libs (from obs) that have `gst` in their name if they are also in SUSE:SLFO:Main and adds `pipewire` and `wireplumber` to the list. Note that since this expression uses projects from obs and ibs, this assumes ibs was set to True.
+
+
+### source_projects
+
+The `source_projects` variable is a list of strings containing the source projects for the packages evaluated from the `packages` expression. For each package, the first project will be tested to be used as source, if it doesn't contain the package, the second project will be used. If it doesn't contain the package, the third project will be used, and so on. Example:
+
+```
+source_projects = ['openSUSE.org:openSUSE:Factory']
+```
+Uses openSUSE:Factory as source (assuming ibs is True)
+
+```
+source_projects = ['KDE:Frameworks', 'KDE:Applications', 'KDE:Extra']
+```
+Use KDE:Frameworks, KDE:Applications and KDE:Extra as source project.
+
+### target_project
+
+The `target_project` variable is the project used as target for the copypacs/submitrequests commands.
+
+### only_packages_developed_in_project
+
+This variable is used to filter the packages list and keep only the packages that are developed in a given project. This variable can be `None` or a project. 
+
+```
+only_packages_developed_in_project = None
+```
+Don't use this variable.
+
+```
+only_packages_developed_in_project = 'KDE:Frameworks5'
+```
+Only keep packages whose devel project is KDE:Frameworks.
+
+### reference
+
+This can be used to specify a jira ticket, bugzilla ID or any other reference that should be added to every message text in submit requests or copypac commands.
+
+```
+reference = 'jsc#PED-2635'
+```
+Adds a reference to that jira ID
+
+```
+reference = ''
+```
+Don't add any reference.
+
+### set_bugowner
+
+When submitting a new package to a project, you sometimes need to set a bugowner, this variable can be used to set this. The variable contains the username of the bugowner that should be set.
+
+```
+set_bugowner = False
+```
+Do not set a bugowner (an empty string can also be used)
+
+```
+set_bugowner = 'geeko'
+```
+Set the geeko obs userid as bugowner
+
+```
+set_bugowner = 'group:gnome-maintainers'
+```
+Set the gnome-maintainers group as bugowner
+
+### force_write_message_file
+
+This is a boolean that can be used to force to write a message file that can be edited instead of just using a generated text.
+
+### previous_logfile
+
+This variable contains the name of a logfile generated in a previous run of osc-batch-submit (actually, by the generated shell script) to allow osc-batch-submit to supersede previous SRs.
+Imagine that you submit all of GNOME to Factory and while the SRs are being reviewed you notice there's a new version in the source project that you want to submit, superseding all previous SRs.
+This would be quite tedious to do manually, so you can just set `previous_logfile` to the name of the logfile and the new shell script generated by osc-batch-submit will generate `osc sr` commands
+including the `-s superseded_SR` parameter to supersede old SRs.
+
+### use_cache
+
+osc-batch-submit always write all information it gets from obs to a cache, but doesn't use the contents in this cache unless the use_cache variable is set in the config file or the --cache parameter is passed on the command line. This is done this way because packages might have been added, removed or modified in obs and using the cache implicitly would give unexpected results so it's better to only use it when explicitly requested (like when you re-run osc-batch-submit just after changing something in the configuration file).
+
+# list-srs
+
+After running the shell script generated by osc-batch-submit, it will generate a logfile under a logs directory containing the result of the run. You can use the `list-srs` command to check the SRs generated and the status of them. Some examples:
+
+```
+> list-srs gstreamer-from-Factory-to-SLFO.log
+376234 376235 376236 376237 376238 376239 376240 376241 376242 376243
+```
+
+```
+> list-srs -v gstreamer-from-Factory-to-SLFO.log
+376234 gstreamer
+376235 gstreamer-devtools
+376236 gstreamer-docs
+376237 gstreamer-plugins-bad
+376238 gstreamer-plugins-base
+376239 gstreamer-plugins-good
+376240 gstreamer-plugins-libav
+376241 gstreamer-plugins-rs
+376242 gstreamer-plugins-ugly
+376243 gstreamer-rtsp-server
+```
+
+```
+> list-srs -vv gstreamer-from-Factory-to-SLFO.log
+https://build.suse.de/request/show/376234 gstreamer
+https://build.suse.de/request/show/376235 gstreamer-devtools
+https://build.suse.de/request/show/376236 gstreamer-docs
+https://build.suse.de/request/show/376237 gstreamer-plugins-bad
+https://build.suse.de/request/show/376238 gstreamer-plugins-base
+https://build.suse.de/request/show/376239 gstreamer-plugins-good
+https://build.suse.de/request/show/376240 gstreamer-plugins-libav
+https://build.suse.de/request/show/376241 gstreamer-plugins-rs
+https://build.suse.de/request/show/376242 gstreamer-plugins-ugly
+https://build.suse.de/request/show/376243 gstreamer-rtsp-server
+```
+
+```
+> list-srs -s qt6-6.8.2-from-KDE_Qt6-to-SLFO.log
+363017 superseded :  python3-pyside6
+363018 accepted : Auto accept qt6-3d
+363019 accepted : Auto accept qt6-base
+363020 accepted : Auto accept qt6-charts
+...
+```
--- a/osc_batch_submit/examples/ffmpeg-7-from-obs-to-SLFO.conf
+++ b/osc_batch_submit/examples/ffmpeg-7-from-obs-to-SLFO.conf
@@ -2,7 +2,7 @@ ibs = True

 use_cache = False

-packages_to_copy_expr = '{ffmpeg-7, geany, glfw, libdovi, libplacebo, librist, python-glad2}'
+packages = '{ffmpeg-7, geany, glfw, libdovi, libplacebo, librist, python-glad2}'

 source_projects = ['openSUSE.org:multimedia:libs', 'openSUSE.org:games', 'openSUSE.org:openSUSE:Factory']
               
@@ -16,4 +16,4 @@ previous_logfile = ''

 set_bugowner = 'group:gnome-maintainers'

-force_recheck_bugowner = False
+force_write_message_file = False
--- a/osc_batch_submit/examples/gstreamer-from-Factory-to-SLFO.conf
+++ b/osc_batch_submit/examples/gstreamer-from-Factory-to-SLFO.conf
@@ -2,7 +2,7 @@ ibs = True

 use_cache = False

-packages_to_copy_expr = '((openSUSE.org:multimedia:libs * {.*gst.*}) ∩ SUSE:SLFO:Main)'
+packages = '((openSUSE.org:multimedia:libs * {.*gst.*}) ∩ SUSE:SLFO:Main)'

 source_projects = ['openSUSE.org:multimedia:libs']
               
@@ -12,4 +12,4 @@ only_packages_developed_in_project = None

 set_bugowner = False

-force_recheck_bugowner = False
+force_write_message_file = False
--- a/osc_batch_submit/examples/gstreamer-from-Factory-to-home-SLE-15-SP6.conf
+++ b/osc_batch_submit/examples/gstreamer-from-Factory-to-home-SLE-15-SP6.conf
@@ -2,7 +2,7 @@ ibs = True

 use_cache = False

-packages_to_copy_expr = '((openSUSE.org:multimedia:libs * {.*gst.*}) ∩ SUSE:SLFO:Main) ∪ {pipewire, wireplumber}'
+packages = '((openSUSE.org:multimedia:libs * {.*gst.*}) ∩ SUSE:SLFO:Main) ∪ {pipewire, wireplumber}'

 source_projects = ['openSUSE.org:multimedia:libs']
               
--- a/osc_batch_submit/examples/kf6-to-Leap16.conf
+++ b/osc_batch_submit/examples/kf6-to-Leap16.conf
@@ -2,7 +2,7 @@ ibs = False

 use_cache = True

-packages_to_copy_expr = 'openSUSE:Factory ∩ KDE:Frameworks'
+packages = 'openSUSE:Factory ∩ KDE:Frameworks'

 source_projects = ['KDE:Frameworks']
               
@@ -16,4 +16,4 @@ previous_logfile = ''

 set_bugowner = False

-force_recheck_bugowner = False
+force_write_message_file = False
--- a/osc_batch_submit/examples/latest-plasma-and-kde-gear-to-Leap16.conf
+++ b/osc_batch_submit/examples/latest-plasma-and-kde-gear-to-Leap16.conf
@@ -2,7 +2,7 @@ ibs = False

 use_cache = True

-packages_to_copy_expr = 'openSUSE:Factory ∩ (KDE:Frameworks ∪ KDE:Applications ∪ KDE:Extra)'
+packages = 'openSUSE:Factory ∩ (KDE:Frameworks ∪ KDE:Applications ∪ KDE:Extra)'

 source_projects = ['KDE:Frameworks', 'KDE:Applications', 'KDE:Extra']
               
@@ -16,4 +16,4 @@ previous_logfile = ''

 set_bugowner = False

-force_recheck_bugowner = False
+force_write_message_file = False
--- a/osc_batch_submit/examples/plasma-to-Backports-SLE15-SP6.conf
+++ b/osc_batch_submit/examples/plasma-to-Backports-SLE15-SP6.conf
@@ -2,7 +2,7 @@ ibs = False

 use_cache = True

-packages_to_copy_expr = '(openSUSE:Backports:SLE-15-SP6 ∪ openSUSE:Leap:15.6) ∩ (KDE:Frameworks5)'
+packages = '(openSUSE:Backports:SLE-15-SP6 ∪ openSUSE:Leap:15.6) ∩ (KDE:Frameworks5)'

 source_projects = ['KDE:Frameworks5']
               
--- a/osc_batch_submit/main.py
+++ b/osc_batch_submit/main.py
@@ -33,14 +33,14 @@ class OSCBatchSubmit:
        # Default values
        self.use_cache = False
        self.ibs = False
-        self.packages_to_copy_expr = ''
+        self.packages = ''
        self.source_projects = []
        self.target_project = ''
        self.only_packages_developed_in_project = None
        self.reference = None  # (like a jira or bugzilla refence)
        self.previous_logfile = None  # To be able to supersede SRs
        self.set_bugowner = False
-        self.force_recheck_bugowner = False
+        self.force_write_message_file = False

        self.config_file = None
        self.verbose = False
@@ -60,17 +60,20 @@ copypac
 list-packages
 list-bugowners-in-src
 list-bugowners-in-tgt
+remove-cache
 ''')

        parser = sps.add_parser('submitrequest', aliases=['sr'],
                                description='create submit requests')
        parser.add_argument('config_file')
        parser.add_argument('--verbose', '-v', action='store_true')
+        parser.add_argument('--cache', action='store_true')
        parser.add_argument('--progress', action='store_true')

        parser = sps.add_parser('copypac', description='copy packages')
        parser.add_argument('config_file')
        parser.add_argument('--verbose', '-v', action='store_true')
+        parser.add_argument('--cache', action='store_true')
        parser.add_argument('--progress', action='store_true')

        parser = sps.add_parser('list-packages',
@@ -88,10 +91,14 @@ list-bugowners-in-tgt
                                'target projects')
        parser.add_argument('config_file')

+        sps.add_parser('remove-cache',
+                       description='Remove cache contents')
+
        options = main_parser.parse_args()

        self.command = options.command
-        if getattr(options, 'config_file'):
+
+        if getattr(options, 'config_file', None):
            if not options.config_file.endswith('.conf'):
                print(f'Please rename {options.config_file} to end in .conf')
                sys.exit(1)
@@ -100,12 +107,15 @@ list-bugowners-in-tgt
            self.base_name = self.config_file.removesuffix('.conf')
            self.load_data_from_config_file()

-        if getattr(options, 'verbose'):
+        if getattr(options, 'verbose', None):
            self.verbose = options.verbose

-        if getattr(options, 'progress'):
+        if getattr(options, 'progress', None):
            self.print_progress = options.progress

+        if getattr(options, 'cache', None):
+            self.use_cache = options.cache
+
    def load_data_from_config_file(self):
        spec = spec_from_loader(self.base_name,
                                SourceFileLoader(self.base_name,
@@ -117,7 +127,13 @@ list-bugowners-in-tgt
            setattr(self, var, getattr(data, var))

    def run(self):
-        print(f'Package list expression: {self.packages_to_copy_expr}')
+        osc = OSC(self.use_cache, ibs=self.ibs)
+
+        if self.command == 'remove-cache':
+            osc.remove_cache_files()
+            return
+
+        print(f'Package list expression: {self.packages}')
        print(f'From: {self.source_projects}')
        print(f'To: {self.target_project}')

@@ -133,11 +149,9 @@ list-bugowners-in-tgt
            idx += 1
            new_logfile = f'logs/{self.base_name}-{idx}.log'

-        osc = OSC(self.use_cache, ibs=self.ibs)
-
        print('Obtaining the list of packages...')
        osc.set_params(self.source_projects, self.target_project,
-                       self.packages_to_copy_expr)
+                       self.packages)

        packagesToCopy = osc.packages_to_copy
        if self.ibs:
@@ -206,6 +220,7 @@ list-bugowners-in-tgt
        output.write('#!/bin/sh')
        os.chmod(commands_file, 0o764)
        output.write(f'logfile="{new_logfile}"')
+        output.write(f'mkdir -p "{os.path.dirname(new_logfile)}"')

        msg_log_files = []

@@ -266,7 +281,7 @@ list-bugowners-in-tgt
                                                 package, srcPrj, package)
                if not diff_log:
                    output.write(f'# {package} is the same version and exactly'
-                                 f'the same in {srcPrj} and '
+                                 f' the same in {srcPrj} and '
                                 f'{self.target_project} ({srcVersion})')
                    continue
            elif (srcVersion and tgtVersion and versions_can_be_parsed and
@@ -329,10 +344,9 @@ list-bugowners-in-tgt
                              'that should be the bugowner')
                        sys.exit(1)

-                write_message_file = self.force_recheck_bugowner
                write_diff = False

-            if write_message_file:
+            if write_message_file or self.force_write_message_file:
                filename = (f'log-messages/{srcPrj}__{self.target_project}'
                            f'__{package}.txt')
                separator = '\n' + '-' * 72 + '\n'
@@ -372,7 +386,7 @@ list-bugowners-in-tgt

        if msg_log_files:
            output.write('')
-            output.write('# Check following msg log files:')
+            output.write('# Check the following message log files:')
            output.write(f'{" ".join(msg_log_files)}')

        output.write('')
--- a/osc_batch_submit/osc.py
+++ b/osc_batch_submit/osc.py
@@ -7,6 +7,7 @@ import subprocess
 import json
 import re
 import os
+import glob
 from osc_batch_submit.progress import Progress

 # project_with_packages_to_copy = '(((SUSE:SLE-15-SP4:GA ∪ SUSE:SLE-15-SP3:Update ∪ SUSE:SLE-15-SP3:GA ∪ SUSE:SLE-15-SP2:Update ∪ SUSE:SLE-15-SP2:GA) ∩ openSUSE.org:KDE:Frameworks5) ∪ home:alarrosa:branches:SUSE:SLE-15-SP4:GA:kf5) ∖ SUSE:SLE-15-SP4:GA:Staging:B'  # noqa
@@ -142,18 +143,19 @@ class OSC:
        else:
            return self.packagesIn[prj_set_expr]

-    def cache_path_for(self, cache_type, project):
+    def cache_path_for(self, cache_type, project, create_directory=True):
        if not self.cache_dirpath:
            cache_path = os.getenv('XDG_CACHE_HOME')
            if not cache_path:
                cache_path = os.path.expanduser('~/.cache')
-            self.cache_dirpath = f'{cache_path}/osc_batch_tool'
-            try:
-                os.mkdir(self.cache_dirpath)
-            except FileExistsError:
-                pass
+            self.cache_dirpath = f'{cache_path}/osc-batch-submit'
+            if create_directory:
+                try:
+                    os.mkdir(self.cache_dirpath)
+                except FileExistsError:
+                    pass

-        return f'{self.cache_dirpath}/{cache_type}-{project}'
+        return f'{self.cache_dirpath}/{cache_type}{"-" if cache_type and project else ""}{project}'

    def get_project_packages(self, project):
        filename = self.cache_path_for('packages-in', project)
@@ -259,8 +261,12 @@ class OSC:
                          f'/source/{project}/{package}?view=info&parse=1']
        proc = subprocess.run(cmd, capture_output=True)
        if proc.returncode != 0:
+            stderr = proc.stderr.decode('utf-8', errors='replace')
+            if (proc.returncode == 1 and
+                    'HTTP Error 400: download in progress' in stderr):
+                return 'unknown'
            raise RuntimeError('osc command returned an error when getting the'
-                               f'version of {project}/{package}:\n' +
+                               f' version of {project}/{package}:\n' +
                               proc.stderr.decode('utf-8', errors='replace'))
        root = ET.fromstring(proc.stdout)
        try:
@@ -286,7 +292,7 @@ class OSC:
        proc = subprocess.run(cmd, capture_output=True)
        if proc.returncode != 0:
            raise RuntimeError('osc command returned an error when getting the'
-                               f'state of request {requestid}:\n' +
+                               f' state of request {requestid}:\n' +
                               proc.stderr.decode('utf-8', errors='replace'))
        root = ET.fromstring(proc.stdout)
        try:
@@ -445,16 +451,16 @@ class OSC:
        return versions

    def set_params(self, source_projects, target_project,
-                   packages_to_copy_expression):
+                   packages_expression):
        self.source_projects = source_projects
        self.target_project = target_project
-        self.packages_to_copy_expression = packages_to_copy_expression
+        self.packages_expression = packages_expression
        self.projects_inherited_from_tgt = \
            self.get_all_projects_inherited_from_project(
                target_project, repository_name='standard')

        tmp_prjs = self.get_all_projects_from_projects_set_expression(
-                packages_to_copy_expression)
+                packages_expression)

        self.log('Projects inherited from tgt')
        self.log(self.projects_inherited_from_tgt)
@@ -465,7 +471,7 @@ class OSC:

        self.packages_to_copy = \
            self.get_packages_from_projects_in_set_expression(
-                packages_to_copy_expression)
+                packages_expression)

    def add_packages_to_packages_to_copy_list(self, packages):
        self.packages_to_copy.extend(p for p in packages
@@ -641,3 +647,11 @@ class OSC:
                               f'{tgtPrj}/{tgtPackage}:\n' +
                               proc.stderr.decode('utf-8', errors='replace'))
        return proc.stdout.decode('utf-8', errors='replace').strip('\n')
+
+    def remove_cache_files(self):
+        dirpath = self.cache_path_for('', '')
+        cache_types = ['packages-in', 'meta-prj', 'versions-in', 'devel-projects']
+        for cache_type in cache_types:
+            path = f'{dirpath}/{cache_type}-*'
+            for filename in glob.glob(path):
+                os.unlink(filename)
Author	SHA256	Message	Date
Antonio Larrosa	78421b6d7e	Add documentation for osc-batch-submit	2025-05-21 17:47:27 +02:00
Antonio Larrosa	792e9a6b20	Improve the wording of a message	2025-05-21 17:46:36 +02:00
Antonio Larrosa	766e22205e	Add mkdir command in the generated shell script to create the logs directory	2025-05-21 17:46:36 +02:00
Antonio Larrosa	c39e324883	Add remove-cache command	2025-05-21 17:46:36 +02:00
Antonio Larrosa	b8706b551f	Allow using commands that don't have a config_file argument (or others)	2025-05-21 17:46:36 +02:00
Antonio Larrosa	34763a5add	Add --cache command line parameter This makes osc-batch-submit read information from the cache if available instead of connecting to obs.	2025-05-21 17:46:36 +02:00
Antonio Larrosa	ded1d962de	Rename packages_to_copy_expr and force_recheck_bugowner variables Rename packages_to_copy_expr and force_recheck_bugowner variables to packages and force_write_message_file which are clearer and more readable.	2025-05-21 17:46:36 +02:00
Antonio Larrosa	9fec0ef7ed	Return 'unknown' when getting the version while obs is updating it	2025-05-21 17:46:36 +02:00
Antonio Larrosa	3210fd9fa3	Fix missing spaces in stringa	2025-05-21 17:46:36 +02:00