Treat all group*.yml like groups.yml

Fixes #2655

docs/pkglistgen.md

@@ -1,33 +1,92 @@
-# The package list generator
+# Package List Generator
 
-## Sources
+pkglistgen.py is a self contained script to generate and update OBS products for openSUSE and SLE. 
+It works on the products and its staging projects and ports.
 
-The package list generator is contained within `pkglistgen.py` and consists of
+The main input is a package named 000package-groups and it will update the content of other packages
-two phases.
+from that. For that it will read [YAML](https://en.wikipedia.org/wiki/YAML) input from e.g. 000package-groups/groups.yml and generate .group files into 000product. The rest of 000package-groups is copied into 000product as well and it runs the OBS product converter service (See [OBS Documentation](https://en.opensuse.org/openSUSE:Build_Service_product_definition) for details)
+The generated release spec files are split into 000release-packages to avoid needless rebuilds. 
 
-- `update`: updates local package cache
+## Input
-- `solve`: uses libsolv to determine list of packages
 
-The two phases can be run together via the `update_and_solve` subcommand which
+The package list generator reads several files. The most important are group*.yml (tradionally only groups.yml) within 000package-groups.
-also performs necessary initialization and post-processing. By default it will
-run against all _scopes_, but a specific scope such as `rings` or `staging` can
-be specified.
 
-## Input and Output
+### supportstatus.txt
+ TODO
  
-There is one input package and two output packages:
+### group*.yml
+The file is a list of package lists and the special hash 'OUTPUT'. OUTPUT contains an entry for every group file that needs to be written out. The group name of it needs to exist as package list as well. OUTPUT also contains flags for the groups.
 
-- 000package-groups: This is the input. It contains the *.product files, as
+We currently support:
-  well as release package templates (e.g openSUSE-release.spec.in). The special
+ * default-support
-  files groups.yml is read by pkglistgen.py to output group files that are
+ Sets the support level in case there is no explicitly entry in [supportstatus.txt](#supportstatus.txt), defaults to 'unsupported'
-  meant to be included by the product files.
+ * recommends
-- 000product: This is the output container where resulting kiwi files are put
+ If the solver should take recommends into account when solving the package list, defaults to false.
-- 000release-packages: This is the output container for release package spec files.
+ * includes
+ Adds package lists to the group to be solved. Allows to organize different topics into the same group. By default there are no package lists added - the package list with the group name is always there.
+ * excludes
+ Removes all packages from the __solved__ groups listed. Used to build addons to main products.
+ * conflicts
+ Sets package groups not to be part of the same product. Influences the [overlap calculation](#overlap-calculation) only.
 
-## The groups.yml file
+Be aware that group names must not contain a '-'.
 
-The groups.yml is the input for the solver. It contains dictionaries with lists
+You can also adapt the solving on a package level by putting a hash into the package list. Normally the package name is a string, in case it's a hash the key needs to be the package name and the value is a list of following modifiers: 
-of packages. The special key OUTPUT contains a dictionary that lists the group
+
-files to generate.
+ * recommended
+ Evaluate also 'Recommends' in package to determine dependencies. Otherwise only 'required' are considered. Used mainly for patterns in SLE. It can not be combined with platforms, For architecture specific recommends, use patterns.
+ * suggested
+ Evaluate also 'Suggests' in package to determine dependencies. This implies recommended
+ * architecture (e.g. x86_64,s390x,ppc64le,aarch64)
+ Makes the entry specific to the listed architectures. Will get ignored if used in combination with 'recommended'.
+ * locked
+ Do not put the package into this group. Used to *force* certain packages into other modules
+ * silent
+ Use this package for dependency solving of groups "on top", but do not output the package for this group. Mainly to mark the product to use by adding release packages. Use with care, this breaks dependency chains!
+ * required
+ If the package is missing or is uninstallable, don't leave a comment but put the error as package entry for OBS to create unresolvable error to avoid building a DVD
+
+Note that you can write yaml lists in 2 ways. You can put the modifier lists as multiple lines starting with -, but it's recommended to put them as [M1,M2] behind the package name. See the difference between pkg4 and pkg5 in the example. 
+
+#### Example:
+
+```
+OUTPUT:
+  - group1:
+    includes:
+    - list1
+    - list2
+  - group2:
+    default-support: l3
+    recommends: true
+    includes:
+    - list3
+    excludes:
+    - group1
+    conflicts:
+    - group3
+  - group3:
+    includes:
+    - list2
+    
+group1:
+  - pkg1
+  
+group2:
+  - pkg2: [locked]
+  - pkg3
+  
+group3:
+  - pkg4: [x86_64]
+  
+list1:
+  - pkg5:
+    - x86_64
+  
+list2:
+  - pkg6: [recommended]
+``` 
+
+## Overlap calculcation
+ TODO 
 
-FIXME: continue here

pkglistgen.md

@@ -1,92 +0,0 @@
-# Package List Generator
-
-pkglistgen.py is a self contained script to generate and update OBS products for openSUSE and SLE. 
-It works on the products and its staging projects and ports.
-
-The main input is a package named 000package-groups and it will update the content of other packages
-from that. For that it will read [YAML](https://en.wikipedia.org/wiki/YAML) input from e.g. 000package-groups/groups.yml and generate .group files into 000product. The rest of 000package-groups is copied into 000product as well and it runs the OBS product converter service (See [OBS Documentation](https://en.opensuse.org/openSUSE:Build_Service_product_definition) for details)
-The generated release spec files are split into 000release-packages to avoid needless rebuilds. 
-
-## Input
-
-The package list generator reads several files. The most important is groups.yml within 000package-groups
-
-### supportstatus.txt
- TODO
- 
-### groups.yml
-The file is a list of package lists and the special hash 'OUTPUT'. OUTPUT contains an entry for every group file that needs to be written out. The group name of it needs to exist as package list as well. OUTPUT also contains flags for the groups.
-
-We currently support:
- * default-support
- Sets the support level in case there is no explicitly entry in [supportstatus.txt](#supportstatus.txt), defaults to 'unsupported'
- * recommends
- If the solver should take recommends into account when solving the package list, defaults to false.
- * includes
- Adds package lists to the group to be solved. Allows to organize different topics into the same group. By default there are no package lists added - the package list with the group name is always there.
- * excludes
- Removes all packages from the __solved__ groups listed. Used to build addons to main products.
- * conflicts
- Sets package groups not to be part of the same product. Influences the [overlap calculation](#overlap-calculation) only.
-
-Be aware that group names must not contain a '-'.
-
-You can also adapt the solving on a package level by putting a hash into the package list. Normally the package name is a string, in case it's a hash the key needs to be the package name and the value is a list of following modifiers: 
-
- * recommended
- Evaluate also 'Recommends' in package to determine dependencies. Otherwise only 'required' are considered. Used mainly for patterns in SLE. It can not be combined with platforms, For architecture specific recommends, use patterns.
- * suggested
- Evaluate also 'Suggests' in package to determine dependencies. This implies recommended
- * architecture (e.g. x86_64,s390x,ppc64le,aarch64)
- Makes the entry specific to the listed architectures. Will get ignored if used in combination with 'recommended'.
- * locked
- Do not put the package into this group. Used to *force* certain packages into other modules
- * silent
- Use this package for dependency solving of groups "on top", but do not output the package for this group. Mainly to mark the product to use by adding release packages. Use with care, this breaks dependency chains!
- * required
- If the package is missing or is uninstallable, don't leave a comment but put the error as package entry for OBS to create unresolvable error to avoid building a DVD
-
-Note that you can write yaml lists in 2 ways. You can put the modifier lists as multiple lines starting with -, but it's recommended to put them as [M1,M2] behind the package name. See the difference between pkg4 and pkg5 in the example. 
-
-#### Example:
-
-```
-OUTPUT:
-  - group1:
-    includes:
-    - list1
-    - list2
-  - group2:
-    default-support: l3
-    recommends: true
-    includes:
-    - list3
-    excludes:
-    - group1
-    conflicts:
-    - group3
-  - group3:
-    includes:
-    - list2
-    
-group1:
-  - pkg1
-  
-group2:
-  - pkg2: [locked]
-  - pkg3
-  
-group3:
-  - pkg4: [x86_64]
-  
-list1:
-  - pkg5:
-    - x86_64
-  
-list2:
-  - pkg6: [recommended]
-``` 
-
-## Overlap calculcation
- TODO 
-

pkglistgen/file_utils.py

@@ -32,7 +32,7 @@ def unlink_all_except(path, ignore_list=['_service'], ignore_hidden=True):
 def copy_directory_contents(source, destination, ignore_list=[]):
     for name in os.listdir(source):
         name_path = os.path.join(source, name)
-        if name in ignore_list or not os.path.isfile(name_path):
+        if name in ignore_list or name_path in ignore_list or not os.path.isfile(name_path):
             continue
 
         shutil.copy(name_path, os.path.join(destination, name))

pkglistgen/tool.py

@@ -96,8 +96,11 @@ class PkgListGen(ToolBase.ToolBase):
                 g.parse_yml(group)
         return output, unwanted
 
+    def group_input_files(self):
+        return glob.glob(os.path.join(self.input_dir, 'group*.yml'))
+
     def load_all_groups(self):
-        for fn in glob.glob(os.path.join(self.input_dir, 'group*.yml')):
+        for fn in self.group_input_files():
             o, u = self._load_group_file(fn)
             if o:
                 if self.output is not None:
@@ -646,6 +649,9 @@ class PkgListGen(ToolBase.ToolBase):
         release_dir = os.path.join(cache_dir, release)
         oldrepos_dir = os.path.join(cache_dir, oldrepos)
 
+        self.input_dir = group_dir
+        self.output_dir = product_dir
+
         for package in checkout_list:
             if no_checkout:
                 print('Skipping checkout of {}/{}'.format(project, package))
@@ -656,16 +662,12 @@ class PkgListGen(ToolBase.ToolBase):
         file_utils.unlink_all_except(release_dir, ['weakremovers.inc'])
         if not only_release_packages:
             file_utils.unlink_all_except(product_dir)
-        file_utils.copy_directory_contents(group_dir, product_dir,
+        ignore_list = ['supportstatus.txt', 'reference-unsorted.yml', 'reference-summary.yml', 'package-groups.changes']
-                                           ['supportstatus.txt', 'groups.yml',
+        ignore_list += self.group_input_files()
-                                            'reference-unsorted.yml', 'reference-summary.yml',
+        file_utils.copy_directory_contents(group_dir, product_dir, ignore_list)
-                                            'package-groups.changes'])
         file_utils.change_extension(product_dir, '.spec.in', '.spec')
         file_utils.change_extension(product_dir, '.product.in', '.product')
 
-        self.input_dir = group_dir
-        self.output_dir = product_dir
-
         print('-> do_update')
         # make sure we only calculcate existant architectures
         self.filter_architectures(target_archs(api.apiurl, project, main_repo))