127 Commits

Author SHA1 Message Date
Stephan Kulow
fb4d63ec3d Fix overlap calculation: modules only have filtered archs 2018-10-10 06:45:14 +02:00
Stephan Kulow
a96e864c06 pkglistgen: Make sure we don't calculate non-existant architectures in stagings
Otherwise the package list is not staging specific
2018-10-04 07:00:56 +02:00
Stephan Kulow
c4fa49aef2 Refactor conditional_PUT as core function
Introduce source_file_ensure in osclib.core following review in #1717
2018-10-03 07:36:42 +02:00
Stephan Kulow
2d4148f3f2 pkglistgen: only PUT new data if it changed
Otherwise we collect quickly a million revisions with no gain
2018-10-02 06:48:58 +02:00
Stephan Kulow
f3961e4857 pkglistgen: Only check repository check per arch
(also indent 2 lines due to calling autopep)
2018-09-28 08:05:15 +02:00
Stephan Kulow
5a1307b98b Move creation of the Solver object out of loop
After time for sle_module_basesystem: before 16.5s, after 9.7s
2018-09-27 07:07:05 +02:00
Stephan Kulow
ea0e6d4aaa Don't take unsorted.group as reference for SP1 product changes
The release managers are actually supposed to maintain the filter list
in unneeded.yml, but that only applies to unsorted.yml not to the unsorted.group
(which is basically unfiltered to be published into OBS's SLE-15)
2018-09-24 15:33:10 +02:00
Jimmy Berry
a074572c45 pkglistgen: compare repository state to determine when to update solv.
Since both pkglistgen and repo_checker share the same rpm metadata cache
either may update it first and thus bs_mirrorfull will only contain output
for one of the tools. As such the check in pkglistgen could cause the solv
file to remain stale. Instead write a secondary file containing the
repository state hash and check to see if the file already exists.
2018-09-20 16:38:05 -05:00
Jimmy Berry
c47bb4c902 Utilize CacheManager for all existing caches. 2018-09-04 15:19:02 -05:00
Jimmy Berry
6069245350 Remove SUSE copyright, warranty, and license headers.
Distinct copyrights were left as I do not wish to track down commit
history to ensure it properly documents the copyright holders. Also left
non-GPLv2 licenses and left bs_copy untouched as a mirror from OBS.

Already have a mix of with and without headers and even OBS does not place
on majority of files. If SUSE lawyers have an issue it will come up in
legal review for Factory.
2018-08-23 19:18:06 -05:00
Jimmy Berry
36a5a6c2e4 pkglistgen: utilize osclib.core.repository_path_expand(). 2018-08-21 20:49:31 -05:00
Jimmy Berry
760ddf39e6 osclib/conf: decouple from StagingAPI and always fetch remote config.
As the remote config is no longer optional for SLE and is utilized by
openSUSE to the point were it is dangerous not to load the remote config
it should be required. Currently only certain users call apply_remote()
while this will make it built-in during construction and thus makes the
usage consistent and no longer require StagingAPI.
2018-08-16 21:53:40 -05:00
8129bef009 pkglistgen: fix calling expand_repos() in CommandLineInterface 2018-08-03 16:21:16 +08:00
ec019c42ce pkglistgen: respect main_repo setup for other variant
Respect main_repo setup for other variant eg. ARM. Also removed ports
from SCOPES.
2018-08-03 16:21:16 +08:00
Stephan Kulow
af1252ef3e pkglistgen: Support passing multiple scopes
For SLE I need -s target -s staging, but it only did the later - as
I didn't figure the concept of commandline :)
2018-08-01 15:46:50 +02:00
Stephan Kulow
fb114a75d4 pkglistgen: Commit a summary.yml into 000product-summary if existant 2018-07-19 21:55:52 +02:00
Stephan Kulow
57fe0871dc pkglist: Review the proper package (000release-packages) 2018-07-19 21:55:52 +02:00
Stephan Kulow
9e23eab573 pkglistgen: allow to pass staging:a as scope 2018-07-19 21:55:50 +02:00
Stephan Kulow
715d94c01a pkglistgen: remove default-support-status option
The OUTPUT groups need to specify the status now or it will default to
unsupported (which should still be fine for leap/factory)
2018-07-19 21:55:14 +02:00
Stephan Kulow
9951b454fb
Merge pull request #1620 from coolo/fix_repo_checker
repo_checker: Fix the review loop
2018-07-19 21:54:39 +02:00
Stephan Kulow
4f275a5af8 Move expanded_repos into staging api
and make pkglistgen and repo_checker use it. We will have more
cases in the future of tools requiring it :)
2018-07-19 09:14:30 +02:00
4f58794f8c Remove debug code accidentally part of #1615
Apparently accidentally did a commit --amend --all on last commit,
revert.
2018-07-17 10:03:00 +02:00
d113e01bbb Add Leap 15.1 ARM support 2018-07-16 15:51:47 +02:00
Stephan Kulow
29d6152fa4 pkglistgen: Don't crash on empty groups (unsorted is one) 2018-07-03 15:35:44 +02:00
Stephan Kulow
91a8b5dfa6 pkglistgen: Add --stop-after-solve option to ease debugging 2018-07-03 15:35:44 +02:00
Stephan Kulow
c1177170d5 pkglistgen: Add --no-checkout option to accelerate reruns 2018-07-03 15:35:44 +02:00
Stephan Kulow
121cec768b pkglistgen: Calculate the repos list instead of hardcoding 2018-07-03 15:35:44 +02:00
Stephan Kulow
3efedd4096 pkglistgen: drop ring 2 2018-07-03 14:10:10 +02:00
Stephan Kulow
03ea996ba4 pkglistgen: Don't warn about locked packages not found
We sometimes have to lock packages that only exist on one arch - and
that makes debug output unreadable. So be careful with locked packages
instead :)
2018-06-19 07:21:52 +02:00
Stephan Kulow
f8ff45896a pkglistgen: Copy away inc files too 2018-06-19 07:21:52 +02:00
Stephan Kulow
e452271e5e pkglistgen: Output more infos about overlap 2018-06-19 07:21:52 +02:00
Stephan Kulow
a68be43d04 Generate an unsorted.group if configured
We need this to have the remains of SLE15 in OBS to build against
2018-06-15 17:13:10 +02:00
Ludwig Nussel
a3f320c8be
Merge pull request #1540 from lnussel/pkglistgen
pkglistgen: log unsorted locale packages
2018-06-11 09:56:29 +02:00
086bd38980 Skip nonfree for ports scope
We don't actually have any nonfree items for ports, this is a x86_64
specific thing. so skip it for ports - makes no sense only eats time.
2018-05-15 11:58:39 +02:00
Ludwig Nussel
5311708270 pkglistgen: log unsorted locale packages
For some yet to be determined reason, the namespaceprovider does not
pick up package that just provide eg. locale(ja). So find and and log
them (boo#1092055)
2018-05-14 15:21:46 +02:00
Jimmy Berry
547703d92a pkglistgen: do_update_and_solve(): print exception and continue for all projects. 2018-04-12 00:08:43 -05:00
Ludwig Nussel
5d0c024ffe
Merge pull request #1453 from coolo/add_sle_droplist
Add create_sle_weakremovers command
2018-03-23 11:21:43 +01:00
Jimmy Berry
9a26912ab5
Merge pull request #1442 from jberry-suse/pkglistgen-update-hash
pkglistgen: do_dump_solv(): include hash in update solv to avoid download.
2018-03-12 15:55:44 -05:00
Stephan Kulow
78b107e6e5 Add create_sle_weakremovers command
It's similiar but not the same as the droplist for openSUSE, so I created
a fork of the openSUSE command

To be called with the target project as first argument and then a
list of old projects

e.g. create_sle_weakremovers SUSE:SLE-15:GA SUSE:SLE-12-SP3:GA \
SUSE:SLE-12-SP3:Update SUSE:SLE-12-SP2:GA SUSE:SLE-12-SP2:Update ...
2018-03-12 08:20:42 +01:00
Ludwig Nussel
ad6056258f
Merge pull request #1447 from lnussel/pkglistgen
pkglistgen: Fix reading boolean values
2018-03-11 09:22:51 +01:00
Jimmy Berry
f9b7832e8b pkglistgen: do_dump_solv(): include hash in update solv to avoid download. 2018-03-09 11:32:27 -06:00
Jimmy Berry
28b467f4e8 pkglistgen: do_dump_solv(): utilize os.path.join(). 2018-03-09 11:32:27 -06:00
Ludwig Nussel
0f5d25f463 pkglistgen: Fix reading boolean values 2018-03-09 13:50:08 +01:00
Stephan Kulow
fdef876580 Treat stagings differently
This is specific to SLE stagings, but won't hurt in openSUSE stagings
as the products there are hand made anyway
2018-03-09 11:56:21 +01:00
Ludwig Nussel
7484722b2a
Merge pull request #1439 from jberry-suse/pkglistgen-dump_solv-hash-and-flush
pkglistgen: do_dump_solv: flush after write before reading back and verify primary repo gziped content sha256.
2018-03-09 11:00:05 +01:00
9370efc8e4 pkglistgen: Do not run local service check when committing to release package
That check is aiming to check request rather than package, for release package
we indeed had multispecs and multibuild file together, just do not run
local service when committing to release package.
2018-03-09 16:45:37 +08:00
Jimmy Berry
1968a594f7 pkglistgen: do_dump_solv: flush after write before reading back.
Ended up consistently creating a short XML file that created a null section
in solv file that cause pkglistgen drop list generator to create a <NULL>
drop entry.

Always flush() before you read(). :(

It certainly would have been nice if libsolv would have complained about
an invalid XML file rather than just producing garbage as the file just
ended. Then of course mergesolv did not complain and comes back in to
python as '<NULL>'. Failure to validate and multiple levels.
2018-03-08 12:35:36 -06:00
Jimmy Berry
e353d513a6 pkglistgen: do_dump_solv: verify primary repo gziped content sha256. 2018-03-08 12:35:30 -06:00
Jimmy Berry
a5775e6e54 pkglistgen: do_dump_solv: use BytesIo instead of StringIO.
request.content is not a string like request.text.
2018-03-08 12:22:02 -06:00
Jimmy Berry
f56ccb878c
Merge pull request #1406 from jberry-suse/pkglistgen-dump-include-update
pkglistgen: solv_cache_update(): support update repo and 4 way merge (and merge skipping)
2018-02-20 20:47:24 -06:00