196 Commits

Author SHA1 Message Date
Jimmy Berry
cff5befed3 Provide cache for expensive and cache-able staging requests.
The two slowest staging API calls are for information that rarely changes.
By caching the result the commands typically execute over twice as fast.
Going further can see improvements of an order of magnitude or more by
caching almost all the GET requests.

In contrast to osclib/memoize.py this cache operates at the HTTP request
level. This has several advantages:

- Caches the expensive part (ie the HTTP request). There are a number of
  functions in osc.core and elsewhere that make the same API request, but
  process the result differently which would require multiple API calls
  using memoize.
- Handles cases were a loader function uses class attributes as input and
  output and thus no relevant method parameters or return. An important
  example is StagingAPI._generate_ring_packages().
- Storage is project aware which allows caches to be deleted when a project
  is known to have changed.
- Due to project awareness, can utilize OBS /statistics/latest_updated API
  call to determine which projects need to be expired.

The cache file structure is as follows:

- hostname(apiurl)
  - project
    - sha1(url)
  - sha1(url)

See Cache.PATTERNS for changing the time to live (ttl) or add patterns to
be cached.
2017-01-11 10:23:54 -06:00
Ludwig Nussel
3119bec1e1 Allow to force individual package queries for devel project 2016-10-17 11:35:21 +02:00
Ludwig Nussel
7a19cf19fb Merge pull request #526 from lnussel/master
Handle separate lookup file in :NonFree
2016-06-14 14:29:31 +02:00
Ludwig Nussel
b505e7e7d8 Create empty dict even for projects without packages
cross OBS instance search doesn't seem to work so get_devel_project would
crash

closes #529
2016-06-14 14:05:17 +02:00
8c649111d9 Supeseding the request only when it was an reasonable type
Supeseding in stagin tool mainly works for submit/delete type request,
ie. supesede each other. The rest of type should not the case here.
2016-06-13 17:30:08 +08:00
335089ac61 Displays as Ring0 package if the subpackage was linked to Ring1
If the package was linked to Ring1 from Ring1, staging list command
should display it as Ring0 package ie. it's only be able to puts in
ring0 staging project. This is work out
https://progress.opensuse.org/issues/8266
2016-06-13 16:54:01 +08:00
Ludwig Nussel
bddcc12c8d Fix accept logic to actually work 2016-06-09 13:15:01 +02:00
Ludwig Nussel
4c43fe5c87 Group staging list by devel project 2016-06-07 15:12:02 +02:00
Ludwig Nussel
f2db5cdb5e Use implicit whitespace split 2016-06-07 15:12:02 +02:00
Ludwig Nussel
8857e79813 Print warning when staging with build disabled 2016-06-07 15:11:26 +02:00
Ludwig Nussel
082f3a9a52 Fixup cleanup_rings a bit 2016-06-07 14:44:42 +02:00
d70f80cbd2 Autocleanup: Make list of packages to retain configurable 2016-06-02 16:37:32 +02:00
5c8aa3944f stagingapi: Add list_packages method
* Lists all the packages in a project

and use the new API call in check_pkgs
2016-05-31 15:05:36 +02:00
Ludwig Nussel
28831300fc Defer initialization of ring packages
some users of the staging api don't need to access rings, so defer
initialization to first use.
2016-05-09 17:32:40 +02:00
8fa6783c3f Do not check totest on SLE
If release-notes-sles is existed then it is not openSUSE Factory but
SLE, hence do not check totest and openSUSE-release.rpm on it.

And add try/except protection in get_binary_version otherwise it will
crashes in case the package is not existed.
2016-03-25 18:55:36 +08:00
9a03a60f62 Fix PUT data never send 2016-02-18 18:25:51 +08:00
Ludwig Nussel
8c53f9f8f0 don't use recursive retry
don't use recursion and don't hammer server
2016-02-05 16:26:16 +01:00
Ludwig Nussel
11b25be352 Merge pull request #444 from DimStar77/accept-ready_check
staging accept: check if /totest is properly synced before changing /…
2015-12-21 16:29:05 +01:00
ce218979c4 staging accept: check if /totest is properly synced before changing /standard 2015-11-30 11:22:50 +01:00
37af83e927 Check subproject is exist before delete package
It's workaound for https://progress.opensuse.org/issues/9646, suppose it
is adi staging in case current project doesn't have ":DVD" subproject,
and the package is exist in current project.
2015-11-24 19:36:04 +08:00
0daad0fd1c auto-superseding in adi command 2015-11-19 20:23:10 +08:00
a3f59150af Do not superseding requests with list command
Do not superseding requests with list command, but if had the permission
of staging, can executing with --supersede argument, or even against
specific requests.
2015-10-01 19:59:40 +08:00
Stephan Kulow
51ad423ec0 fetch crash 2015-09-30 05:47:53 +02:00
Stephan Kulow
a3259f56e0 Merge pull request #407 from openSUSE/retry
retry 5xx - especially needed for view=info calls
2015-08-30 18:20:08 +02:00
Stephan Kulow
dccb5d39c4 avoid division for 5XX 2015-08-30 17:00:21 +02:00
Alberto Planas
f0cb13e9b9 Add method to recover status of a flag in _meta 2015-08-28 14:49:15 +02:00
Alberto Planas
f5e1d88dcf Use better parameter name. 2015-08-28 10:46:04 +02:00
69e1771a11 API: add a is_user_member_of() function 2015-08-06 15:54:56 +02:00
Alberto Planas
d7f445264f Memoize get_prj_pseudometa and invalidate in set_prj_pseudometa 2015-08-03 13:21:44 +02:00
Stephan Kulow
7df8a13c8d don't build disable adi projects 2015-07-27 11:16:53 +02:00
Stephan Kulow
1253661500 Merge pull request #372 from openSUSE/coolo_adi
add adi command to create an ad interim staging prj for all non-ring …
2015-07-20 10:52:04 +02:00
Stephan Kulow
555c23e10c allow projects (42) to exclude archs (ppc64le) 2015-07-19 17:50:46 +02:00
Stephan Kulow
78eec38b4f check adi stagings and allow arguments to adi 2015-07-19 09:32:32 +02:00
Stephan Kulow
595e78ebbc fix sorting of adi projects 2015-07-18 14:44:41 +02:00
Stephan Kulow
5de87f84e4 add adi command to create an ad interim staging prj for all non-ring packages 2015-07-16 15:09:26 +02:00
Alberto Planas
85c9cd239c Add create_adi_project method in stagingapi. 2015-07-16 13:42:23 +02:00
Stephan Kulow
77e9276340 no longer accept non-ring packages but show them in list 2015-07-16 12:42:25 +02:00
Stephan Kulow
7baeda10f0 only pass the group to the API
needs deploying openqa.oo though
2015-04-08 10:33:17 +02:00
Dinar Valeev
521cec9c00 Add group to the openQA query
Query openQA results by Job group

Signed-off-by: Dinar Valeev <dvaleev@suse.com>
2015-03-31 15:38:04 +02:00
Alberto Planas
8308eaab14 Small changes for SLE integration. 2015-03-24 14:46:36 +01:00
Stephan Kulow
a02076d8f8 we need ppc64le test dvd too 2015-03-17 16:35:04 +01:00
Stephan Kulow
9ae2850e66 remove openqa call no longer used 2015-03-12 11:47:42 +01:00
Alberto Planas
37c7e2a6c6 Add a test for accept command when there are multiple SPECs 2015-03-02 11:15:24 +01:00
dfa6a9ac38 given the correct ring_dvd path 2015-03-02 15:54:42 +08:00
Alberto Planas
c9d1e06ca8 Refactor configuration parameters into configuration file. 2015-02-19 11:39:05 +01:00
Alberto Planas
7fa1c06565 Rewrite DimStar work, using the OBSLock class. 2015-02-11 13:21:19 +01:00
12c2f5618c Locking: Let's get some locking mechanism so we hopefully step less on
each others toes.
2015-02-11 13:16:57 +01:00
87c8916620 Merge sync-rebuild into staging accept command 2015-02-09 17:17:47 +01:00
0968ffc6ff staging accept: create new links for packages when a new .spec file appears
https://progress.opensuse.org/issues/5902
2015-01-30 06:14:12 +01:00
Stephan Kulow
f095600f9b import re at the right places 2015-01-20 15:02:46 +01:00