Using build percentage is sub-optimal since many events can reset the
percentage which can result in additional requests being merged into the
staging. A better metric is the time since the first request was added to
the staging (the age of the staging). Unfortunately, this is not trivial to
determine especially given that the original request may be superseded or
unstaged entirely. As such the datetime at which the staging was activated
is stored in the staging pseudometa. A max age is then used to allow for
merging up until that point.
- handle build disable in select command just like comment instead of
stagingapi move_between_project() for consistency and to reduce double
checking the same information via extra API calls
- build_switch_staging_project() provides target_flag parameter rather
than trying to figure out flag value since caller has the context to
know and again reduces API calls
- build_switch_staging_project() checking conditions are essentially
checking if True is True since non-adi projects will always have ring
packages or rings disabled entirely. The condition of interest is
non-empty. Additionally adi projects are removed if empty so not a
terribly useful distinction to make.
- provide helper update_status_or_deactivate() to handle common logic
and replace in calling locations
Primarily an issue for Leap in that requests originating from different
source projects may be submitted for the same package at once. Instead of
always superseding the previous request the request will only be
superseded if it originates from the same project.
If different sources md5s are found the request will be ignored and left
for manual review and an identical request will be declined.
Often packages will fail to build after becoming stuck or other false
negative cases and need to have a rebuild triggered. The process can be
tedious if several packages failed in various stages.
- reworked internal representation of stagings to be more flexible
- new considerable staging state of mergeable based on set of conditions
- multiple proposal stages can be combined to create one proposal
- proposals may now fail to find every group a home without stopping
- splitter_info is stored when first request in an empty staging is staged
to allow for automated merging into that staging
- strategy concept introduced along with several strategies
- strategy_try() concept for applying multiple strategies and keeping
desirable groups which provides a solid basis for making smart decisions
- merge() introduced for merging new requests into existing stagings
- various internal cleanups along the way
functionality exposed via osc-staging:
- new options: --merge, --try-strategies, --strategy
- splitter info written when not merging
stagingapi: add set_splitter_info_in_prj_pseudometa().
The search/package query is typically very slow and can spend several
minutes timing out. The concept of loading all package meta data at once
is attractive, but given that the query regularly increases the runtime
of the script by an order of magnitude or more making individual calls
seems advantageous. These calls are already included in the cache which
means the initial request avoids the 10+ minute wait and repetitive calls
have no additional cost.
- re-implement list and adi commands using RequestSplitter
- numerous small cleanups and clarity improvements
- notably, adi now prints similar output to select when adding requests
- lxml is needed to provide more fully-featured xpath implementation
It is not uncommon for a request to be in a pending state which requires
action beyond the scope of the staging workflow, but does not make sense
to deny the request. For lack of a "postponed" or "pending" state on OBS
a sudo-state of "ignore" is provided for the staging workflow. The ignore
state will remove a request from the "list" and "adi" commands so as not
to be accidentally staged. This avoids the need for keeping a context of
what requests should be ignored in one's memory.
It is expected that an ignored request will have comments reflecting what
is to be done or one should be added via the -m option of ignore command.
The two slowest staging API calls are for information that rarely changes.
By caching the result the commands typically execute over twice as fast.
Going further can see improvements of an order of magnitude or more by
caching almost all the GET requests.
In contrast to osclib/memoize.py this cache operates at the HTTP request
level. This has several advantages:
- Caches the expensive part (ie the HTTP request). There are a number of
functions in osc.core and elsewhere that make the same API request, but
process the result differently which would require multiple API calls
using memoize.
- Handles cases were a loader function uses class attributes as input and
output and thus no relevant method parameters or return. An important
example is StagingAPI._generate_ring_packages().
- Storage is project aware which allows caches to be deleted when a project
is known to have changed.
- Due to project awareness, can utilize OBS /statistics/latest_updated API
call to determine which projects need to be expired.
The cache file structure is as follows:
- hostname(apiurl)
- project
- sha1(url)
- sha1(url)
See Cache.PATTERNS for changing the time to live (ttl) or add patterns to
be cached.
