This allows to utilise support for systemd-nspawn backend in build engine.
Like LXC, systemd-nspawn creates isolated lightweight container.
Signed-off-by: Oleg Girko <ol@infoserver.lv>
If no "dest" argument is specified when calling CpioRead.copyin or
CpioRead.copyin_file, a TypeError occurs in CpioRead._copyin_file
because os.getcwd(), which returns a str, is used as dest and, hence,
the subsequent os.path.join(...) fails (because it tries to join a
str and a bytes).
In order to avoid this, encode the result of os.getcwd().
Note that the existing
archive.copyin_file(hdr.filename,
os.path.dirname(tmpfile),
os.path.basename(tmpfile))
was OK because CpioRead._copyin_file os.path.join()s "dest" and
"new_fn", which are both str. It is just changed to stress that
CpioRead is a bytes-only API.
Fixes: #865 ("Traceback in osc/util/cpio.py line 128: TypeError:
Can't mix strings and bytes in path components")
Currently, when trying to initialize a non existent (server-side)
project via "osc init <prj>", osc errors out (after creating the wc)
because it fails to retrieve the package list. However, there is no
need to retrieve the package list in the "osc init <prj>" case. Hence,
skip the package list retrieval. As a result, osc does not error out.
For the background, see the discussion in #858 ("osc fails to check
out an empty project as project") [1].
[1] https://github.com/openSUSE/osc/issues/858#issuecomment-722330024
If meta=True is passed to checkRevision, the meta parameter is used
as a revision in the show_upstream_rev call. Instead, it should be
bound to show_upstream_rev's meta parameter.
Some services expect "old" service files (that is, files from a
previous service run) to be present in an ".old" dir. Hence, osc
should support that.
Instead of removing all files from a previous service run, move them
to the ".old" dir, run the services, and, finally, remove the ".old"
dir.
Unfortunately, the location of the ".old" dir is hardcoded in the
specific services. That is, we have to be careful if an ".old" dir
exists (in this case, we error out).
Based on [1].
[1] https://github.com/openSUSE/osc/pull/846
Currently, if the --offline option is passed to "osc build ...", a
preinstallimage is not used (even if it exists). Instead, a
preinstallimage should be used (if it exists) even if the --offline
option is specified.
This is faster in best case since the binary search does not need
to be executed on the server.
It also finds package names where no binary with that name exists.
(as for some multibuild cases)
Replace usage with better explanation. It was missing that it requires a
prefixed hostarch. Also workername is instead called workerid in the
API.
Usage help was before: osc workerinfo WORKER
Add actual example.
See also the fix for this in OBS API docs:
https://github.com/openSUSE/open-build-service/pull/10024
In the API a new request action release was implemented. This changes
enables the user to create a release request for non-maintenance projects
and to review / view the release requests
Without this patch, running an individual service that has parameters
defined in the _service file fails:
$ osc service run obs_scm
Please specify valid --scm=... options
Aborting: service call failed: /usr/lib/obs/service/obs_scm --outdir [snipped]
This is because although the service is defined in the _service file and
the "scm" parameter is set in it, the service wasn't being found in the
data structure and so the service executable wasn't being called with
the parameters supplied in the _service file. This patch corrects the
issue with the services data structure so that the service data isn't
overridden if it is defined in the _service file.
A side effect of this correction for services defined in the _service
file is that instead of overriding the service mode with '', the mode is
taken from the _service file. When using the "run" command, this would
mean that the call mode of None may not be in agreement with the service
mode defined in the file, e.g. "manual", and so the "run" command would
no longer cause it to run when it would before. We can take this
opportunity to define this as the correct behavior - the "run" command
now only runs services with "trylocal", "localonly", or no mode set -
and also ensure that other call mode commands result in sensible
behavior when called with a service name, for instance "osc service
manualrun download_files" will run only services with mode="manual" and
name="download"files" instead of all services with mode="manual".
Additionally, services that aren't defined in the _service file can be
called with a call mode command and will use that call mode rather than
None.
Add a "manual" service mode. A service with mode "manual" is not executed
by default (that is, via "osc service run"). As of now, "manual" is in
some sense just an alias for "disabled". However, this might change in the
future [1]. Also, "localrun" now executes services with mode "serveronly".
Moreover, the documentation of "disabledrun" is updated ("disabledrun"
never executed services with mode "serveronly"). Additionally, "localrun"
and "disabledrun" are marked as "[n]ot for common usage anymore" in the
service command's description.
The rationale for these changes is (partly) described in [1]. The main
motivation is to add some clarity (in contrast to the "disabled" mode,
it is probably easier to get/guess the semantics of the "manual" mode).
[1] https://github.com/openSUSE/osc/pull/826
The "disabledrun" service commands is marked as deprecated but has no
explicit replacement. It is still a useful command for updating packages
manually or through a CI system without being forced to run all defined
services with the "runall" command. This change adds a new command
"manualrun" and a new mode "manual" which behave the same as the
deprecated "disabledrun" command and "disabled" mode but have clearer
meaning. "manualrun" does not attempt backwards-compatible behavior with
the "disabledrun" mode for "disabled" services because "disabled" mode
may eventually be removed or change meaning. The "localrun" command is
enhanced to consider the "serveronly" mode. Since "disabledrun" never
executed services with mode "serveronly", its docs are updated
accordingly.
Improve error message in do_service a bit. The old "Too few arguments."
was misleading (for instance, if a non-remote command was not executed
in a package wc).
Note: with the new logic we could also get rid of the
"raise oscerr.WrongArgs('Local directory is no package')" statement.
Add a --status-filter option to "osc results" that can be used to
show, for instance, only the repos where a package failed to build. As
a short circuit, a -f/--failed option is added, too.
Add a --brief option to "osc prjresults" and "osc results" that can be
used to get a more compact representation of the results. In case of
"osc results", --brief is ignored if the results for a package are
requested.
in do_results:
* add --brief option on prj level:
[packagename] [repo] [arch] [buildstatus]
* filter by --status-filter <long status name>
works on prj and pkg level
in do_prjresults:
* --brief
* assume len(state)>1 as long state
core.py
* filter packages by build status
* long status handling in get_prj_results
* brief output generation in get_prj_results
Interprete unicode escape characters in a "--message" option. In some
cases, this breaks the existing UI (but that's OK because it can be
fixed by properly escaping the escape character(s)).
Note: if we are going to do more advanced stuff in the future, we should
move the logic into a separate function.
In commit 276d6e2439 ("Do not use the
chardet module in util.helper.decode_it") util.helper.decode_it was
changed to always decode the passed object if it has a decode method.
Since a python2 str has a decode method, the new code tries to utf-8
decode the passed str. As a result, a unicode object is returned (if
the decoding worked). Since a unicode object is not an instance of
type str, all subsequent isinstance(decoded_obj, str) checks evaluate
to False, which break some codepaths.
In order to fix this, restore the old python2 behavior (that is, if
the passed object is a str, it is not decode it). This change does not
affect the python3 codepaths.
Fixes: #814 ("osc log | fails")
Improve "osc rdiff --issues-only ..." output: now, it shows the added,
deleted and changed issues. Also, add a new "osc rdiff --xml ..." option,
which only works in combination with the "--issues-only" option: it prints
the raw xml.
Note: server_diff_noex has no option for the "full" parameter. Hence,
with the addition of the "xml=False" parameter, the signatures of the
server_diff_noex and server_diff functions are going to differ "forever".
That's OK (IMHO) because it is probably more sane to simply specify the
additional args via the kwargs syntax.
Remove dead code from the fetch module. Actually, it should have
been removed in commit 95ec7dee7b
('- fixed#590606 ("osc/fetch.py does not support authenticated URLs")').
In general, decode_it is used to get a str from an arbitrary bytes
instance. For this, decode_it used the chardet module (if present)
to detect the underlying encoding (if the bytes instance corresponds
to a "supported" encoding). The drawback of this detection is that
it can take quite some time in case of a large bytes instance, which
represents no "supported" encoding (see #669 and #746).
Instead of doing a potentially "time consuming" detection, either
assume an utf-8 encoding or a latin-1 encoding. Rationale: it is just
not worth the effort to detect a _potential_ encoding because we have
no clue what the _correct_ encoding is. For instance, consider the
following bytes instance:
b'This character group is not supported: [abc\xc3\xbf]'
It represents a valid utf-8 and latin-1 encoding. What is the "correct"
one? We don't know... Even if you interpret the bytes instance as a
human you cannot give a definite answer (implicit assumption: there is
no additional context available).
That is, if we cannot give a definite answer in case of two potential
encodings, there is no point in bringing even more potential encodings
into play. Hence, do not use the chardet module.
Note: the rationale for trying utf-8 first is that utf-8 is pretty
much in vogue these days and, hence, the chances are "high" that we
guess the "correct" encoding.
Fixes: #669 ("check in huge shell archives is insanely slow")
Fixes: #746 ("Very slow local buildlog parsing")
Importing `cElementTree` has been deprecated since Python 3.3 -
importing `ElementTree` automatically uses the fastest
implementation available - and is finally removed in Python 3.9.
Importing cElementTree directly (not as part of xml) is an even
older relic, it's for Ye Time Before ElementTree Was Added To
Python and it was instead an external module...which was before
Python 2.5.
We still need to work with Python 2.7 for now, so we use a try/
except to handle both 2.7 and 3.9 cases. Also, let's not repeat
this import 12 times in one file for some reason.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This checks if the filename of a downloaded file has
been modified (for example by a MITM attack) to contain
slashes. This could mean that the file is compromised
and that the attacker tries to overwrite system files.
split the code of do_dependson into two separate commands (just for
the osc help overview)
They are doing the opposite of each other.
Duplicate code was moved to _dependson()
do_whatdependson and do_dependson just call _dependson with an option
reverse set to None or 1.
add new regex and check for missing arguments.
The error message in python3 differs from the one in python2.
python3:
do_api() missing 1 required positional argument: 'url'
python2:
do_api() takes exactly 4 arguments (3 given)
To be compatible with python2 two checks are needed.
The repodata.RepoDataQueryResult is supposed to be a bytes API and
that's what our users (see build module) expect.
Note that the repodata.RepoDataQueryResult.path method still returns
a str. That's what the rpmquery.RpmQuery, debquery.DebQuery, and
archquery.ArchQuery classes also do (if the "path" was initially
passed as a str).
Fixes: #760 ("osc build fails when called with --prefer-pkgs where the
passed directory is a repodata repository or a subdirectory of one")
The packagequery.PackageQueryResult class is supposed to provide a
bytes API. Hence, packagequery.PackageQueryResult.evr() should return
bytes instead of a str. Also, adjust the single caller in the build
module.
This is a follow-up commit for commit
6dbf103e10 ("Use html.escape instead
removed cgi.escape"), which breaks the python2 backward compatibility
(since the "html" module is not available by default) and also breaks
the code in general (due to missing html imports).
The fix is based on the proposed fix in [1].
Fixes: boo#1166537 ("osc rq accept - forwarding request causes backtrace")
[1] https://github.com/openSUSE/osc/pull/764
Fixes:
`Traceback (most recent call last):
File "/usr/bin/osc", line 41, in <module>
r = babysitter.run(osccli)
File "/usr/lib/python3.8/site-packages/osc/babysitter.py", line 64, in run
return prg.main(argv)
File "/usr/lib/python3.8/site-packages/osc/cmdln.py", line 344, in main
return self.cmd(args)
File "/usr/lib/python3.8/site-packages/osc/cmdln.py", line 367, in cmd
retval = self.onecmd(argv)
File "/usr/lib/python3.8/site-packages/osc/cmdln.py", line 501, in onecmd
return self._dispatch_cmd(handler, argv)
File "/usr/lib/python3.8/site-packages/osc/cmdln.py", line 1232, in _dispatch_cmd
return handler(argv[0], opts, *args)
File "/usr/lib/python3.8/site-packages/osc/commandline.py", line 1458, in do_submitrequest
result = create_submit_request(apiurl,
File "/usr/lib/python3.8/site-packages/osc/core.py", line 4244, in create_submit_request
cgi.escape(message))
AttributeError: module 'cgi' has no attribute 'escape'
`
`cgi.escape` was deprecated in python 3.2
On Tumbleweed, `zypper in python-keyring` installs python2 version, while `osc` runs on python3.
After this change, user will be pointed to the correct version.
The correct zst magic is b'(\xb5/\xfd' (4 bytes) (that's what obs-build
is also using).
Kudos to Tobias Ellinghaus for spotting this.
Fixes: #756 ("zst detection fails")
osc importsrcpkg -n <pacname> does not work. If the option is supplied, osc
mistakenly trys to "decode" the pac object. This patch limit the decode
call when pac is not a string.
Refactored fix based on suggestions from marcus-h
Old keyring versions of the KeyringBackend class have no name method (the
name method was added in 2016). In this case, the class name is used as the
backend name.
Use configparser.ConfigParser instead of configparser.SafeConfigParser,
since the latter will be removed in future python versions. No functional
changes because SafeConfigParser is a ConfigParser except that its __init__
prints a DeprecationWarning.
configparser.SafeConfigParser has been changed to configparser.ConfigParser
upstream in Python 3.2. Warnings are saying that the alias will be
removed in future versions.
Signed-off-by: Scott Bahling <sbahling@suse.com>
Add core.parse_meta_to_string helper to work around the insane
implementation of core.meta_exists. Since core.meta_exists may return
a list of bytes, a str, a list of str etc., we ultimately convert the
data to str before passing it ET.fromstring(...).
In case of bytes, the explicit decoding is OK because it is assumed to
be a valid utf-8 encoding (the data represents an xml).
Note: at the moment core.parse_meta_to_string is also called even if it
is not necessary (it is only necessary if the "create" parameter of a
corresponding core.meta_exists call is True).
Note 2: this is just a temporary workaround and, eventually, we will make
the implementation of core.meta_exists more reasonable. When doing so,
we will also remove "public" function core.parse_meta_to_string again.
(Yes, this breaks API but the core.meta_exists change will also break the
API in some sense - so that's OK.)
Do not pass None as a release to RpmQuery.filename. The release might
be None in case of a kiwi build (the returned canonname is not used in
case of kiwi build).
In all the cases where meta_exists returns either
string data, bytes data or a list, the output needs
to be parsed correctly.
Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>
This is very unlikely but in very rare cases this
can happen.
Builing kiwi images containing debian is one case.
In this case we do not know what is inside the kiwi
file and osc build assumes buildtype 'rpm' to generate
a package list which get's thrown away anyway.
Now we just check for release
Add a "print_web_links" option. If enabled, a webui url is printed after
certain operations. For now, it is only used in "do_submitrequest" in order
to print the webui url that can be used to view the newly created request
(note: the print_web_links option is not considered in the "cd prj; osc sr"
case).
This will print the direct url to show the created request.
New general bool option 'print_web_links' must be set to enable
this.
Right now this is only for creating requests. More to follow.
Improve the SSLError message if a version mismatch (TLSv1) is
detected and give the user a hint what is wrong.
(Basically the API does not supprt TLS <= 1.2 and the python
version does not support TLS >= 1.2)
In case of an error, core.get_user_data returns an empty list.
None is never returned. Hence, only pop data from the returned list,
if it is non-empty.
In interactive review mode:
If a diff is issued and the request is accepted with 'a -m ok'
the tmpfile with the diff will be read. This tmpfile.read() call
is now decoded properly.
When creating a submit reqeust against a project that does not
accept SR anymore a maintenance request via create_maintenance_request
is generated. With this commit the orev will be honored and the
appropriate revision will be submitted.
This fixes some regressions with osc chroot:
- osc chroot --wipe --root=/dir/ can now be called outside
a working copy
- osc chroot --noinit --root=/dir/ can now be called outside
a working copy and behaves like the old code (Just entering
the chroot without any modifications)
- The confirmation of the deletion is implemented again and thus
the --force option was implemented too.
without this patch, `importsrcpkg` breaks with the following output:
```
File "/usr/lib/python3.7/site-packages/osc/commandline.py", line 7868, in do_importsrcpkg
createPackageDir(os.path.join(project.dir, pac), project)
File "/usr/lib64/python3.7/posixpath.py", line 94, in join
genericpath._check_arg_types('join', a, *p)
File "/usr/lib64/python3.7/genericpath.py", line 151, in _check_arg_types
raise TypeError("Can't mix strings and bytes in path components") from None
TypeError: Can't mix strings and bytes in path components
```
Fixes#712
fix the lastbuildroot handling if the package is being build oustide of
a package directory and thus no .osc/ dir can be found.
We then just skip the store_write_last_buildroot and store_read_last_buildroot
calls.
Use the distro module (if available) for guessing the linux distribution.
In case of python < 3.8, use the platform.linux_distribution() fallback if
no distro module is available.
Rationale: platform.linux_distribution() was dropped in python3.8
Existing passx enries resulted in a stacktrace because
the cp.get() call for the passx entries was missing.
Also added a delete_password function for ObfuscatedPasswordManager
to delete passx entries
If a python-keyring based backend is configured, but
python-keyring is not installed osc fails without giving
the user the opportunity to continue.
This introduces a new class method `create` for the AbstractCredentialsManager.
The CredentialsManagers for the backends that use a 3rd party software can
now check if the software is present in its own create method.
The error message if the call of _commit is just:
"ERROR: service run failed"
One option why this can fail is that the user Editor in env('EDITOR')
is not present.
We check now if e.msg gives a hint about "not found file" and then
error out with a better error message
building with alternative project (pac='_repository')
and multibuild did not work correctly, because the buildflavor
was not submitted to the src server.
With commit 2390823d649a3b0b6bf3b7bd07713c4426932bed in open-build-service
it is now possible to submit the build flavor like this: _repository:<flavor>
The obs commit also enables osc buildinfo --alternative-prject -M <flavor>
to show the correct buildinfo for the flavor.
There is a config option 'include_request_from_project' which
can be set to 0 to exclude all requests originating from the
project in osc [rq|review] list.
To force this behaviour just once (without always editing oscrc
the '--incoming' option can be used.
Make "osc buildinfo --alternative-project <project> ..." consistent with
"osc build --alternative-project <project> ...". This change breaks some
invocations where a build_descr is omitted (but in this case, simply
use "osc buildinfo <project> <package> <repo> <arch>).
Pass the filename to the vc script instead of passing its content via
the "-m" parameter. If "osc vc --file <fname>" is issued, it might be
possible that $EDITOR is still opened (if the "vc" script is too old),
which contradicts the --file option's help text. This "break" is acceptable
(just update osc in tandem with the vc script).
In case diff3 returns something except 0 or 1, osc errors out because the
variable merge_cmd is not defined. It used to exist, but got removed in the
past and is reintroduced via this commit.
Simplify the project sr handling in do_submitrequest a bit. It also fixes
a bug in case a target project and --separate-request is specified.
Other bugs are retained for now (for instance, --update-link is ignored).
Most of the stuff is now done in the API.
For just submitting all in one request the following is sufficient:
<action type="submit"> <source project="%s" /> %s </action>
If a target project is given it looks like this:
<action type="submit"> <source project="%s" /> <target project="%s"> %s </action>
We don't need package name or source package and targets anymore.
This is all handled by the API now.
For the --seperate-requests case the check for the entries in the linkinfo
is not working on most cases. So we handle this in the do_submitrequest
and skip the package if X-Opensuse-Errorcode is missing_action which means
basically nothing to do for this package
redesign target_project handling
* Adapt do_config to use the new credentials manager implementation
and add a --change-password option which can be used to change the
password.
* Adapt config_set_option to follow the change in do_config.
* Split selection of the credentials manager descriptor to reuse it
in do_config and interactive_config_setup.
* Introduce new ConfigMissingCredentialsError which is raised in case
of missing credentials (user or password). In this case the user will
be asked to enter the new credentials.
* new module credentials.py which contains classes
and methods to set and get passwords for different
backends:
- python-keyring
- gnomekeyring
- ConfigFile based storage
The new code should be backward compatible except a minor
change in add_section (pass and passx are not removed from
the config parser). This affects only callers that do not pass
a creds_mgr_descriptor.
On initial osc call or initial osc call on new API Url
the user now can decide where to store the password (based
on the backends available on his system)
The upcoming credentials manager abstraction supersedes the old
plaintext_passwd behavior. This commits "breaks" the old behavior:
Passwords are not rewritten anymore (plaintext <-> obfuscated)
Handle multiple revisions the same as a single revision in terms of what
is allowed (digit, or 32 character string [md5]). Additionally, support
either blank or "latest" to mean latest revision (None).
This allows for new revision to be specified without old revision like the
following example:
osc rdiff ... -r :17
Move logic to conf module in new function interactive_config_setup
for new config file and missing apiurl section.
Dropped keyring code, because it was never used in write_initial_config()
python3 "workaround" for the quite fragile code in
RawCmdln._help_preprocess: It depends on the "non-deterministic" order
of preprocessor.items(). Hence, the results may differ on python2 and
python3. The real fix is probably to enforce a fixed iteration order.
See also the discussion in [1].
[1] https://github.com/openSUSE/osc/pull/604
prevent call of preprocessor if ${cmd_name} marker is accidentaly set,
but cmdname is None.
Also for cmd_option_list and cmd_usage
overwrite _help_preprocess_* functions in Osc class
After (successfully) running an 'osc vc' the following exception trace
comes up, when no rpm python module is available:
```
no changes made
Traceback (most recent call last):
File "/home/mgerstner/.local/lib64/python3.6/site-packages/osc/babysitter.py", line 62, in run
return prg.main(argv)
File "/home/mgerstner/.local/lib64/python3.6/site-packages/osc/cmdln.py", line 344, in main
return self.cmd(args)
File "/home/mgerstner/.local/lib64/python3.6/site-packages/osc/cmdln.py", line 367, in cmd
retval = self.onecmd(argv)
File "/home/mgerstner/.local/lib64/python3.6/site-packages/osc/cmdln.py", line 501, in onecmd
return self._dispatch_cmd(handler, argv)
File "/home/mgerstner/.local/lib64/python3.6/site-packages/osc/cmdln.py", line 1232, in _dispatch_cmd
return handler(argv[0], opts, *args)
File "/home/mgerstner/.local/lib64/python3.6/site-packages/osc/commandline.py", line 8924, in do_vc
sys.exit(vc.returncode)
SystemExit: 0
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/mgerstner/bin/osc", line 41, in <module>
r = babysitter.run(osccli)
File "/home/mgerstner/.local/lib64/python3.6/site-packages/osc/babysitter.py", line 172, in run
except RPMError as e:
TypeError: catching classes that do not inherit from BaseException is not allowed
```
To fix this change the fallback RPMError from None to an actual
Exception-derived type.
Since we do not know the files' encoding, the diff functions/methods
have to return bytes. Note: decoding the diff result is wrong in
general (see the discussion in [1]).
[1] https://github.com/openSUSE/osc/pull/554
The files are now opened as rb for diffing. In python2 nothing changes.
In python3 the returned diff is bytes now.
The following changes were made:
* commandline.py: The returned diff is now bytes
* get_diff now returs the diff as a bytes-like object
* run_pager writes with sys.stdout.buffer.write if message
is not a string
* for the commit message the returned diff needs to be decoded
now. Otherwise it will just producce garbage. For the commit
message the diff on decoded bytes-objects is ok. (nothing harmfull
can happen here)
* fixed submit_action_diff
* fixed request_interactive_review
use own implementation of HTTPSConnection (myHTTPSConnection)
instead the one provided by M2Crypto (httpslib.HTTPConnection)
And in proxy case use myProxyHTTPSConnection.
all credits go to wfrisch
Do not guess the decoding of the returned output on osc api calls.
This could lead to unexptected behavior. Just use sys.stdout.buffer.write()
to print the bytes-like object in python3.
if prjconfig meta is empty the class metafile init
fails with
if isinstance(input[0], str):
IndexError: list index out of range
This is because input is an empty list and has no
[0] index.
This can be fixed by changing this line to:
if input and isinstance(input[0], str):
simplified and fixed the __init__ method of the metafile class.
input can be string, bytes-like object, list of strings or list
of bytes-like objects now.
Based on the input now always a list is generated and joined to
a string for writing in the fd. (This is ugly but needed for
compat reasons)
this makes sure not to mix objects to prevent a TypeError
TypeError: Can't mix strings and bytes in path components
Fixes#567
Signed-off-by: Maximilian Meister <mmeister@suse.de>
