If an error occurs while sending a file to the API (e.g. 401), the io handle
is reused and the current postion is at the end of the file. This results in an
empty value.
This patch takes care that the current file postion gets resetted
if a retransmission is required.
Additional changes:
* rename getbinaries --debug to --debuginfo to avoid conflicts
* switch conf['verbose'] from int to bool
The changes were needed because the new argparser behaves differently
and commands such as `osc ls` ran in verbose mode by default.
Cmdln.py is unmaintained for years and uses deprecated optparse.
Let's replace it with a simpler custom code. Also remove code
that generates man page, we'll replace it with a 3rd party tool.
If `sshkey` config option is set, then osc prefers it over password auth.
If `sshkey` config option is not set and the server supports both basic
and signature auth, basic auth is used and ssh key is NOT auto-detected.
Users who want to use ssh auth with ssh key auto-detection can now leave
the `pass` config option empty to trigger ssh key auto-detection.
The ssh-key autodetection picks the first key that matches:
- key loaded to ssh-agent (`ssh-add -l`) that has a public key in ~/.ssh
- ~/.ssh/{id_ed25519,id_rsa}
It is also recommended to use Obfuscated or Plaintext credentials manager.
Please be aware that storing passwords using these credentials managers
is unsafe, because they're stored in plain text on disk.
Example:
[<apiurl>]
user=<username>
pass=
# ssh key is auto-detected because `pass` is empty
sshkey=
credentials_mgr_class=osc.credentials.ObfuscatedConfigFileCredentialsManager
When using ssh keys from gpg, there are no private key files on
disk. The public keys are available from "ssh-add -L". Conveniently,
users store the public keys in some ".pub" file under ~/.ssh
(see e.g. https://serverfault.com/questions/906871/force-the-use-of-a-gpg-key-as-an-ssh-key-for-a-given-server;
this is also necessary to use IdentityFile= in ssh itself).
Thus public key files can't be ignored any more in list_ssh_dir_keys().
"ssh-keygen -Y sign" works nicely with a public key file if the agent
has access to the private key.
Emojis were broken due to this commit.
The revert causes a change in behavior:
"\n" no longer gets resolved to a newline.
It is suggested to do the following instead:
$ osc <command> ... -m "first line
second line"
This reverts commit 16fda3115c.
* src/noarch rpm packaages needs to be stored in scheduler architecture to avoid
conflicts of the multiple versions
* avoid removal of every downloaded file
There seem to be a bug in how GitHub generates archives.
"Format:" and "$" characters get removed from the version string,
setting it to:
version = "%(describe:tags=true)"
Fixes the following error:
```
% osc
Traceback (most recent call last):
File "/usr/bin/osc", line 45, in <module>
r = babysitter.run(osccli)
File "/usr/lib/python3.10/site-packages/osc/babysitter.py", line 67, in run
return prg.main(argv)
File "/usr/lib/python3.10/site-packages/osc/cmdln.py", line 341, in main
self.postoptparse()
File "/usr/lib/python3.10/site-packages/osc/commandline.py", line 137, in postoptparse
if self._get_canonical_cmd_name(self.args[0]) == "help":
IndexError: list index out of range
```
* init command is working inside of a git repository
* downloadassets command fetches references assets from build description
* checkout is cloning from git
This changes the code back to retrying up to 5 times for old
python version 2.6.6-2.7.9. The complete backport of the basic auth
changes clutters up the code way to much for such a little gain.
(This basically reverts commit 326abe0c8b)
It is possible that the self._pwfunc() call returns a callable. For
instance, if the keyutils.osc.OscKernelKeyringBackend is configured
in the oscrc. Hence, check in credentials._LazyPassword.__str__
if the returned password is a callable and, if so, call it. Moreover,
a deprecation warning is printed. Eventually, this compat code will
be removed again.
This is a follow-up commit for commit
784d330f20 ("Only prompt for a password
if the server asks for it") (actually, it is a regression that was
not caught during the review...).
Only ask for a password if it is really needed for authentication.
The new lazy password approach is much smarter than the old callable
hack. That's why we deprecate returning a callable from
AbstractCredentialsManager.get_password. The current compatibility code
for a callable will be removed in the near future.
Minor nitpick: actually it would have been "cleaner" to introduce a new
subclass like an AbstractLazyPasswordCredentialsManager that encapsulates
the lazy password behavior. Currently, if, for instance, a credentials
manager is always non-lazy it would just override get_password but still
inherits the abstract (and unused) _get_password method.
In many cases the session cookie is already available, so there
is no need to ask for a password. To make this work with the
python authentication implementation, we add a small proxy object
for the password and only ask the credential manager if the
stringify method is called.
This approach also makes it possible to offer a non-password based
authorization type if the server allows multiple authentication
methods.
This applies when downloading multiple packages, typically the whole repo.
When downloading a single package, everything works as usual
and the subdir is not created.
This allows a format to be specified in the sccache uri, specifically
the file uri so that a per-package cache can be created. This way
an osc build locally doesn't ruin your cache moving between different
packages.
They have identical names for all downloaded packages
and get overwritten by the last downloaded file.
Unless we dowload them into subdirs or prefix them with package name,
it makes no sense to download them.
The ':' character is used as a separator in Open Build Service
and constantly appears in directory names after running osc commands.
Windows do not support ':' as a valid character on file system.
This breaks not only osc but also basic commands such
as 'git clone' on a project that contains colons in paths.
That's why we decided to make osc unsupported on Windows.
Do not fail with a traceback if the config file parsing fails.
Hence, catch the configparser.Error exception and print its
"message" attribute to the user.
Fixes: #985 ("Type error on any osc call")
Since a recent backend change, a bdep has a hdrmd5 by default. That
is, osc always downloads these bdeps from the API (unless they are
cached) instead of a mirror. This is not intended.
Using a mirror is no problem because the hdrmd5s are verified in
the build module.
Note: If this causes a problem, one could also use "osc build
--download-api-only" to mimic the old behavior.
If the oscrc is a symlink, follow the symlink when writing the
configuration file. The old code replaced the symlink with a
regular file (see #390 ("symlinked $HOME/.oscrc gets replaced
with a ordinary file")).
Implementation note: if the directory, which contains the resolved
config file, has a <config file>.new file, the file is overwritten.
When a user creates a symlink pointing from ~/.config/osc/oscrc
to a different location, don't overwrite the symlink but follow
it when writing configuration on disk.
Without escaping the % character, the download URL could be subject
to string formatting (depending on the subsequent characters). For
instance, if the url attribute's value of a buildinfo's path element
contains the substring "c_c%2B%2B", the "%2B" is interpreted as a
format string (see issue #965), which is wrong ("B" is not a valid
format character at all). In order to avoid this, escape all
% characters in the download urls.
Note: escaping the % characters in the download url itself is OK
because we only intend to "format" the path.
Note: we do not escape the % characters for urls from the config
file (implicit assumption: the user already correctly escaped the
urls (whether this assumption is sensible or not is debatable, of
course)).
Fixes: #965 ("unsupported format character 'B' (0x42) at index 66")
The old code passes try_again=False to the recursive postoptparse
call when calling it from one of the exception handlers. This is
wrong because it can result in an incomplete conf.config dict (for
instance, if two apiurl sections have no user and no password and
no credentials_mgr_class option - see #761 ("Traceback config with
two backends and no username")).
Hence, Osc.postoptparse should only return if the conf.get_config
call succeeds. For this, unconditionally call Osc.postoptparse from
within the exception handlers. Note: this could potentially (although
quite unlikely) result in an endless recursion but in each recursive
call "user" interaction is required (that is, the user could simply
press CTRL+c) - so this should not be a problem.
Implementation note: this change breaks the API. Rationale: the
semantics of Osc.postoptparse changed. Hence, "pretending" to
honor the try_again parameter could result in unexpected behavior
(from the API consumer's POV). Hence, a traceback might be more
sensible.
Fixes: #761 ("Traceback config with two backends and no username")
Support a zst compressed control.tar in debquery.DebQuery. A zst compressed
control tar is used, for instance, in Ubuntu 21.10.
Note: this requires the 3rd-party python-zstandard module.
Current OBS is delivering hdrmd5 in buildinfo. It turns out
that osc has already code for validating cached files, but it
invalidates all local files atm with python 3.x
Current OBS is delivering hdrmd5 in buildinfo. It turns out
that osc has already code for validating cached files, but it
invalidates all local files atm with python 3.x
Do not create an MR for the entire project if "osc mr" is invoked in
a package wc (only create an MR for the specific package instead).
Strictly speaking, the "breaks" the existing UI - but this rather seems
to be a "fix" than a "break";)
Some of our repositories have specific download urls. osc is ignoring
this so far and just tries to use the generic downloadurl
This code prefers definitions for individual path elements if they exist.
We could IMHO remove the old code, since old OBS instances would still
work via the api download fallback.
Real life examples for repo specific configs are on openSUSE all
repositories outside of the /repositories/ directory. eg.
<path project="openSUSE:Tumbleweed" repository="dod" url="http://download.opensuse.org/tumbleweed/repo/oss/"/>
Co-Author: Marcus Hüwe <suse-tux@gmx.de>
The old code does not print any information about the host, for
which the access failed, in case of an URLError. In order to fix
this, add information about the host (and port) to the URLError
instance in core.http_request and use this information in the
babysitter to print out a more detailed error message (which includes
the host (and port)).
For now, we simply add a "private" "_osc_host_port" attribute to
the URLError instance (this way we avoid potential name clashes (due
to the "_osc" prefix) and could come up with a different/more clever
way in the future (due to its privateness)).
Fixes: #954 ("Better diagnostic for domain name issues")
os.sysconf is not available on all platforms (like Windows) but it
is used to retrieve the number of online processors. If missing,
assume one processor (building on such a platform will most likely
not work, though).
Fixes: #948 ("Windows compatibility") (at least it improves the
Windows support a bit)
A workflow token can be created via "osc token --create --operation
workflow --scm-token <SCM_TOKEN>".
Triggering a workflow token via osc is probably unlikely - that's
why it is not yet implemented (it would also make the UI a bit
awkward because one has to specify a concrete http header).
Fixes: #943 ("implement osc token --operation=workflow")
The use of makeurl makes the code more readable/maintainable (IMHO)
and it also does proper percentage encoding of the query string (not
that the osc codebase cares much about it, though:/).
Newer rexml Ruby gem used on OBS server side uses stricter XPath parsing.
This change fixes incorrect XPath that was accepted by older rexml,
but not accepted by newer one.
Signed-off-by: Oleg Girko <ol@infoserver.lv>
Offer a force ("f") choice if, for instance, "osc meta prj foobar -e"
fails due to a HTTPError in metafile.edit. If the force choice is
selected, a new url is constructed by invoking the metafile._URLFactory
instance with a "force='1'" argument (this adds a "force=1" to the
original url's query string (*)) and the corresponding file is PUTed
to the new url. If this PUT fails again and now the "y" choice is
selected, the file is PUTed to the original url (*).
(*): Stricly speaking, from metafile.edit's POV, the concrete url
depends on the passed in metafile._URLFactory instance, though.
Note: the metafile._URLFactory class and its is_force_supported method
is a gross hack. That's why this class is marked as private (that is,
we can remove it at any point in time again without breaking the
API/3rd party applications). An alternative to the metafile._URLFactory
approach would be manual URL parsing and manual URL construction
(adding "force=1" to the query string)... but this is also pretty
awkward (if done properly).
Fixes: #916 ("for osc meta edit change y/n to y/n/f")
Fixes: #942 ("Offer -f when prjmeta change leads to repo_dependency")
The order is now:
- ~/.osc_cookiejar, if it exists
- $XDG_STATE_HOME/osc/cookiejar if XDG_STATE_HOME neither null nor empty
- ~/.local/state/osc/cookiejar
Do not try to run source services when building in a non package wc. This
is the behavior we had prior commit c39c3b8cae
("Cleanup the source services execution code in do_build").
There is no "sane" way to execute the source services in case of a
non package wc build because we cannot export the OBS_SERVICE_PACKAGE
env variable with a meaningful value.
Fixes: #936 ("'osc build --local-package ...' fails with 'not an osc
package working copy'")
When building a package from a directory that is not a checked-out
OBS working, the error message:
"Error: "<directory>" is not an osc package working copy."
is generated.
This occurs when build.main() attempts to run source services which
is probably not a good idea as these are part of the core.Package
infrastructure which cannot be initialized for such packages.
It is probably best to disable the source services in this case.
See Issue#936.
Suggested-by: Marcus Huewe <suse-tux@gmx.de>
Signed-off-by: Egbert Eich <eich@suse.com>
The old code does not support the --binary option in combination
with the --verbose option. Specifying --binary and --verbose at
the same time results in a crash (because the binary listing
contains no <title>...</title> element).
In order to fix this, do not try to access a <title>...</title>
element when --binary and --verbose are both specified. Instead,
in this case, include information about the repo, arch, version,
and release of the corresponding binary element.
Fixes: #933 ("osc se -v -B crash")
The old code uses a variable .rXYZ suffix (where XYZ is the revision
of the package wc during the merge operation). Now, if Package.mergefile
is invoked during an update, XYZ represents the "old" revision. That
is, if a merge conflict happens, then a subsequent "osc resolved <file>"
will not unlink the <file>.rXYZ file (because
Package.clear_from_conflictlist only takes the current rev into account).
In order to fix this, use a fixed ".new" suffix. This way,
Package.clear_from_conflictlist can properly unlink the corresponding
*.new file. This naming scheme for the "upfilename" is in line with
"osc pull" and "osc repairlink".
Note: if a working copy was updated with an "old" osc version (without
this commit) and a "new" osc version (with this commit) is used to run
"osc resolved <file>", then the <file>.rXYZ file is _NOT_ removed (it
is not worth the effort to add compat code for this).
A password can be deleted via "osc config -d <apiurl> pass". Actually,
if we really want to support password deletion, we should introduce
a --delete-password option because the "pass" config option can be
considered as an implementation detail, which we should not expose
to our users.
The password store can be changed (without entering the password
again) via "osc config <apiurl> --select-password-store". This
command deletes the password from the current password store and
stores it in the selected password store.
Previously, the --select-password-store option had no meaningful
semantics. In order to use it, one always had to provide a password
and explicitly pass "pass" as the config option (the same could be
achieved by using --change-password). Hence, in a strict sense,
this change breaks the UI.
Without the slash splitting, "osc browse prj/pkg" interprets the
argument as a project, which is wrong. Hence, perform the slash
splitting (as most commands do).
Always send the sha256sums of all tracked files in case of a
frozen package wc. For instance, this is needed if the package is
a plain link (no branch) because in this case the backend might
request a sha256sum for a tracked but unmodified file (this can
happen because the backend cannot expand the link).
The new behavior is in line with a pulled/linkrepair package wc.
Fixes: #924 ("Transmitting file data There is no sha256 sum for
file")
When trying to commit a non-existent package via Project.commit it
is treated as an external package (because a non-existent package
has no "state" inside the project). That is, Project.commitExtPackage
is called, which fails with a FileNotFoundError in case of a
non-existent package (and the traceback is printed to the user).
In order to fix this, treat a non-existent package as broken package.
That is, simply print an info message and do not error out with a
traceback (note: the commit is _not_ aborted).
Fixes: #920 ("osc commit should fail gracefully in case of
nonexistent filename")
Sccache is an alternate build caching system to ccache/icecream. It
supports C, C++ and Rust. It can optionally have distributed or remote
caches via redis, s3 object stores, memcached, azure storage or
google cloud storage.
This can help to significantly improve the performance of Rust rebuilds.
For example, Kanidm changes from 400s to 122s on a rebuild, and rust-lang
rebuilds improve from 7200s to 4770s. With some changes to the rust
packages especially this will be possible to speed up over version
changes as well.
See also: obs-build PR https://github.com/openSUSE/obs-build/pull/680
Do not use a preinstallimage if the local build is executed as a non-root
(the preinstallimage contains device nodes which usually cannot be created
by a non-root user - this is not a problem in the non-preinstallimage
codepath (see [1])).
[1] https://github.com/openSUSE/osc/pull/908#issuecomment-806903856
The old code only supports a file whose size is less then or equal
to INT_MAX (due to a reasonable(!) limit in M2Crypto). The actual
issue is in core.http_request which mmap(...)s the file, wraps it
into a memoryview/buffer and then passes the memoryview/buffer to
urlopen. Eventually, the whole memoryview/buffer is read into memory
(see m2_PyObject_GetBufferInt). If the file is too large (> INT_MAX),
m2_PyObject_GetBufferInt raises a ValueError (which is perfectly
fine!).
Reading a whole file into memory is completely insane. In order to
avoid this, we now simply pass a file-like object to urlopen (more
precisely, the file-like object is associated with the Request
instance that is passed to urlopen). The advantange is that the
file-like object is processed in chunks of 8192 bytes (see
http.client.HTTPConnection) (that is, only 8192 bytes are read into
memory (instead of the whole file)).
There are two pitfalls when passing a file-like object to urlopen:
* By default, a chunked Transfer-Encoding is applied. It seems that
some servers (like api.o.o) do not like this (PUTing a file with
a chunked Transfer-Encoding to api.o.o results in status 400). In
order to avoid a chunked Transfer-Encoding, we explicitly set a
Content-Length header (we also do this in the non-file case (just
for the sake of completeness)).
* If the request fails with status 401, it is retried with an
appropriate Authorization header. When retrying the request, the
file's offset has to be repositioned to the beginning of the file
(otherwise, a 0-length body is sent which most likely does not
match the Content-Length header).
Note: core.http_request's "data" and "file" parameters are now mutually
exclusive because specifying both makes no sense (only one of them
is considered) and it simplifies the implementation a bit.
Fixes: #202 ("osc user authentification seems to be broken with last
commit")
Fixes: #304 ("osc ci - cannot handle more than 2 GB file uploads")
This kind of guessing can not really work here and leads to failing
builds when using KVM. (eg. when using a preinstallimage)
Removing the code, since we have a now a way to allow the user to
specify building as user via su-wrapper config
Element.getchildren is deprecated and not available on python39
anymore. Instead, iterate over the element itself (which iterates
over the element's children).
Fixes: #903 ("AttributeError: 'xml.etree.ElementTree.Element' object
has no attribute 'getchildren'")
Most osc commands support slash notation for the specification of
a project package pair. That is, "osc <cmd> prj/pkg" has the same
semantics as "osc <cmd> prj pkg" (in most cases).
For consistency reasons, "osc creq" should also support the slash
notation for the action type's arguments. That is, for instance,
"osc creq -a submit src_prj/src_pkg dst_prj/dst_pkg" should have the
same effect as "osc creq -a submit src_prj src_pkg dst_prj dst_pkg".
Proposed-by: darix
If there are existing requests that should be superseded, the old
code stores the Request instances in the myreqs list, which is
returned to the caller. However, the caller expects only request
ids instead of instances of class Request. Eventually, this results
in a type error - excerpt:
...
File "/usr/lib/python3.8/site-packages/osc/commandline.py", line 1892, in do_createrequest
change_request_state(apiurl, srid, 'superseded',
File "/usr/lib/python3.8/site-packages/osc/core.py", line 4322, in change_request_state
u = makeurl(apiurl,
File "/usr/lib/python3.8/site-packages/osc/core.py", line 3326, in makeurl
return urlunsplit((scheme, netloc, '/'.join([path] + list(l)), query, ''))
TypeError: sequence item 2: expected str instance, Request found
Hence, simply return the request ids instead of the Request instances.
Note: this changes the API of the Osc._submit_request method but
this is OK because it is not part of the public API.
When calling "osc creq -a prj1 foo prj2 bar -a submit prj1 bar prj2 bar",
the requests that could be superseded are calculated two times for the
prj2/bar package. Hence, they could end up two times in the "supersede"
list (see do_createrequest) In order to avoid duplicates, use a set
instead of a list.
Kudos to darix for pointing this out!
Note: it is a bit questionable if osc's current semantics makes sense
in the above example.
When creating a new request via the core.Request.create method, there is
no need to escape the data that is assigned to the "description" attribute
of a core.Request instance. Internally, core.Request.create ensures that
the data, which is POSTed to the api, is correctly escaped (the escaping
is implicitly done by ET (see core.Request.to_str)). Manually escaping the
description results in a double escaping (the escaped description is
escaped by ET again) - this is not the desired behavior.
Analogously, there is no need to escape the data that is passed to the
message parameter of the core.create_submit_request function because
core.create_submit_request takes care of escaping it.
Fixes: #869 ("Silly encoding of htmlencodable entities")
So far, core.get_prj_results only supports a substring based name
filtering mechanism. Now, a regex based name filtering mechanism
is used. That is, if the regex matches a package name, the package
is not filtered out.
This is an API incompatible change:
- 3rd party code which looks like this is going to break:
class Foo(str):
...
def __eq__(self, other):
return ...
... = core.get_prj_results(..., name_filter=Foo())
(My gut feeling says there are no such callers. However, if this
really breaks any serious code, we can fix it in a follow-up commit)
- "osc prjresults openSUSE:Leap:15.2:Update --name-filter zypper." will
now also show "zypper-docker" etc. because the dot (".") matches any
character (except a newline). Previously, only packages that contained
the str "zypper" followed by a dot (".") were shown. The old behavior
can be restored, if the dot is escaped: "osc prjresults
openSUSE:Leap:15.2:Update --name-filter 'zypper\.'".
Of course, this affects all other special characters, too.
Additionally, if an illegal regex is passed to the --name-filter option,
an exception is raised. The previous code did not fail.
This can break existing workflows and scripts. We could avoid this by
introducing a --name-filter-regex option but this would clutter the
UI (IMHO).
A regex based name filtering feature was requested by darix.
Using os.getcwd() in combination with a subsequent .encode() is error
prone:
marcus@linux:~> mkdir illegal_utf-8_encoding_$'\xff'_dir
marcus@linux:~> cd illegal_utf-8_encoding_$'\xff'_dir/
marcus@linux:~/illegal_utf-8_encoding_ÿ_dir> python3
Python 3.8.6 (default, Nov 09 2020, 12:09:06) [GCC] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.getcwd().encode()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcff' in position 36: surrogates not allowed
>>>
Hence, use os.getcwdb(), which returns a bytes, instead of
os.getcwd().encode().
Fixes: commit 36f7b8ffe9 ("Fix a
potential TypeError in CpioRead.copyin and CpioRead.copyin_file")
If no dir is passed to util.ArFile.saveTo, dir is set to os.getcwd(),
which returns a str. Since self.name is a bytes, the subsequent
os.path.join(dir, self.name) results in a TypeError.
To fix this, use os.getcwdb(), which returns a bytes instead of a
str.
This allows to utilise support for systemd-nspawn backend in build engine.
Like LXC, systemd-nspawn creates isolated lightweight container.
Signed-off-by: Oleg Girko <ol@infoserver.lv>
If no "dest" argument is specified when calling CpioRead.copyin or
CpioRead.copyin_file, a TypeError occurs in CpioRead._copyin_file
because os.getcwd(), which returns a str, is used as dest and, hence,
the subsequent os.path.join(...) fails (because it tries to join a
str and a bytes).
In order to avoid this, encode the result of os.getcwd().
Note that the existing
archive.copyin_file(hdr.filename,
os.path.dirname(tmpfile),
os.path.basename(tmpfile))
was OK because CpioRead._copyin_file os.path.join()s "dest" and
"new_fn", which are both str. It is just changed to stress that
CpioRead is a bytes-only API.
Fixes: #865 ("Traceback in osc/util/cpio.py line 128: TypeError:
Can't mix strings and bytes in path components")
Currently, when trying to initialize a non existent (server-side)
project via "osc init <prj>", osc errors out (after creating the wc)
because it fails to retrieve the package list. However, there is no
need to retrieve the package list in the "osc init <prj>" case. Hence,
skip the package list retrieval. As a result, osc does not error out.
For the background, see the discussion in #858 ("osc fails to check
out an empty project as project") [1].
[1] https://github.com/openSUSE/osc/issues/858#issuecomment-722330024
If meta=True is passed to checkRevision, the meta parameter is used
as a revision in the show_upstream_rev call. Instead, it should be
bound to show_upstream_rev's meta parameter.
Some services expect "old" service files (that is, files from a
previous service run) to be present in an ".old" dir. Hence, osc
should support that.
Instead of removing all files from a previous service run, move them
to the ".old" dir, run the services, and, finally, remove the ".old"
dir.
Unfortunately, the location of the ".old" dir is hardcoded in the
specific services. That is, we have to be careful if an ".old" dir
exists (in this case, we error out).
Based on [1].
[1] https://github.com/openSUSE/osc/pull/846
Currently, if the --offline option is passed to "osc build ...", a
preinstallimage is not used (even if it exists). Instead, a
preinstallimage should be used (if it exists) even if the --offline
option is specified.
This is faster in best case since the binary search does not need
to be executed on the server.
It also finds package names where no binary with that name exists.
(as for some multibuild cases)
Replace usage with better explanation. It was missing that it requires a
prefixed hostarch. Also workername is instead called workerid in the
API.
Usage help was before: osc workerinfo WORKER
Add actual example.
See also the fix for this in OBS API docs:
https://github.com/openSUSE/open-build-service/pull/10024
In the API a new request action release was implemented. This changes
enables the user to create a release request for non-maintenance projects
and to review / view the release requests
Without this patch, running an individual service that has parameters
defined in the _service file fails:
$ osc service run obs_scm
Please specify valid --scm=... options
Aborting: service call failed: /usr/lib/obs/service/obs_scm --outdir [snipped]
This is because although the service is defined in the _service file and
the "scm" parameter is set in it, the service wasn't being found in the
data structure and so the service executable wasn't being called with
the parameters supplied in the _service file. This patch corrects the
issue with the services data structure so that the service data isn't
overridden if it is defined in the _service file.
A side effect of this correction for services defined in the _service
file is that instead of overriding the service mode with '', the mode is
taken from the _service file. When using the "run" command, this would
mean that the call mode of None may not be in agreement with the service
mode defined in the file, e.g. "manual", and so the "run" command would
no longer cause it to run when it would before. We can take this
opportunity to define this as the correct behavior - the "run" command
now only runs services with "trylocal", "localonly", or no mode set -
and also ensure that other call mode commands result in sensible
behavior when called with a service name, for instance "osc service
manualrun download_files" will run only services with mode="manual" and
name="download"files" instead of all services with mode="manual".
Additionally, services that aren't defined in the _service file can be
called with a call mode command and will use that call mode rather than
None.
Add a "manual" service mode. A service with mode "manual" is not executed
by default (that is, via "osc service run"). As of now, "manual" is in
some sense just an alias for "disabled". However, this might change in the
future [1]. Also, "localrun" now executes services with mode "serveronly".
Moreover, the documentation of "disabledrun" is updated ("disabledrun"
never executed services with mode "serveronly"). Additionally, "localrun"
and "disabledrun" are marked as "[n]ot for common usage anymore" in the
service command's description.
The rationale for these changes is (partly) described in [1]. The main
motivation is to add some clarity (in contrast to the "disabled" mode,
it is probably easier to get/guess the semantics of the "manual" mode).
[1] https://github.com/openSUSE/osc/pull/826
The "disabledrun" service commands is marked as deprecated but has no
explicit replacement. It is still a useful command for updating packages
manually or through a CI system without being forced to run all defined
services with the "runall" command. This change adds a new command
"manualrun" and a new mode "manual" which behave the same as the
deprecated "disabledrun" command and "disabled" mode but have clearer
meaning. "manualrun" does not attempt backwards-compatible behavior with
the "disabledrun" mode for "disabled" services because "disabled" mode
may eventually be removed or change meaning. The "localrun" command is
enhanced to consider the "serveronly" mode. Since "disabledrun" never
executed services with mode "serveronly", its docs are updated
accordingly.
Improve error message in do_service a bit. The old "Too few arguments."
was misleading (for instance, if a non-remote command was not executed
in a package wc).
Note: with the new logic we could also get rid of the
"raise oscerr.WrongArgs('Local directory is no package')" statement.
Add a --status-filter option to "osc results" that can be used to
show, for instance, only the repos where a package failed to build. As
a short circuit, a -f/--failed option is added, too.
Add a --brief option to "osc prjresults" and "osc results" that can be
used to get a more compact representation of the results. In case of
"osc results", --brief is ignored if the results for a package are
requested.
in do_results:
* add --brief option on prj level:
[packagename] [repo] [arch] [buildstatus]
* filter by --status-filter <long status name>
works on prj and pkg level
in do_prjresults:
* --brief
* assume len(state)>1 as long state
core.py
* filter packages by build status
* long status handling in get_prj_results
* brief output generation in get_prj_results
Interprete unicode escape characters in a "--message" option. In some
cases, this breaks the existing UI (but that's OK because it can be
fixed by properly escaping the escape character(s)).
Note: if we are going to do more advanced stuff in the future, we should
move the logic into a separate function.
In commit 276d6e2439 ("Do not use the
chardet module in util.helper.decode_it") util.helper.decode_it was
changed to always decode the passed object if it has a decode method.
Since a python2 str has a decode method, the new code tries to utf-8
decode the passed str. As a result, a unicode object is returned (if
the decoding worked). Since a unicode object is not an instance of
type str, all subsequent isinstance(decoded_obj, str) checks evaluate
to False, which break some codepaths.
In order to fix this, restore the old python2 behavior (that is, if
the passed object is a str, it is not decode it). This change does not
affect the python3 codepaths.
Fixes: #814 ("osc log | fails")
Improve "osc rdiff --issues-only ..." output: now, it shows the added,
deleted and changed issues. Also, add a new "osc rdiff --xml ..." option,
which only works in combination with the "--issues-only" option: it prints
the raw xml.
Note: server_diff_noex has no option for the "full" parameter. Hence,
with the addition of the "xml=False" parameter, the signatures of the
server_diff_noex and server_diff functions are going to differ "forever".
That's OK (IMHO) because it is probably more sane to simply specify the
additional args via the kwargs syntax.
Remove dead code from the fetch module. Actually, it should have
been removed in commit 95ec7dee7b
('- fixed#590606 ("osc/fetch.py does not support authenticated URLs")').
In general, decode_it is used to get a str from an arbitrary bytes
instance. For this, decode_it used the chardet module (if present)
to detect the underlying encoding (if the bytes instance corresponds
to a "supported" encoding). The drawback of this detection is that
it can take quite some time in case of a large bytes instance, which
represents no "supported" encoding (see #669 and #746).
Instead of doing a potentially "time consuming" detection, either
assume an utf-8 encoding or a latin-1 encoding. Rationale: it is just
not worth the effort to detect a _potential_ encoding because we have
no clue what the _correct_ encoding is. For instance, consider the
following bytes instance:
b'This character group is not supported: [abc\xc3\xbf]'
It represents a valid utf-8 and latin-1 encoding. What is the "correct"
one? We don't know... Even if you interpret the bytes instance as a
human you cannot give a definite answer (implicit assumption: there is
no additional context available).
That is, if we cannot give a definite answer in case of two potential
encodings, there is no point in bringing even more potential encodings
into play. Hence, do not use the chardet module.
Note: the rationale for trying utf-8 first is that utf-8 is pretty
much in vogue these days and, hence, the chances are "high" that we
guess the "correct" encoding.
Fixes: #669 ("check in huge shell archives is insanely slow")
Fixes: #746 ("Very slow local buildlog parsing")
Importing `cElementTree` has been deprecated since Python 3.3 -
importing `ElementTree` automatically uses the fastest
implementation available - and is finally removed in Python 3.9.
Importing cElementTree directly (not as part of xml) is an even
older relic, it's for Ye Time Before ElementTree Was Added To
Python and it was instead an external module...which was before
Python 2.5.
We still need to work with Python 2.7 for now, so we use a try/
except to handle both 2.7 and 3.9 cases. Also, let's not repeat
this import 12 times in one file for some reason.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This checks if the filename of a downloaded file has
been modified (for example by a MITM attack) to contain
slashes. This could mean that the file is compromised
and that the attacker tries to overwrite system files.
split the code of do_dependson into two separate commands (just for
the osc help overview)
They are doing the opposite of each other.
Duplicate code was moved to _dependson()
do_whatdependson and do_dependson just call _dependson with an option
reverse set to None or 1.
add new regex and check for missing arguments.
The error message in python3 differs from the one in python2.
python3:
do_api() missing 1 required positional argument: 'url'
python2:
do_api() takes exactly 4 arguments (3 given)
To be compatible with python2 two checks are needed.
The repodata.RepoDataQueryResult is supposed to be a bytes API and
that's what our users (see build module) expect.
Note that the repodata.RepoDataQueryResult.path method still returns
a str. That's what the rpmquery.RpmQuery, debquery.DebQuery, and
archquery.ArchQuery classes also do (if the "path" was initially
passed as a str).
Fixes: #760 ("osc build fails when called with --prefer-pkgs where the
passed directory is a repodata repository or a subdirectory of one")
The packagequery.PackageQueryResult class is supposed to provide a
bytes API. Hence, packagequery.PackageQueryResult.evr() should return
bytes instead of a str. Also, adjust the single caller in the build
module.
This is a follow-up commit for commit
6dbf103e10 ("Use html.escape instead
removed cgi.escape"), which breaks the python2 backward compatibility
(since the "html" module is not available by default) and also breaks
the code in general (due to missing html imports).
The fix is based on the proposed fix in [1].
Fixes: boo#1166537 ("osc rq accept - forwarding request causes backtrace")
[1] https://github.com/openSUSE/osc/pull/764
Fixes:
`Traceback (most recent call last):
File "/usr/bin/osc", line 41, in <module>
r = babysitter.run(osccli)
File "/usr/lib/python3.8/site-packages/osc/babysitter.py", line 64, in run
return prg.main(argv)
File "/usr/lib/python3.8/site-packages/osc/cmdln.py", line 344, in main
return self.cmd(args)
File "/usr/lib/python3.8/site-packages/osc/cmdln.py", line 367, in cmd
retval = self.onecmd(argv)
File "/usr/lib/python3.8/site-packages/osc/cmdln.py", line 501, in onecmd
return self._dispatch_cmd(handler, argv)
File "/usr/lib/python3.8/site-packages/osc/cmdln.py", line 1232, in _dispatch_cmd
return handler(argv[0], opts, *args)
File "/usr/lib/python3.8/site-packages/osc/commandline.py", line 1458, in do_submitrequest
result = create_submit_request(apiurl,
File "/usr/lib/python3.8/site-packages/osc/core.py", line 4244, in create_submit_request
cgi.escape(message))
AttributeError: module 'cgi' has no attribute 'escape'
`
`cgi.escape` was deprecated in python 3.2
On Tumbleweed, `zypper in python-keyring` installs python2 version, while `osc` runs on python3.
After this change, user will be pointed to the correct version.
The correct zst magic is b'(\xb5/\xfd' (4 bytes) (that's what obs-build
is also using).
Kudos to Tobias Ellinghaus for spotting this.
Fixes: #756 ("zst detection fails")
osc importsrcpkg -n <pacname> does not work. If the option is supplied, osc
mistakenly trys to "decode" the pac object. This patch limit the decode
call when pac is not a string.
Refactored fix based on suggestions from marcus-h
Old keyring versions of the KeyringBackend class have no name method (the
name method was added in 2016). In this case, the class name is used as the
backend name.
Use configparser.ConfigParser instead of configparser.SafeConfigParser,
since the latter will be removed in future python versions. No functional
changes because SafeConfigParser is a ConfigParser except that its __init__
prints a DeprecationWarning.
configparser.SafeConfigParser has been changed to configparser.ConfigParser
upstream in Python 3.2. Warnings are saying that the alias will be
removed in future versions.
Signed-off-by: Scott Bahling <sbahling@suse.com>
Add core.parse_meta_to_string helper to work around the insane
implementation of core.meta_exists. Since core.meta_exists may return
a list of bytes, a str, a list of str etc., we ultimately convert the
data to str before passing it ET.fromstring(...).
In case of bytes, the explicit decoding is OK because it is assumed to
be a valid utf-8 encoding (the data represents an xml).
Note: at the moment core.parse_meta_to_string is also called even if it
is not necessary (it is only necessary if the "create" parameter of a
corresponding core.meta_exists call is True).
Note 2: this is just a temporary workaround and, eventually, we will make
the implementation of core.meta_exists more reasonable. When doing so,
we will also remove "public" function core.parse_meta_to_string again.
(Yes, this breaks API but the core.meta_exists change will also break the
API in some sense - so that's OK.)
Do not pass None as a release to RpmQuery.filename. The release might
be None in case of a kiwi build (the returned canonname is not used in
case of kiwi build).
In all the cases where meta_exists returns either
string data, bytes data or a list, the output needs
to be parsed correctly.
Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>
This is very unlikely but in very rare cases this
can happen.
Builing kiwi images containing debian is one case.
In this case we do not know what is inside the kiwi
file and osc build assumes buildtype 'rpm' to generate
a package list which get's thrown away anyway.
Now we just check for release
Add a "print_web_links" option. If enabled, a webui url is printed after
certain operations. For now, it is only used in "do_submitrequest" in order
to print the webui url that can be used to view the newly created request
(note: the print_web_links option is not considered in the "cd prj; osc sr"
case).
This will print the direct url to show the created request.
New general bool option 'print_web_links' must be set to enable
this.
Right now this is only for creating requests. More to follow.
Improve the SSLError message if a version mismatch (TLSv1) is
detected and give the user a hint what is wrong.
(Basically the API does not supprt TLS <= 1.2 and the python
version does not support TLS >= 1.2)
In case of an error, core.get_user_data returns an empty list.
None is never returned. Hence, only pop data from the returned list,
if it is non-empty.
In interactive review mode:
If a diff is issued and the request is accepted with 'a -m ok'
the tmpfile with the diff will be read. This tmpfile.read() call
is now decoded properly.
When creating a submit reqeust against a project that does not
accept SR anymore a maintenance request via create_maintenance_request
is generated. With this commit the orev will be honored and the
appropriate revision will be submitted.
This fixes some regressions with osc chroot:
- osc chroot --wipe --root=/dir/ can now be called outside
a working copy
- osc chroot --noinit --root=/dir/ can now be called outside
a working copy and behaves like the old code (Just entering
the chroot without any modifications)
- The confirmation of the deletion is implemented again and thus
the --force option was implemented too.
without this patch, `importsrcpkg` breaks with the following output:
```
File "/usr/lib/python3.7/site-packages/osc/commandline.py", line 7868, in do_importsrcpkg
createPackageDir(os.path.join(project.dir, pac), project)
File "/usr/lib64/python3.7/posixpath.py", line 94, in join
genericpath._check_arg_types('join', a, *p)
File "/usr/lib64/python3.7/genericpath.py", line 151, in _check_arg_types
raise TypeError("Can't mix strings and bytes in path components") from None
TypeError: Can't mix strings and bytes in path components
```
Fixes#712
fix the lastbuildroot handling if the package is being build oustide of
a package directory and thus no .osc/ dir can be found.
We then just skip the store_write_last_buildroot and store_read_last_buildroot
calls.
Use the distro module (if available) for guessing the linux distribution.
In case of python < 3.8, use the platform.linux_distribution() fallback if
no distro module is available.
Rationale: platform.linux_distribution() was dropped in python3.8
