The old code does not support the --binary option in combination
with the --verbose option. Specifying --binary and --verbose at
the same time results in a crash (because the binary listing
contains no <title>...</title> element).
In order to fix this, do not try to access a <title>...</title>
element when --binary and --verbose are both specified. Instead,
in this case, include information about the repo, arch, version,
and release of the corresponding binary element.
Fixes: #933 ("osc se -v -B crash")
The old code uses a variable .rXYZ suffix (where XYZ is the revision
of the package wc during the merge operation). Now, if Package.mergefile
is invoked during an update, XYZ represents the "old" revision. That
is, if a merge conflict happens, then a subsequent "osc resolved <file>"
will not unlink the <file>.rXYZ file (because
Package.clear_from_conflictlist only takes the current rev into account).
In order to fix this, use a fixed ".new" suffix. This way,
Package.clear_from_conflictlist can properly unlink the corresponding
*.new file. This naming scheme for the "upfilename" is in line with
"osc pull" and "osc repairlink".
Note: if a working copy was updated with an "old" osc version (without
this commit) and a "new" osc version (with this commit) is used to run
"osc resolved <file>", then the <file>.rXYZ file is _NOT_ removed (it
is not worth the effort to add compat code for this).
A password can be deleted via "osc config -d <apiurl> pass". Actually,
if we really want to support password deletion, we should introduce
a --delete-password option because the "pass" config option can be
considered as an implementation detail, which we should not expose
to our users.
The password store can be changed (without entering the password
again) via "osc config <apiurl> --select-password-store". This
command deletes the password from the current password store and
stores it in the selected password store.
Previously, the --select-password-store option had no meaningful
semantics. In order to use it, one always had to provide a password
and explicitly pass "pass" as the config option (the same could be
achieved by using --change-password). Hence, in a strict sense,
this change breaks the UI.
Without the slash splitting, "osc browse prj/pkg" interprets the
argument as a project, which is wrong. Hence, perform the slash
splitting (as most commands do).
Always send the sha256sums of all tracked files in case of a
frozen package wc. For instance, this is needed if the package is
a plain link (no branch) because in this case the backend might
request a sha256sum for a tracked but unmodified file (this can
happen because the backend cannot expand the link).
The new behavior is in line with a pulled/linkrepair package wc.
Fixes: #924 ("Transmitting file data There is no sha256 sum for
file")
When trying to commit a non-existent package via Project.commit it
is treated as an external package (because a non-existent package
has no "state" inside the project). That is, Project.commitExtPackage
is called, which fails with a FileNotFoundError in case of a
non-existent package (and the traceback is printed to the user).
In order to fix this, treat a non-existent package as broken package.
That is, simply print an info message and do not error out with a
traceback (note: the commit is _not_ aborted).
Fixes: #920 ("osc commit should fail gracefully in case of
nonexistent filename")
Sccache is an alternate build caching system to ccache/icecream. It
supports C, C++ and Rust. It can optionally have distributed or remote
caches via redis, s3 object stores, memcached, azure storage or
google cloud storage.
This can help to significantly improve the performance of Rust rebuilds.
For example, Kanidm changes from 400s to 122s on a rebuild, and rust-lang
rebuilds improve from 7200s to 4770s. With some changes to the rust
packages especially this will be possible to speed up over version
changes as well.
See also: obs-build PR https://github.com/openSUSE/obs-build/pull/680
Do not use a preinstallimage if the local build is executed as a non-root
(the preinstallimage contains device nodes which usually cannot be created
by a non-root user - this is not a problem in the non-preinstallimage
codepath (see [1])).
[1] https://github.com/openSUSE/osc/pull/908#issuecomment-806903856
The old code only supports a file whose size is less then or equal
to INT_MAX (due to a reasonable(!) limit in M2Crypto). The actual
issue is in core.http_request which mmap(...)s the file, wraps it
into a memoryview/buffer and then passes the memoryview/buffer to
urlopen. Eventually, the whole memoryview/buffer is read into memory
(see m2_PyObject_GetBufferInt). If the file is too large (> INT_MAX),
m2_PyObject_GetBufferInt raises a ValueError (which is perfectly
fine!).
Reading a whole file into memory is completely insane. In order to
avoid this, we now simply pass a file-like object to urlopen (more
precisely, the file-like object is associated with the Request
instance that is passed to urlopen). The advantange is that the
file-like object is processed in chunks of 8192 bytes (see
http.client.HTTPConnection) (that is, only 8192 bytes are read into
memory (instead of the whole file)).
There are two pitfalls when passing a file-like object to urlopen:
* By default, a chunked Transfer-Encoding is applied. It seems that
some servers (like api.o.o) do not like this (PUTing a file with
a chunked Transfer-Encoding to api.o.o results in status 400). In
order to avoid a chunked Transfer-Encoding, we explicitly set a
Content-Length header (we also do this in the non-file case (just
for the sake of completeness)).
* If the request fails with status 401, it is retried with an
appropriate Authorization header. When retrying the request, the
file's offset has to be repositioned to the beginning of the file
(otherwise, a 0-length body is sent which most likely does not
match the Content-Length header).
Note: core.http_request's "data" and "file" parameters are now mutually
exclusive because specifying both makes no sense (only one of them
is considered) and it simplifies the implementation a bit.
Fixes: #202 ("osc user authentification seems to be broken with last
commit")
Fixes: #304 ("osc ci - cannot handle more than 2 GB file uploads")
This kind of guessing can not really work here and leads to failing
builds when using KVM. (eg. when using a preinstallimage)
Removing the code, since we have a now a way to allow the user to
specify building as user via su-wrapper config
Element.getchildren is deprecated and not available on python39
anymore. Instead, iterate over the element itself (which iterates
over the element's children).
Fixes: #903 ("AttributeError: 'xml.etree.ElementTree.Element' object
has no attribute 'getchildren'")
Most osc commands support slash notation for the specification of
a project package pair. That is, "osc <cmd> prj/pkg" has the same
semantics as "osc <cmd> prj pkg" (in most cases).
For consistency reasons, "osc creq" should also support the slash
notation for the action type's arguments. That is, for instance,
"osc creq -a submit src_prj/src_pkg dst_prj/dst_pkg" should have the
same effect as "osc creq -a submit src_prj src_pkg dst_prj dst_pkg".
Proposed-by: darix
If there are existing requests that should be superseded, the old
code stores the Request instances in the myreqs list, which is
returned to the caller. However, the caller expects only request
ids instead of instances of class Request. Eventually, this results
in a type error - excerpt:
...
File "/usr/lib/python3.8/site-packages/osc/commandline.py", line 1892, in do_createrequest
change_request_state(apiurl, srid, 'superseded',
File "/usr/lib/python3.8/site-packages/osc/core.py", line 4322, in change_request_state
u = makeurl(apiurl,
File "/usr/lib/python3.8/site-packages/osc/core.py", line 3326, in makeurl
return urlunsplit((scheme, netloc, '/'.join([path] + list(l)), query, ''))
TypeError: sequence item 2: expected str instance, Request found
Hence, simply return the request ids instead of the Request instances.
Note: this changes the API of the Osc._submit_request method but
this is OK because it is not part of the public API.
When calling "osc creq -a prj1 foo prj2 bar -a submit prj1 bar prj2 bar",
the requests that could be superseded are calculated two times for the
prj2/bar package. Hence, they could end up two times in the "supersede"
list (see do_createrequest) In order to avoid duplicates, use a set
instead of a list.
Kudos to darix for pointing this out!
Note: it is a bit questionable if osc's current semantics makes sense
in the above example.
When creating a new request via the core.Request.create method, there is
no need to escape the data that is assigned to the "description" attribute
of a core.Request instance. Internally, core.Request.create ensures that
the data, which is POSTed to the api, is correctly escaped (the escaping
is implicitly done by ET (see core.Request.to_str)). Manually escaping the
description results in a double escaping (the escaped description is
escaped by ET again) - this is not the desired behavior.
Analogously, there is no need to escape the data that is passed to the
message parameter of the core.create_submit_request function because
core.create_submit_request takes care of escaping it.
Fixes: #869 ("Silly encoding of htmlencodable entities")
Using os.getcwd() in combination with a subsequent .encode() is error
prone:
marcus@linux:~> mkdir illegal_utf-8_encoding_$'\xff'_dir
marcus@linux:~> cd illegal_utf-8_encoding_$'\xff'_dir/
marcus@linux:~/illegal_utf-8_encoding_ÿ_dir> python3
Python 3.8.6 (default, Nov 09 2020, 12:09:06) [GCC] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.getcwd().encode()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcff' in position 36: surrogates not allowed
>>>
Hence, use os.getcwdb(), which returns a bytes, instead of
os.getcwd().encode().
Fixes: commit 36f7b8ffe9 ("Fix a
potential TypeError in CpioRead.copyin and CpioRead.copyin_file")
If no dir is passed to util.ArFile.saveTo, dir is set to os.getcwd(),
which returns a str. Since self.name is a bytes, the subsequent
os.path.join(dir, self.name) results in a TypeError.
To fix this, use os.getcwdb(), which returns a bytes instead of a
str.
This allows to utilise support for systemd-nspawn backend in build engine.
Like LXC, systemd-nspawn creates isolated lightweight container.
Signed-off-by: Oleg Girko <ol@infoserver.lv>
If no "dest" argument is specified when calling CpioRead.copyin or
CpioRead.copyin_file, a TypeError occurs in CpioRead._copyin_file
because os.getcwd(), which returns a str, is used as dest and, hence,
the subsequent os.path.join(...) fails (because it tries to join a
str and a bytes).
In order to avoid this, encode the result of os.getcwd().
Note that the existing
archive.copyin_file(hdr.filename,
os.path.dirname(tmpfile),
os.path.basename(tmpfile))
was OK because CpioRead._copyin_file os.path.join()s "dest" and
"new_fn", which are both str. It is just changed to stress that
CpioRead is a bytes-only API.
Fixes: #865 ("Traceback in osc/util/cpio.py line 128: TypeError:
Can't mix strings and bytes in path components")
