Always send the sha256sums of all tracked files in case of a
frozen package wc. For instance, this is needed if the package is
a plain link (no branch) because in this case the backend might
request a sha256sum for a tracked but unmodified file (this can
happen because the backend cannot expand the link).
The new behavior is in line with a pulled/linkrepair package wc.
Fixes: #924 ("Transmitting file data There is no sha256 sum for
file")
When trying to commit a non-existent package via Project.commit it
is treated as an external package (because a non-existent package
has no "state" inside the project). That is, Project.commitExtPackage
is called, which fails with a FileNotFoundError in case of a
non-existent package (and the traceback is printed to the user).
In order to fix this, treat a non-existent package as broken package.
That is, simply print an info message and do not error out with a
traceback (note: the commit is _not_ aborted).
Fixes: #920 ("osc commit should fail gracefully in case of
nonexistent filename")
Sccache is an alternate build caching system to ccache/icecream. It
supports C, C++ and Rust. It can optionally have distributed or remote
caches via redis, s3 object stores, memcached, azure storage or
google cloud storage.
This can help to significantly improve the performance of Rust rebuilds.
For example, Kanidm changes from 400s to 122s on a rebuild, and rust-lang
rebuilds improve from 7200s to 4770s. With some changes to the rust
packages especially this will be possible to speed up over version
changes as well.
See also: obs-build PR https://github.com/openSUSE/obs-build/pull/680
Do not use a preinstallimage if the local build is executed as a non-root
(the preinstallimage contains device nodes which usually cannot be created
by a non-root user - this is not a problem in the non-preinstallimage
codepath (see [1])).
[1] https://github.com/openSUSE/osc/pull/908#issuecomment-806903856
The old code only supports a file whose size is less then or equal
to INT_MAX (due to a reasonable(!) limit in M2Crypto). The actual
issue is in core.http_request which mmap(...)s the file, wraps it
into a memoryview/buffer and then passes the memoryview/buffer to
urlopen. Eventually, the whole memoryview/buffer is read into memory
(see m2_PyObject_GetBufferInt). If the file is too large (> INT_MAX),
m2_PyObject_GetBufferInt raises a ValueError (which is perfectly
fine!).
Reading a whole file into memory is completely insane. In order to
avoid this, we now simply pass a file-like object to urlopen (more
precisely, the file-like object is associated with the Request
instance that is passed to urlopen). The advantange is that the
file-like object is processed in chunks of 8192 bytes (see
http.client.HTTPConnection) (that is, only 8192 bytes are read into
memory (instead of the whole file)).
There are two pitfalls when passing a file-like object to urlopen:
* By default, a chunked Transfer-Encoding is applied. It seems that
some servers (like api.o.o) do not like this (PUTing a file with
a chunked Transfer-Encoding to api.o.o results in status 400). In
order to avoid a chunked Transfer-Encoding, we explicitly set a
Content-Length header (we also do this in the non-file case (just
for the sake of completeness)).
* If the request fails with status 401, it is retried with an
appropriate Authorization header. When retrying the request, the
file's offset has to be repositioned to the beginning of the file
(otherwise, a 0-length body is sent which most likely does not
match the Content-Length header).
Note: core.http_request's "data" and "file" parameters are now mutually
exclusive because specifying both makes no sense (only one of them
is considered) and it simplifies the implementation a bit.
Fixes: #202 ("osc user authentification seems to be broken with last
commit")
Fixes: #304 ("osc ci - cannot handle more than 2 GB file uploads")
This kind of guessing can not really work here and leads to failing
builds when using KVM. (eg. when using a preinstallimage)
Removing the code, since we have a now a way to allow the user to
specify building as user via su-wrapper config
Element.getchildren is deprecated and not available on python39
anymore. Instead, iterate over the element itself (which iterates
over the element's children).
Fixes: #903 ("AttributeError: 'xml.etree.ElementTree.Element' object
has no attribute 'getchildren'")
Most osc commands support slash notation for the specification of
a project package pair. That is, "osc <cmd> prj/pkg" has the same
semantics as "osc <cmd> prj pkg" (in most cases).
For consistency reasons, "osc creq" should also support the slash
notation for the action type's arguments. That is, for instance,
"osc creq -a submit src_prj/src_pkg dst_prj/dst_pkg" should have the
same effect as "osc creq -a submit src_prj src_pkg dst_prj dst_pkg".
Proposed-by: darix
If there are existing requests that should be superseded, the old
code stores the Request instances in the myreqs list, which is
returned to the caller. However, the caller expects only request
ids instead of instances of class Request. Eventually, this results
in a type error - excerpt:
...
File "/usr/lib/python3.8/site-packages/osc/commandline.py", line 1892, in do_createrequest
change_request_state(apiurl, srid, 'superseded',
File "/usr/lib/python3.8/site-packages/osc/core.py", line 4322, in change_request_state
u = makeurl(apiurl,
File "/usr/lib/python3.8/site-packages/osc/core.py", line 3326, in makeurl
return urlunsplit((scheme, netloc, '/'.join([path] + list(l)), query, ''))
TypeError: sequence item 2: expected str instance, Request found
Hence, simply return the request ids instead of the Request instances.
Note: this changes the API of the Osc._submit_request method but
this is OK because it is not part of the public API.
When calling "osc creq -a prj1 foo prj2 bar -a submit prj1 bar prj2 bar",
the requests that could be superseded are calculated two times for the
prj2/bar package. Hence, they could end up two times in the "supersede"
list (see do_createrequest) In order to avoid duplicates, use a set
instead of a list.
Kudos to darix for pointing this out!
Note: it is a bit questionable if osc's current semantics makes sense
in the above example.
When creating a new request via the core.Request.create method, there is
no need to escape the data that is assigned to the "description" attribute
of a core.Request instance. Internally, core.Request.create ensures that
the data, which is POSTed to the api, is correctly escaped (the escaping
is implicitly done by ET (see core.Request.to_str)). Manually escaping the
description results in a double escaping (the escaped description is
escaped by ET again) - this is not the desired behavior.
Analogously, there is no need to escape the data that is passed to the
message parameter of the core.create_submit_request function because
core.create_submit_request takes care of escaping it.
Fixes: #869 ("Silly encoding of htmlencodable entities")
Using os.getcwd() in combination with a subsequent .encode() is error
prone:
marcus@linux:~> mkdir illegal_utf-8_encoding_$'\xff'_dir
marcus@linux:~> cd illegal_utf-8_encoding_$'\xff'_dir/
marcus@linux:~/illegal_utf-8_encoding_ÿ_dir> python3
Python 3.8.6 (default, Nov 09 2020, 12:09:06) [GCC] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.getcwd().encode()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcff' in position 36: surrogates not allowed
>>>
Hence, use os.getcwdb(), which returns a bytes, instead of
os.getcwd().encode().
Fixes: commit 36f7b8ffe9 ("Fix a
potential TypeError in CpioRead.copyin and CpioRead.copyin_file")
If no dir is passed to util.ArFile.saveTo, dir is set to os.getcwd(),
which returns a str. Since self.name is a bytes, the subsequent
os.path.join(dir, self.name) results in a TypeError.
To fix this, use os.getcwdb(), which returns a bytes instead of a
str.
This allows to utilise support for systemd-nspawn backend in build engine.
Like LXC, systemd-nspawn creates isolated lightweight container.
Signed-off-by: Oleg Girko <ol@infoserver.lv>
If no "dest" argument is specified when calling CpioRead.copyin or
CpioRead.copyin_file, a TypeError occurs in CpioRead._copyin_file
because os.getcwd(), which returns a str, is used as dest and, hence,
the subsequent os.path.join(...) fails (because it tries to join a
str and a bytes).
In order to avoid this, encode the result of os.getcwd().
Note that the existing
archive.copyin_file(hdr.filename,
os.path.dirname(tmpfile),
os.path.basename(tmpfile))
was OK because CpioRead._copyin_file os.path.join()s "dest" and
"new_fn", which are both str. It is just changed to stress that
CpioRead is a bytes-only API.
Fixes: #865 ("Traceback in osc/util/cpio.py line 128: TypeError:
Can't mix strings and bytes in path components")
Currently, when trying to initialize a non existent (server-side)
project via "osc init <prj>", osc errors out (after creating the wc)
because it fails to retrieve the package list. However, there is no
need to retrieve the package list in the "osc init <prj>" case. Hence,
skip the package list retrieval. As a result, osc does not error out.
For the background, see the discussion in #858 ("osc fails to check
out an empty project as project") [1].
[1] https://github.com/openSUSE/osc/issues/858#issuecomment-722330024
If meta=True is passed to checkRevision, the meta parameter is used
as a revision in the show_upstream_rev call. Instead, it should be
bound to show_upstream_rev's meta parameter.
Some services expect "old" service files (that is, files from a
previous service run) to be present in an ".old" dir. Hence, osc
should support that.
Instead of removing all files from a previous service run, move them
to the ".old" dir, run the services, and, finally, remove the ".old"
dir.
Unfortunately, the location of the ".old" dir is hardcoded in the
specific services. That is, we have to be careful if an ".old" dir
exists (in this case, we error out).
Based on [1].
[1] https://github.com/openSUSE/osc/pull/846
Currently, if the --offline option is passed to "osc build ...", a
preinstallimage is not used (even if it exists). Instead, a
preinstallimage should be used (if it exists) even if the --offline
option is specified.
This is faster in best case since the binary search does not need
to be executed on the server.
It also finds package names where no binary with that name exists.
(as for some multibuild cases)
