This allows a format to be specified in the sccache uri, specifically
the file uri so that a per-package cache can be created. This way
an osc build locally doesn't ruin your cache moving between different
packages.
When a user creates a symlink pointing from ~/.config/osc/oscrc
to a different location, don't overwrite the symlink but follow
it when writing configuration on disk.
os.sysconf is not available on all platforms (like Windows) but it
is used to retrieve the number of online processors. If missing,
assume one processor (building on such a platform will most likely
not work, though).
Fixes: #948 ("Windows compatibility") (at least it improves the
Windows support a bit)
The order is now:
- ~/.osc_cookiejar, if it exists
- $XDG_STATE_HOME/osc/cookiejar if XDG_STATE_HOME neither null nor empty
- ~/.local/state/osc/cookiejar
A password can be deleted via "osc config -d <apiurl> pass". Actually,
if we really want to support password deletion, we should introduce
a --delete-password option because the "pass" config option can be
considered as an implementation detail, which we should not expose
to our users.
The password store can be changed (without entering the password
again) via "osc config <apiurl> --select-password-store". This
command deletes the password from the current password store and
stores it in the selected password store.
Previously, the --select-password-store option had no meaningful
semantics. In order to use it, one always had to provide a password
and explicitly pass "pass" as the config option (the same could be
achieved by using --change-password). Hence, in a strict sense,
this change breaks the UI.
Sccache is an alternate build caching system to ccache/icecream. It
supports C, C++ and Rust. It can optionally have distributed or remote
caches via redis, s3 object stores, memcached, azure storage or
google cloud storage.
This can help to significantly improve the performance of Rust rebuilds.
For example, Kanidm changes from 400s to 122s on a rebuild, and rust-lang
rebuilds improve from 7200s to 4770s. With some changes to the rust
packages especially this will be possible to speed up over version
changes as well.
See also: obs-build PR https://github.com/openSUSE/obs-build/pull/680
The old code only supports a file whose size is less then or equal
to INT_MAX (due to a reasonable(!) limit in M2Crypto). The actual
issue is in core.http_request which mmap(...)s the file, wraps it
into a memoryview/buffer and then passes the memoryview/buffer to
urlopen. Eventually, the whole memoryview/buffer is read into memory
(see m2_PyObject_GetBufferInt). If the file is too large (> INT_MAX),
m2_PyObject_GetBufferInt raises a ValueError (which is perfectly
fine!).
Reading a whole file into memory is completely insane. In order to
avoid this, we now simply pass a file-like object to urlopen (more
precisely, the file-like object is associated with the Request
instance that is passed to urlopen). The advantange is that the
file-like object is processed in chunks of 8192 bytes (see
http.client.HTTPConnection) (that is, only 8192 bytes are read into
memory (instead of the whole file)).
There are two pitfalls when passing a file-like object to urlopen:
* By default, a chunked Transfer-Encoding is applied. It seems that
some servers (like api.o.o) do not like this (PUTing a file with
a chunked Transfer-Encoding to api.o.o results in status 400). In
order to avoid a chunked Transfer-Encoding, we explicitly set a
Content-Length header (we also do this in the non-file case (just
for the sake of completeness)).
* If the request fails with status 401, it is retried with an
appropriate Authorization header. When retrying the request, the
file's offset has to be repositioned to the beginning of the file
(otherwise, a 0-length body is sent which most likely does not
match the Content-Length header).
Note: core.http_request's "data" and "file" parameters are now mutually
exclusive because specifying both makes no sense (only one of them
is considered) and it simplifies the implementation a bit.
Fixes: #202 ("osc user authentification seems to be broken with last
commit")
Fixes: #304 ("osc ci - cannot handle more than 2 GB file uploads")
On Tumbleweed, `zypper in python-keyring` installs python2 version, while `osc` runs on python3.
After this change, user will be pointed to the correct version.
This will print the direct url to show the created request.
New general bool option 'print_web_links' must be set to enable
this.
Right now this is only for creating requests. More to follow.
If a python-keyring based backend is configured, but
python-keyring is not installed osc fails without giving
the user the opportunity to continue.
This introduces a new class method `create` for the AbstractCredentialsManager.
The CredentialsManagers for the backends that use a 3rd party software can
now check if the software is present in its own create method.
* Adapt do_config to use the new credentials manager implementation
and add a --change-password option which can be used to change the
password.
* Adapt config_set_option to follow the change in do_config.
* Split selection of the credentials manager descriptor to reuse it
in do_config and interactive_config_setup.
* Introduce new ConfigMissingCredentialsError which is raised in case
of missing credentials (user or password). In this case the user will
be asked to enter the new credentials.
* new module credentials.py which contains classes
and methods to set and get passwords for different
backends:
- python-keyring
- gnomekeyring
- ConfigFile based storage
The new code should be backward compatible except a minor
change in add_section (pass and passx are not removed from
the config parser). This affects only callers that do not pass
a creds_mgr_descriptor.
On initial osc call or initial osc call on new API Url
the user now can decide where to store the password (based
on the backends available on his system)
The upcoming credentials manager abstraction supersedes the old
plaintext_passwd behavior. This commits "breaks" the old behavior:
Passwords are not rewritten anymore (plaintext <-> obfuscated)
Move logic to conf module in new function interactive_config_setup
for new config file and missing apiurl section.
Dropped keyring code, because it was never used in write_initial_config()
write oscrc to the default location for user-specific configuration.
If XDG_CONFIG_HOME is not set use ~/.config/osc/oscrc which is basically the same.
If there is already a ~/.oscrc use this one (for compat reasons). Existing user
installations should not get affected by this commit.
The order is the following:
Given config with -c
config defined in OSC_CONFIG
existing ~/.oscrc
default XDG_CONFIG_HOME/osc/oscrc
In Debian and Ubuntu build is renamed to obs-build for disambiguation
purposes.
Add a simple check to use the correct paths if running on Debian and
use /usr/bin/obs-build and /usr/lib/obs-build if so.
This change makes basic authentication fail after first unsuccessful
attempt (instead of automatically retrying) and thus prevents unnecessary
failed login attempts.
This change also allows printing the contents of an HTTP 401 message to
the user.
Since python >= 2.7.9 urllib2/httplib verifies the ssl cert by default - so
make sure that this "builtin" ssl check is disabled as well if
"sslcertck" is disabled. Fixes#179.
Note: eventually, we should abandon m2crypto and use urllib2/httplib to
verify the ssl cert.
Currently osc supports API URLs without path only,
like https://api.example.com (if there is pathname, it's just ignored).
With this change API URLS with path, like https://example.com/api
are supported correctly.
This is useful for those who can't have OBS api, webui and main site
on different domains and buy separate X.509 certificates for them,
or just can't afford to have separate IP addresses for them
(please note that currently osc doesn't support TLS SNI).
Signed-off-by: Oleg Girko <ol@infoserver.lv>
Some distributions have no initrd images compatible with KVM builds and
sometimes we need to use not system kernel version.
New options in configuration file:
- build-kernel -- kernel used for VM builds
- build-initrd -- initrd image used for VM builds
The string.encode('bz2') does not work in python3. Implement
passx_encode/passx_decode functions compatible with python 2.6, 2.7 and
3.3.
Add a simple unit test.
this is left-over from 87d354e1a0
Addressing:
Traceback (most recent call last):
File "/usr/bin/osc", line 26, in <module>
r = babysitter.run(osccli)
File "/usr/lib/python2.7/site-packages/osc/babysitter.py", line 60, in run
return prg.main()
File "/usr/lib/python2.7/site-packages/osc/cmdln.py", line 335, in main
self.postoptparse()
File "/usr/lib/python2.7/site-packages/osc/commandline.py", line 136, in postoptparse
override_verbose = self.options.verbose)
File "/usr/lib/python2.7/site-packages/osc/conf.py", line 873, in get_config
add_section(conffile, url, user, passwordx)
File "/usr/lib/python2.7/site-packages/osc/conf.py", line 712, in add_section
except OscConfigParser.ConfigParser.DuplicateSectionError:
AttributeError: class OscConfigParser has no attribute 'ConfigParser'
If http_debug is set we redirect sys.stdout to an StringIO
instance in order to do some header filtering (see conf module)
so we have to use the "original" stdout for printing the certificate
information.
The most visible change in python3 - removal of print statement and all
the crufty
print >> sys.stderr, foo,
The from __future__ import print_function makes it available in python
2.6
Some modules (httplib, StringIO, ...) were renamed in python3. This
patch try to import the proper symbols from python3 and then fallback to
python2 in a case ImportError will appear.
There is one exception, python 2.7 got the io module with StringIO, but
it allow unicode arguments only. Therefor the old module is poked before
new one.
This is more useful and meaningful error message than M2Crypto's
"ValueError: cafile and capath can not both be None." when neither
of cafile and capath is defined for load_verify_locations call.
This way it's possible to hide requests which have a given project
as a source project when running "osc rq list project".
Original patch by darix (just a bit beautified)
