Additional changes:
* rename getbinaries --debug to --debuginfo to avoid conflicts
* switch conf['verbose'] from int to bool
The changes were needed because the new argparser behaves differently
and commands such as `osc ls` ran in verbose mode by default.
If `sshkey` config option is set, then osc prefers it over password auth.
If `sshkey` config option is not set and the server supports both basic
and signature auth, basic auth is used and ssh key is NOT auto-detected.
Users who want to use ssh auth with ssh key auto-detection can now leave
the `pass` config option empty to trigger ssh key auto-detection.
The ssh-key autodetection picks the first key that matches:
- key loaded to ssh-agent (`ssh-add -l`) that has a public key in ~/.ssh
- ~/.ssh/{id_ed25519,id_rsa}
It is also recommended to use Obfuscated or Plaintext credentials manager.
Please be aware that storing passwords using these credentials managers
is unsafe, because they're stored in plain text on disk.
Example:
[<apiurl>]
user=<username>
pass=
# ssh key is auto-detected because `pass` is empty
sshkey=
credentials_mgr_class=osc.credentials.ObfuscatedConfigFileCredentialsManager
When using ssh keys from gpg, there are no private key files on
disk. The public keys are available from "ssh-add -L". Conveniently,
users store the public keys in some ".pub" file under ~/.ssh
(see e.g. https://serverfault.com/questions/906871/force-the-use-of-a-gpg-key-as-an-ssh-key-for-a-given-server;
this is also necessary to use IdentityFile= in ssh itself).
Thus public key files can't be ignored any more in list_ssh_dir_keys().
"ssh-keygen -Y sign" works nicely with a public key file if the agent
has access to the private key.
* init command is working inside of a git repository
* downloadassets command fetches references assets from build description
* checkout is cloning from git
This changes the code back to retrying up to 5 times for old
python version 2.6.6-2.7.9. The complete backport of the basic auth
changes clutters up the code way to much for such a little gain.
(This basically reverts commit 326abe0c8b)
Only ask for a password if it is really needed for authentication.
The new lazy password approach is much smarter than the old callable
hack. That's why we deprecate returning a callable from
AbstractCredentialsManager.get_password. The current compatibility code
for a callable will be removed in the near future.
Minor nitpick: actually it would have been "cleaner" to introduce a new
subclass like an AbstractLazyPasswordCredentialsManager that encapsulates
the lazy password behavior. Currently, if, for instance, a credentials
manager is always non-lazy it would just override get_password but still
inherits the abstract (and unused) _get_password method.
In many cases the session cookie is already available, so there
is no need to ask for a password. To make this work with the
python authentication implementation, we add a small proxy object
for the password and only ask the credential manager if the
stringify method is called.
This approach also makes it possible to offer a non-password based
authorization type if the server allows multiple authentication
methods.
This allows a format to be specified in the sccache uri, specifically
the file uri so that a per-package cache can be created. This way
an osc build locally doesn't ruin your cache moving between different
packages.
When a user creates a symlink pointing from ~/.config/osc/oscrc
to a different location, don't overwrite the symlink but follow
it when writing configuration on disk.
os.sysconf is not available on all platforms (like Windows) but it
is used to retrieve the number of online processors. If missing,
assume one processor (building on such a platform will most likely
not work, though).
Fixes: #948 ("Windows compatibility") (at least it improves the
Windows support a bit)
The order is now:
- ~/.osc_cookiejar, if it exists
- $XDG_STATE_HOME/osc/cookiejar if XDG_STATE_HOME neither null nor empty
- ~/.local/state/osc/cookiejar
A password can be deleted via "osc config -d <apiurl> pass". Actually,
if we really want to support password deletion, we should introduce
a --delete-password option because the "pass" config option can be
considered as an implementation detail, which we should not expose
to our users.
The password store can be changed (without entering the password
again) via "osc config <apiurl> --select-password-store". This
command deletes the password from the current password store and
stores it in the selected password store.
Previously, the --select-password-store option had no meaningful
semantics. In order to use it, one always had to provide a password
and explicitly pass "pass" as the config option (the same could be
achieved by using --change-password). Hence, in a strict sense,
this change breaks the UI.
Sccache is an alternate build caching system to ccache/icecream. It
supports C, C++ and Rust. It can optionally have distributed or remote
caches via redis, s3 object stores, memcached, azure storage or
google cloud storage.
This can help to significantly improve the performance of Rust rebuilds.
For example, Kanidm changes from 400s to 122s on a rebuild, and rust-lang
rebuilds improve from 7200s to 4770s. With some changes to the rust
packages especially this will be possible to speed up over version
changes as well.
See also: obs-build PR https://github.com/openSUSE/obs-build/pull/680
The old code only supports a file whose size is less then or equal
to INT_MAX (due to a reasonable(!) limit in M2Crypto). The actual
issue is in core.http_request which mmap(...)s the file, wraps it
into a memoryview/buffer and then passes the memoryview/buffer to
urlopen. Eventually, the whole memoryview/buffer is read into memory
(see m2_PyObject_GetBufferInt). If the file is too large (> INT_MAX),
m2_PyObject_GetBufferInt raises a ValueError (which is perfectly
fine!).
Reading a whole file into memory is completely insane. In order to
avoid this, we now simply pass a file-like object to urlopen (more
precisely, the file-like object is associated with the Request
instance that is passed to urlopen). The advantange is that the
file-like object is processed in chunks of 8192 bytes (see
http.client.HTTPConnection) (that is, only 8192 bytes are read into
memory (instead of the whole file)).
There are two pitfalls when passing a file-like object to urlopen:
* By default, a chunked Transfer-Encoding is applied. It seems that
some servers (like api.o.o) do not like this (PUTing a file with
a chunked Transfer-Encoding to api.o.o results in status 400). In
order to avoid a chunked Transfer-Encoding, we explicitly set a
Content-Length header (we also do this in the non-file case (just
for the sake of completeness)).
* If the request fails with status 401, it is retried with an
appropriate Authorization header. When retrying the request, the
file's offset has to be repositioned to the beginning of the file
(otherwise, a 0-length body is sent which most likely does not
match the Content-Length header).
Note: core.http_request's "data" and "file" parameters are now mutually
exclusive because specifying both makes no sense (only one of them
is considered) and it simplifies the implementation a bit.
Fixes: #202 ("osc user authentification seems to be broken with last
commit")
Fixes: #304 ("osc ci - cannot handle more than 2 GB file uploads")
On Tumbleweed, `zypper in python-keyring` installs python2 version, while `osc` runs on python3.
After this change, user will be pointed to the correct version.
This will print the direct url to show the created request.
New general bool option 'print_web_links' must be set to enable
this.
Right now this is only for creating requests. More to follow.
If a python-keyring based backend is configured, but
python-keyring is not installed osc fails without giving
the user the opportunity to continue.
This introduces a new class method `create` for the AbstractCredentialsManager.
The CredentialsManagers for the backends that use a 3rd party software can
now check if the software is present in its own create method.
* Adapt do_config to use the new credentials manager implementation
and add a --change-password option which can be used to change the
password.
* Adapt config_set_option to follow the change in do_config.
* Split selection of the credentials manager descriptor to reuse it
in do_config and interactive_config_setup.
* Introduce new ConfigMissingCredentialsError which is raised in case
of missing credentials (user or password). In this case the user will
be asked to enter the new credentials.
* new module credentials.py which contains classes
and methods to set and get passwords for different
backends:
- python-keyring
- gnomekeyring
- ConfigFile based storage
The new code should be backward compatible except a minor
change in add_section (pass and passx are not removed from
the config parser). This affects only callers that do not pass
a creds_mgr_descriptor.
On initial osc call or initial osc call on new API Url
the user now can decide where to store the password (based
on the backends available on his system)
The upcoming credentials manager abstraction supersedes the old
plaintext_passwd behavior. This commits "breaks" the old behavior:
Passwords are not rewritten anymore (plaintext <-> obfuscated)
Move logic to conf module in new function interactive_config_setup
for new config file and missing apiurl section.
Dropped keyring code, because it was never used in write_initial_config()
write oscrc to the default location for user-specific configuration.
If XDG_CONFIG_HOME is not set use ~/.config/osc/oscrc which is basically the same.
If there is already a ~/.oscrc use this one (for compat reasons). Existing user
installations should not get affected by this commit.
The order is the following:
Given config with -c
config defined in OSC_CONFIG
existing ~/.oscrc
default XDG_CONFIG_HOME/osc/oscrc
In Debian and Ubuntu build is renamed to obs-build for disambiguation
purposes.
Add a simple check to use the correct paths if running on Debian and
use /usr/bin/obs-build and /usr/lib/obs-build if so.
This change makes basic authentication fail after first unsuccessful
attempt (instead of automatically retrying) and thus prevents unnecessary
failed login attempts.
This change also allows printing the contents of an HTTP 401 message to
the user.
Since python >= 2.7.9 urllib2/httplib verifies the ssl cert by default - so
make sure that this "builtin" ssl check is disabled as well if
"sslcertck" is disabled. Fixes#179.
Note: eventually, we should abandon m2crypto and use urllib2/httplib to
verify the ssl cert.
Currently osc supports API URLs without path only,
like https://api.example.com (if there is pathname, it's just ignored).
With this change API URLS with path, like https://example.com/api
are supported correctly.
This is useful for those who can't have OBS api, webui and main site
on different domains and buy separate X.509 certificates for them,
or just can't afford to have separate IP addresses for them
(please note that currently osc doesn't support TLS SNI).
Signed-off-by: Oleg Girko <ol@infoserver.lv>
Some distributions have no initrd images compatible with KVM builds and
sometimes we need to use not system kernel version.
New options in configuration file:
- build-kernel -- kernel used for VM builds
- build-initrd -- initrd image used for VM builds
The string.encode('bz2') does not work in python3. Implement
passx_encode/passx_decode functions compatible with python 2.6, 2.7 and
3.3.
Add a simple unit test.
this is left-over from 87d354e1a0
Addressing:
Traceback (most recent call last):
File "/usr/bin/osc", line 26, in <module>
r = babysitter.run(osccli)
File "/usr/lib/python2.7/site-packages/osc/babysitter.py", line 60, in run
return prg.main()
File "/usr/lib/python2.7/site-packages/osc/cmdln.py", line 335, in main
self.postoptparse()
File "/usr/lib/python2.7/site-packages/osc/commandline.py", line 136, in postoptparse
override_verbose = self.options.verbose)
File "/usr/lib/python2.7/site-packages/osc/conf.py", line 873, in get_config
add_section(conffile, url, user, passwordx)
File "/usr/lib/python2.7/site-packages/osc/conf.py", line 712, in add_section
except OscConfigParser.ConfigParser.DuplicateSectionError:
AttributeError: class OscConfigParser has no attribute 'ConfigParser'
If http_debug is set we redirect sys.stdout to an StringIO
instance in order to do some header filtering (see conf module)
so we have to use the "original" stdout for printing the certificate
information.
The most visible change in python3 - removal of print statement and all
the crufty
print >> sys.stderr, foo,
The from __future__ import print_function makes it available in python
2.6
