Simplify the project sr handling in do_submitrequest a bit. It also fixes
a bug in case a target project and --separate-request is specified.
Other bugs are retained for now (for instance, --update-link is ignored).
Most of the stuff is now done in the API.
For just submitting all in one request the following is sufficient:
<action type="submit"> <source project="%s" /> %s </action>
If a target project is given it looks like this:
<action type="submit"> <source project="%s" /> <target project="%s"> %s </action>
We don't need package name or source package and targets anymore.
This is all handled by the API now.
For the --seperate-requests case the check for the entries in the linkinfo
is not working on most cases. So we handle this in the do_submitrequest
and skip the package if X-Opensuse-Errorcode is missing_action which means
basically nothing to do for this package
redesign target_project handling
* Adapt do_config to use the new credentials manager implementation
and add a --change-password option which can be used to change the
password.
* Adapt config_set_option to follow the change in do_config.
* Split selection of the credentials manager descriptor to reuse it
in do_config and interactive_config_setup.
* Introduce new ConfigMissingCredentialsError which is raised in case
of missing credentials (user or password). In this case the user will
be asked to enter the new credentials.
* new module credentials.py which contains classes
and methods to set and get passwords for different
backends:
- python-keyring
- gnomekeyring
- ConfigFile based storage
The new code should be backward compatible except a minor
change in add_section (pass and passx are not removed from
the config parser). This affects only callers that do not pass
a creds_mgr_descriptor.
On initial osc call or initial osc call on new API Url
the user now can decide where to store the password (based
on the backends available on his system)
The upcoming credentials manager abstraction supersedes the old
plaintext_passwd behavior. This commits "breaks" the old behavior:
Passwords are not rewritten anymore (plaintext <-> obfuscated)
Handle multiple revisions the same as a single revision in terms of what
is allowed (digit, or 32 character string [md5]). Additionally, support
either blank or "latest" to mean latest revision (None).
This allows for new revision to be specified without old revision like the
following example:
osc rdiff ... -r :17
Move logic to conf module in new function interactive_config_setup
for new config file and missing apiurl section.
Dropped keyring code, because it was never used in write_initial_config()
python3 "workaround" for the quite fragile code in
RawCmdln._help_preprocess: It depends on the "non-deterministic" order
of preprocessor.items(). Hence, the results may differ on python2 and
python3. The real fix is probably to enforce a fixed iteration order.
See also the discussion in [1].
[1] https://github.com/openSUSE/osc/pull/604
prevent call of preprocessor if ${cmd_name} marker is accidentaly set,
but cmdname is None.
Also for cmd_option_list and cmd_usage
overwrite _help_preprocess_* functions in Osc class
After (successfully) running an 'osc vc' the following exception trace
comes up, when no rpm python module is available:
```
no changes made
Traceback (most recent call last):
File "/home/mgerstner/.local/lib64/python3.6/site-packages/osc/babysitter.py", line 62, in run
return prg.main(argv)
File "/home/mgerstner/.local/lib64/python3.6/site-packages/osc/cmdln.py", line 344, in main
return self.cmd(args)
File "/home/mgerstner/.local/lib64/python3.6/site-packages/osc/cmdln.py", line 367, in cmd
retval = self.onecmd(argv)
File "/home/mgerstner/.local/lib64/python3.6/site-packages/osc/cmdln.py", line 501, in onecmd
return self._dispatch_cmd(handler, argv)
File "/home/mgerstner/.local/lib64/python3.6/site-packages/osc/cmdln.py", line 1232, in _dispatch_cmd
return handler(argv[0], opts, *args)
File "/home/mgerstner/.local/lib64/python3.6/site-packages/osc/commandline.py", line 8924, in do_vc
sys.exit(vc.returncode)
SystemExit: 0
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/mgerstner/bin/osc", line 41, in <module>
r = babysitter.run(osccli)
File "/home/mgerstner/.local/lib64/python3.6/site-packages/osc/babysitter.py", line 172, in run
except RPMError as e:
TypeError: catching classes that do not inherit from BaseException is not allowed
```
To fix this change the fallback RPMError from None to an actual
Exception-derived type.
Since we do not know the files' encoding, the diff functions/methods
have to return bytes. Note: decoding the diff result is wrong in
general (see the discussion in [1]).
[1] https://github.com/openSUSE/osc/pull/554
The files are now opened as rb for diffing. In python2 nothing changes.
In python3 the returned diff is bytes now.
The following changes were made:
* commandline.py: The returned diff is now bytes
* get_diff now returs the diff as a bytes-like object
* run_pager writes with sys.stdout.buffer.write if message
is not a string
* for the commit message the returned diff needs to be decoded
now. Otherwise it will just producce garbage. For the commit
message the diff on decoded bytes-objects is ok. (nothing harmfull
can happen here)
* fixed submit_action_diff
* fixed request_interactive_review
use own implementation of HTTPSConnection (myHTTPSConnection)
instead the one provided by M2Crypto (httpslib.HTTPConnection)
And in proxy case use myProxyHTTPSConnection.
all credits go to wfrisch
Do not guess the decoding of the returned output on osc api calls.
This could lead to unexptected behavior. Just use sys.stdout.buffer.write()
to print the bytes-like object in python3.
if prjconfig meta is empty the class metafile init
fails with
if isinstance(input[0], str):
IndexError: list index out of range
This is because input is an empty list and has no
[0] index.
This can be fixed by changing this line to:
if input and isinstance(input[0], str):
simplified and fixed the __init__ method of the metafile class.
input can be string, bytes-like object, list of strings or list
of bytes-like objects now.
Based on the input now always a list is generated and joined to
a string for writing in the fd. (This is ugly but needed for
compat reasons)
this makes sure not to mix objects to prevent a TypeError
TypeError: Can't mix strings and bytes in path components
Fixes#567
Signed-off-by: Maximilian Meister <mmeister@suse.de>
The offline mode does not really work offline since
parse_repoarchdescr() uses get_buildconfig to store the
buildconfig in a temporary file.
Solution:
Use the same logic as in build.py. If in offline mode(noinit = True)
try to use the local _buildconfig file. If not in offline mode just
download the buildconfig from the server via get_buildconfig
On osc meta attribute --attribue <name> --delete the returned
data is encoded. Therefore the sys.stdout.write(data) call fails.
Solution: Decode data
--> sys.stdout.write(decode_it(data))
opts.quiet is passed to get_binary_file which is False by default.
So the following if progress_meter is always False unless -q is given.
So it needs to be if not progress_meter.
When running osc triggerreason on newly created (empty) packages
the command failes with AttributeError: 'NoneType' object has no attribute 'text'
because root.find('explain') is NoneType.
Solution:
Check if root.find('explain') is None and print "No triggerreaseon found".
In this case also do not try to get the triggertime. It will result in the same error.
The content in the todo dict are strings. In python3 the communicate
method expects a bytes-like object not a string.
Solution: Encode every element in todo to a new dict (enc_todo) and
pass this instead of todo
osc se did not sort the output anymore. The logic
in the existing results.sort(key=...) was wrong. Now it is using
key=itemgetter(0,1) has two columns and key=itemgetter(0) if there
is only one column.
When adding attribute with osc meta attribute <prj> -a <attr> -s '<val>'
the api call gets executed successfull but the return value needs to be
decoded to print it correctly.
In some rare cases the chardet encoding detection detects
a wrong encoding standard. Then we switch to latin-1 which
covers most if utf-8 does not work.
* use cmp_to_key from functools for python 2.7 and higer
* use self written cmp_to_key for python 2.6
* new functions compare und cmp (used in python2 and python3)
* a lot of bytestring handling and decoding
* fix slow rbl based on f.readline(bufsize)
In that case, the packages to setup the build environment are
taken from the repositories defined in the kiwi file. Osc did
not take into account that the build config must match this
path. So it cannot just get the build config like with normal
builds, but must use a different path.
This led to build errors on some projects like CentOS 7 which
rely on 'Order' statements from the project config.
The OBS backend already had support for this: the 'path' parameter
can be used to overwrite the project path in the _buildconfig
query. We now use this to provide the correct path if we
detect this case. (The detection is currently a heuristic
because OBS does not provide us with a clear indicator.)
Always error out if the source services execution failed (the old
code only errored out if the local_service_run config option was
set (this behavior was probably unintentionally introduced in commit
d3dd8539d9 ('- fallback to "build
--local-package" if it does not exist instead of complaining'))).
Also, do not suppress all exceptions (if something failed in the
services execution code path, the user should be informed).
The bug was reported by darix.
The vc_export_env call was introduced in commit
37ca5535ce ("Export vc env vars when
running a source service") and broke the testsuite. The potential
http request is useless if there are no services. Hence, avoid it.
Thanks to Marco Strigl for catching the testsuite error!
Some services (like the obs_scm) will use them. Note: if realname
or email is not set in the oscrc, an additional http request will
be performed.
This change was requested by darix.
No functional changes. Note that we cannot simply decode the control's
fields as ascii/utf-8 because a field is not necessarily a valid
ascii/utf-8 encoding (it is possible to register _arbitrary_ custom
fields via a 'register-custom-fields' hook when building a deb
package).
Note: DebQuery.debvercmp really deserves a cleanup:/
cmp(a, b) returns
-1 if a < b
0 if a == 0
1 if a > b
This is needed since python3 has no cmp function anymore.
All credits for this go to Marco Strigl <mstrigl@suse.com> (see
PR#483 [1]).
[1] https://github.com/openSUSE/osc/pull/483
The None argument is always <= than the other argument. We need this
in case of a broken/pathological package where version() or release()
return None (see vercmp (which calls rpmvercmp)).
Returning None breaks ArchQuery.vercmp. Returning b'0' is ok because
an epoch, if present, is always supposed to be an integer (at least
in a "valid" arch package (see scripts/libmakepkg/lint_pkgbuild/epoch.sh.in
in the pacman sources)). Hence, if we compare the epoch of a package,
which has no explicit epoch set, with the epoch of a package, which
has an explicit epoch set, we always have a <= relation.
If size is 0, using pb.Percentage will result in a
ZeroDivisionException. Note: the output in case of
size == 0 looks a bit "strange" - for a consistent
output we should probably subclass pb.Percentage.
* new function create_text_meter with fallback selection
* NoPBTextMeter.start() will print the basename (if not stated otherise with
basename = None)
* The callers that should use an alternare TextMeter class now call create_text_meter()
* The callers that should not use and alternate TextMeter (because of different handling,
like build.py) call create_text_meter(use_pb_fallback=False)
* the warning 'Please install the progressbar module' is now only shown once
improvements
Now, CpioWrite provides a bytes-only API. It would be also possible
that the API accepts bytes and str (we would need to explicitly
encode the latter) but this would be a bit inconsistent wrt.
cpio.CpioRead (which is bytes-only).
Also, by using a bytesarray instead of a [] we avoid several
intermediate ''.join(...)s.
This is a bytes only API because a filename in a cpio archive can
contain, for instance, illegal utf-8 sequences. A user can decode
the filename/content as she wishes.
A ValueError is more appropriate because there is no issue with the
ar archive itself. Also, the old codepath never worked because the
fn parameter was missing.
Since an ar archive can contain arbitary filenames (that is a
filename can be an invalid utf-8 encoding (for instance,
"foo\xff\xffbar")), the ar module provides a bytes only API. A
user can decode filenames as she wishes.
Note: if a "fn" parameter is passed to Ar.__init__ it should be a
bytes (a str is also ok, but then be aware that an ArError's file
attribute might be a str or a bytes).
There is no need to unpack a single byte because it is not
affected by (byte) endianness (and that's what struct.unpack is
about). Moreover, rpmquery.unpack_string now supports an optional
encoding parameter, which could be used by the python3 port to
decode a string. Note: in general we cannot assume that all strings
in a rpm are utf-8 encoded (it is possible to build a rpm that
contains illegal utf-8 sequences).
With the NoPBTextMeter class the build view gets broken.
Old view:
1/11 (repo) filename
new view:
Please install the progressbar module...
Please install the progressbar module...
Please install the progressbar module...
With this commit the old behavior is restored.
The getbinaries call now lists the file he downloads instead of just
stating "Please install the progressbar module..." several times.
(but only if not called with the option quiet)
Since commit 8a6abe3a6c ("warn user
if python-progressbar is not installed") it is always safe to import
the meter module (even if the progressbar module is not installed).
On ImportError have_pb_module is false and the class NoPBTextMeter gets
returned which prints "Please install progressbar module..." on TextMeter.start()
This functions are used in the whole code and are
mandatory for the python3 support to work. In python2
case nothing is touched.
* cmp_to_key:
converts a cmp= into a key= function
* decode_list:
decodes each element of a list. This is needed if
we have a mixed list with strings and bytes.
* decode_it:
Takes the input and checks if it is not a string.
Then it uses chardet to get the encoding.
If urlgrab returns a URLError (for example if the Network is unreachable)
the for loop did not continue and the osc build aborts.
Now we also catch the URLError and try the next mirror and return False
correctly if no mirror could be reached. And then try to download it from
api
Get rid of the urlgrabber dependency. The current implementation of the
progress bar is quite "noisy" if the line length exceeds the size of
the terminal window, but that's something we could fix later. (The
superfluous error message will be fixed in a subsequent commit.)
This is useful so one can wait with a following osc prjresult until
everything done. Without this one can not know if a release job hasn't
started yet or is already finished.
* altered command do_checkout
new option -D | --deleted. Can only be used with PACKAGE and needs
-o to work
* what happens:
core function checkout_deleted_package fetches the file list of the deleted
package and saves it to given destdir. If destdir is not existent it
creates the destination directory. Otherwise the files will be written
in the existing directory.
Print error and exit if osc -A <apiurl> sr is issued on a
package with a different apiurl in its working copy. This leads
to unexpected behavior, because the SR is created for the package
apiurl, not the apiurl given by the -A command line option.
The old code could potentially yield to a use-after-free situation,
which results in UB. For this, consider the following scenario, where
osc performs several HTTPS requests (assumption: the server supports
ssl session resumption):
- HTTPS Request 1:
* a new SSL *s connection is established, which also creates a new
SSL_SESSION *ss => ss->references == 1
* once the handshake is done, the ss is put into the session cache
(see ssl_update_cache) => ss->references == 2
- osc saves the session ss in a class variable
- s is SSL_free()d, which calls SSL_SESSION_free => ss->references == 1
- HTTPS Request 2:
* setup a new SSL *s connection that reuses the saved session ss
=> ss->references == 2
* once the handshake is done, ssl_update_cache is called, which is a
NOP, because s->hit == 1 (that is, the session was resumed)
* osc saves the session ss in a class variable
* s is SSL_free()d, which calls SSL_SESSION_free => ss->references == 1
...
> 2 hours later (see tls1_default_timeout)
...
- HTTPS Request 256:
* setup a new SSL *s connection that reuses the saved session ss
=> ss->references == 2
* once the handshake is done, ssl_update_cache is called, but is
_no_ NOP anymore
* ssl_update_cache flushes the session cache (this is done every
255/256 (depending on the way we count) connections) => ss is
SSL_SESSION_free()d => ss->references == 1
* osc saves the session ss in a class variable
* s is SSL_free()d, which calls SSL_SESSION_free:
since ss->references == 1, ss is eventually free()d
- HTTPS Request 257:
* setup a new SSL *s connection that reuses the saved session ss
Since ss does not exist anymore, the remaining program execution is UB.
(Note: SSL_free(...) is _NOT_ called, if M2Crypto 0.29 is used.
M2Crypto 0.30 calls SSL_free(...) again.)
Due to a bug in OpenSSL_1_1_0h (see openssl commit 8e405776858) the
scenario from above can be triggered with exactly 2 HTTPS requests (the
SSL_SESSION is not cached, because we configured SSL_VERIFY_PEER, but
no sid_ctx was set). This is fixed in openssl commit c4fa1f7fc01.
In order to reliably reuse a session, we probably need to listen to the
session cache changes. Such callbacks could be registered via
SSL_CTX_sess_set_new_cb and/or SSL_CTX_sess_set_remove_cb, but both
functions are not provided by M2Crypto. Another idea is to directly utilize
the session cache, but this also has to be implemented in M2Crypto first.
Yet another approach is to retrieve the session via SSL_get1_session, which
increases the session's refcnt, but this also needs to be implemented in
M2Crypto first (if we choose to use this approach, we also have to make
sure that we eventually free the session manually...).
Fixes: #398 ("SIGSEGV on \"osc commit\"")
Always pass the "--statistics" option to the build script (only affects
a vm build). It is not implemented as an option, because we already have
so many of them... (see also https://github.com/openSUSE/osc/pull/412).
yet another option, but
* only very old build scripts don't know it, we should just require a recent one
* build script is ignoring it for chroot case
so why bother with another option?
Note that the diff is only shown, if it was requested before (that is
the "i" command was issued). The new behavior is consistent with the
other commands like "a", "d" etc.
Storing the error encoding in an "encoding" attribute "breaks" the
python3 "input" function: In essence, builtin_input_impl does a
getattr(sys.stdout, 'encoding'), which returns our error encoding
instead of the "real" stdout encoding. In order to avoid this, we
store the error encoding in an "_encoding" attribute.
Making SafeWriter a new-style class simplifies the code a lot.
This is a fix for issue #385. osc commit breaks due to
the use of sfilelist.findall('.//entry[@hash]')
I now will iterate through the sfilelist and use
for entry in sfilelist.findall('entry'):
if entry.get('hash'):
... execute hash code ...
This is a little bit slower, but should not break
on SLE11 anymore
Without this change, using "--alternative-project <prj>" in combination
with "--multibuild-package <flavor>" yields to unexpected results (from
a user's point of view). Note that this may break existing (artificial)
workflows (e.g., using --alternative-project to ignore the package
meta's debuginfo flag), but these workflows should be rare and there
are options to achieve the same.
Fixes: #376 ("osc build -M something does not work with
--alternative-project")
Only include a tag if it "has" text in get_request_issues. Note
that the code in get_formatted_issues always assumes the presence
of the "label" tag.
Fixes: #369 ("crash trying to view diff of a request")
The retrun at this point breaks the call, because in most
cases <prj> is not a binary. And the code always checks for the
binary first and then returns if no binary with the name <prj>
is found.
The following abstract methods are added to the PackageQueryResult
class: recommends(), suggests(), supplements(), and enhances().
Note that not all package/metadata formats have a notion of these
weak dependencies.
rpm rpmmd deb arch
recommends x x x
suggests x x x x
supplements x x
enhances x x x
(where "x" represents "supported"). In case of an unsupported weak
dependency, the implementation returns an empty list.
We need the weak dependency support in order to fix#363 ("osc build
-p ../rpms/tw doesnt send recommends to the server which makes client
side build behave differently to server side build").
There is no good reason why "--revision <rev>" and "--expand-link" or
"--revision <rev>" and "--unexpand-link" should be mutually exclusive
during an "osc up" of a package wc.
Introduce the new "--linkrev <rev>" option to specify a rev of the link
target that is used during link expansion.
In case of a pulled/linkrepair wc, it is possible that the backend
requests a hash for a tracked file, which is neither added, restored,
nor modified. For instance, this can happen if a new file was added
to the link target. Hence, for a pulled/linkrepair wc always send
the sha256 hashes of the tracked files.
This is needed for a new validation of the source server.
The source server will 'ask' for the sha256 sum of files which are new or
modified and osc calculates the sha256 sums for those files and sends them
back to the server.
The server checks the sha256 sums and if dies if something is wrong.
Mount sysfs during "osc chroot". The current implementation
of "osc chroot" is a major pain for plain "su" users, because the
root password has to be entered several times - we should fix this.
Fixes: #354 ("Mount sysfs in chroot")
At the moment just repo.name is considered. So if
the repo is disabled for s390 all other repo / arch
combination are not shown in the repo list.
To be able to change this r is now a list of dicts
containing the name and arch of the disabled repo.
None for repo if a complete arch gets disabled
None for arch if a complete repo gets disabled
Store a newly created config file in $XDG_CONFIG_HOME/osc/. For backward
compatibility, ~/.oscrc is used, if present.
Fixes: #313 ("oscrc should be stored in $XDG_CONFIG_HOME on linux")
write oscrc to the default location for user-specific configuration.
If XDG_CONFIG_HOME is not set use ~/.config/osc/oscrc which is basically the same.
If there is already a ~/.oscrc use this one (for compat reasons). Existing user
installations should not get affected by this commit.
The order is the following:
Given config with -c
config defined in OSC_CONFIG
existing ~/.oscrc
default XDG_CONFIG_HOME/osc/oscrc
Support an xz compressed control.tar file. In case of a control.tar.xz and
a missing lzma module, an exception is thrown at runtime (for now, in order
to avoid a hard depedency to the lzma module, which is no standard module).
Similar to recent fixes in libsolv and obs-build. Since tarfile
on python2 doesn't do lzma, decompress the file into memory and
feed it as a fake file via StringIO to tarfile
Multiple context expressions are only supported since version 2.7.
It was introduced in commit f6f879d ("Fix potential shell injection
when running rpm2cpio").
Actually, there is nothing that can be injected, except the "-h"
option. However, in case rpm2cpio evolves, we are on the safe side.
Also, document the potential shell injection in the cpio call
(the comment was accidentally removed in commit dbdc712) (the
current osc code is not affected, because we never pass filenames
via *files to core.unpack_srcrpm).
It seems that the "find" binary has no way to indicate an
end of options for its arguments. Hence, we use os.walk to mimic
"find"'s behavior, which is also the cleaner solution.
Fixes: #340 ("osc add of directories does not quote the argument")
This basically reverts commit b2b59ca, because the old code performed
a "no" instead of a "yes" (see also the discussion in
https://github.com/openSUSE/osc/pull/269).
Fixes: #343 ("'osc sr --yes ...' doesn't supersede existing requests
as promised")
The old code was flawed, because, for instance,
core.show_results_meta(apiurl, project, arch=['x86_64']) resulted
in a wrong http request: GET <apiurl>/build/<project>/_result&arch=x86_64
(note the "&" instead of the correct "?"). The drawback of the new
implementation is that we have to do the proper quoting manually.
This harmonizes "osc prjresults --help" with osc's actual behavior.
Also, core.get_prj_results expects lists (or None) instead of strings
for the corresponding repo/arch parameters.
This is a follow-up commit for commit c9c0f8a. Using core.run_external
with shell=True is too error-prone.
Fixes: #340 ("osc add of directories does not quote the argument")
Also, document a potential shell injection in core.unpack_srcrpm
(via the "files" parameter), which cannot be exploited, because
"files" is not used by the current osc code.
Fixes: #340 ("osc add of directories does not quote the argument")
Provide function to show the content of rpmlint.log.
The core function get_rpmlint_log will later be used in the interactive
request mode to provide the rpmlint logs to the reviewer.
I add the osc rpmlint | rpmlintlog | lint command also.
This will filter the requests you get to review based on
the tgt_package in the action element (if not action type is 'group').
This helps reviewers who want to just review a subset of requests or
do _not_ want to review some packages.
Examples:
osc review list --interactive --target-package-filter='^python.*' -G opensuse-review-team
osc review list --interactive --target-package-filter='^(?!ghc).*' -G opensuse-review-team
by using mtime metadata
before checking digests.
This slightly changes the semantic by assuming that modified files
will always have updated timestamps.
With this change it is possible to do osc status
on a checkout of openSUSE:Factory that contains 40GB of source tarballs
in seconds instead of minutes:
time .../osc/osc-wrapper.py status > /dev/null
real 0m33.652s
user 0m32.590s
sys 0m1.060s
Without the patch it took 22x as long:
real 12m14.545s
user 1m50.084s
sys 0m20.566s
In Debian and Ubuntu build is renamed to obs-build for disambiguation
purposes.
Add a simple check to use the correct paths if running on Debian and
use /usr/bin/obs-build and /usr/lib/obs-build if so.
For now, "osc results --xml..." ignores the "--show-excluded" option
(that's what we did in the past).
Fixes: PR#297 ("results --xml causes a stack trace")
Old obs versions have no creator attribute. In this case, string
formatting operations will fail if the "creator" attribute is
initialized with None (affects some codepaths).
An alternative would be to officially resurrect the deprecated
"Request.get_creator" method, which could return something like
"unknown" if no creator information is present.
This is a follow-up fix for the commits d68507f and 7d54b5c.
don't do a second wipe all after doing the intended wipe by a filter.
Eg: osc wipebinaries home:adrianSuSE -r standard -a i586 --build-disabled
wiped first --build-disabled only, but did a second wipe all afterwards
request creator is only delivered by OBS 2.8 and newer. Makeing this not a hard
condition therefor. (introduced in d68507fa95)
Also fixes test suite failure
Request.get_creator is used by some factory bots (see comment
in issue #286).
Note: Request.get_creator is deprecated and the "creator" attribute
should be used instead.
This implements the Request api change that was proposed in commit
6965dc5 ("Adjust request testcases to the upcoming Request api
change").
Fixes: #286 ("get_creator() does not return request creator")
Deleting a conflict file (state 'C') during an update operation results
in an inconsistent working copy. To fix this, we remove the conflict
file from the _in_conflict list.
If a package does not exist on the server, the
Serviceinfo.getProjectGlobalServices call results in a 404. In this
case, we try it again with the _project package (note: this does not
take potential "linked" services into account, if the local package
is a _link).
Fixes: #277 ("404 on running service for non commited package")
Do not pass a family parameter to SSL.Connection's constructor if it
does not support it. If the family parameter is not supported, we
_try_ to fallback to socket.AF_INET, which is implicitly used by
older versions of the SSL.Connection class.
Fixes: #274 ("osc 0.157 Exception AttributeError: "Connection
instance has no attribute 'ssl_close_flag'"")
Adds a "--multibuild-package" option to the following commands:
buildlog, remotebuildlog, buildinfo, build, buildhistory, jobhistory,
rebuild, restartbuild/abortbuild, wipebinaries, getbinaries
This change makes basic authentication fail after first unsuccessful
attempt (instead of automatically retrying) and thus prevents unnecessary
failed login attempts.
This change also allows printing the contents of an HTTP 401 message to
the user.
If build-root contains %(package) substitution, --local-package builds
would substitute absolute path there. This is different than the rule used
in osc chroot (uses relative path), causing the chroot to fail by default.
This commit removes the directory part from both build-root substitutions.
Follow-up commit for 8c45eb64ade699ce3d8717f460b3064ca3ab3a58 (just to
make sure that the "apiurl" attribute is always part of the Serviceinfo
class...)
At the moment only the project status is taken into account when determining when to stop watching a build.
This leads to wrong behavior when a package is in 'blocked' for a longer time.
In this state the project status and code is 'published' but the package remains at 'blocked'.
With this additional check this problem is fixed.
This is used to turn "dir" into a package and add it to the version
control (got broken in commit b6f7d1be6c).
Integrated the obscpio code that was introduced in commit
b6f7d1be6c into core.addFiles (XXX: we
should get rid of the run_external(..., shell=True) code).
The bogus check is twofold: the first error is due to a wrong
refactoring (repostate was intended to be the "code" attribute
and "code" was supposed to the "code" attribute of the status
node) (see commit f3a1d12a). The second error is a logic error
in the original code, because the package state "succeeded" and
the repostate "unpublished" is a valid combination. Consequently,
the check didn't make any sense...
This is up for discussion, but I try to simplify the interface here:
- obsolete "localrun" and "disabledrun". Still implemented, but not anymore
recommended and documented. can hopefully removed later.
- "runall" is running all services local, also buildtime services
- files get replaced by the service
- can be used to get final tar balls to be used with tools like quilt
This reverts commit c53a7681ef (for now!).
It seems to break local obs instances (see issue #202) (this needs
further debugging). Moreover, it breaks the python 3.4 - excerpt
from a travis run:
======================================================================
ERROR: test_added_missing2 (test_commit.TestCommit)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/opt/python/3.4.2/lib/python3.4/urllib/request.py", line 1111, in do_request_
mv = memoryview(data)
TypeError: memoryview: _io.BufferedReader object does not have the buffer interface
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/travis/build/openSUSE/osc/tests/common.py", line 122, in wrapped_test_method
test_method(*args)
File "/home/travis/build/openSUSE/osc/tests/common.py", line 122, in wrapped_test_method
test_method(*args)
File "/home/travis/build/openSUSE/osc/tests/common.py", line 122, in wrapped_test_method
test_method(*args)
File "/home/travis/build/openSUSE/osc/tests/common.py", line 122, in wrapped_test_method
test_method(*args)
File "/home/travis/build/openSUSE/osc/tests/common.py", line 122, in wrapped_test_method
test_method(*args)
File "/home/travis/build/openSUSE/osc/tests/test_commit.py", line 290, in test_added_missing2
p.commit()
File "/home/travis/build/openSUSE/osc/tests/osc/core.py", line 1471, in commit
self.put_source_file(filename, tdir)
File "/home/travis/build/openSUSE/osc/tests/osc/core.py", line 1319, in put_source_file
http_PUT(u, file = tfilename)
File "/home/travis/build/openSUSE/osc/tests/osc/core.py", line 3243, in http_PUT
def http_PUT(*args, **kwargs): return http_request('PUT', *args, **kwargs)
File "/home/travis/build/openSUSE/osc/tests/osc/core.py", line 3231, in http_request
fd = urlopen(req, data=data)
File "/opt/python/3.4.2/lib/python3.4/urllib/request.py", line 153, in urlopen
return opener.open(url, data, timeout)
File "/opt/python/3.4.2/lib/python3.4/urllib/request.py", line 453, in open
req = meth(req)
File "/opt/python/3.4.2/lib/python3.4/urllib/request.py", line 1116, in do_request_
data))
ValueError: Content-Length should be specified for iterable data of type <class '_io.BufferedReader'> <_io.BufferedReader name='/tmp/osc_test571whun4/osctest/added_missing/.osc/_in_commit/bar'>
This is up for discussion, but I try to simplify the interface here:
- obsolete "localrun" and "disabledrun". Still implemented, but not anymore
recommended and documented. can hopefully removed later.
- "runall" is running all services local, also buildtime services
- files get replaced by the service
- can be used to get final tar balls to be used with tools like quilt
Note: this commit "breaks" the existing get_package_results api, because
it returns a generator, which yields a result xml, instead of a list of
result dicts.
In this case, "osc bl repo" behaves more or less the same as "osc rbl repo"
(in case no corresponding .osc/_buildinfo file exists, we default to
repo and hostarch).
Since python >= 2.7.9 urllib2/httplib verifies the ssl cert by default - so
make sure that this "builtin" ssl check is disabled as well if
"sslcertck" is disabled. Fixes#179.
Note: eventually, we should abandon m2crypto and use urllib2/httplib to
verify the ssl cert.
The syntax to create a maintnancerequest for a single currently package
is:
osc mr SOURCEPROJECT SOURCEPACKAGES RELEASEPROJECT
which means that the source project, the package name and the release
project have to be specified on the command line.
Often times the workflow is such that the user will already be inside
of the subdirectory containing the checked out package.
To simplify the submission when the user is in a package subdirectory
this patch adds the syntax:
osc mr .
to indicate that the source project and source target is to be taken
from the meta information in this package directory.
Signed-off-by: Egbert Eich <eich@freedesktop.org>
Currently osc can't access API URLs which share the same IP address
with other SSL-enabled sites, complaining about certificate
not matching hostname.
This change solves this problem by instructing M2Crypto.SSL.Connection
to send the desired hostname to https server using TLS SNI extension,
thus allowing the server to present the right certificate and choose
the right virtual site.
This is useful for those who can't afford to have a separate IP address
for OBS API.
For TLS SNI to work correctly, M2Crypto should be patched:
https://bugzilla.osafoundation.org/show_bug.cgi?id=13073
Some distributions (like Fedora) already include this patch.
For unpatched M2Crypto osc degrades to operation without TLS SNI.
Signed-off-by: Oleg Girko <ol@infoserver.lv>
Updating a whole project against an old API server (experienced with
2.3.5) leads to:
Server returned an error: HTTP Error 400: Bad Request
unknown parameter 'nofilename'
So just retry without nofilename if an 400 is thrown.
This is using the request collection api call now instead of xpath query.
More searches should be adapted similar most likely.
"osc my sr" is not a good name for this since it is not limited to
submit actions. But "osc my rq" is used for incoming requests.
Now, the fullfilename is calculated using the canonname of a
bdep instead of using the bdep's binary attribute (the canonname
and binary attribute can differ (e.g. ConsoleKit-64bit-0.4.6-3.2.ppc.rpm
vs. ::import::ppc64::ConsoleKit-64bit-0.4.6-3.2.ppc.rpm))
Currently osc supports API URLs without path only,
like https://api.example.com (if there is pathname, it's just ignored).
With this change API URLS with path, like https://example.com/api
are supported correctly.
This is useful for those who can't have OBS api, webui and main site
on different domains and buy separate X.509 certificates for them,
or just can't afford to have separate IP addresses for them
(please note that currently osc doesn't support TLS SNI).
Signed-off-by: Oleg Girko <ol@infoserver.lv>
Currently --jobs sets only -smp flag for VM, in some cases we want
to pass threads as well. So the command line would like -smp 4,threads=4
Signed-off-by: Dinar Valeev <dvaleev@suse.com>
This is needed if mark_frozen is (indirectly) called from Project.update
(otherwise it is not clear for which project an "osc pull" has to be performed).
Some distributions have no initrd images compatible with KVM builds and
sometimes we need to use not system kernel version.
New options in configuration file:
- build-kernel -- kernel used for VM builds
- build-initrd -- initrd image used for VM builds
Renamed osc chroot's "--root" option to "--login-as-root" (kept the short
option "-r"). Added new "--root /path/to/buildroot" option (this is consistent
with "osc build --root...").
for the same projects. So do not use project "maintains", but the
attribute to be able to switch between maintenance, security and PTF
working contexts for example.
Commands like "osc service disabledrun" would always return exitcode 0
even when the source service failed. This broke any scripts which
wrapped around osc service.
This commit allows for 'osc build' to do local builds of type livebuild.
Debian livebuild is the native Debian live image building system.
Signed-off-by: Jan Blunck <jblunck@infradead.org>
It is possible that two different build types use the same package type.
Therefore we need to make get_built_files() work on the build type.
Signed-off-by: Jan Blunck <jblunck@infradead.org>
The new code tries to reduce the number of http requests for unchanged
packages (that is, packages that do not need an update).
In the worst case (the project wc's packages are unchanged and contain
services, for example), the new code needs #http requests of the old code + 1
http requests. But we can improve, once the "noservice" patch
(see http://lists.opensuse.org/opensuse-buildservice/2014-06/msg00067.html)
is merged.
Note: the semantics of the update process itself did not change.
If a time field is not complete within the same read block
then it can't be found by time_regex in buildlog_strip_time().
Fixed by simply reading line by line. I couldn't measure any
performance difference neither for real nor user time. IMO no
need to optimize for more lines per data chunk. Maybe it's
even more fluent now for interactive users.
BTW we can safely simplify time_regex.
Signed-off-by: Ruediger Meier <ruediger.meier@ga-group.nl>
The purpose of the default is to freeze the link
to the current version in the link target. It did
entirely drop the rev attribute which basically
unfreeze the link (same as -u).
* Fix osc ignoring -A apiurl command option when arguments are
less than 2 and executed within local working copy
* Enhance handling of meta command within local working copy.
- meta prj: Try to use project and apiurl of local working copy
if no arguments are passed
- meta pkg: Try to use project and apiurl of local working copy
if one argument is passed (single argument assumed to be package
name), and try to use project, package and apiurl if no
arguments are passed
If the package is linked to a different project, osc will ask
permision to branch the package in the current project before
commiting.
Show a message in osc meta pkg -e in the same case.
It seems that os.utime raises EINVAL if the file resides on a nfs
share. Since osc itself does not rely on the correct utime, we simply
ignore this exception.
The string.encode('bz2') does not work in python3. Implement
passx_encode/passx_decode functions compatible with python 2.6, 2.7 and
3.3.
Add a simple unit test.
The reason of the crash is that not all targets are always defined for
all packages. In our environment it happens when linked project is
created and sources have just been uploaded to one package.
Here is the crash:
Traceback (most recent call last):
File "../tools-testing/build-package", line 313, in <module>
sys.exit(main(sys.argv))
File "../tools-testing/build-package", line 309, in main
return build(apiurl, tproject, package, timeout)
File "../tools-testing/build-package", line 226, in build
hide_legend=True, csv=True)
File "/usr/lib/python2.7/site-packages/osc/core.py", line 5135, in get_prj_results
row = [pac] + [status[pac][tg] for tg in targets]
status[pac]]
KeyError: ('Fedora_18', 'i586', 'outdated')
Please note that this fix fixes only csv mode. For other modes osc still
can crash, but it's hard to reproduce here as we're not using them.
Signed-off-by: Ed Bartosh <bartosh@gmail.com>
Let "osc develproject" (with new alias "dp") display the devel package
if it doesn't match the package name. Deprecate "--raw" option which
only was a crude workaround for that anyway.
Add "osc setdevelproject" (alias "sdp") command to change a package's
devel project / package.
As long as the timeout is >= 0, Python's _ssl module will still use
non-blocking I/O but not poll() but select(). poll(&fd, 1, TIMEOUT)
seems to block _at least_ TIMEOUT time regardless of any errors.
Currently, with SUSE's internal IPv6 layout, this meant a hang of a
second per binary to fetch during "osc build".
Note: it is probably better to raise a ServiceRuntimeError in
Serviceinfo.execute instead of Package.commit because "execute"
has all the information about the failing service...
This is needed for backward compatibility. New plugins
(which do not care about "old" osc versions) should not
use "self.<imported modname>.<something>" anymore
to refer to the imported module. Instead use
"<imported modname>.<something>" (this will only work with
osc > 0.140.1).
This way other python scripts can execute osc commands via the
babysitter. Example:
>>> from osc import babysitter, commandline
>>> cli = commandline.Osc()
>>> babysitter.run(cli, ['osc', 'ls', '<project>'])
this is left-over from 87d354e1a0
Addressing:
Traceback (most recent call last):
File "/usr/bin/osc", line 26, in <module>
r = babysitter.run(osccli)
File "/usr/lib/python2.7/site-packages/osc/babysitter.py", line 60, in run
return prg.main()
File "/usr/lib/python2.7/site-packages/osc/cmdln.py", line 335, in main
self.postoptparse()
File "/usr/lib/python2.7/site-packages/osc/commandline.py", line 136, in postoptparse
override_verbose = self.options.verbose)
File "/usr/lib/python2.7/site-packages/osc/conf.py", line 873, in get_config
add_section(conffile, url, user, passwordx)
File "/usr/lib/python2.7/site-packages/osc/conf.py", line 712, in add_section
except OscConfigParser.ConfigParser.DuplicateSectionError:
AttributeError: class OscConfigParser has no attribute 'ConfigParser'
If http_debug is set we redirect sys.stdout to an StringIO
instance in order to do some header filtering (see conf module)
so we have to use the "original" stdout for printing the certificate
information.
Using the "old" exec approach isn't possible anymore because it'll break all
plugins which aren't compatible with python3 (for instance the usage of
python2's "print" statement will lead to an error). In order to circumvent
this problem we do the following:
- import the plugin/module
- update the module's global symbol table with the "globals()" of the
commandline module
- bind the module's "do_*" functions to the "Osc" class
This basically mimics the old "exec" semantics.
Two issues:
- There is no _find() member in ConfigLineOrder. Use _find_section()
instead
- Use 'key' instead of 'line' as argument for _find_section() since
'line' is used before assignment.
Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
We still have to support older urlgrabber versions where the class
"URLGrabber" is an old-style class and therefore we cannot use a "super(...)"
call. This fixes issue #32
given urlgrabber version 3.9.1, MirrorGroup trys to access
grabber.opts, but the derived grabber OscFileGrabber hasn't
this attribute
Signed-off-by: Gui Chen <gui.chen@intel.com>
Unfortunately the class URLGrabError does not define constants for the
different error codes therefore we have to hardcode the errno (alternatively
we can define them in our fetch module).
There are many places can't be covered by 2to3, especially the
str/unicode -> str/bytes change done in python3. This is a big patch
incorporating all changes made in order to make python3 suite.py run
without any single failure.
It
* adapt the introspect_handler_3 for case there are no __defaults__
* adds the ET_ENCODING variable for ET.fromstring ("unicode" in py3,
"utf-8" in py2)
* (re)adds various builtins to both python versions
- memoryview to python 2.6
- bytes compatible with py3 to 2.6 and 2.7
and it changes few parts of tests/common.py in order to be compatible
with python3
* new urlcompare method compares all components or url + parsed query
string in a dictionary, so the ordering, neither quoting does not matter
* bytes builtin has been added to 2.x and used in assertEqualMultiline
reintroduce unicode function - despite the fact I am pretty sure the
usage is plain wrong, because code says - return a unicode string from
utf-8 encoded one, which is probably not what was intended. I bet
something like encode the input as utf-8 was wanted.
raw_input has been removed and equals to input in py3. Unfortunatelly no
__future__ statement exists for that. Ensure all modules uses
osc.core.raw_input except osc.cmdln, where there is NameError way
implemented.
The most visible change in python3 - removal of print statement and all
the crufty
print >> sys.stderr, foo,
The from __future__ import print_function makes it available in python
2.6
Some modules (httplib, StringIO, ...) were renamed in python3. This
patch try to import the proper symbols from python3 and then fallback to
python2 in a case ImportError will appear.
There is one exception, python 2.7 got the io module with StringIO, but
it allow unicode arguments only. Therefor the old module is poked before
new one.
this patch
1.) removes the iteritems/itervalues, which were dropped in py3
items/values are used instead
2.) add an extra list() in a cases the list-based access is needed
(included appending, indexing and so)
3.) changes a sorting idiom in few places
instead of
foo = dict.keys()
foo.sort()
for i in foo:
there is a recommended
for i in sorted(dict.keys()):
4.) in one occassion it removes a if dict.has_key() by simpler
dict.get(key, default)
With the current implementation it's possible to detect if the user saved
the message file or if he/she just quits the editor (e.g. to abort the commit).
Basically it's just a wrapper around subprocess.call which raises an ExtRuntimeError
exception if subprocess.call raised an OSError with errno set to ENOENT (unfortunately
the OSError's filename attribute is set to None therefore we cannot print a meaningful
error message (that's why an ExtRuntimeError is raised)).
Replaced all occurrences of subprocess.call with a corresponding run_external call.
If the user didn't change the default template it either means that
he wants to use the default message or that he wants to abort the process.
So if the template wasn't modified osc will prompt:
Commit template was not changed
a)bort, c)ontinue, e)dit:
The new argument for osc build --host will perform the build on a remote
host. It is a shortcut for
rsync -az -e ssh `pwd` user@hostname:/remote/dir
rsync -az -e ssh prefer-dir \
user@hotname:/remote/dir/__prefer-rpms__/prefer-dir
ssh -t user@hostname "cd /remote/dir/package; osc build *build-args"
rsync -az -e ssh user@hostname:/remote/dir/__keep-pkgs__ keep-pkgs
iow it copy the current directory to the /remote/directory on hostname
(if not specified, the ~/ is supplied) and then run the osc build on
hostname. All global and local arguments are supplied to the remote osc
build, but arguments points to local path (--host, --keep-pkgs, --prefer-pkgs,
--rsync-dest, --rsync-src and --overlay) are modified to match to remote
host.
It support the OSC_ variables, so running with
OSC_BUILD_ROOT=/somewhere osc build --host ... will push the variable to
the remove osc process
This is more useful and meaningful error message than M2Crypto's
"ValueError: cafile and capath can not both be None." when neither
of cafile and capath is defined for load_verify_locations call.
Prevents a crash in osc chroot when %(apihost)s is defined in the
"build-root" config option.
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Hi,
"declined" was some time ago changed to be considered an "open" request state.
the default behavior of "osc rq list" is IMHO to show open requests (requests
in state new or review), so I'm attaching a patch that adds "declined" as
well.
Tia,
Dirk
From 73fd10a2203be8e9dc215106648b2d9686d8fdc5 Mon Sep 17 00:00:00 2001
From: Dirk Mueller <dmueller@suse.com>
Date: Mon, 10 Sep 2012 10:40:48 +0200
Subject: [PATCH] add 'declined' to list of default states for rq list
osc rq list should show all currently "open" requests. "declined"
is an open request state, so it should be included by default.
This patch workarounds the BEAST attack in the client side
making OSC to prefer TLS v1.1 or v1.2 ciphers/key exchanges
when available.
Now, as long as openSUSE API webservers do not support
these protocols it will prefer the RC4-SHA1 combination to
be in the safe side.
