Get rid of the urlgrabber dependency. The current implementation of the
progress bar is quite "noisy" if the line length exceeds the size of
the terminal window, but that's something we could fix later. (The
superfluous error message will be fixed in a subsequent commit.)
This is useful so one can wait with a following osc prjresult until
everything done. Without this one can not know if a release job hasn't
started yet or is already finished.
* altered command do_checkout
new option -D | --deleted. Can only be used with PACKAGE and needs
-o to work
* what happens:
core function checkout_deleted_package fetches the file list of the deleted
package and saves it to given destdir. If destdir is not existent it
creates the destination directory. Otherwise the files will be written
in the existing directory.
Print error and exit if osc -A <apiurl> sr is issued on a
package with a different apiurl in its working copy. This leads
to unexpected behavior, because the SR is created for the package
apiurl, not the apiurl given by the -A command line option.
The old code could potentially yield to a use-after-free situation,
which results in UB. For this, consider the following scenario, where
osc performs several HTTPS requests (assumption: the server supports
ssl session resumption):
- HTTPS Request 1:
* a new SSL *s connection is established, which also creates a new
SSL_SESSION *ss => ss->references == 1
* once the handshake is done, the ss is put into the session cache
(see ssl_update_cache) => ss->references == 2
- osc saves the session ss in a class variable
- s is SSL_free()d, which calls SSL_SESSION_free => ss->references == 1
- HTTPS Request 2:
* setup a new SSL *s connection that reuses the saved session ss
=> ss->references == 2
* once the handshake is done, ssl_update_cache is called, which is a
NOP, because s->hit == 1 (that is, the session was resumed)
* osc saves the session ss in a class variable
* s is SSL_free()d, which calls SSL_SESSION_free => ss->references == 1
...
> 2 hours later (see tls1_default_timeout)
...
- HTTPS Request 256:
* setup a new SSL *s connection that reuses the saved session ss
=> ss->references == 2
* once the handshake is done, ssl_update_cache is called, but is
_no_ NOP anymore
* ssl_update_cache flushes the session cache (this is done every
255/256 (depending on the way we count) connections) => ss is
SSL_SESSION_free()d => ss->references == 1
* osc saves the session ss in a class variable
* s is SSL_free()d, which calls SSL_SESSION_free:
since ss->references == 1, ss is eventually free()d
- HTTPS Request 257:
* setup a new SSL *s connection that reuses the saved session ss
Since ss does not exist anymore, the remaining program execution is UB.
(Note: SSL_free(...) is _NOT_ called, if M2Crypto 0.29 is used.
M2Crypto 0.30 calls SSL_free(...) again.)
Due to a bug in OpenSSL_1_1_0h (see openssl commit 8e405776858) the
scenario from above can be triggered with exactly 2 HTTPS requests (the
SSL_SESSION is not cached, because we configured SSL_VERIFY_PEER, but
no sid_ctx was set). This is fixed in openssl commit c4fa1f7fc01.
In order to reliably reuse a session, we probably need to listen to the
session cache changes. Such callbacks could be registered via
SSL_CTX_sess_set_new_cb and/or SSL_CTX_sess_set_remove_cb, but both
functions are not provided by M2Crypto. Another idea is to directly utilize
the session cache, but this also has to be implemented in M2Crypto first.
Yet another approach is to retrieve the session via SSL_get1_session, which
increases the session's refcnt, but this also needs to be implemented in
M2Crypto first (if we choose to use this approach, we also have to make
sure that we eventually free the session manually...).
Fixes: #398 ("SIGSEGV on \"osc commit\"")
Always pass the "--statistics" option to the build script (only affects
a vm build). It is not implemented as an option, because we already have
so many of them... (see also https://github.com/openSUSE/osc/pull/412).
yet another option, but
* only very old build scripts don't know it, we should just require a recent one
* build script is ignoring it for chroot case
so why bother with another option?
Note that the diff is only shown, if it was requested before (that is
the "i" command was issued). The new behavior is consistent with the
other commands like "a", "d" etc.
Storing the error encoding in an "encoding" attribute "breaks" the
python3 "input" function: In essence, builtin_input_impl does a
getattr(sys.stdout, 'encoding'), which returns our error encoding
instead of the "real" stdout encoding. In order to avoid this, we
store the error encoding in an "_encoding" attribute.
Making SafeWriter a new-style class simplifies the code a lot.
This is a fix for issue #385. osc commit breaks due to
the use of sfilelist.findall('.//entry[@hash]')
I now will iterate through the sfilelist and use
for entry in sfilelist.findall('entry'):
if entry.get('hash'):
... execute hash code ...
This is a little bit slower, but should not break
on SLE11 anymore
Without this change, using "--alternative-project <prj>" in combination
with "--multibuild-package <flavor>" yields to unexpected results (from
a user's point of view). Note that this may break existing (artificial)
workflows (e.g., using --alternative-project to ignore the package
meta's debuginfo flag), but these workflows should be rare and there
are options to achieve the same.
Fixes: #376 ("osc build -M something does not work with
--alternative-project")
Only include a tag if it "has" text in get_request_issues. Note
that the code in get_formatted_issues always assumes the presence
of the "label" tag.
Fixes: #369 ("crash trying to view diff of a request")
The retrun at this point breaks the call, because in most
cases <prj> is not a binary. And the code always checks for the
binary first and then returns if no binary with the name <prj>
is found.
The following abstract methods are added to the PackageQueryResult
class: recommends(), suggests(), supplements(), and enhances().
Note that not all package/metadata formats have a notion of these
weak dependencies.
rpm rpmmd deb arch
recommends x x x
suggests x x x x
supplements x x
enhances x x x
(where "x" represents "supported"). In case of an unsupported weak
dependency, the implementation returns an empty list.
We need the weak dependency support in order to fix#363 ("osc build
-p ../rpms/tw doesnt send recommends to the server which makes client
side build behave differently to server side build").
There is no good reason why "--revision <rev>" and "--expand-link" or
"--revision <rev>" and "--unexpand-link" should be mutually exclusive
during an "osc up" of a package wc.
Introduce the new "--linkrev <rev>" option to specify a rev of the link
target that is used during link expansion.
In case of a pulled/linkrepair wc, it is possible that the backend
requests a hash for a tracked file, which is neither added, restored,
nor modified. For instance, this can happen if a new file was added
to the link target. Hence, for a pulled/linkrepair wc always send
the sha256 hashes of the tracked files.
This is needed for a new validation of the source server.
The source server will 'ask' for the sha256 sum of files which are new or
modified and osc calculates the sha256 sums for those files and sends them
back to the server.
The server checks the sha256 sums and if dies if something is wrong.
Mount sysfs during "osc chroot". The current implementation
of "osc chroot" is a major pain for plain "su" users, because the
root password has to be entered several times - we should fix this.
Fixes: #354 ("Mount sysfs in chroot")
At the moment just repo.name is considered. So if
the repo is disabled for s390 all other repo / arch
combination are not shown in the repo list.
To be able to change this r is now a list of dicts
containing the name and arch of the disabled repo.
None for repo if a complete arch gets disabled
None for arch if a complete repo gets disabled
Store a newly created config file in $XDG_CONFIG_HOME/osc/. For backward
compatibility, ~/.oscrc is used, if present.
Fixes: #313 ("oscrc should be stored in $XDG_CONFIG_HOME on linux")
write oscrc to the default location for user-specific configuration.
If XDG_CONFIG_HOME is not set use ~/.config/osc/oscrc which is basically the same.
If there is already a ~/.oscrc use this one (for compat reasons). Existing user
installations should not get affected by this commit.
The order is the following:
Given config with -c
config defined in OSC_CONFIG
existing ~/.oscrc
default XDG_CONFIG_HOME/osc/oscrc
Support an xz compressed control.tar file. In case of a control.tar.xz and
a missing lzma module, an exception is thrown at runtime (for now, in order
to avoid a hard depedency to the lzma module, which is no standard module).
Similar to recent fixes in libsolv and obs-build. Since tarfile
on python2 doesn't do lzma, decompress the file into memory and
feed it as a fake file via StringIO to tarfile
Multiple context expressions are only supported since version 2.7.
It was introduced in commit f6f879d ("Fix potential shell injection
when running rpm2cpio").
Actually, there is nothing that can be injected, except the "-h"
option. However, in case rpm2cpio evolves, we are on the safe side.
Also, document the potential shell injection in the cpio call
(the comment was accidentally removed in commit dbdc712) (the
current osc code is not affected, because we never pass filenames
via *files to core.unpack_srcrpm).
It seems that the "find" binary has no way to indicate an
end of options for its arguments. Hence, we use os.walk to mimic
"find"'s behavior, which is also the cleaner solution.
Fixes: #340 ("osc add of directories does not quote the argument")
This basically reverts commit b2b59ca, because the old code performed
a "no" instead of a "yes" (see also the discussion in
https://github.com/openSUSE/osc/pull/269).
Fixes: #343 ("'osc sr --yes ...' doesn't supersede existing requests
as promised")
The old code was flawed, because, for instance,
core.show_results_meta(apiurl, project, arch=['x86_64']) resulted
in a wrong http request: GET <apiurl>/build/<project>/_result&arch=x86_64
(note the "&" instead of the correct "?"). The drawback of the new
implementation is that we have to do the proper quoting manually.
This harmonizes "osc prjresults --help" with osc's actual behavior.
Also, core.get_prj_results expects lists (or None) instead of strings
for the corresponding repo/arch parameters.
This is a follow-up commit for commit c9c0f8a. Using core.run_external
with shell=True is too error-prone.
Fixes: #340 ("osc add of directories does not quote the argument")
Also, document a potential shell injection in core.unpack_srcrpm
(via the "files" parameter), which cannot be exploited, because
"files" is not used by the current osc code.
Fixes: #340 ("osc add of directories does not quote the argument")
Provide function to show the content of rpmlint.log.
The core function get_rpmlint_log will later be used in the interactive
request mode to provide the rpmlint logs to the reviewer.
I add the osc rpmlint | rpmlintlog | lint command also.
This will filter the requests you get to review based on
the tgt_package in the action element (if not action type is 'group').
This helps reviewers who want to just review a subset of requests or
do _not_ want to review some packages.
Examples:
osc review list --interactive --target-package-filter='^python.*' -G opensuse-review-team
osc review list --interactive --target-package-filter='^(?!ghc).*' -G opensuse-review-team
by using mtime metadata
before checking digests.
This slightly changes the semantic by assuming that modified files
will always have updated timestamps.
With this change it is possible to do osc status
on a checkout of openSUSE:Factory that contains 40GB of source tarballs
in seconds instead of minutes:
time .../osc/osc-wrapper.py status > /dev/null
real 0m33.652s
user 0m32.590s
sys 0m1.060s
Without the patch it took 22x as long:
real 12m14.545s
user 1m50.084s
sys 0m20.566s
In Debian and Ubuntu build is renamed to obs-build for disambiguation
purposes.
Add a simple check to use the correct paths if running on Debian and
use /usr/bin/obs-build and /usr/lib/obs-build if so.
For now, "osc results --xml..." ignores the "--show-excluded" option
(that's what we did in the past).
Fixes: PR#297 ("results --xml causes a stack trace")
Old obs versions have no creator attribute. In this case, string
formatting operations will fail if the "creator" attribute is
initialized with None (affects some codepaths).
An alternative would be to officially resurrect the deprecated
"Request.get_creator" method, which could return something like
"unknown" if no creator information is present.
This is a follow-up fix for the commits d68507f and 7d54b5c.
don't do a second wipe all after doing the intended wipe by a filter.
Eg: osc wipebinaries home:adrianSuSE -r standard -a i586 --build-disabled
wiped first --build-disabled only, but did a second wipe all afterwards
request creator is only delivered by OBS 2.8 and newer. Makeing this not a hard
condition therefor. (introduced in d68507fa95)
Also fixes test suite failure
Request.get_creator is used by some factory bots (see comment
in issue #286).
Note: Request.get_creator is deprecated and the "creator" attribute
should be used instead.
This implements the Request api change that was proposed in commit
6965dc5 ("Adjust request testcases to the upcoming Request api
change").
Fixes: #286 ("get_creator() does not return request creator")
Deleting a conflict file (state 'C') during an update operation results
in an inconsistent working copy. To fix this, we remove the conflict
file from the _in_conflict list.
If a package does not exist on the server, the
Serviceinfo.getProjectGlobalServices call results in a 404. In this
case, we try it again with the _project package (note: this does not
take potential "linked" services into account, if the local package
is a _link).
Fixes: #277 ("404 on running service for non commited package")
Do not pass a family parameter to SSL.Connection's constructor if it
does not support it. If the family parameter is not supported, we
_try_ to fallback to socket.AF_INET, which is implicitly used by
older versions of the SSL.Connection class.
Fixes: #274 ("osc 0.157 Exception AttributeError: "Connection
instance has no attribute 'ssl_close_flag'"")
Adds a "--multibuild-package" option to the following commands:
buildlog, remotebuildlog, buildinfo, build, buildhistory, jobhistory,
rebuild, restartbuild/abortbuild, wipebinaries, getbinaries
This change makes basic authentication fail after first unsuccessful
attempt (instead of automatically retrying) and thus prevents unnecessary
failed login attempts.
This change also allows printing the contents of an HTTP 401 message to
the user.
If build-root contains %(package) substitution, --local-package builds
would substitute absolute path there. This is different than the rule used
in osc chroot (uses relative path), causing the chroot to fail by default.
This commit removes the directory part from both build-root substitutions.
Follow-up commit for 8c45eb64ade699ce3d8717f460b3064ca3ab3a58 (just to
make sure that the "apiurl" attribute is always part of the Serviceinfo
class...)
At the moment only the project status is taken into account when determining when to stop watching a build.
This leads to wrong behavior when a package is in 'blocked' for a longer time.
In this state the project status and code is 'published' but the package remains at 'blocked'.
With this additional check this problem is fixed.
This is used to turn "dir" into a package and add it to the version
control (got broken in commit b6f7d1be6c).
Integrated the obscpio code that was introduced in commit
b6f7d1be6c into core.addFiles (XXX: we
should get rid of the run_external(..., shell=True) code).
The bogus check is twofold: the first error is due to a wrong
refactoring (repostate was intended to be the "code" attribute
and "code" was supposed to the "code" attribute of the status
node) (see commit f3a1d12a). The second error is a logic error
in the original code, because the package state "succeeded" and
the repostate "unpublished" is a valid combination. Consequently,
the check didn't make any sense...
This is up for discussion, but I try to simplify the interface here:
- obsolete "localrun" and "disabledrun". Still implemented, but not anymore
recommended and documented. can hopefully removed later.
- "runall" is running all services local, also buildtime services
- files get replaced by the service
- can be used to get final tar balls to be used with tools like quilt
This reverts commit c53a7681ef (for now!).
It seems to break local obs instances (see issue #202) (this needs
further debugging). Moreover, it breaks the python 3.4 - excerpt
from a travis run:
======================================================================
ERROR: test_added_missing2 (test_commit.TestCommit)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/opt/python/3.4.2/lib/python3.4/urllib/request.py", line 1111, in do_request_
mv = memoryview(data)
TypeError: memoryview: _io.BufferedReader object does not have the buffer interface
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/travis/build/openSUSE/osc/tests/common.py", line 122, in wrapped_test_method
test_method(*args)
File "/home/travis/build/openSUSE/osc/tests/common.py", line 122, in wrapped_test_method
test_method(*args)
File "/home/travis/build/openSUSE/osc/tests/common.py", line 122, in wrapped_test_method
test_method(*args)
File "/home/travis/build/openSUSE/osc/tests/common.py", line 122, in wrapped_test_method
test_method(*args)
File "/home/travis/build/openSUSE/osc/tests/common.py", line 122, in wrapped_test_method
test_method(*args)
File "/home/travis/build/openSUSE/osc/tests/test_commit.py", line 290, in test_added_missing2
p.commit()
File "/home/travis/build/openSUSE/osc/tests/osc/core.py", line 1471, in commit
self.put_source_file(filename, tdir)
File "/home/travis/build/openSUSE/osc/tests/osc/core.py", line 1319, in put_source_file
http_PUT(u, file = tfilename)
File "/home/travis/build/openSUSE/osc/tests/osc/core.py", line 3243, in http_PUT
def http_PUT(*args, **kwargs): return http_request('PUT', *args, **kwargs)
File "/home/travis/build/openSUSE/osc/tests/osc/core.py", line 3231, in http_request
fd = urlopen(req, data=data)
File "/opt/python/3.4.2/lib/python3.4/urllib/request.py", line 153, in urlopen
return opener.open(url, data, timeout)
File "/opt/python/3.4.2/lib/python3.4/urllib/request.py", line 453, in open
req = meth(req)
File "/opt/python/3.4.2/lib/python3.4/urllib/request.py", line 1116, in do_request_
data))
ValueError: Content-Length should be specified for iterable data of type <class '_io.BufferedReader'> <_io.BufferedReader name='/tmp/osc_test571whun4/osctest/added_missing/.osc/_in_commit/bar'>
This is up for discussion, but I try to simplify the interface here:
- obsolete "localrun" and "disabledrun". Still implemented, but not anymore
recommended and documented. can hopefully removed later.
- "runall" is running all services local, also buildtime services
- files get replaced by the service
- can be used to get final tar balls to be used with tools like quilt
Note: this commit "breaks" the existing get_package_results api, because
it returns a generator, which yields a result xml, instead of a list of
result dicts.
In this case, "osc bl repo" behaves more or less the same as "osc rbl repo"
(in case no corresponding .osc/_buildinfo file exists, we default to
repo and hostarch).
Since python >= 2.7.9 urllib2/httplib verifies the ssl cert by default - so
make sure that this "builtin" ssl check is disabled as well if
"sslcertck" is disabled. Fixes#179.
Note: eventually, we should abandon m2crypto and use urllib2/httplib to
verify the ssl cert.
The syntax to create a maintnancerequest for a single currently package
is:
osc mr SOURCEPROJECT SOURCEPACKAGES RELEASEPROJECT
which means that the source project, the package name and the release
project have to be specified on the command line.
Often times the workflow is such that the user will already be inside
of the subdirectory containing the checked out package.
To simplify the submission when the user is in a package subdirectory
this patch adds the syntax:
osc mr .
to indicate that the source project and source target is to be taken
from the meta information in this package directory.
Signed-off-by: Egbert Eich <eich@freedesktop.org>
