1
0
mirror of https://github.com/openSUSE/osc.git synced 2024-12-27 02:16:12 +01:00
Commit Graph

3317 Commits

Author SHA1 Message Date
Marcus Huewe
bf42c70f56 Use a fixed suffix for the upfilename in Package.mergefile
The old code uses a variable .rXYZ suffix (where XYZ is the revision
of the package wc during the merge operation). Now, if Package.mergefile
is invoked during an update, XYZ represents the "old" revision. That
is, if a merge conflict happens, then a subsequent "osc resolved <file>"
will not unlink the <file>.rXYZ file (because
Package.clear_from_conflictlist only takes the current rev into account).

In order to fix this, use a fixed ".new" suffix. This way,
Package.clear_from_conflictlist can properly unlink the corresponding
*.new file. This naming scheme for the "upfilename" is in line with
"osc pull" and "osc repairlink".

Note: if a working copy was updated with an "old" osc version (without
this commit) and a "new" osc version (with this commit) is used to run
"osc resolved <file>", then the <file>.rXYZ file is _NOT_ removed (it
is not worth the effort to add compat code for this).
2021-07-20 15:19:16 +02:00
Marcus Huewe
fd4d8d726a Fix typo in the password deletion codepath in conf.config_set_option
A password can be deleted via "osc config -d <apiurl> pass". Actually,
if we really want to support password deletion, we should introduce
a --delete-password option because the "pass" config option can be
considered as an implementation detail, which we should not expose
to our users.
2021-07-14 11:31:56 +02:00
Marcus Huewe
7e23743239 Add support for changing the password store via osc config
The password store can be changed (without entering the password
again) via "osc config <apiurl> --select-password-store". This
command deletes the password from the current password store and
stores it in the selected password store.

Previously, the --select-password-store option had no meaningful
semantics. In order to use it, one always had to provide a password
and explicitly pass "pass" as the config option (the same could be
achieved by using --change-password). Hence, in a strict sense,
this change breaks the UI.
2021-07-14 11:28:57 +02:00
Marcus Huewe
0963bc690f Slash split arguments in do_browse
Without the slash splitting, "osc browse prj/pkg" interprets the
argument as a project, which is wrong. Hence, perform the slash
splitting (as most commands do).
2021-07-13 15:57:58 +02:00
Marcus Huewe
e71b79c6d0 Fix the commit of a frozen package wc
Always send the sha256sums of all tracked files in case of a
frozen package wc. For instance, this is needed if the package is
a plain link (no branch) because in this case the backend might
request a sha256sum for a tracked but unmodified file (this can
happen because the backend cannot expand the link).
The new behavior is in line with a pulled/linkrepair package wc.

Fixes: #924 ("Transmitting file data There is no sha256 sum for
file")
2021-07-02 23:20:16 +02:00
90d9402a05 fix local product builds using obsrepositories:/ directives
We need an absolute path since osc calls build in user home directory.

Also extend the file name to hint to osc (and not any service
configuration)
2021-06-28 09:26:27 +02:00
Marcus Huewe
fb44630ad7 Handle "cd prj; osc ci non_existent_pkg" more gracefully
When trying to commit a non-existent package via Project.commit it
is treated as an external package (because a non-existent package
has no "state" inside the project). That is, Project.commitExtPackage
is called, which fails with a FileNotFoundError in case of a
non-existent package (and the traceback is printed to the user).
In order to fix this, treat a non-existent package as broken package.
That is, simply print an info message and do not error out with a
traceback (note: the commit is _not_ aborted).

Fixes: #920 ("osc commit should fail gracefully in case of
nonexistent  filename")
2021-06-04 13:06:00 +02:00
Marcus Huewe
c94ea04eb7 Open 0.174.0 development 2021-05-18 22:39:39 +02:00
Marcus Huewe
60e06ad96d Release 0.173.0
For the details, see the NEWS file (which is mostly about user
visible changes).
2021-05-18 22:36:09 +02:00
209686a16f support cross compile using a sysroot
We need to hand over the sysroot rpms in rpmlist with a prefix. And we
need to take care of the "crossarch" definition in buildinfo
2021-05-18 07:57:02 +02:00
William Brown
c12e0b5cda Add support for sccache.
Sccache is an alternate build caching system to ccache/icecream. It
supports C, C++ and Rust. It can optionally have distributed or remote
caches via redis, s3 object stores, memcached, azure storage or
google cloud storage.

This can help to significantly improve the performance of Rust rebuilds.

For example, Kanidm changes from 400s to 122s on a rebuild, and rust-lang
rebuilds improve from 7200s to 4770s. With some changes to the rust
packages especially this will be possible to speed up over version
changes as well.

See also: obs-build PR https://github.com/openSUSE/obs-build/pull/680
2021-05-14 12:57:40 +10:00
Marcus Huewe
1e94757498 Merge branch 'build_as_user' of https://github.com/adrianschroeter/osc
Do not use a preinstallimage if the local build is executed as a non-root
(the preinstallimage contains device nodes which usually cannot be created
by a non-root user - this is not a problem in the non-preinstallimage
codepath (see [1])).

[1] https://github.com/openSUSE/osc/pull/908#issuecomment-806903856
2021-04-28 16:55:34 +02:00
08979fb0eb do not use preinstallimages when building as non-root
it can not work and build script will direct complain in future
2021-04-28 16:53:17 +02:00
6632ca2d2e support _keyinfo route with signkey command by default 2021-04-28 13:42:06 +02:00
Pedro Monreal
67ac287f55 Adjust the description size of 'osc service run' 2021-04-28 12:11:56 +02:00
320adda399 simpleimage uses OTHER directory to deliver files 2021-04-26 08:02:38 +02:00
Marcus Huewe
c932f95d46 Support an arbitrary sized file in core.http_request
The old code only supports a file whose size is less then or equal
to INT_MAX (due to a reasonable(!) limit in M2Crypto). The actual
issue is in core.http_request which mmap(...)s the file, wraps it
into a memoryview/buffer and then passes the memoryview/buffer to
urlopen. Eventually, the whole memoryview/buffer is read into memory
(see m2_PyObject_GetBufferInt). If the file is too large (> INT_MAX),
m2_PyObject_GetBufferInt raises a ValueError (which is perfectly
fine!).
Reading a whole file into memory is completely insane. In order to
avoid this, we now simply pass a file-like object to urlopen (more
precisely, the file-like object is associated with the Request
instance that is passed to urlopen). The advantange is that the
file-like object is processed in chunks of 8192 bytes (see
http.client.HTTPConnection) (that is, only 8192 bytes are read into
memory (instead of the whole file)).

There are two pitfalls when passing a file-like object to urlopen:
* By default, a chunked Transfer-Encoding is applied. It seems that
  some servers (like api.o.o) do not like this (PUTing a file with
  a chunked Transfer-Encoding to api.o.o results in status 400). In
  order to avoid a chunked Transfer-Encoding, we explicitly set a
  Content-Length header (we also do this in the non-file case (just
  for the sake of completeness)).
* If the request fails with status 401, it is retried with an
  appropriate Authorization header. When retrying the request, the
  file's offset has to be repositioned to the beginning of the file
  (otherwise, a 0-length body is sent which most likely does not
  match the Content-Length header).

Note: core.http_request's "data" and "file" parameters are now mutually
exclusive because specifying both makes no sense (only one of them
is considered) and it simplifies the implementation a bit.

Fixes: #202 ("osc user authentification seems to be broken with last
commit")
Fixes: #304 ("osc ci - cannot handle more than 2 GB file uploads")
2021-04-10 22:14:25 +02:00
Marcus Huewe
3c9ee7545e Merge branch 'remove_etree_getchildren' of https://github.com/marcus-h/osc
Do not use the deprecated Element.getchildren anymore.
2021-03-28 21:10:30 +02:00
6ac7855f4d remove "need root" autodetection
This kind of guessing can not really work here and leads to failing
builds when using KVM. (eg. when using a preinstallimage)

Removing the code, since we have a now a way to allow the user to
specify building as user via su-wrapper config
2021-03-25 09:06:48 +01:00
Marcus Huewe
4e9dde71b8 Merge branch 'build-as-user' of https://github.com/adrianschroeter/osc
Allow a local build as the current user/support an empty su-wrapper
config option.
2021-03-22 15:22:13 +01:00
4907982d42 allow to run build script as user
works only with kvm atm, we should maybe point the user to it in that
case?

Requires: https://github.com/openSUSE/obs-build/pull/678/files
2021-03-22 15:16:35 +01:00
Marcus Huewe
21274b6981 Do not use the deprecated Element.getchildren anymore
Element.getchildren is deprecated and not available on python39
anymore. Instead, iterate over the element itself (which iterates
over the element's children).

Fixes: #903 ("AttributeError: 'xml.etree.ElementTree.Element' object
has no attribute 'getchildren'")
2021-03-21 19:19:23 +01:00
Marcus Huewe
f65f08dfbb Add "--force" option to the "osc add" command
"osc add --force <filename>" adds the filename to the working copy
even if it is excluded by the exclude_glob config option.
2021-03-18 15:05:31 +01:00
2bc186d0ea add new stage option for the build script
(minor code improvement for handing over release number)
2021-02-18 09:01:18 +01:00
c06b794ddf support osc build --shell-after-fail from build script 2021-02-12 08:20:51 +01:00
bc25ac769e add showlinked command 2021-02-08 10:50:45 +01:00
Marcus Huewe
fc7d3d91fc Merge branch 'creq_slash_split' of https://github.com/marcus-h/osc
Support slash notation in "osc creq -a <action type> args".
2021-02-01 23:06:57 +01:00
Marcus Huewe
609990ba62 Open 0.173.0 development 2021-01-27 13:30:43 +01:00
Marcus Huewe
5df5465b1a Release 0.172.0
For the details see the NEWS file.
2021-01-27 13:21:32 +01:00
Marcus Huewe
2371a6c91b Support slash notation in "osc creq -a <action type> args"
Most osc commands support slash notation for the specification of
a project package pair. That is, "osc <cmd> prj/pkg" has the same
semantics as "osc <cmd> prj pkg" (in most cases).
For consistency reasons, "osc creq" should also support the slash
notation for the action type's arguments. That is, for instance,
"osc creq -a submit src_prj/src_pkg dst_prj/dst_pkg" should have the
same effect as "osc creq -a submit src_prj src_pkg dst_prj dst_pkg".

Proposed-by: darix
2021-01-25 21:16:17 +01:00
Ludwig Nussel
224ec6eef5
Merge pull request #847 from marcus-h/service_old_dir
Add .old dir support for source services
2021-01-20 15:08:27 +01:00
ca080d2118 add --lastsucceeded option also for buildlog command
introduce --last-succeeded alias
2021-01-18 10:21:58 +01:00
9ad555ee24 mention flatpack as well 2021-01-18 10:21:50 +01:00
Marcus Huewe
6104560050 Merge branch 'creq_no_supersede_dups' of https://github.com/marcus-h/osc
Do not supersede the same requests several times in osc creq.
2020-12-22 16:07:43 +01:00
Marcus Huewe
0926e37f1e Do not return Request instances in Osc._submit_request
If there are existing requests that should be superseded, the old
code stores the Request instances in the myreqs list, which is
returned to the caller. However, the caller expects only request
ids instead of instances of class Request. Eventually, this results
in a type error - excerpt:

...
  File "/usr/lib/python3.8/site-packages/osc/commandline.py", line 1892, in do_createrequest
    change_request_state(apiurl, srid, 'superseded',
  File "/usr/lib/python3.8/site-packages/osc/core.py", line 4322, in change_request_state
    u = makeurl(apiurl,
  File "/usr/lib/python3.8/site-packages/osc/core.py", line 3326, in makeurl
    return urlunsplit((scheme, netloc, '/'.join([path] + list(l)), query, ''))
TypeError: sequence item 2: expected str instance, Request found

Hence, simply return the request ids instead of the Request instances.

Note: this changes the API of the Osc._submit_request method but
this is OK because it is not part of the public API.
2020-12-22 16:04:13 +01:00
Marcus Huewe
1aab0a8ed9 Do not supersede the same requests several times in osc creq
When calling "osc creq -a prj1 foo prj2 bar -a submit prj1 bar prj2 bar",
the requests that could be superseded are calculated two times for the
prj2/bar package. Hence, they could end up two times in the "supersede"
list (see do_createrequest) In order to avoid duplicates, use a set
instead of a list.

Kudos to darix for pointing this out!

Note: it is a bit questionable if osc's current semantics makes sense
in the above example.
2020-12-09 21:41:37 +01:00
Marcus Huewe
745dc1180d Merge branch 'fix__html_escape_usage' of https://github.com/marcus-h/osc
Avoid superfluous/excessive usage of _html_escape.
2020-12-05 19:26:59 +01:00
Marcus Huewe
fbea0ea729 Avoid superfluous/excessive usage of _html_escape
When creating a new request via the core.Request.create method, there is
no need to escape the data that is assigned to the "description" attribute
of a core.Request instance. Internally, core.Request.create ensures that
the data, which is POSTed to the api, is correctly escaped (the escaping
is implicitly done by ET (see core.Request.to_str)). Manually escaping the
description results in a double escaping (the escaped description is
escaped by ET again) - this is not the desired behavior.
Analogously, there is no need to escape the data that is passed to the
message parameter of the core.create_submit_request function because
core.create_submit_request takes care of escaping it.

Fixes: #869 ("Silly encoding of htmlencodable entities")
2020-11-28 18:03:24 +01:00
Marcus Huewe
b647521f81 Support a regex based name filtering in core.get_prj_results
So far, core.get_prj_results only supports a substring based name
filtering mechanism. Now, a regex based name filtering mechanism
is used. That is, if the regex matches a package name, the package
is not filtered out.

This is an API incompatible change:
- 3rd party code which looks like this is going to break:
  class Foo(str):
      ...
      def __eq__(self, other):
          return ...

  ... = core.get_prj_results(..., name_filter=Foo())

  (My gut feeling says there are no such callers. However, if this
  really breaks any serious code, we can fix it in a follow-up commit)

- "osc prjresults openSUSE:Leap:15.2:Update --name-filter zypper." will
  now also show "zypper-docker" etc. because the dot (".") matches any
  character (except a newline). Previously, only packages that contained
  the str "zypper" followed by a dot (".") were shown. The old behavior
  can be restored, if the dot is escaped: "osc prjresults
  openSUSE:Leap:15.2:Update --name-filter 'zypper\.'".
  Of course, this affects all other special characters, too.
  Additionally, if an illegal regex is passed to the --name-filter option,
  an exception is raised. The previous code did not fail.

  This can break existing workflows and scripts. We could avoid this by
  introducing a --name-filter-regex option but this would clutter the
  UI (IMHO).

A regex based name filtering feature was requested by darix.
2020-11-25 22:51:02 +01:00
Marcus Huewe
1933da5bcc Use os.getcwdb() instead of os.getcwd().encode() in util.cpio.CpioRead
Using os.getcwd() in combination with a subsequent .encode() is error
prone:

marcus@linux:~> mkdir illegal_utf-8_encoding_$'\xff'_dir
marcus@linux:~> cd illegal_utf-8_encoding_$'\xff'_dir/
marcus@linux:~/illegal_utf-8_encoding_ÿ_dir> python3
Python 3.8.6 (default, Nov 09 2020, 12:09:06) [GCC] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.getcwd().encode()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcff' in position 36: surrogates not allowed
>>>

Hence, use os.getcwdb(), which returns a bytes, instead of
os.getcwd().encode().

Fixes: commit 36f7b8ffe9 ("Fix a
potential TypeError in CpioRead.copyin and CpioRead.copyin_file")
2020-11-22 17:39:54 +01:00
Marcus Huewe
674ea78815 Avoid a potential TypeError in util.ArFile.saveTo
If no dir is passed to util.ArFile.saveTo, dir is set to os.getcwd(),
which returns a str. Since self.name is a bytes, the subsequent
os.path.join(dir, self.name) results in a TypeError.
To fix this, use os.getcwdb(), which returns a bytes instead of a
str.
2020-11-22 17:36:17 +01:00
Oleg Girko
74846ea83b Add support for nspawn VM type.
This allows to utilise support for systemd-nspawn backend in build engine.
Like LXC, systemd-nspawn creates isolated lightweight container.

Signed-off-by: Oleg Girko <ol@infoserver.lv>
2020-11-22 14:34:37 +00:00
Marcus Huewe
36f7b8ffe9 Fix a potential TypeError in CpioRead.copyin and CpioRead.copyin_file
If no "dest" argument is specified when calling CpioRead.copyin or
CpioRead.copyin_file, a TypeError occurs in CpioRead._copyin_file
because os.getcwd(), which returns a str, is used as dest and, hence,
the subsequent os.path.join(...) fails (because it tries to join a
str and a bytes).
In order to avoid this, encode the result of os.getcwd().

Note that the existing

archive.copyin_file(hdr.filename,
                    os.path.dirname(tmpfile),
                    os.path.basename(tmpfile))

was OK because CpioRead._copyin_file os.path.join()s "dest" and
"new_fn", which are both str. It is just changed to stress that
CpioRead is a bytes-only API.

Fixes: #865 ("Traceback in osc/util/cpio.py line 128: TypeError:
Can't mix strings and bytes in path components")
2020-11-20 09:55:09 +01:00
Anatoli Babenia
48e8a1e802 Pretty print dists table with two spaces between columns 2020-11-14 19:45:29 +03:00
Anatoli Babenia
c000e0e372 Simplify looping over /distributions XML 2020-11-14 18:31:03 +03:00
Marcus Huewe
d7594ddeda Merge branch 'do_init_non_existent_prj' of https://github.com/marcus-h/osc into master
Fix "osc init" in case of a non-existent project.
2020-11-12 22:41:51 +01:00
Michael Schroeder
5d2542522d Support flatpak building 2020-11-12 14:08:10 +01:00
lethliel
a47a0a5f3c open 0.172.0 development 2020-11-10 14:31:25 +01:00
lethliel
528991829f release 0.171.0 2020-11-10 14:25:02 +01:00
Marcus Huewe
0d78560e5d Do not retrieve the package list when initializing a project via osc init
Currently, when trying to initialize a non existent (server-side)
project via "osc init <prj>", osc errors out (after creating the wc)
because it fails to retrieve the package list. However, there is no
need to retrieve the package list in the "osc init <prj>" case. Hence,
skip the package list retrieval. As a result, osc does not error out.

For the background, see the discussion in #858 ("osc fails to check
out an empty project as project") [1].

[1] https://github.com/openSUSE/osc/issues/858#issuecomment-722330024
2020-11-05 13:13:07 +01:00
Bernhard M. Wiedemann
c3522dbb28 Use verifymd5 to determine freshness
to speed up osc up for large repo checkouts by a factor of 10 or more

The verifymd5 value is not (yet) stored
so we quickly compute it as needed

https://github.com/openSUSE/open-build-service/issues/10250
2020-10-17 21:14:12 +02:00
Marcus Huewe
b573f02eb4 Fix show_upstream_rev call in checkRevision
If meta=True is passed to checkRevision, the meta parameter is used
as a revision in the show_upstream_rev call. Instead, it should be
bound to show_upstream_rev's meta parameter.
2020-10-13 19:35:08 +02:00
lethliel
8085aa6012 use sr_ids[0] for superseding
result is not assigned in the case of many packages per request
2020-10-05 19:53:45 +02:00
Marcus Huewe
89bb15d3b6 Add .old dir support for source services
Some services expect "old" service files (that is, files from a
previous service run) to be present in an ".old" dir. Hence, osc
should support that.
Instead of removing all files from a previous service run, move them
to the ".old" dir, run the services, and, finally, remove the ".old"
dir.
Unfortunately, the location of the ".old" dir is hardcoded in the
specific services. That is, we have to be careful if an ".old" dir
exists (in this case, we error out).

Based on [1].

[1] https://github.com/openSUSE/osc/pull/846
2020-10-01 14:03:10 +02:00
lethliel
eaf50dd13d add proper reporuls
for use in Arch, Debian and Ubuntu
2020-09-22 08:52:57 +02:00
Marcus Huewe
e58abf1527 Support a preinstallimage in combination with the --offline option
Currently, if the --offline option is passed to "osc build ...", a
preinstallimage is not used (even if it exists). Instead, a
preinstallimage should be used (if it exists) even if the --offline
option is specified.
2020-09-07 18:54:59 +02:00
9b245db224 catch URLs to github.com and gitlab.com as git repos
yes, ugly, but I am tired to explain people to add a .git suffix
to get the right mode for "osc add"
2020-09-01 13:50:18 +02:00
Marco Strigl
5bf40aa9a4
Merge pull request #830 from adrianschroeter/owner
maintainer search: lookup via package name by default and binary as fallback
2020-09-01 13:25:21 +02:00
36fc925ee0 maintainer search: lookup via package name by default and binary as fallback
This is faster in best case since the binary search does not need
to be executed on the server.

It also finds package names where no binary with that name exists.
(as for some multibuild cases)
2020-08-31 13:53:41 +02:00
bf7899da83
Ignore signal.SIGWINCH.
Fixes #584
2020-08-27 16:56:36 +02:00
Marcus Huewe
a57b8a60b2 Merge branch 'help-rdiff-p-in-archives' of https://github.com/JanZerebecki/osc
Improve the documentation of the "rdiff" command + its "--plain" option.
2020-08-25 18:44:56 +02:00
294aaa0dd7
help rdiff: -p shows diff of files in archives
This was undocumented.

Also remove the part that incorrectly says this is the default.
2020-08-25 18:14:02 +02:00
75934c995c
fix content of osc workerinfo --help
Replace usage with better explanation. It was missing that it requires a
prefixed hostarch. Also workername is instead called workerid in the
API.

Usage help was before: osc workerinfo WORKER

Add actual example.

See also the fix for this in OBS API docs:
https://github.com/openSUSE/open-build-service/pull/10024
2020-08-25 16:42:01 +02:00
lethliel
a91b8f051e implement releaserequest for non-maintenance prj
In the API a new request action release was implemented. This changes
enables the user to create a release request for non-maintenance projects
and to review / view the release requests
2020-08-06 21:52:53 +02:00
Colleen Murphy
cb6eaf3720 Fix single service disabled run
Without this patch, running an individual service that has parameters
defined in the _service file fails:

  $ osc service run obs_scm
  Please specify valid --scm=... options
  Aborting: service call failed:  /usr/lib/obs/service/obs_scm --outdir [snipped]

This is because although the service is defined in the _service file and
the "scm" parameter is set in it, the service wasn't being found in the
data structure and so the service executable wasn't being called with
the parameters supplied in the _service file. This patch corrects the
issue with the services data structure so that the service data isn't
overridden if it is defined in the _service file.

A side effect of this correction for services defined in the _service
file is that instead of overriding the service mode with '', the mode is
taken from the _service file. When using the "run" command, this would
mean that the call mode of None may not be in agreement with the service
mode defined in the file, e.g. "manual", and so the "run" command would
no longer cause it to run when it would before. We can take this
opportunity to define this as the correct behavior - the "run" command
now only runs services with "trylocal", "localonly", or no mode set -
and also ensure that other call mode commands result in sensible
behavior when called with a service name, for instance "osc service
manualrun download_files" will run only services with mode="manual" and
name="download"files" instead of all services with mode="manual".
Additionally, services that aren't defined in the _service file can be
called with a call mode command and will use that call mode rather than
None.
2020-08-03 18:01:30 -07:00
Marcus Huewe
7612fe1614 Merge branch 'undeprecate-disabledrun' of https://github.com/cmurphy/osc
Add a "manual" service mode. A service with mode "manual" is not executed
by default (that is, via "osc service run"). As of now, "manual" is in
some sense just an alias for "disabled". However, this might change in the
future [1]. Also, "localrun" now executes services with mode "serveronly".
Moreover, the documentation of "disabledrun" is updated ("disabledrun"
never executed services with mode "serveronly"). Additionally, "localrun"
and "disabledrun" are marked as "[n]ot for common usage anymore" in the
service command's description.
The rationale for these changes is (partly) described in [1]. The main
motivation is to add some clarity (in contrast to the "disabled" mode,
it is probably easier to get/guess the semantics of the "manual" mode).

[1] https://github.com/openSUSE/osc/pull/826
2020-07-31 15:01:23 +02:00
Colleen Murphy
894b97d471 Add command "manualrun" to replace "disabledrun"
The "disabledrun" service commands is marked as deprecated but has no
explicit replacement. It is still a useful command for updating packages
manually or through a CI system without being forced to run all defined
services with the "runall" command. This change adds a new command
"manualrun" and a new mode "manual" which behave the same as the
deprecated "disabledrun" command and "disabled" mode but have clearer
meaning. "manualrun" does not attempt backwards-compatible behavior with
the "disabledrun" mode for "disabled" services because "disabled" mode
may eventually be removed or change meaning. The "localrun" command is
enhanced to consider the "serveronly" mode. Since "disabledrun" never
executed services with mode "serveronly", its docs are updated
accordingly.
2020-07-30 14:26:47 -07:00
Kevin Tee
783544d2da
Update build.py
Fix repeated word.
2020-07-26 11:57:18 +00:00
lethliel
0ef77b8cd7 print detailed error message in interactive review
Print the details of a Bad Request also in interactive review mode.
2020-07-22 20:20:09 +02:00
lethliel
55d3f46ac0 open 0.171.0 development 2020-07-09 10:27:10 +02:00
lethliel
ead43243db release 0.170.0 2020-07-09 09:57:53 +02:00
Marcus Huewe
76eafc92b5 Merge branch 'print_links_in_mr' of https://github.com/lethliel/osc
Print a link to webui if a new maintenance request is created and
the "print_web_links" config option is enabled.
2020-07-07 17:50:48 +02:00
lethliel
ce794cb37d print web links on osc mr
print web_links also on maintenance requests when print_web_links=1
is configured in oscrc
2020-07-07 17:31:20 +02:00
Marcus Huewe
ea8c6e1c45 Merge branch 'overhaul_do_service' of https://github.com/lethliel/osc
Improve error message in do_service a bit. The old "Too few arguments."
was misleading (for instance, if a non-remote command was not executed
in a package wc).

Note: with the new logic we could also get rid of the
"raise oscerr.WrongArgs('Local directory is no package')" statement.
2020-07-07 17:19:38 +02:00
Marcus Huewe
aa855b1c69 Merge branch 'filter_results_based_on_code' of https://github.com/lethliel/osc
Add a --status-filter option to "osc results" that can be used to
show, for instance, only the repos where a package failed to build. As
a short circuit, a -f/--failed option is added, too.
Add a --brief option to "osc prjresults" and "osc results" that can be
used to get a more compact representation of the results. In case of
"osc results", --brief is ignored if the results for a package are
requested.
2020-07-07 17:08:32 +02:00
lethliel
f2f80a1498 fix error message if osc service not in cwd
fix the error message if osc servce is run without [proj] [pack] arguments
in a non-valid osc working directory.
2020-07-07 08:31:35 +02:00
lethliel
2f0918b93e add brief results/prjresults with filtered code
in do_results:
  * add --brief option on prj level:
    [packagename] [repo] [arch] [buildstatus]
  * filter by --status-filter <long status name>
    works on prj and pkg level

in do_prjresults:
  * --brief
  * assume len(state)>1 as long state

core.py
  * filter packages by build status
  * long status handling in get_prj_results
  * brief output generation in get_prj_results
2020-07-07 08:23:02 +02:00
Marcus Huewe
ce291abac6 Merge branch 'fix_string_literal_in_commit_msg' of https://github.com/lethliel/osc
Interprete unicode escape characters in a "--message" option. In some
cases, this breaks the existing UI (but that's OK because it can be
fixed by properly escaping the escape character(s)).
Note: if we are going to do more advanced stuff in the future, we should
move the logic into a separate function.
2020-06-29 23:53:32 +02:00
lethliel
16fda3115c interpretation of string literals in messages
string literals are not interpreted correctly when using -m "my\ntwo line message"
2020-06-26 17:42:35 +02:00
Marcus Huewe
d85030b72d Fix python2 regression in util.helper.decode_it
In commit 276d6e2439 ("Do not use the
chardet module in util.helper.decode_it") util.helper.decode_it was
changed to always decode the passed object if it has a decode method.
Since a python2 str has a decode method, the new code tries to utf-8
decode the passed str. As a result, a unicode object is returned (if
the decoding worked). Since a unicode object is not an instance of
type str, all subsequent isinstance(decoded_obj, str) checks evaluate
to False, which break some codepaths.
In order to fix this, restore the old python2 behavior (that is, if
the passed object is a str, it is not decode it). This change does not
affect the python3 codepaths.

Fixes: #814 ("osc log | fails")
2020-06-25 15:38:14 +02:00
Marcus Huewe
050c94dcf3 Merge branch 'fix_issues_diff' of https://github.com/lethliel/osc
Improve "osc rdiff --issues-only ..." output: now, it shows the added,
deleted and changed issues. Also, add a new "osc rdiff --xml ..." option,
which only works in combination with the "--issues-only" option: it prints
the raw xml.

Note: server_diff_noex has no option for the "full" parameter. Hence,
with the addition of the "xml=False" parameter, the signatures of the
server_diff_noex and server_diff functions are going to differ "forever".
That's OK (IMHO) because it is probably more sane to simply specify the
additional args via the kwargs syntax.
2020-06-15 22:42:16 +02:00
lethliel
a1df48145b Add more infos to issues rdiff
Show the state of the issues (added, deleted, changed)
of the issues in a list.

Also --xml can now be given to get the raw xml output from the server.
2020-06-15 20:53:38 +02:00
Marcus Huewe
dd814054ee Merge branch 'remove_dead_code' of https://github.com/lethliel/osc
Remove dead code from the fetch module. Actually, it should have
been removed in commit 95ec7dee7b
('- fixed #590606 ("osc/fetch.py does not support authenticated URLs")').
2020-06-08 21:32:42 +02:00
lethliel
297b050f1e remove dead opener code
The opener is generated but never used.
Since commit 95ec7dee7b
2020-06-08 20:31:28 +02:00
Marcus Huewe
a87d2c1202 Document a potential decoding issue in print_buildlog
We should reconsider this in the future (in practice, it does not
seem to cause too much trouble (at least I'm not aware of any bug
reports)).
2020-06-04 15:09:03 +02:00
Marcus Huewe
276d6e2439 Do not use the chardet module in util.helper.decode_it
In general, decode_it is used to get a str from an arbitrary bytes
instance. For this, decode_it used the chardet module (if present)
to detect the underlying encoding (if the bytes instance corresponds
to a "supported" encoding). The drawback of this detection is that
it can take quite some time in case of a large bytes instance, which
represents no "supported" encoding (see #669 and #746).
Instead of doing a potentially "time consuming" detection, either
assume an utf-8 encoding or a latin-1 encoding. Rationale: it is just
not worth the effort to detect a _potential_ encoding because we have
no clue what the _correct_ encoding is. For instance, consider the
following bytes instance:

b'This character group is not supported: [abc\xc3\xbf]'

It represents a valid utf-8 and latin-1 encoding. What is the "correct"
one? We don't know... Even if you interpret the bytes instance as a
human you cannot give a definite answer (implicit assumption: there is
no additional context available).
That is, if we cannot give a definite answer in case of two potential
encodings, there is no point in bringing even more potential encodings
into play. Hence, do not use the chardet module.

Note: the rationale for trying utf-8 first is that utf-8 is pretty
much in vogue these days and, hence, the chances are "high" that we
guess the "correct" encoding.

Fixes: #669 ("check in huge shell archives is insanely slow")
Fixes: #746 ("Very slow local buildlog parsing")
2020-06-04 13:12:22 +02:00
Adam Williamson
13a13a87c4 Fix ElementTree imports for Python 3.9
Importing `cElementTree` has been deprecated since Python 3.3 -
importing `ElementTree` automatically uses the fastest
implementation available - and is finally removed in Python 3.9.
Importing cElementTree directly (not as part of xml) is an even
older relic, it's for Ye Time Before ElementTree Was Added To
Python and it was instead an external module...which was before
Python 2.5.

We still need to work with Python 2.7 for now, so we use a try/
except to handle both 2.7 and 3.9 cases. Also, let's not repeat
this import 12 times in one file for some reason.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-06-02 15:13:10 -07:00
lethliel
e99b2942ef increase version number in core.py 2020-05-29 20:35:00 +02:00
lethliel
0e70579113 release 0.169.1 2020-05-29 20:28:42 +02:00
Marcus Huewe
1d26cb3079 Merge branch 'ccache' of https://github.com/sjamgade/osc
Add "--ccache" option to "osc getbinaries".
2020-05-27 16:09:06 +02:00
lethliel
22ac336f8b open 0.170.0 development 2020-05-27 14:50:37 +02:00
lethliel
42b4992ecb release 0.169.0 2020-05-27 14:49:10 +02:00
lethliel
a79c54418b fix security issue (bsc#1122675) no / in filename
This checks if the filename of a downloaded file has
been modified (for example by a MITM attack) to contain
slashes. This could mean that the file is compromised
and that the attacker tries to overwrite system files.
2020-05-27 11:17:40 +02:00
Sumit Jamgade
5557a06e5b add and implement ccache option for getbinaries
If this option is used, the api is asked to list ccache archive if
present, osc will then download the archive.
2020-05-25 09:03:33 +02:00
Marco Strigl
3967133022
Merge pull request #793 from sjamgade/ccache_path
add option pkg-ccache to be passed to build-cmd
2020-05-22 15:55:48 +02:00
Fabian Vogt
f3d07eab3f Fix kiwi build with --prefer-pkgs
The buildinfo contains packages from the local dir, so it needs the list of
local packages to work correctly.
2020-05-20 11:34:27 +02:00
Sumit Jamgade
5dd853e302 add option pkg-ccache to be passed to build-cmd
this option is a /path/to/_ccache.tar. The patch just forwards that
option to the build script.
2020-05-19 16:15:49 +02:00
Marcus Huewe
6b5b0fd55d Merge branch 'fix_help_depends_on' of https://github.com/lethliel/osc
Split dependson and whatdependson into two commands. This way, both
commands are part of "osc --help".
2020-05-19 12:56:55 +02:00
lethliel
798e3e4f63 split dependson/whatdependon in two commands
split the code of do_dependson into two separate commands (just for
the osc help overview)

They are doing the opposite of each other.

Duplicate code was moved to _dependson()

do_whatdependson and do_dependson just call _dependson with an option
reverse set to None or 1.
2020-05-19 12:50:15 +02:00
Marcus Huewe
9b2cbfead0 Merge branch 'fix_osc_api_without_arguments' of https://github.com/lethliel/osc
Fix handling of incorrect CLI arguments (python3).
2020-05-18 20:05:25 +02:00
lethliel
69b1233316 add regex for python3 missing arguments err
add new regex and check for missing arguments.
The error message in python3 differs from the one in python2.

python3:
do_api() missing 1 required positional argument: 'url'

python2:
do_api() takes exactly 4 arguments (3 given)

To be compatible with python2 two checks are needed.
2020-05-18 19:46:22 +02:00
12e0b67117 complete --shell-cmd support parts to get it working 2020-05-12 13:16:29 +02:00
5d6351b1bd Support git@ (private github) or git:// URLs correctly 2020-05-05 10:54:43 +02:00
lethliel
9a3fda8471 quote prj, repository, arch and package
quote the arguments before creating the apiurl.
This prevents weird behaviour when any has an '#'
at the end.
2020-04-28 14:47:40 +02:00
William Brown
bdb0c8033c Add ccache argument for oscrc 2020-04-14 14:50:24 +10:00
lethliel
81acc39c9f e.reason is a string not a integer decimal 2020-04-01 12:48:12 +02:00
Marcus Huewe
894f1c281a Merge branch 'packagequery_bytes_fixes' of https://github.com/marcus-h/osc
bytes fixes/cleanups in the packagequery and repodata modules.
2020-03-25 19:39:11 +01:00
lethliel
b99c4486e8 update news and open new development milestone 2020-03-23 13:50:01 +01:00
Marcus Huewe
55aef1a014 Convert repodata.RepoDataQueryResult to a bytes API
The repodata.RepoDataQueryResult is supposed to be a bytes API and
that's what our users (see build module) expect.
Note that the repodata.RepoDataQueryResult.path method still returns
a str. That's what the rpmquery.RpmQuery, debquery.DebQuery, and
archquery.ArchQuery classes also do (if the "path" was initially
passed as a str).

Fixes: #760 ("osc build fails when called with --prefer-pkgs where the
       passed directory is a repodata repository or a subdirectory of one")
2020-03-15 18:30:28 +01:00
Marcus Huewe
cd51f47a77 Return bytes in packagequery.PackageQueryResult.evr() instead of a str
The packagequery.PackageQueryResult class is supposed to provide a
bytes API. Hence, packagequery.PackageQueryResult.evr() should return
bytes instead of a str. Also, adjust the single caller in the build
module.
2020-03-15 18:30:00 +01:00
lethliel
3914b8c5e8 release 0.168.2 2020-03-13 07:44:34 +01:00
Marcus Huewe
33bbc57b5f Fix the previously introduced escaping via the html module
This is a follow-up commit for commit
6dbf103e10 ("Use html.escape instead
removed cgi.escape"), which breaks the python2 backward compatibility
(since the "html" module is not available by default) and also breaks
the code in general (due to missing html imports).

The fix is based on the proposed fix in [1].

Fixes: boo#1166537 ("osc rq accept - forwarding request causes backtrace")

[1] https://github.com/openSUSE/osc/pull/764
2020-03-12 23:00:47 +01:00
6dbf103e10
Use html.escape instead removed cgi.escape
Fixes:

`Traceback (most recent call last):
  File "/usr/bin/osc", line 41, in <module>
    r = babysitter.run(osccli)
  File "/usr/lib/python3.8/site-packages/osc/babysitter.py", line 64, in run
    return prg.main(argv)
  File "/usr/lib/python3.8/site-packages/osc/cmdln.py", line 344, in main
    return self.cmd(args)
  File "/usr/lib/python3.8/site-packages/osc/cmdln.py", line 367, in cmd
    retval = self.onecmd(argv)
  File "/usr/lib/python3.8/site-packages/osc/cmdln.py", line 501, in onecmd
    return self._dispatch_cmd(handler, argv)
  File "/usr/lib/python3.8/site-packages/osc/cmdln.py", line 1232, in _dispatch_cmd
    return handler(argv[0], opts, *args)
  File "/usr/lib/python3.8/site-packages/osc/commandline.py", line 1458, in do_submitrequest
    result = create_submit_request(apiurl,
  File "/usr/lib/python3.8/site-packages/osc/core.py", line 4244, in create_submit_request
    cgi.escape(message))
AttributeError: module 'cgi' has no attribute 'escape'
`
`cgi.escape` was deprecated in python 3.2
2020-03-11 16:58:04 +01:00
lethliel
a8641a47c8 release 0.168.0 2020-03-04 10:21:17 +01:00
Jan Chren
fa205be798
Suggest correct python-keyring package version
On Tumbleweed, `zypper in python-keyring` installs python2 version, while `osc` runs on python3.
After this change, user will be pointed to the correct version.
2020-02-29 00:28:20 +00:00
Marcus Huewe
c3e7428631 Merge branch 'master' of https://github.com/matthewdva/osc
Fix "osc importsrcpkg --name <name>". Do not try to decode a str.
2020-02-26 21:46:42 +01:00
Marcus Huewe
4e8e0492e8 Fix arch zst magic in util.packagequery
The correct zst magic is b'(\xb5/\xfd' (4 bytes) (that's what obs-build
is also using).

Kudos to Tobias Ellinghaus for spotting this.

Fixes: #756 ("zst detection fails")
2020-02-26 20:04:26 +01:00
Matthew D
9ba0d8adcd Fix -n / --name flag on importsrcpkg command.
osc importsrcpkg -n <pacname> does not work.  If the option is supplied, osc
mistakenly trys to "decode" the pac object.  This patch limit the decode
call when pac is not a string.

Refactored fix based on suggestions from marcus-h
2020-02-24 00:23:14 -05:00
175bcb4613
Remove compat code for Python < 2.6 2020-02-21 15:09:58 +01:00
Marcus Huewe
20f2f40614 Merge branch 'fix_748' of https://github.com/lethliel/osc
Add missing oscerr import in util.helper. Oops.
2020-02-20 09:10:57 +01:00
lethliel
95c68dc3f0 import oscerr in helper.py 2020-02-20 08:45:02 +01:00
Nicholas Brown
75ba922eb2 custom exception if importing m2crypto fails
fixes #743
2020-02-18 17:35:29 +00:00
Marcus Huewe
5185026084 Merge branch 'fix_name_display' of https://github.com/lethliel/osc
Old keyring versions of the KeyringBackend class have no name method (the
name method was added in 2016). In this case, the class name is used as the
backend name.
2020-02-14 09:52:20 +01:00
lethliel
6a20fd8bf5 fix list of backends for old python-keyring
old python-keyring classes have no name method.
This is used instead:

return self._keyring_backend.__class__.__name__
2020-02-14 09:35:07 +01:00
Marcus Huewe
9b0dcf3535 Merge branch 'master' of https://github.com/sbahling/osc
Use configparser.ConfigParser instead of configparser.SafeConfigParser,
since the latter will be removed in future python versions. No functional
changes because SafeConfigParser is a ConfigParser except that its __init__
prints a DeprecationWarning.
2020-02-09 15:57:49 +01:00
Scott Bahling
b9adde96b6 Rename SafeConfigParser to ConfigParser
configparser.SafeConfigParser has been changed to configparser.ConfigParser
upstream in Python 3.2. Warnings are saying that the alias will be
removed in future versions.

Signed-off-by: Scott Bahling <sbahling@suse.com>
2020-02-07 12:04:49 +01:00
Marcus Huewe
32859d6803 Merge branch 'handle-string-conversion' of https://github.com/krig/osc
Add core.parse_meta_to_string helper to work around the insane
implementation of core.meta_exists. Since core.meta_exists may return
a list of bytes, a str, a list of str etc., we ultimately convert the
data to str before passing it ET.fromstring(...).

In case of bytes, the explicit decoding is OK because it is assumed to
be a valid utf-8 encoding (the data represents an xml).

Note: at the moment core.parse_meta_to_string is also called even if it
is not necessary (it is only necessary if the "create" parameter of a
corresponding core.meta_exists call is True).

Note 2: this is just a temporary workaround and, eventually, we will make
the implementation of core.meta_exists more reasonable. When doing so,
we will also remove "public" function core.parse_meta_to_string again.
(Yes, this breaks API but the core.meta_exists change will also break the
API in some sense - so that's OK.)
2020-02-07 10:27:10 +01:00
Marcus Huewe
fb9e64c2f1 Merge branch 'handle_empty_releas_in_rpmquery' of https://github.com/lethliel/osc
Do not pass None as a release to RpmQuery.filename. The release might
be None in case of a kiwi build (the returned canonname is not used in
case of kiwi build).
2020-02-07 10:23:35 +01:00
Kristoffer Grönlund
16a3fcfabf Handle bytes vs. str error when parsing meta (#683)
In all the cases where meta_exists returns either
string data, bytes data or a list, the output needs
to be parsed correctly.

Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>
2020-02-07 10:02:27 +01:00
lethliel
f5aa389e38 handle empty release in build.py
This is very unlikely but in very rare cases this
can happen.

Builing kiwi images containing debian is one case.
In this case we do not know what is inside the kiwi
file and osc build assumes buildtype 'rpm' to generate
a package list which get's thrown away anyway.

Now we just check for release
2020-02-07 09:41:23 +01:00
Marco Strigl
9449745b5e
Merge pull request #673 from lethliel/fix_non_colon_checkout_on_prj_level
fix checkout_no_colon on project level
2020-02-07 09:27:00 +01:00
lethliel
cf6939e68a fix checkout_no_colon on project level
prj_dir was not altered when issuing osc co on
project level.

Solution: Replace ":" with "/" on project level when
no output_dir is given
2020-02-07 09:21:36 +01:00
Marcus Huewe
4d0c9f2dc6 Merge branch 'print_additional_APIURL_links' of https://github.com/lethliel/osc
Add a "print_web_links" option. If enabled, a webui url is printed after
certain operations. For now, it is only used in "do_submitrequest" in order
to print the webui url that can be used to view the newly created request
(note: the print_web_links option is not considered in the "cd prj; osc sr"
case).
2020-02-06 21:26:20 +01:00
lethliel
f079be17c2 print web url links for creating requests
This will print the direct url to show the created request.
New general bool option 'print_web_links' must be set to enable
this.

Right now this is only for creating requests. More to follow.
2020-02-06 13:57:38 +01:00
Marco Strigl
d279dfbedb
Merge pull request #723 from lethliel/improve_error_message_on_ssl_version_mismatch
improve SSLError message
2020-02-06 13:31:54 +01:00
Marco Strigl
ff7b64d5eb
Merge pull request #727 from lethliel/version_in_osc_maintained
print verion in osc maintained if --version is given
2020-02-06 11:05:23 +01:00
lethliel
90dd172af0 print verion in osc maintained if -v is given
Determine the version of the package for maintained project
and print version or 'unknown' if version unknown.
2020-02-06 10:55:51 +01:00
lethliel
1ee7b638cc improve SSLError message
Improve the SSLError message if a version mismatch (TLSv1) is
detected and give the user a hint what is wrong.
(Basically the API does not supprt TLS <= 1.2 and the python
version does not support TLS >= 1.2)
2020-02-06 10:50:22 +01:00
Marcus Huewe
73d880e138 Fix bogus None check in core.vc_export_env
In case of an error, core.get_user_data returns an empty list.
None is never returned. Hence, only pop data from the returned list,
if it is non-empty.
2020-02-06 10:09:52 +01:00
Marcus Huewe
d22ca55975 Merge branch 'vertical' of https://github.com/adrianschroeter/osc
Add a "--vertical" option to "osc results <project>" to list the
packages vertically instead of horizontally.
2020-02-04 20:24:06 +01:00
Marcus Huewe
55dcfa72c6 Merge branch 'add_release_project_to_print_if_exists' of https://github.com/lethliel/osc
Print out the release project (if known) when creating a maintenance
request (just to be a bit more verbose).
2020-02-04 19:50:27 +01:00
Marcus Huewe
d94e457632 Merge branch 'fix_build_user_determination' of https://github.com/lethliel/osc
By default, pass --norootforbuild to the build script when running osc
shell/chroot (unless --userootforbuild is passed to osc).
2020-02-04 19:43:24 +01:00
8716dcc24a use signdummy for product builds
avoids build failure, but leads to unsigned media
2020-02-04 17:26:24 +01:00
37f19f7de9 fix counter handling for obsrepositories:/ in products
all entries had the same name, so repos got ignored.
2020-02-04 17:25:14 +01:00
2978b37fa7 - allow "osc r --vertical" for projects
dunno about others, but the default horizontal listing is not really
usable. This is at least true if you have more packages then
repostories.
2020-01-30 14:41:01 +01:00
lethliel
0b7b515f11 fix decoding in interactive request mode
In interactive review mode:

If a diff is issued and the request is accepted with 'a -m ok'
the tmpfile with the diff will be read. This tmpfile.read() call
is now decoded properly.
2020-01-24 09:23:24 +01:00
lethliel
82ca311eb3 print release_project on maintenancerequests
When creating maintenance requests print out the relese_project
if already known at this point.
2020-01-23 13:28:40 +01:00
lethliel
9849112278 append --norootforbuild as default
pass opts to run_build and check if norootforbuild
needs to be appended to cmd. (if opts.userootforbuild is not set)
2020-01-22 14:24:58 +01:00
Marcus Huewe
efbc60f147 Merge branch 'fix_deletereq_for_repo' of https://github.com/lethliel/osc
Support "osc deletereq <project> -r <REPO>" (previously, the -r option
could only be used if a <package> was specified as well).
2020-01-16 22:26:43 +01:00
lethliel
897c23c0ab Fix deletereq for repository
Do not require --all if --repository is given. There
are no packages touched when removing a repository for
a home project.
2020-01-16 09:01:43 +01:00