1
0
mirror of https://github.com/openSUSE/osc.git synced 2024-11-14 08:16:15 +01:00
Commit Graph

115 Commits

Author SHA1 Message Date
feb53212dd Modernize code with pyupgrade
pyupgrade --keep-percent-format --py36-plus `find -name '*.py'`
2022-07-28 19:14:12 +02:00
229913a77f Clean imports up, drop python 2 fallbacks 2022-07-28 13:17:43 +02:00
194f829297
fix product build rpm caching
* src/noarch rpm packaages needs to be stored in scheduler architecture to avoid
  conflicts of the multiple versions
* avoid removal of every downloaded file
2022-07-07 12:30:55 +02:00
William Brown
1ba375d81f Show the md5s that are failing to validate 2022-05-06 12:27:43 +10:00
fd6634e320 Re-download file from API when hdrmd5 doesn't match 2022-02-28 09:28:32 +01:00
d76ec31ef4 Honor --download-api-only option
Used Marcus Huewe's patch from:
https://github.com/openSUSE/open-build-service/issues/12034#issuecomment-1003051674
2022-02-18 12:00:53 +01:00
Marcus Huewe
7da451c87d Do not download a bdep with a hdrmd5 from the api by default
Since a recent backend change, a bdep has a hdrmd5 by default. That
is, osc always downloads these bdeps from the API (unless they are
cached) instead of a mirror. This is not intended.
Using a mirror is no problem because the hdrmd5s are verified in
the build module.
Note: If this causes a problem, one could also use "osc build
--download-api-only" to mimic the old behavior.
2021-12-08 10:41:04 +01:00
4d7dd3d46c
fix hdmrd5 check of local cached files
Current OBS is delivering hdrmd5 in buildinfo. It turns out
that osc has already code for validating cached files, but it
invalidates all local files atm with python 3.x
2021-10-19 16:01:24 +02:00
3b90480dfc
fix downloading from mirrors
Some of our repositories have specific download urls. osc is ignoring
this so far and just tries to use the generic downloadurl

This code prefers definitions for individual path elements if they exist.
We could IMHO remove the old code, since old OBS instances would still
work via the api download fallback.

Real life examples for repo specific configs are on openSUSE all
repositories outside of the /repositories/ directory. eg.

  <path project="openSUSE:Tumbleweed" repository="dod" url="http://download.opensuse.org/tumbleweed/repo/oss/"/>

Co-Author: Marcus Hüwe <suse-tux@gmx.de>
2021-10-06 13:16:40 +02:00
Marcus Huewe
36f7b8ffe9 Fix a potential TypeError in CpioRead.copyin and CpioRead.copyin_file
If no "dest" argument is specified when calling CpioRead.copyin or
CpioRead.copyin_file, a TypeError occurs in CpioRead._copyin_file
because os.getcwd(), which returns a str, is used as dest and, hence,
the subsequent os.path.join(...) fails (because it tries to join a
str and a bytes).
In order to avoid this, encode the result of os.getcwd().

Note that the existing

archive.copyin_file(hdr.filename,
                    os.path.dirname(tmpfile),
                    os.path.basename(tmpfile))

was OK because CpioRead._copyin_file os.path.join()s "dest" and
"new_fn", which are both str. It is just changed to stress that
CpioRead is a bytes-only API.

Fixes: #865 ("Traceback in osc/util/cpio.py line 128: TypeError:
Can't mix strings and bytes in path components")
2020-11-20 09:55:09 +01:00
lethliel
297b050f1e remove dead opener code
The opener is generated but never used.
Since commit 95ec7dee7b
2020-06-08 20:31:28 +02:00
lethliel
a79c54418b fix security issue (bsc#1122675) no / in filename
This checks if the filename of a downloaded file has
been modified (for example by a MITM attack) to contain
slashes. This could mean that the file is compromised
and that the attacker tries to overwrite system files.
2020-05-27 11:17:40 +02:00
Marco Strigl
3d2f1a4ffb
Merge pull request #718 from adrianschroeter/arch_zst
- support zstd arch linux files in local build
2020-01-10 14:06:56 +01:00
5f2721d8f6 - support zstd arch linux files in local build
Note: This requires a tar executable supporting zstd
2020-01-09 15:49:54 +01:00
Marco Strigl
40abc51657
Merge pull request #698 from adrianschroeter/fix_api
fix Fetch api to become compatible again.
2019-12-10 11:43:59 +01:00
lethliel
67358c9746 remove decoding of hdr.filename
It is wrong in this case.
2019-12-10 11:24:17 +01:00
ef4e0411e9 fix Fetch api to become compatible again.
Broke in db795c8121 as reported by Marcus
2019-12-10 07:54:01 +01:00
db795c8121 support local building using RedHat rpm-md modules 2019-12-05 13:06:30 +01:00
lethliel
41158412f8 fix decoding in fetch_cpio 2019-12-05 13:02:24 +01:00
lethliel
50891392ce fix output in http_debug mode for pubkey not found
Variable e is not known at the time of the call and
e.strerror is None nevertheless. So just ommit e.strerror.
2019-11-13 11:40:22 +01:00
Maximilian Meister
1c825ec8b6
cast bytes object to string during os.path.join
this makes sure not to mix objects to prevent a TypeError

TypeError: Can't mix strings and bytes in path components

Fixes#567

Signed-off-by: Maximilian Meister <mmeister@suse.de>
2019-05-28 13:29:12 +02:00
lethliel
96ce14221b [python3] fetch.py python3 ready
* the result of a division is a float
  but we need int to continue.
* use decode_it to decode hdr.filename
2019-04-07 10:06:24 -05:00
lethliel
2c905ad94f do not add key to buildinfo if not existent
fixes issue: https://github.com/openSUSE/osc/issues/471

Add the key_path only to buildinfo(bi) if it exists (not try_parent)
2018-11-20 20:41:03 +01:00
lethliel
65b053abb3 replace urlgrabber to enable python3 compatibility
- new module grabber.py
  * OscMirrorGroup to keep urlgrabber.mirrorgroup behavior
  * OscFileGrabber moved here

- meter.py
  * reworked to use progressbar module instead of progressbar
    of urlgrabber and simplified the module.

- babysitter.py
  * removed URLGrabErrorr

- build.py
  * removed adding of url_local to urllist. (not needed anymore)
  * removed URLGrabError

- commandline.py
  * switched from urlgrabber.urlgrab to OscFileGrabber().urlgrab

- core.py
  * reworked progressbar behavior

- fetch.py
  * removed join_url (not needed anymore)
  * moved OscFileGrabber to grabber.py
  * removed failureReport (not needed anymore)
2018-10-19 09:31:37 +02:00
Michael Schroeder
c3e0bbccb2 Container support: use multiple container extensions when checking the cache
The container may end in .tar.xz, .tar.gz, or .tar
2017-08-02 11:02:14 +02:00
Michael Schroeder
2b7cef3b33 Prepare for md5sum checking of containers
We do not want to use an old container if a new version is built
on the server. They don't have a rebuild count or something like
that (yet).
2017-05-17 11:20:03 +02:00
Michael Schroeder
7dae421d77 Support derived images for kiwi docker builds
Kiwi currently rejects the --set-container-derived-from option,
but this will hopefully change.
2017-05-16 14:02:37 +02:00
Marcus Huewe
89ac49f659 fetch.py: reworded Fetcher.failureReport message a bit
Based on PR#182.
2016-02-05 11:28:47 +01:00
Marcus Huewe
84ba8c6e17 - fixed fullfilename calculation in the Pac class
Now, the fullfilename is calculated using the canonname of a
bdep instead of using the bdep's binary attribute (the canonname
and binary attribute can differ (e.g. ConsoleKit-64bit-0.4.6-3.2.ppc.rpm
vs. ::import::ppc64::ConsoleKit-64bit-0.4.6-3.2.ppc.rpm))
2015-06-16 17:37:40 +02:00
Michael Schroeder
f47c39521d Verify hdrmd5 of packages if specified in the buildinfo
Needed for buildenv builds that want specific packages.
2014-11-17 12:40:54 +01:00
2be979361e All rename cases require delete=False 2013-08-13 11:54:11 +02:00
531251a86c Fix file descriptor leaks during download
This makes it possible to build packages which
require more than 512 packages for building.
2013-08-12 18:31:27 +02:00
Danny Al-Gaaf
6a6b25f9cb fetch.py: fix "More than one statement on a single line"
Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
2013-05-27 14:58:15 +02:00
Danny Al-Gaaf
6db0431cb7 fetch.py: fix bad indentation
Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
2013-05-27 14:16:58 +02:00
Marcus Huewe
cdd73ba983 - fetch.OscFileGrabber: fix for commit 9ab5a37113
We still have to support older urlgrabber versions where the class
"URLGrabber" is an old-style class and therefore we cannot use a "super(...)"
call. This fixes issue #32
2013-05-23 15:04:20 +02:00
Gui Chen
9ab5a37113 fix OscFileGrabber missing 'opts' attribute issue
given urlgrabber version 3.9.1, MirrorGroup trys to access
grabber.opts, but the derived grabber OscFileGrabber hasn't
this attribute

Signed-off-by: Gui Chen <gui.chen@intel.com>
2013-05-14 10:16:10 +02:00
Marcus Huewe
51e6fca88f - fetch: check if errno == 14 before accessing the URLGrabError's code attribute
Unfortunately the class URLGrabError does not define constants for the
different error codes therefore we have to hardcode the errno (alternatively
we can define them in our fetch module).
2013-04-22 15:54:55 +02:00
Michal Vyskocil
419367fca3 python3 compatibility: urllib
fixes all renames in urllib, urllib2, urlparse modules in python3
2013-04-16 10:51:42 +02:00
Michal Vyskocil
7f2031558c python3 compatibility: print function
The most visible change in python3 - removal of print statement and all
the crufty
   print >> sys.stderr, foo,

The from __future__ import print_function makes it available in python
2.6
2013-04-16 10:51:41 +02:00
Michal Vyskocil
0dcbddbafe python3 compatibility: new syntax for literals
The octal literals got a new syntax 0755 -> 0o755
2013-04-16 10:51:18 +02:00
Michal Vyskocil
c612e8d47e python3 compatibility: use relative imports
Use relative imports when using module's own modules - this makes a
clear distinction between already developed copy and installed package.
2013-04-16 10:51:18 +02:00
Michal Vyskocil
3a93ac6d10 python3 compatibility: except
changes 'except FooError, fe' to 'except FooError as fe'

available in python 2.6
2013-04-16 10:51:17 +02:00
Michal Vyskocil
d3648be24b python3 compatibility: dict
this patch
 1.) removes the iteritems/itervalues, which were dropped in py3
     items/values are used instead
 2.) add an extra list() in a cases the list-based access is needed
     (included appending, indexing and so)
 3.) changes a sorting idiom in few places
     instead of
     foo = dict.keys()
     foo.sort()
     for i in foo:

     there is a recommended

     for i in sorted(dict.keys()):
 4.) in one occassion it removes a if dict.has_key() by simpler
   dict.get(key, default)
2013-04-16 10:51:17 +02:00
b0ddb63432 - Abort when the server is not answering with 200 or 404 when downloading _pubkey files
- really use the internal rpm signature check
2013-03-19 17:17:14 +01:00
Marcus Huewe
161b9ca274 - Fetcher.__fetch_cpio: split cpio request into smaller requests if the server returns code 414 2012-10-10 17:05:26 +02:00
Michael Schroeder
8df38af7e3 support archlinux builds 2012-04-03 16:00:06 +02:00
Stephan Kulow
b1502a5715 support buildinfos with binary present - and don't die with XML 2012-02-22 13:28:19 +01:00
511ebf9d89 - don't say to download cpio (why just one package ?;) 2011-09-30 07:42:57 +02:00
Egbert Eich
db2e538efd Sanitize file permissions for downloaded files.
Downloaded files may not have read permission for everybody.
This doesn't seem to be a problem since the user as well as
root should be able to read them anyhow. This is true unless
the directory where the cache resides is mounted over nfs
with root_squash.

Signed-off-by: Egbert Eich <eich@suse.de>
2011-08-16 11:33:56 +02:00
Ludwig Nussel
208230bc58 fix --no-init 2011-02-22 11:17:26 +01:00