mirror of
https://github.com/openSUSE/osc.git
synced 2024-11-10 06:46:15 +01:00
The Command Line Interface to work with an Open Build Service
c932f95d46
The old code only supports a file whose size is less then or equal to INT_MAX (due to a reasonable(!) limit in M2Crypto). The actual issue is in core.http_request which mmap(...)s the file, wraps it into a memoryview/buffer and then passes the memoryview/buffer to urlopen. Eventually, the whole memoryview/buffer is read into memory (see m2_PyObject_GetBufferInt). If the file is too large (> INT_MAX), m2_PyObject_GetBufferInt raises a ValueError (which is perfectly fine!). Reading a whole file into memory is completely insane. In order to avoid this, we now simply pass a file-like object to urlopen (more precisely, the file-like object is associated with the Request instance that is passed to urlopen). The advantange is that the file-like object is processed in chunks of 8192 bytes (see http.client.HTTPConnection) (that is, only 8192 bytes are read into memory (instead of the whole file)). There are two pitfalls when passing a file-like object to urlopen: * By default, a chunked Transfer-Encoding is applied. It seems that some servers (like api.o.o) do not like this (PUTing a file with a chunked Transfer-Encoding to api.o.o results in status 400). In order to avoid a chunked Transfer-Encoding, we explicitly set a Content-Length header (we also do this in the non-file case (just for the sake of completeness)). * If the request fails with status 401, it is retried with an appropriate Authorization header. When retrying the request, the file's offset has to be repositioned to the beginning of the file (otherwise, a 0-length body is sent which most likely does not match the Content-Length header). Note: core.http_request's "data" and "file" parameters are now mutually exclusive because specifying both makes no sense (only one of them is considered) and it simplifies the implementation a bit. Fixes: #202 ("osc user authentification seems to be broken with last commit") Fixes: #304 ("osc ci - cannot handle more than 2 GB file uploads") |
||
|---|---|---|
| dist | ||
| docs | ||
| fuse | ||
| osc | ||
| tests | ||
| .gitignore | ||
| .travis.yml | ||
| AUTHORS | ||
| COPYING | ||
| MANIFEST.in | ||
| NEWS | ||
| osc_expand_link.pl | ||
| osc-wrapper.py | ||
| osc.fish | ||
| osc.ico | ||
| osc.png | ||
| PROJ_PACK.txt | ||
| README | ||
| run_bandit.sh | ||
| setup.py | ||
| TODO | ||
osc -- opensuse-commander with svn like handling
Patches can be submitted via
* mail to opensuse-buildservice@opensuse.org
* Bugzilla: https://bugzilla.opensuse.org/enter_bug.cgi?product=openSUSE.org&component=BuildService
* or the official Git repository on Github:
https://github.com/openSUSE/osc
INSTALLATION:
RPM packages are here (rpm-md repository):
http://download.opensuse.org/repositories/openSUSE:/Tools/
To install from git, do
python setup.py build
python setup.py install
# create a symlink 'osc' in your path pointing to osc.py.
ln -s osc-wrapper.py /usr/bin/osc
Alternatively, you can directly use osc-wrapper.py from the source dir
(which is easier if you develop on osc).
CONFIGURATION:
When you use it for the first time, it will ask you for your username and
password, and store it in ~/.oscrc.
CONFIGURATION MIGRATION (only affects versions >= 0.114):
Version 0.114 got some cleanups for the configfile handling and therefore some
options are now deprecated, namely:
* apisrv
* scheme
One new option was added:
* apiurl = <protocol>://<somehost> # use this as the default apiurl. If this
option isn't specified the default (https://api.opensuse.org) is used.
So far osc still has some backward compatibility for these options but it might
get removed in the future that's why it issues a deprecation warning in case
one of those options is still in use.
The new configuration scheme looks like the following:
# entry for an apiurl
[<protocol>://<apiurl>]
user = <username>
password = <password>
...
'''Before starting the migration please save your ~/.oscrc file!'''
If the migration doesn't work for whatever reason feel free to send me an email
or ask on the opensuse-buildservice mailinglist or in the #opensuse-buildservice
irc channel.
=== Migration case I (apisrv only) ===
The apisrv option is used to specify the default apihost. If apisrv isn't
specified at all the default ("api.opensuse.org") is used.
The current [general] section looks like this:
[general]
...
apisrv = <somehost>
# or
apisrv = <protocol>://<somehost>
apisrv got superseded by the new apiurl option which looks like this:
[general]
...
apiurl = <protocol>://<somehost>
If apisrv has no "<protocol>" https is used. Make sure all apiurl sections have
the new format which is described above. Afterwards apisrv can be removed.
=== Migration case II (scheme only) ===
The current [general] section looks like this:
[general]
...
scheme = <protocol>
This means every apiurl section which don't have the new format which is
described above for instance
[<somehost>]
user = <username>
password = <password>
...
has to be converted to
[<protocol>://<somehost>]
user = <username>
password = <password>
...
Afterwards the scheme option can be removed from the [general] section (it
might be the case that some sections already have the correct format).
=== Migration case III (apisrv and scheme) ===
The current [general] section looks like this:
[general]
...
apisrv = <somehost>
scheme = <protocol>
Both options can be removed if all apiurl sections have the new format which is
described above. So basically just adjust all apiurl sections (it might be the
case that some sections already have the correct format).
KEYRING USAGE
Osc now can store passwords in keyrings instead of ~/.oscrc. To use it,
you need python-keyring and either python-keyring-kde or -gnome.
If you want to switch to using a keyring you need to delete apiurl section
from ~/.oscrc and you will be asked for credentials again, which will be then
stored in the keyring application.
WORKING COPY INCONSISTENT (only affects version >= 0.130)
osc's working copy handling was rewritten in 0.130. Thus some
consistency checks were added. As a result osc might complain
that some old working copies are in an inconsistent state:
Your working copy '.' is in an inconsistent state.
Please run 'osc repairwc .' (Note this might _remove_
files from the .osc/ dir). Please check the state
of the working copy afterwards (via 'osc status .')
To fix this simply run "osc repairwc ." as suggested in the
error message. Note that "osc repairwc ." might need to contact
the api in order to fetch some missing files. Also it might remove
some files from the storedir (.osc/) but it won't touch any locally
modified files.
If it DOES NOT fix the problem please create a bug report and attach
your working copy to the bug (if possible).
USAGE EXAMPLES:
(online at http://en.opensuse.org/openSUSE:OSC )
To list existing content on the server
osc ls # list projects
osc ls Apache # list packages in a project
osc ls Apache subversion # list files of package of a project
Check out content
osc co Apache # entire project
osc co Apache subversion # a package
osc co Apache subversion foo # single file
Update a working copy
osc up
osc up [pac_dir] # update a single package by its path
osc up * # from within a project dir, update all packages
osc up # from within a project dir, update all packages
# AND check out all newly added packages
If an update can't be merged automatically, a file is in 'C' (conflict)
state, and conflicts are marked with special <<<<<<< and >>>>>>> lines.
After manually resolving the problem, use
osc resolved foo
Upload change content
osc ci # current dir
osc ci <dir>
osc ci file1 file2 ...
Show the status (which files have been changed locally)
osc st
osc st <directory>
osc st file1 file2 ...
Mark files to be added or removed on the next 'checkin'
osc add file1 file2 ...
osc rm file1 file2 ...
Adds all new files in local copy and removes all disappeared files.
osc addremove
Generates a diff, to view the changes
osc diff # current dir
osc diff file1 file2 ...
Shows the build results of the package
osc results
osc results [repository]
Shows the log file of a package (you need to be inside a package directory)
osc log <repository> <arch>
Shows the URLs of .repo files which are packages sources for Yum/YaST/smart
osc repourls [dir]
Triggers a package rebuild for all repositories/architectures of a package
osc rebuildpac [dir]
Shows available repository/build targets
osc repository
Shows the configured repository/build targets of a project
osc repository <project>
Shows meta information
osc meta Apache
osc meta Apache subversion
osc id username
Edit meta information
(Creates new package/project if it doesn't exist)
osc editmeta Apache
osc editmeta Apache subversion
Update package meta data with metadata taken from spec file
osc updatepacmetafromspec <dir>
There are other commands, which you may not need (they may be useful in scripts):
osc repos
osc buildconfig
osc buildinfo
Locally build a package (see 'osc help build' for more info):
osc build <repo> <arch> specfile [--clean|--noinit]
Update a package to a different sources (directory foo_package_source):
cp -a foo_package_source foo; cd foo; osc init <prj> <pac>; osc addremove; osc ci; cd $OLDPWD; rm -r foo
HINT FOR W3M USERS
Putting the following in the file ~/.w3m/passwd will make
w3m know the credentials for the buildservice servers:
"""
host api.opensuse.org
port 80
realm Authentication required
login foo
password bar
host build.opensuse.org
port 80
realm openSUSE Build Service
login foo
password bar
"""
chmod 0600 ~/.w3m/passwd
NOTES about the testsuite
A new test suite has been created and should run via doing
# cd tests
# python suite.py