If size is 0, using pb.Percentage will result in a
ZeroDivisionException. Note: the output in case of
size == 0 looks a bit "strange" - for a consistent
output we should probably subclass pb.Percentage.
* new function create_text_meter with fallback selection
* NoPBTextMeter.start() will print the basename (if not stated otherise with
basename = None)
* The callers that should use an alternare TextMeter class now call create_text_meter()
* The callers that should not use and alternate TextMeter (because of different handling,
like build.py) call create_text_meter(use_pb_fallback=False)
* the warning 'Please install the progressbar module' is now only shown once
improvements
Now, CpioWrite provides a bytes-only API. It would be also possible
that the API accepts bytes and str (we would need to explicitly
encode the latter) but this would be a bit inconsistent wrt.
cpio.CpioRead (which is bytes-only).
Also, by using a bytesarray instead of a [] we avoid several
intermediate ''.join(...)s.
This is a bytes only API because a filename in a cpio archive can
contain, for instance, illegal utf-8 sequences. A user can decode
the filename/content as she wishes.
A ValueError is more appropriate because there is no issue with the
ar archive itself. Also, the old codepath never worked because the
fn parameter was missing.
Since an ar archive can contain arbitary filenames (that is a
filename can be an invalid utf-8 encoding (for instance,
"foo\xff\xffbar")), the ar module provides a bytes only API. A
user can decode filenames as she wishes.
Note: if a "fn" parameter is passed to Ar.__init__ it should be a
bytes (a str is also ok, but then be aware that an ArError's file
attribute might be a str or a bytes).
There is no need to unpack a single byte because it is not
affected by (byte) endianness (and that's what struct.unpack is
about). Moreover, rpmquery.unpack_string now supports an optional
encoding parameter, which could be used by the python3 port to
decode a string. Note: in general we cannot assume that all strings
in a rpm are utf-8 encoded (it is possible to build a rpm that
contains illegal utf-8 sequences).
With the NoPBTextMeter class the build view gets broken.
Old view:
1/11 (repo) filename
new view:
Please install the progressbar module...
Please install the progressbar module...
Please install the progressbar module...
With this commit the old behavior is restored.
The getbinaries call now lists the file he downloads instead of just
stating "Please install the progressbar module..." several times.
(but only if not called with the option quiet)
Since commit 8a6abe3a6c ("warn user
if python-progressbar is not installed") it is always safe to import
the meter module (even if the progressbar module is not installed).
On ImportError have_pb_module is false and the class NoPBTextMeter gets
returned which prints "Please install progressbar module..." on TextMeter.start()
This functions are used in the whole code and are
mandatory for the python3 support to work. In python2
case nothing is touched.
* cmp_to_key:
converts a cmp= into a key= function
* decode_list:
decodes each element of a list. This is needed if
we have a mixed list with strings and bytes.
* decode_it:
Takes the input and checks if it is not a string.
Then it uses chardet to get the encoding.
If urlgrab returns a URLError (for example if the Network is unreachable)
the for loop did not continue and the osc build aborts.
Now we also catch the URLError and try the next mirror and return False
correctly if no mirror could be reached. And then try to download it from
api
Get rid of the urlgrabber dependency. The current implementation of the
progress bar is quite "noisy" if the line length exceeds the size of
the terminal window, but that's something we could fix later. (The
superfluous error message will be fixed in a subsequent commit.)
This is useful so one can wait with a following osc prjresult until
everything done. Without this one can not know if a release job hasn't
started yet or is already finished.
* altered command do_checkout
new option -D | --deleted. Can only be used with PACKAGE and needs
-o to work
* what happens:
core function checkout_deleted_package fetches the file list of the deleted
package and saves it to given destdir. If destdir is not existent it
creates the destination directory. Otherwise the files will be written
in the existing directory.
The old code could potentially yield to a use-after-free situation,
which results in UB. For this, consider the following scenario, where
osc performs several HTTPS requests (assumption: the server supports
ssl session resumption):
- HTTPS Request 1:
* a new SSL *s connection is established, which also creates a new
SSL_SESSION *ss => ss->references == 1
* once the handshake is done, the ss is put into the session cache
(see ssl_update_cache) => ss->references == 2
- osc saves the session ss in a class variable
- s is SSL_free()d, which calls SSL_SESSION_free => ss->references == 1
- HTTPS Request 2:
* setup a new SSL *s connection that reuses the saved session ss
=> ss->references == 2
* once the handshake is done, ssl_update_cache is called, which is a
NOP, because s->hit == 1 (that is, the session was resumed)
* osc saves the session ss in a class variable
* s is SSL_free()d, which calls SSL_SESSION_free => ss->references == 1
...
> 2 hours later (see tls1_default_timeout)
...
- HTTPS Request 256:
* setup a new SSL *s connection that reuses the saved session ss
=> ss->references == 2
* once the handshake is done, ssl_update_cache is called, but is
_no_ NOP anymore
* ssl_update_cache flushes the session cache (this is done every
255/256 (depending on the way we count) connections) => ss is
SSL_SESSION_free()d => ss->references == 1
* osc saves the session ss in a class variable
* s is SSL_free()d, which calls SSL_SESSION_free:
since ss->references == 1, ss is eventually free()d
- HTTPS Request 257:
* setup a new SSL *s connection that reuses the saved session ss
Since ss does not exist anymore, the remaining program execution is UB.
(Note: SSL_free(...) is _NOT_ called, if M2Crypto 0.29 is used.
M2Crypto 0.30 calls SSL_free(...) again.)
Due to a bug in OpenSSL_1_1_0h (see openssl commit 8e405776858) the
scenario from above can be triggered with exactly 2 HTTPS requests (the
SSL_SESSION is not cached, because we configured SSL_VERIFY_PEER, but
no sid_ctx was set). This is fixed in openssl commit c4fa1f7fc01.
In order to reliably reuse a session, we probably need to listen to the
session cache changes. Such callbacks could be registered via
SSL_CTX_sess_set_new_cb and/or SSL_CTX_sess_set_remove_cb, but both
functions are not provided by M2Crypto. Another idea is to directly utilize
the session cache, but this also has to be implemented in M2Crypto first.
Yet another approach is to retrieve the session via SSL_get1_session, which
increases the session's refcnt, but this also needs to be implemented in
M2Crypto first (if we choose to use this approach, we also have to make
sure that we eventually free the session manually...).
Fixes: #398 ("SIGSEGV on \"osc commit\"")
Always pass the "--statistics" option to the build script (only affects
a vm build). It is not implemented as an option, because we already have
so many of them... (see also https://github.com/openSUSE/osc/pull/412).
yet another option, but
* only very old build scripts don't know it, we should just require a recent one
* build script is ignoring it for chroot case
so why bother with another option?
Note that the diff is only shown, if it was requested before (that is
the "i" command was issued). The new behavior is consistent with the
other commands like "a", "d" etc.
Storing the error encoding in an "encoding" attribute "breaks" the
python3 "input" function: In essence, builtin_input_impl does a
getattr(sys.stdout, 'encoding'), which returns our error encoding
instead of the "real" stdout encoding. In order to avoid this, we
store the error encoding in an "_encoding" attribute.
Making SafeWriter a new-style class simplifies the code a lot.
