The general pattern is that a C function that participates in `errno`
leaves `errno` unspecified on success, or sets it to a defined value
on failure. Unfortunately, it is not always clear what it means for
a `strtoll()`-like function to have failed.
If an extreme value (maximum positive or minimum negative) is returned,
then we can be confident that checking for `errno == ERANGE` is the
only way to distinguish between "the true value is the max/min value"
(`errno == 0`) and "the true value would be out of range and cannot
be returned" (`errno == ERANGE`).
Otherwise, ignore `errno`, and instead rely on checking `endptr`.
This matters because `g_ascii_strtoull()` does not *only* call a
function resembling `strtoull()` (`strtoull_l()` or its reimplementation
`g_parse_long_long()`): it also calls `get_C_locale()`, which wraps
`newlocale()`. Even if `newlocale()` succeeds (which in practice we
expect and assume that it will), it is valid for it to clobber `errno`.
For example, it might attempt to open a file that only conditionally
exists, which would leave `errno` set to `ENOENT`.
This is difficult to reproduce in practice: I encountered what I
believe to be this bug when compiling GLib-based software for i386 in a
Debian 12 derivative via an Open Build Service instance, but I could
not reproduce the bug in a similar chroot environment locally, and I
also could not reproduce the bug when compiling for x86_64 or for a
Debian 10, 11 or 13 derivative on the same Open Build Service instance.
It also cannot be reproduced via the GTest framework, because
`g_test_init()` indirectly calls `g_ascii_strtoull()`, resulting in
the call to `newlocale()` already having happened by the time we enter
test code.
Resolves: https://gitlab.gnome.org/GNOME/glib/-/issues/3418
Signed-off-by: Simon McVittie <smcv@collabora.com>
b685b1f0c4