We’re essentially trying to build a minimal perfect hash function, and
`vals` is the map which represents that function. If we redefine a
member of `vals`, the map is no longer a partial function — one input
value (a Unicode codepoint) has two output values (compose table
indices).
So it’s bad if a member of `vals` gets redefined, and we want to be
notified if that happens.
As it happens, some of the new codepoints in Unicode 16.0 cause these
checks to fail. For example, U+16121 Gurung Khema Vowel Sign U
decomposes to U+1611E U+1611E. This causes `vals{U+1611E}` to be defined
to an index from the `first` map, and then redefined to an index from
the `second` map.
The following few commits will fix this, but let’s get the checks in
first.
Signed-off-by: Philip Withnall <pwithnall@gnome.org>
Because how these big tables of numbers work is perhaps a bit hard to
figure out, and it would be useful to document the design decisions
involved in it.
Signed-off-by: Philip Withnall <pwithnall@gnome.org>
All changes mechanically generated with:
```
./tools/update-unicode-data.sh ~/Downloads/UCD 16.0.0
```
using the data from https://www.unicode.org/Public/16.0.0/ucd/UCD.zip.
Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
Fixes: #3470
Manually added from the data in
https://www.unicode.org/Public/16.0.0/ucd/UCD.zip.
The following commit will mechanically update the Unicode tables to use
them.
Signed-off-by: Philip Withnall <pwithnall@gnome.org>
Helps: #3470
Only one other previous author, and my contribution just now is so
simple as to not be copyrightable.
Signed-off-by: Philip Withnall <pwithnall@gnome.org>
Helps: #1415
The time has finally come when Unicode has specified a codepoint above
U+FFFF which has a decomposition: U+16125 GURUNG KHEMA VOWEL SIGN AI, in
Unicode 16 which the following commits will add support for.
So far, we’ve managed to store the reverse-lookup from decomposed pairs
to their composed form using a 16-bit integer. Now we have to switch to
storing the composed form in a 32-bit `gunichar` as U+16125 won’t fit
otherwise.
This introduces no functional changes, but does double the in-memory
size of the `compose_array` table from 9176 bytes to 19932 bytes.
The code which uses this lookup table, in `gunidecomp.c`, was already
implicitly converting the loaded value to a `gunichar`, so needs no
changes.
When we update to Unicode 16, the new `NormalizationTest.txt` file
contains a test which will check that composed codepoints > U+FFFF work.
Specifically, U+11391 TULU-TIGALARI LETTER AU is tested.
Signed-off-by: Philip Withnall <pwithnall@gnome.org>
Helps: #3470
On recent versions of Debian, PST8PDT is part of the tzdata-legacy
package, which is not always installed and might disappear in future.
Successfully tested with and without tzdata-legacy on Debian unstable.
Signed-off-by: Simon McVittie <smcv@debian.org>
Instead of using timestamp 0 as a magic number (in this case interpreted
as 1970-01-01T00:00:00-08:00), calculate a timestamp from a recent
year/month/day in winter, in this case 2024-01-01T00:00:00-08:00.
Similarly, instead of using a timestamp 15 million seconds later
(1970-06-23T15:40:00-07:00), calculate a timestamp from a recent
year/month/day in summer, in this case 2024-07-01T00:00:00-07:00.
Signed-off-by: Simon McVittie <smcv@debian.org>
In newer tzdata, it is an alias for America/Los_Angeles, which has a
slightly different meaning: DST did not exist there before 1883. As a
result, we can no longer hard-code the knowledge that interval 0 is
standard time and interval 1 is summer time, and instead we need to look
up the correct intervals from known timestamps.
Resolves: https://gitlab.gnome.org/GNOME/glib/-/issues/3502
Bug-Debian: https://bugs.debian.org/1084190
[smcv: expand commit message, fix whitespace]
Signed-off-by: Simon McVittie <smcv@debian.org>
We need to write a Meson cross-compilation file on the fly here, hence the
additions in test-msvc.bat to set up the paths.
Like the 32-bit VS2019 CI job this is only run manually or weekly.
Why they’re necessary, why we _think_ the optimised implementation of
`g_utf8_validate()` is OK despite what valgrind and asan are telling us,
and how they work.
Signed-off-by: Philip Withnall <pwithnall@gnome.org>
Helps: #3493
Commit 760a6f647 rearranged how the lengths are calculated for the test
data and added `escape_data_string()` so they could be printed safely.
Unfortunately there was a miscount in the length of the first test
vector in `test_read_upto()`: there are 31 bytes in the string literal,
plus one nul terminator which is added by the compiler. The quoted
string length was 32 bytes. This should be fine (explicitly including
the nul delimiter), but then `escape_data_string()` adds another byte to
the length because it assumes the nul delimiter has *not* been included
in the count.
Changing the string length from 32 to 31 breaks the tests, as the final
component of the data is then the wrong length, so add an additional
explicit nul byte to the string literal so that it matches the length.
Signed-off-by: Philip Withnall <pwithnall@gnome.org>
This is a follow up to commit e7e5ddd2a. oss-fuzz found a case where
performance was pathologically bad with a long `stop_chars` string.
Since our inner loop in that case was iterating over `stop_chars` and
comparing each of them to `buffer[i]`, we can use `memchr()` the
opposite way round to in commit e7e5ddd2a to speed that up, using
`buffer[i]` as the needle in a `stop_chars` haystack.
From some brief testing, this doesn’t impact on the performance of a
more normal use case of having a short (<10 bytes long) `stop_chars`. I
was slightly concerned that the function call overhead of calling out to
`memchr()` would have an impact there, but apparently not.
Signed-off-by: Philip Withnall <pwithnall@gnome.org>
oss-fuzz#372994443
I wish people would stop moving their documentation around without
adding redirects. This is not how the internet is supposed to work.
Signed-off-by: Philip Withnall <pwithnall@gnome.org>
The existing g-io-module-default-singleton-calloc suppression
seems applicable to definite _g_io_module_get_default_type leaks
seen with Valgrind 3.23.0.
This check is necessary for Solaris & illumos, where 32-bit libelf
is incompatible with large-file mode, which meson forces to be enabled,
but 64-bit libelf works fine.
Signed-off-by: Alan Coopersmith <alan.coopersmith@oracle.com>
For historical reasons, pid_t & mode_t are defined as long instead
of int for 32-bit processes in the Solaris headers, and even though
they are the same size, gcc issues -Wformat headers if you try to
print them with "%d" and "%u" instead of "%ld" & "%lu".
Signed-off-by: Alan Coopersmith <alan.coopersmith@oracle.com>
Previously the build was requesting interfaces matching SUSv1/Unix95,
as implemented in Solaris 2.6 and later. This changes it to try the
most recent version supported, and limits to the versions supported
by OS versions that meson supports. This includes these _XOPEN_SOURCE
versions:
800 (2024): supported by illumos starting in July 2024
700 (2008): supported by Solaris 11.4 & illumos from 2014-2024
600 (2001): supported by Solaris 10-11.3 & illumos prior to 2014
Signed-off-by: Alan Coopersmith <alan.coopersmith@oracle.com>
Like _GNU_SOURCE on glibc, this tells the header to define functions
not included in the requested standards versions. This is needed to
build glib/tests/utils-c-89 with -std=c89 and utils-c-89 with -std=c99
and still be able to call functions like isnan() and realpath().
Signed-off-by: Alan Coopersmith <alan.coopersmith@oracle.com>
The refactoring done by commit 168fd4f2b3
lost the definition of the open_max variable used in the Solaris ifdefs.
Signed-off-by: Alan Coopersmith <alan.coopersmith@oracle.com>
The method was correctly returning an error from
`g_data_input_stream_read_line_utf8()` if the line contained invalid
UTF-8, but it wasn’t correctly setting the returned line length to 0.
This could have caused problems if callers were basing subsequent logic
on the length and not the return value nullness or `GError`.
Signed-off-by: Philip Withnall <pwithnall@gnome.org>
oss-fuzz#372819437
These two data structures are allocated once and live for the lifetime
of the process, and are leaked on exit. That’s fine, and intentional.
Add `g_ignore_leak()` to them to make that a bit clearer, and
communicate the intent to asan.
Signed-off-by: Philip Withnall <pwithnall@gnome.org>
Helps: #3472
The goal of this change is to avoid re-parsing `GVariantTypeInfo` on
every creation or parsing of a `GVariant` byte-buffer. Parsing presents
a non-trivial amount of overhead which can typically be elided.
It was discovered that many applications and tooling are re-generating
this information upon receiving a D-Bus message as they tend to process
messages serially, thus dropping the last reference count.
Previously, when the last reference count for a `GVariantTypeInfo` was
dropped we would finalize the parsed type information.
This change keeps `GVariantTypeInfo` alive in a Garbage Collected array.
The array is collected upon reaching a 32 entries. The number 32 was
chosen because it is larger than what I've seen active on various D-Bus
based applications-or-daemons.
Take a simple test case of using `GVariantBuilder` in a loop with a
debugoptimized build of GLib. A reduction in wallclock time can be
observed in the 35% to more than 70% based on the complexity of the
GVariant being created.
For cases like ibus-daemon, it was previously parsing `GVariantTypeInfo`
up to dozens of times per key-press/release cycle.
Closes: #3472
While reading a single byte or uint16 from an input stream is fairly
simple and uncontroversial, the code to read a line or read up to any of
a set of stop characters is not so trivial. People may be using
`GDataInputStream` to parse untrusted input like this, so we should
probably test that it’s robust against a variety of input conditions.
Signed-off-by: Philip Withnall <pwithnall@gnome.org>
This attempts to use GCC __attribute__((ifunc("resolver_func"))) to check
for valgrind early in the process startup so that the proper function is
dispatched instead of runtime checks within the function.
This should make #3493 less annoying when run under Valgrind.
