We have tests that are failing in some environments, but it's
difficult to handle them because:
- for some environments we just allow all the tests to fail: DANGEROUS
- when we don't allow failures we have flacky tests: A CI pain
So, to avoid this and ensure that:
- New failing tests are tracked in all platforms
- gitlab integration on tests reports is working
- coverage is reported also for failing tests
Add support for `can_fail` keyword on tests that would mark the test as
part of the `failing` test suite.
Not adding the suite directly when defining the tests as this is
definitely simpler and allows to define conditions more clearly (see next
commits).
Now, add a default test setup that does not run the failing and flaky tests
by default (not to bother distributors with testing well-known issues) and
eventually run all the tests in CI:
- Non-flaky tests cannot fail in all platforms
- Failing and Flaky tests can fail
In both cases we save the test reports so that gitlab integration is
preserved.
In principle we could script this so that each max-version.c is compiled
26 times, once per possible MAX_VERSION, but I haven't implemented
that here: just pinning to the oldest possible version is sufficient to
reproduce #2796.
These aren't included in the installed-tests, since they don't really
do anything at runtime (the important thing is that they compile
without warnings).
Reproduces: #2796
Signed-off-by: Simon McVittie <smcv@collabora.com>
Otherwise, the build will fail when the toolchain is static-only, even
with -Ddefault_library=static. I talked to a Meson developer in their
IRC channel, who told me that the correct fix was to ensure that
shared_library is only used if default_library != static.
We've various macros definitions that are depending using C++ features
that may not work in all the standard versions, so recompile the cxx
tests that we have in all the ones we want to support.
We don't need a cpp toolchain for building glib so lets just
automatically disable tests requiring one when not available.
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>