Now that we're using TAP parsing, this will show subtest failures in
details but without showing any logging error, that we'd still need to parse
from actual logs.
That file is created if running the `unstable_tests` suite succeeds. It
can fail, though, leaving that log file nonexistent. There’s no point in
failing the whole test run by bailing out if postprocessing the log file
fails.
Occasionally postprocessing can fail with a `FileNotFoundError`.
Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
This reverts commit 91f14cd058.
The freedesktop SDK, which is used by gnome-build-meta, only has Meson
0.63. Bumping GLib’s Meson dependency to 0.64 means that, at the moment,
GLib is not buildable in gnome-build-meta and hence can’t be tested in
nightly pipelines against other projects, etc.
That’s bad for testing GLib.
It’s arguably bad that we’re restricted to using an older version of
Meson than shipped by Debian Testing, but that’s a separate discussion
to be had.
Revert the Meson 0.64 dependency until the freedesktop SDK ships Meson ≥
0.64. This also means reverting the simplifications to use of
`gnome.mkenum_simple()`.
See https://gitlab.gnome.org/GNOME/glib/-/merge_requests/3077#note_1601064
We have tests that are failing in some environments, but it's
difficult to handle them because:
- for some environments we just allow all the tests to fail: DANGEROUS
- when we don't allow failures we have flacky tests: A CI pain
So, to avoid this and ensure that:
- New failing tests are tracked in all platforms
- gitlab integration on tests reports is working
- coverage is reported also for failing tests
Add support for `can_fail` keyword on tests that would mark the test as
part of the `failing` test suite.
Not adding the suite directly when defining the tests as this is
definitely simpler and allows to define conditions more clearly (see next
commits).
Now, add a default test setup that does not run the failing and flaky tests
by default (not to bother distributors with testing well-known issues) and
eventually run all the tests in CI:
- Non-flaky tests cannot fail in all platforms
- Failing and Flaky tests can fail
In both cases we save the test reports so that gitlab integration is
preserved.
This is what’s available in the new Debian Stable, so we can expect it
to be available pretty much everywhere.
Subsequent commits will clean up old workarounds.
Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
gitlab will drop cmd.exe support with GitLab 13 so I took the opportunity to
add new runners with Windows 2016 and powershell as default.
These runners are tagged with win32-ps instead of win32. The old runners
will be switched off in the coming weeks.
The main difference is that all commands and env expansions use powershell
and Windows 2016 instead of 2012r2.
This effectively renders those tests useless (since realistically nobody
runs tests locally), but it’s better than every other CI run failing for
unrelated reasons. The idea is that the ‘flaky’ tag can be temporarily
applied to a test while a problem is being investigated or fixed, and
then removed later.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
