luc14n0/glib
1
0
Fork 0
mirror of https://gitlab.gnome.org/GNOME/glib.git synced 2025-09-05 15:38:44 +02:00
Code Issues Packages Projects Releases Wiki Activity

meson, ci: Support tests that can fail under certain conditions

Browse Source 
We have tests that are failing in some environments, but it's
difficult to handle them because:
 - for some environments we just allow all the tests to fail: DANGEROUS
 - when we don't allow failures we have flacky tests: A CI pain

So, to avoid this and ensure that:
 - New failing tests are tracked in all platforms
 - gitlab integration on tests reports is working
 - coverage is reported also for failing tests

Add support for `can_fail` keyword on tests that would mark the test as
part of the `failing` test suite.
Not adding the suite directly when defining the tests as this is
definitely simpler and allows to define conditions more clearly (see next
commits).

Now, add a default test setup that does not run the failing and flaky tests
by default (not to bother distributors with testing well-known issues) and
eventually run all the tests in CI:
 - Non-flaky tests cannot fail in all platforms
 - Failing and Flaky tests can fail

In both cases we save the test reports so that gitlab integration is
preserved.
This commit is contained in:
Marco Trevisan (Treviño)
2022-10-19 20:08:15 +02:00
parent 9635fd4e40
commit 62dca6c1cf
10 changed files with 111 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
.gitlab-ci/test-msvc.bat
View File

@@ -18,7 +18,8 @@ python .gitlab-ci/check-missing-install-tag.py _build || goto :error
ninja -C _build || goto :error
:: FIXME: dont ignore test errors
meson test -C _build --timeout-multiplier %MESON_TEST_TIMEOUT_MULTIPLIER% --no-suite flaky
meson test -C _build --timeout-multiplier %MESON_TEST_TIMEOUT_MULTIPLIER%
meson test -C _build --timeout-multiplier %MESON_TEST_TIMEOUT_MULTIPLIER% --setup=unstable_tests --suite=failing --suite=flaky
:: FIXME: can we get code coverage support?