luc14n0/glib
1
0
Fork 0
mirror of https://gitlab.gnome.org/GNOME/glib.git synced 2025-08-31 05:14:11 +02:00
Code Issues Packages Projects Releases Wiki Activity

meson, ci: Support tests that can fail under certain conditions

Browse Source 
We have tests that are failing in some environments, but it's
difficult to handle them because:
 - for some environments we just allow all the tests to fail: DANGEROUS
 - when we don't allow failures we have flacky tests: A CI pain

So, to avoid this and ensure that:
 - New failing tests are tracked in all platforms
 - gitlab integration on tests reports is working
 - coverage is reported also for failing tests

Add support for `can_fail` keyword on tests that would mark the test as
part of the `failing` test suite.
Not adding the suite directly when defining the tests as this is
definitely simpler and allows to define conditions more clearly (see next
commits).

Now, add a default test setup that does not run the failing and flaky tests
by default (not to bother distributors with testing well-known issues) and
eventually run all the tests in CI:
 - Non-flaky tests cannot fail in all platforms
 - Failing and Flaky tests can fail

In both cases we save the test reports so that gitlab integration is
preserved.
This commit is contained in:
Marco Trevisan (Treviño)
2022-10-19 20:08:15 +02:00
parent 9635fd4e40
commit 62dca6c1cf
10 changed files with 111 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
.gitlab-ci/test-msys2.sh
View File

@@ -51,7 +51,9 @@ lcov \
--output-file "${DIR}/_coverage/${CI_JOB_NAME}-baseline.lcov"
# FIXME: fix the test suite
meson test --timeout-multiplier "${MESON_TEST_TIMEOUT_MULTIPLIER}" --no-suite flaky || true
meson test --timeout-multiplier "${MESON_TEST_TIMEOUT_MULTIPLIER}" || true
meson test --timeout-multiplier "${MESON_TEST_TIMEOUT_MULTIPLIER}" \
--setup=unstable_tests --suite=failing --suite=flaky || true
lcov \
--quiet \