meson, ci: Support tests that can fail under certain conditions

We have tests that are failing in some environments, but it's
difficult to handle them because:
 - for some environments we just allow all the tests to fail: DANGEROUS
 - when we don't allow failures we have flacky tests: A CI pain

So, to avoid this and ensure that:
 - New failing tests are tracked in all platforms
 - gitlab integration on tests reports is working
 - coverage is reported also for failing tests

Add support for `can_fail` keyword on tests that would mark the test as
part of the `failing` test suite.
Not adding the suite directly when defining the tests as this is
definitely simpler and allows to define conditions more clearly (see next
commits).

Now, add a default test setup that does not run the failing and flaky tests
by default (not to bother distributors with testing well-known issues) and
eventually run all the tests in CI:
 - Non-flaky tests cannot fail in all platforms
 - Failing and Flaky tests can fail

In both cases we save the test reports so that gitlab integration is
preserved.
This commit is contained in:
Marco Trevisan (Treviño)
2022-10-19 20:08:15 +02:00
parent 9635fd4e40
commit 62dca6c1cf
10 changed files with 111 additions and 31 deletions

View File

@@ -96,5 +96,10 @@ foreach test_name, extra_args : gmodule_tests
suite = ['gmodule'] + extra_args.get('suite', [])
timeout = suite.contains('slow') ? test_timeout_slow : test_timeout
if extra_args.get('can_fail', false)
suite += 'failing'
endif
test(test_name, exe, env : test_env, timeout : timeout, suite : suite)
endforeach