meson, ci: Support tests that can fail under certain conditions

We have tests that are failing in some environments, but it's
difficult to handle them because:
 - for some environments we just allow all the tests to fail: DANGEROUS
 - when we don't allow failures we have flacky tests: A CI pain

So, to avoid this and ensure that:
 - New failing tests are tracked in all platforms
 - gitlab integration on tests reports is working
 - coverage is reported also for failing tests

Add support for `can_fail` keyword on tests that would mark the test as
part of the `failing` test suite.
Not adding the suite directly when defining the tests as this is
definitely simpler and allows to define conditions more clearly (see next
commits).

Now, add a default test setup that does not run the failing and flaky tests
by default (not to bother distributors with testing well-known issues) and
eventually run all the tests in CI:
 - Non-flaky tests cannot fail in all platforms
 - Failing and Flaky tests can fail

In both cases we save the test reports so that gitlab integration is
preserved.
This commit is contained in:
Marco Trevisan (Treviño)
2022-10-19 20:08:15 +02:00
parent 9635fd4e40
commit 62dca6c1cf
10 changed files with 111 additions and 31 deletions

View File

@@ -149,9 +149,9 @@ test_extra_programs = {
'gsubprocess-testprog' : {},
}
python_tests = [
'codegen.py',
]
python_tests = {
'codegen.py' : {},
}
test_env = environment(common_test_env)
test_env.set('G_TEST_SRCDIR', meson.current_source_dir())
@@ -907,6 +907,10 @@ foreach test_name, extra_args : gio_tests
local_test_env.append(var, value)
endforeach
if extra_args.get('can_fail', false)
suite += 'failing'
endif
test(test_name, exe,
env : local_test_env,
timeout : timeout,
@@ -930,13 +934,19 @@ foreach program_name, extra_args : test_extra_programs
)
endforeach
foreach test_name : python_tests
foreach test_name, extra_args : python_tests
suite = ['gio', 'no-valgrind']
if extra_args.get('can_fail', false)
suite += 'failing'
endif
test(
test_name,
python,
args: ['-B', files(test_name)],
env: test_env,
suite: ['gio', 'no-valgrind'],
suite: suite,
)
if installed_tests_enabled