This will make it easier and more obvious for developers to run them
locally: I'm sure I'm not the only developer who had assumed that
`.gitlab-ci/` is private to the CI environment and inappropriate (or
perhaps even destructive) to run on a developer/user system.
The lint checks are automatically skipped (with TAP SKIP syntax) if we
are not in a git checkout, or if git or the lint tool is missing. They
can also be disabled explicitly with `meson test --no-suite=lint`,
which downstream distributions will probably want to do.
By default, most lint checks are reported as an "expected failure"
(with TAP TODO syntax) rather than a hard failure, because they do not
indicate a functional problem with GLib and there is a tendency for
lint tools to introduce additional checks or become more strict over
time. Developers can override this by configuring with `-Dwerror=true`
(which also makes compiler warnings into fatal errors), or by running
the test suite like `LINT_WARNINGS_ARE_ERRORS=1 meson test --suite=lint`.
One exception to this is tests/check-missing-install-tag.py, which is
checking a functionally significant feature of our build system, and
seems like it is unlikely to have false positives: if that one fails,
it is reported as a hard failure.
run-style-check-diff.sh and run-check-todos.sh are not currently given
this treatment, because they require search-common-ancestor.sh, which
uses Gitlab-CI-specific information to find out which commits are in-scope
for checking.
Signed-off-by: Simon McVittie <smcv@collabora.com>
Now that we're using TAP parsing, this will show subtest failures in
details but without showing any logging error, that we'd still need to parse
from actual logs.
We have tests that are failing in some environments, but it's
difficult to handle them because:
- for some environments we just allow all the tests to fail: DANGEROUS
- when we don't allow failures we have flacky tests: A CI pain
So, to avoid this and ensure that:
- New failing tests are tracked in all platforms
- gitlab integration on tests reports is working
- coverage is reported also for failing tests
Add support for `can_fail` keyword on tests that would mark the test as
part of the `failing` test suite.
Not adding the suite directly when defining the tests as this is
definitely simpler and allows to define conditions more clearly (see next
commits).
Now, add a default test setup that does not run the failing and flaky tests
by default (not to bother distributors with testing well-known issues) and
eventually run all the tests in CI:
- Non-flaky tests cannot fail in all platforms
- Failing and Flaky tests can fail
In both cases we save the test reports so that gitlab integration is
preserved.
I haven’t tested any of them. This is entirely mechanical. I used
shellcheck 0.7.0 with default options.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Add a separate CI job which runs memcheck on the unit tests. This is
done as a separate job from the main build, since we don’t want it to
interact with code coverage at all.
Currently, failure of this job is ignored. Issue #333 will eventually
fix that.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Fixes: #487
GitLab can show the results of a CI pipeline if the pipeline generates a
report using the JUnit XML format.
Since Meson provides a machine parseable output for `meson test`, we can
take that and turn it into XML soup.
