Migration fixes 2021-07-26
Peter's fix for a bunch of races
-> Seem to fix the occasional crash seen by Peter
Wei's fix for migration with free page hinting
-> Bug has been around for a while, but makes a huge difference
My fix for OpenBSD test corner case
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
# gpg: Signature made Mon 26 Jul 2021 13:42:16 BST
# gpg: using RSA key 45F5C71B4A0CB7FB977A9FA90516331EBC5BFDE7
# gpg: Good signature from "Dr. David Alan Gilbert (RH2) <dgilbert@redhat.com>" [full]
# Primary key fingerprint: 45F5 C71B 4A0C B7FB 977A 9FA9 0516 331E BC5B FDE7
* remotes/dgilbert-gitlab/tags/pull-migration-20210726a:
migration: clear the memory region dirty bitmap when skipping free pages
migration: Move the yank unregister of channel_close out
migration: Teach QEMUFile to be QIOChannel-aware
migration: Introduce migration_ioc_[un]register_yank()
migration: Make from_dst_file accesses thread-safe
migration: Fix missing join() of rp_thread
tests/qtest/migration-test.c: use 127.0.0.1 instead of 0
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
This will test the PMR functionality.
Signed-off-by: Gollu Appalanaidu <anaidu.gollu@samsung.com>
Reviewed-by: Klaus Jensen <k.jensen@samsung.com>
[k.jensen: replaced memory-backend-file with memory-backend-ram]
Signed-off-by: Klaus Jensen <k.jensen@samsung.com>
From clang-13:
tests/unit/test-iov.c:161:26: error: variable 't' set but not used \
[-Werror,-Wunused-but-set-variable]
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
OpenBSD doesn't like :0 as an address, switch to using 127.0.0.1
in baddest; it's really testing the :0 port number that isn't allowed
on anything.
(The test doesn't currently run anyway because of the userfault
problem that Peter noticed, but this gets us closer to being able to
reenable it)
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Message-Id: <20210719185217.122105-1-dgilbert@redhat.com>
Acked-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Philippe Mathieu-Daude <philmd@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Doc, metadata, plugin and testing updates for 6.1-rc1:
- git ignore some file editor detritus
- add overview on device emulation terminology
- remove needless if leg in configure custom devices logic
- numerous gitdm/mailmap updates
- fix plugin_exit race for linux-user
- fix a few bugs in cache modelling plugin
- fix plugin calculation of physical address
- handle pure assembler/linker tcg tests outside of docker
- add tricore build to gitlab
- remove superfluous MacOSX task
- generalise the OpenBSI gitlab rules
# gpg: Signature made Fri 23 Jul 2021 17:28:26 BST
# gpg: using RSA key 6685AE99E75167BCAFC8DF35FBD0DB095A9E2A44
# gpg: Good signature from "Alex Bennée (Master Work Key) <alex.bennee@linaro.org>" [full]
# Primary key fingerprint: 6685 AE99 E751 67BC AFC8 DF35 FBD0 DB09 5A9E 2A44
* remotes/stsquad/tags/pull-for-6.1-rc1-230721-1: (28 commits)
gitlab-ci: Extract OpenSBI job rules to reusable section
gitlab-ci: Remove the second superfluous macos task
gitlab: enable a very minimal build with the tricore container
tests/tcg/configure.sh: add handling for assembler only builds
plugins: Fix physical address calculation for IO regions
plugins/cache: Fixed "function decl. is not a prototype" warnings
plugins/cache: limited the scope of a mutex lock
plugins/cache: Fixed a bug with destroying FIFO metadata
tcg/plugins: implement a qemu_plugin_user_exit helper
contrib/gitdm: add more individual contributor entries.
contrib/gitdm: add a new interns group-map for GSoC/Outreachy work
contrib/gitdm: add an explicit academic entry for BU
contrib/gitdm: add group-map for Netflix
contrib/gitdm: add domain-map for NVIDIA
contrib/gitdm: add domain-map for Crudebyte
contrib/gitdm: un-ironically add a mapping for LWN
contrib/gitdm: add domain-map/group-map for Wind River
contrib/gitdm: add domain-map for Eldorado
contrib/gitdm: add domain-map/group-map mappings for Samsung
gitdm.config: sort the corporate GroupMap entries
...
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Rather than base of the shared Debian 10 container which would require
us to bring in even more dependencies just bring in what is needed for
building tricore-softmmu in GitLab. We don't even remove the container
from the DOCKER_PARTIAL_IMAGES lest we cause more confusion.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210720232703.10650-28-alex.bennee@linaro.org>
Up until this point we only handled local compilers or assumed we had
everything in the container. This falls down when we are building QEMU
inside the container.
This special handling only affects tricore for now but I put it in a
case just in case we add any other "special" targets. Setting
CROSS_CC_GUEST is a bit of a hack just to ensure the test runs as we
gate on a detected compiler even though the Makefile won't actually
use it. It also means we display something sane in the configure
output.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210720232703.10650-27-alex.bennee@linaro.org>
This new adaptor visitor takes a single field of the adaptee, and exposes it
with a different name.
This will be used for QOM alias properties. Alias targets can of course
have a different name than the alias property itself (e.g. a machine's
pflash0 might be an alias of a property named 'drive'). When the target's
getter or setter invokes the visitor, it will use a different name than
what the caller expects, and the visitor will not be able to find it
(or will consume erroneously).
The solution is for alias getters and setters to wrap the incoming
visitor, and forward the sole field that the target is expecting while
renaming it appropriately.
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
The point of 'qemu-img convert --bitmaps' is to be a convenience for
actions that are already possible through a string of smaller
'qemu-img bitmap' sub-commands. One situation not accounted for
already is that if a source image contains an inconsistent bitmap (for
example, because a qemu process died abruptly before flushing bitmap
state), the user MUST delete those inconsistent bitmaps before
anything else useful can be done with the image.
We don't want to delete inconsistent bitmaps by default: although a
corrupt bitmap is only a loss of optimization rather than a corruption
of user-visible data, it is still nice to require the user to opt in
to the fact that they are aware of the loss of the bitmap. Still,
requiring the user to check 'qemu-img info' to see whether bitmaps are
consistent, then use 'qemu-img bitmap --remove' to remove offenders,
all before using 'qemu-img convert', is a lot more work than just
adding a knob 'qemu-img convert --bitmaps --skip-broken-bitmaps' which
opts in to skipping the broken bitmaps.
After testing the new option, also demonstrate the way to manually fix
things (either deleting bad bitmaps, or re-creating them as empty) so
that it is possible to convert without the option.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1946084
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210709153951.2801666-4-eblake@redhat.com>
[eblake: warning message tweak, test enhancements]
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Waiting until the end of the convert operation (a potentially
time-consuming task) to finally detect that we can't copy a bitmap is
bad, comparing to failing fast up front. Furthermore, this prevents
us from leaving a file behind with a bitmap that is not marked as
inconsistent even though it does not have sane contents.
This fixes the problems exposed in the previous patch to the iotest:
it adds a fast failure up front, and even if we don't fail early, it
ensures that any bitmap we add but do not properly populate is removed
again rather than left behind incomplete.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210709153951.2801666-3-eblake@redhat.com>
[eblake: add a hint to the warning message, simplify name computation]
Reviewed-by: Nir Soffer <nsoffer@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Enhance the test to demonstrate existing less-than-stellar behavior of
qemu-img with a qcow2 image containing an inconsistent bitmap: we
don't diagnose the problem until after copying the entire image (a
potentially long time), and when we do diagnose the failure, we still
end up leaving an empty bitmap in the destination. This mess will be
cleaned up in the next patch.
While at it, rename the test now that we support useful iotest names,
and fix a missing newline in the error message thus exposed.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210709153951.2801666-2-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Nir Soffer <nsoffer@redhat.com>
Passing fixed-iothread=true should make iothread conflicts fatal,
whereas fixed-iothread=false should not.
Combine the second case with an error condition that is checked after
the iothread is handled, to verify that qemu does not crash if there is
such an error after changing the iothread failed.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20210624083825.29224-3-mreitz@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Tested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
It's possible that requests start to wait each other in
mirror_wait_on_conflicts(). To avoid it let's use same technique as in
block/io.c in bdrv_wait_serialising_requests_locked() /
bdrv_find_conflicting_request(): don't wait on intersecting request if
it is already waiting for some other request.
For details of the dead-lock look at testIntersectingActiveIO()
test-case which we actually fixing now.
Fixes: d06107ade0
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210702211636.228981-4-vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
There is a dead-lock in active mirror: when we have parallel
intersecting requests (note that non intersecting requests may be
considered intersecting after aligning to mirror granularity), it may
happen that request A waits request B in mirror_wait_on_conflicts() and
request B waits for A.
Look at the test for details. Test now dead-locks, that's why it's
disabled. Next commit will fix mirror and enable the test.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210702211636.228981-3-vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
pc,pci,virtio: lots of new features
Lots of last minute stuff.
vhost-user-i2c.
vhost-vsock SOCK_SEQPACKET support.
IOMMU bypass.
ACPI based pci hotplug.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
# gpg: Signature made Fri 16 Jul 2021 16:11:27 BST
# gpg: using RSA key 5D09FD0871C8F85B94CA8A0D281F0DB8D28D5469
# gpg: issuer "mst@redhat.com"
# gpg: Good signature from "Michael S. Tsirkin <mst@kernel.org>" [full]
# gpg: aka "Michael S. Tsirkin <mst@redhat.com>" [full]
# Primary key fingerprint: 0270 606B 6F3C DF3D 0B17 0970 C350 3912 AFBE 8E67
# Subkey fingerprint: 5D09 FD08 71C8 F85B 94CA 8A0D 281F 0DB8 D28D 5469
* remotes/mst/tags/for_upstream3:
vhost-vsock: SOCK_SEQPACKET feature bit support
docs: Add documentation for iommu bypass
hw/i386/acpi-build: Add IVRS support to bypass iommu
hw/i386/acpi-build: Add DMAR support to bypass iommu
hw/arm/virt-acpi-build: Add IORT support to bypass SMMUv3
hw/pci: Add pci_bus_range() to get PCI bus number range
hw/i386: Add a default_bus_bypass_iommu pc machine option
hw/arm/virt: Add default_bus_bypass_iommu machine option
hw/pxb: Add a bypass iommu property
hw/pci/pci_host: Allow PCI host to bypass iommu
docs: Add '-device intel-iommu' entry
hw/virtio: add vhost-user-i2c-pci boilerplate
hw/virtio: add boilerplate for vhost-user-i2c device
bios-tables-test: Update golden binaries
hw/acpi/ich9: Set ACPI PCI hot-plug as default on Q35
bios-tables-test: Allow changes in DSDT ACPI tables
hw/pci/pcie: Do not set HPC flag if acpihp is used
hw/acpi/ich9: Enable ACPI PCI hot-plug
hw/i386/acpi-build: Add ACPI PCI hot-plug methods to Q35
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Merge crypto updates and misc fixes
* Introduce a GNUTLS backend for crypto algorithms
* Change crypto library preference gnutls > gcrypt > nettle > built-in
* Remove built-in DES impl
* Remove XTS mode from built-in AES impl
* Fix seccomp rules to allow resource info getters
* Fix migration performance test
* Use GDateTime in io/ and net/rocker/ code
* Improve docs for -smp
# gpg: Signature made Wed 14 Jul 2021 15:08:00 BST
# gpg: using RSA key DAF3A6FDB26B62912D0E8E3FBE86EBB415104FDF
# gpg: Good signature from "Daniel P. Berrange <dan@berrange.com>" [full]
# gpg: aka "Daniel P. Berrange <berrange@redhat.com>" [full]
# Primary key fingerprint: DAF3 A6FD B26B 6291 2D0E 8E3F BE86 EBB4 1510 4FDF
* remotes/berrange-gitlab/tags/crypto-and-more-pull-request: (26 commits)
qemu-options: rewrite help for -smp options
qemu-options: tweak to show that CPU count is optional
qemu-options: re-arrange CPU topology options
docs: fix typo s/Intel/AMD/ in CPU model notes
tests/migration: fix unix socket migration
seccomp: don't block getters for resource control syscalls
io: use GDateTime for formatting timestamp for websock headers
net/rocker: use GDateTime for formatting timestamp in debug messages
crypto: prefer gnutls as the crypto backend if new enough
crypto: add gnutls pbkdf provider
crypto: add gnutls hmac provider
crypto: add gnutls hash provider
crypto: add gnutls cipher provider
crypto: introduce build system for gnutls crypto backend
crypto: flip priority of backends to prefer gcrypt
crypto: replace 'des-rfb' cipher with 'des'
crypto: delete built-in XTS cipher mode support
crypto: delete built-in DES implementation
crypto: add crypto tests for single block DES-ECB and DES-CBC
crypto: drop custom XTS support in gcrypt driver
...
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Testing and plugin updates:
- custom runner playbooks for configuring GitLab runners
- integrate Cirrus jobs into GitLab via cirrus-run
- clean-up docker package lists
- bump NetBSD to 9.2
- bump OpenBSD to 6.9
- make test-mmap more hexagon friendly
- fixup handling of hostaddr for plugins
- disallow some incompatible plugin configurations
- fix handling of -ldl for BSDs
- remove some old unused symbols from the plugin symbol map
- enable plugins by default for most TCG builds
- honour main build -Wall settings for plugins
- new execlog plugin
- new cache modelling plugin
- fix io_uring build regression
- disable modular TCG on Darwin
# gpg: Signature made Wed 14 Jul 2021 15:56:27 BST
# gpg: using RSA key 6685AE99E75167BCAFC8DF35FBD0DB095A9E2A44
# gpg: Good signature from "Alex Bennée (Master Work Key) <alex.bennee@linaro.org>" [full]
# Primary key fingerprint: 6685 AE99 E751 67BC AFC8 DF35 FBD0 DB09 5A9E 2A44
* remotes/stsquad/tags/pull-testing-and-plugins-140721-5: (44 commits)
MAINTAINERS: Added myself as a reviewer for TCG Plugins
docs/devel: Added cache plugin to the plugins docs
plugins/cache: Added FIFO and LRU eviction policies
plugins/cache: Enable cache parameterization
plugins: Added a new cache modelling plugin
docs/devel: tcg-plugins: add execlog plugin description
contrib/plugins: add execlog to log instruction execution and memory access
contrib/plugins: enable -Wall for building plugins
tcg/plugins: enable by default for most TCG builds
configure: stop user enabling plugins on Windows for now
configure: add an explicit static and plugins check
configure: don't allow plugins to be enabled for a non-TCG build
tcg/plugins: remove some stale entries from the symbol list
meson.build: relax the libdl test to one for the function dlopen
meson.build: move TCG plugin summary output
plugins: fix-up handling of internal hostaddr for 32 bit
tests/tcg: make test-mmap a little less aggressive
tests/vm: update openbsd to release 6.9
tests/vm: update NetBSD to 9.2
tests/docker: expand opensuse-leap package list
...
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
New test case enum-dict-no-name.json crashes:
$ python3 scripts/qapi-gen.py tests/qapi-schema/enum-dict-no-name.json
Traceback (most recent call last):
[...]
File "/work/armbru/qemu/scripts/qapi/expr.py", line 458, in check_enum
member_name = member['name']
KeyError: 'name'
Root cause: we try to retrieve member 'name' before we check for
missing members. With that fixed, we get the expected error "'data'
member misses key 'name'".
Fixes: 0825f62c84
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20210616072121.626431-1-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
The check_aligned_anonymous_unfixed_mmaps and
check_aligned_anonymous_unfixed_colliding_mmaps do a lot of mmap's and
copying of data. This is especially unfriendly to targets like hexagon
which have quite large pages and need to do sanity checks on each
memory access.
While we are at it clean-up the white space and style issues from the
legacy code. As we no longer do quite so much needless memory access
we can also remove the hexagon timeout hack.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Taylor Simpson <tsimpson@quicinc.com>
Message-Id: <20210709143005.1554-27-alex.bennee@linaro.org>
librados-dev is not required by QEMU directly, only librbd-dev.
glusterfs-common is not directly needed by QEMU.
QEMU uses ncursesw only on non-Windows hosts.
The clang package is clang 10.
flex and bison are not required by QEMU.
Standardize on nmap ncat implementation to match Fedora/CentOS.
Remove vim since it is not a build pre-requisite and no other containers
include it.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210623142245.307776-10-berrange@redhat.com>
Message-Id: <20210709143005.1554-18-alex.bennee@linaro.org>
Since Docker Hub has started to enforce pull rate limits on clients, it
is preferrable to use project specific container registries where they
are available. Fedora and OpenSUSE projects provide registries.
The images in these registries are also refreshed on a more regular
basis than the ones in docker hub, so the package update should
generally be faster.
While CentOS also has a registry it is considerably outdated compared
to docker.io, and also only provides x86 images, while docker.io images
are multi-arch.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210623142245.307776-4-berrange@redhat.com>
Message-Id: <20210709143005.1554-12-alex.bennee@linaro.org>
The test aborts and error message as the following be throwed:
"No such file or directory: '/var/tmp/qemu-migrate-{pid}.migrate",
when the unix socket migration test nearly done. The reason is
qemu removes the unix socket file after migration before
guestperf.py script do it. So pre-check if the socket file exists
when removing it to prevent the guestperf program from aborting.
See also commit f9cc00346d ("tests/migration: fix unix socket batch
migration").
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Signed-off-by: Hyman <huangy81@chinatelecom.cn>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Currently the crypto layer exposes support for a 'des-rfb'
algorithm which is just normal single-DES, with the bits
in each key byte reversed. This special key munging is
required by the RFB protocol password authentication
mechanism.
Since the crypto layer is generic shared code, it makes
more sense to do the key byte munging in the VNC server
code, and expose normal single-DES support.
Replacing cipher 'des-rfb' by 'des' looks like an incompatible
interface change, but it doesn't matter. While the QMP schema
allows any QCryptoCipherAlgorithm for the 'cipher-alg' field
in QCryptoBlockCreateOptionsLUKS, the code restricts what can
be used at runtime. Thus the only effect is a change in error
message.
Original behaviour:
$ qemu-img create -f luks --object secret,id=sec0,data=123 -o cipher-alg=des-rfb,key-secret=sec0 demo.luks 1G
Formatting 'demo.luks', fmt=luks size=1073741824 key-secret=sec0 cipher-alg=des-rfb
qemu-img: demo.luks: Algorithm 'des-rfb' not supported
New behaviour:
$ qemu-img create -f luks --object secret,id=sec0,data=123 -o cipher-alg=des-rfb,key-secret=sec0 demo.luks 1G
Formatting 'demo.luks', fmt=luks size=1073741824 key-secret=sec0 cipher-alg=des-fish
qemu-img: demo.luks: Invalid parameter 'des-rfb'
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
