[qemu-s390 package document]

SLES 12 SP3 QEMU/KVM RELATED SUPPORT STATEMENTS

Overview
--------

  The QEMU based packages included with SLES 12 SP3 provide a large variety of
  features, from the very latest customer requests to features of questionable
  quality or value. The linux kernel includes components which contribute KVM
  virtualization features as well. This document was created to assist the user
  in deciding which features can be relied upon to build enterprise class
  virtualization solutions. KVM based virtualization for x86 (Intel 64/AMD64)
  and for IBM System z (s390x) are offered at the L3 (full support) level,
  while KVM for the ARM64 architecture (AArch64) is L3 supported for certain
  partner-specific use cases. The bulk of this document deals with L3 supported
  features and is primarily s390x centric. This document should be considered a
  companion to the standard virtualization documentation delivered with the
  product.

  KVM is implemented in linux kernel modules which enable the linux kernel to
  function as an integral part of the KVM hypervisor. The hypervisor-guest
  interaction is managed by QEMU through the /dev/kvm ioctl interface. The linux
  host assists in the virtualization of storage, networking and display
  resources as well as allowing direct hardware passthrough of PCI and USB
  devices. Linux memory and cpu management features are used by QEMU/KVM to
  enable guests to share those host resources as efficiently as possible.

  QEMU is a primary component of KVM based virtualization. The legacy qemu-kvm
  program is provided for continuity with pre SLES 12 usage, including in
  libvirt domain xml references. The QEMU emulator binary qemu-system-s390x is
  now the primary programs to use to access KVM virtualization. When using this
  programs, the -machine option accel=kvm (or its alias -enable-kvm) must be
  specified for KVM acceleration to be used by the guest.

  Libvirt is the preferred means of accessing QEMU/KVM functionality and is
  documented elsewhere. This document focuses on the features and direct usage
  of QEMU/KVM as provided by the QEMU based packages.

Major QEMU/KVM Supported Features
---------------------------------

- KVM virtualization is accomplished by using the QEMU program in KVM
  acceleration mode. KVM acceleration requires that both guest and host have
  the same fundamental architecture.

- Guest images created under previous QEMU versions are supported by machine
  version compatibilities built into more recent QEMU versions.

- For ease of use, the QEMU program has defaults which represent traditional
  usage patterns.

- Guest virtual machine characteristics are specified by a combination of
  internal defaults, options provided on the QEMU program command-line, runtime
  configurations set via the monitor interfaces and optional config files. The
  runtime control of a VM is effected either through the Human Monitor
  "Protocol" (HMP), or the JSON based programmatical QEMU Monitor Protocol (QMP)
  interface. For QMP details, see qemu-qmp-ref man page.
  Since a KVM guest runs in the context of a normal linux process, some types
  of execution controls are managed with linux tools.

- Various standard vCPU types are available, along with the ability to specify
  individual CPU features visible to the guest.

- QEMU incorporates virtualized, 390 specific, ccw bus based firmware for
  booting s390 guests. This firmware is automatically incorporated and
  doesn't need to be explicitly referenced.

- Some QEMU messages have been localized to various languages. This is provided
  by the optional qemu-lang package. Keyboard mappings for various nationalities
  is also provided.

- Virtual machine lifecycle controls include startup through the ccw firmware or
  kernel boot, firmware based shutdown, execution pausing, the saving and
  restoring of machine state or disk state, VM migration to another host, and
  miscellaneous controls available through the "monitors" mentioned above.

- Guest execution state may be "moved" in both time (save/restore) and space
  (static and live migration). These migrations or save/restore operations can
  take place either from certain prior SLES versioned hosts to a SLES 12 SP3
  or between hosts of the same version. Certain other restrictions also apply.

- Security considerations include seccomp2 based sandboxing, privileged
  helpers and a security model which allows running guests as a non-root user.

- QEMU provides best effort reuse of existing disk images, including those with
  systems installed, through geometry probing. Also disk images produced by
  other popular virtualization technologies may be imported into QEMU supported
  storage formats. These QEMU formats include features which exploit the
  benefits of virtualization.

- Memory, cpu and disk space overcommit are possible and can be beneficial when
  done responsibly. Additional management of these resources comes in the form
  of memory ballooning or hotplug, host KSM, vcpu hot-add, online disk resizing,
  trim, discard and hole punching.

- Guest performance is enhanced through the use of virtio devices, various disk
  caching modes, network acceleration via the vhost-net kernel module, multi-
  queue network transmit capabilities, host transparent huge pages (THP) and
  direct hugetlb usage. Physical PCI and USB devices may also be passed through
  to the guest, including SR-IOV VF's.

- The guest UI is accessable via GTK, SDL, VNC, Spice, and serial (including
  curses TUI) interfaces.

- Guest timekeeping is supported in a variety of ways, including a paravirtual
  clocksource, and options for the various guest clocks for how to handle the
  timeslicing of the guest's execution on the host.

- In addition to the para-virtualized devices already mentioned, other devices
  and infrastructure designed to avoid virtualization "problem areas" are
  available such as SPICE graphics, vmmouse emulation, tablet style pointer
  interfaces and virtio based UI interfaces.

- A built-in user-mode network (SLIRP) stack is available.

- Portions of the host file system may be shared with a guest by using virtFS.

- A guest "agent" is available for SLES 12 KVM guests via the qemu-guest-agent
  package. This allows some introspection and control of the guest OS
  environment from the host.

QEMU/KVM Technology Previews
----------------------------

- Specifying and placing PCI devices on a PCI bridge allows for a greater number
  of devices.

- All features indicated as not being supported in this document fall under the
  Technology Preview definition contained in the main product's release notes.

Noteworthy QEMU/KVM Unsupported Features
----------------------------------------

- Note that some features are unsupported simply due to lack of validation. If
  an existing feature is desired, but not marked supported, let SUSE know about
  your requirements.

- The TCG "acceleration" mode may be helpful for problem isolation, but
  otherwise presents insufficient benefit and stability.

- GlusterFS integration is not enabled.

Deprecated, Superseded, Modified and Dropped Features
-----------------------------------------------------

- http://wiki.qemu-project.org/Features/LegacyRemoval
  This website tracks feature deprecation and removal at the upstream
  development level. Our qemu package inherits this community direction, but be
  aware that we can and will deviate as needed. Those deviations and additional
  information can be found in this section.

- The use of "?" as a parameter to "-cpu", "-soundhw", "-device", "-M",
  "-machine", "-d", and "-clock" is now considered deprecated. Use "help"
  instead.

- The use of "if=scsi" as a parameter to "-drive" does not work anymore with PC
  machine types, as it created an obsolete SCSI controller model.

- Use of aio=native without direct cache mode also being specified (cache=none,
  cache=directsync, or cache.direct=on) is no longer allowed.

- The use of image encryption in qcow and qcow2 formats is now considered
  deprecated.
  Analysis has shown it to be weak encryption, in addition to suffering from
  poor design. Images can easily be converted to a non-encrypted format.

- Use of acpi, boot-opts, and smp-opts in a -readconfig config file are now
  considered deprecated. In the future those names will be standardized to
  acpitable, boot, and smp respectively.

- These previously supported command line options are now considered deprecated:
  -device ivshmem (use ivshmem-doorbell or ivshmem-plain instead)

- These previously supported command line options are no longer supported:
  -device pci-assign, -device kvm-pci-assign (use -device vfio-pci instead)

- These previously supported command line options are no longer recognized:
  -device pc-sysfw (no longer needed)

- Specifying a cpu feature with both "+feature/-feature" and "feature=on/off"
  will now cause a warning. The current behavior for this combination where
  "+feature/-feature" wins over "feature=on/off", will be changed going forward
  so that "+feature" and "-feature" will be synonyms for "feature=on" and
  "feature=off" respectively.

- The previously supported blkdev-add QMP command has been flagged as lacking
  and could possibly change syntax in the future.

QEMU Command-Line and Monitor Syntax and Support
------------------------------------------------

- The QEMU program command-line syntax is as follows:
  qemu-system-s390x [options]

  Where 'options' are taken from the options listed below.
  The images used with -drive or -cdrom, may be in the raw (no format), qcow2
  or qed storage formats, and may be located in files within the host
  filesystem, logical volumes, host physical disks, or network based storage.
  Read only media may also be accessed via URL style protocol specifiers.

  Note that as a general rule, as new command line options are added which serve
  to replace an older option or interface, you are strongly encouraged to adapt
  your usage to the new option. The new option is being introduced to provide
  better functionality and usability going forward. In some cases existing
  problems or even bugs in older interfaces cannot be fixed due to functional
  expectations, but are resolved in the newer interface or option.
  This advice includes moving to the most recent machine type (eg
  s390-ccw-virtio-2.9 instead of s390-ccw-virtio-2.6) if possible.

- The following command line options are supported:
  -add-fd ...
  -alt-grab
  -append ...
  -audio-help
  -balloon ...
  -bios ...
  -boot ...
  -cdrom ...
  -chardev ..
  -clock
  -cpu host
  -ctrl-grab
  -d ...
  -daemonize
  -debugcon ...
  -device [virtio-net-pci|virtio-blk-pci|virtio-balloon-pci|virtserialport|
          virtconsole|virtio-serial-pci|virtio-scsi-pci|scsi-cd|scsi-hd|
          scsi-generic|scsi-disk|scsi-block|virtio-rng-pci|pci-bridge|
          megasas-gen2|e1000e|e1000]
          (the following are aliases of these supported devices: virtio-blk|
          virtio-net|virtio-serial|virtio-balloon|virtio-scsi|virtio-rng)
  -dfilter range, ...
  -display ...
  -drive ... (if specified if=[virtio] and format=[qcow2|qed|raw] and
             snapshot=off only)
  -echr ...
  -enable-fips
  -enable-kvm
  -fsdev ...
  -full-screen
  -fw_cfg ...
  -gdb ...
  -global ...
  -h
  -help
  -incoming ...
  -initrd ...
  -iscsi ...
  -k ...
  -kernel ...
  -loadvm ...
  -m ...
  -M [help|?|none|s390-ccw-virtio|s390-ccw-virtio-2.6|s390-ccw-virtio-2.9]
  -machine [help|?|none|s390-ccw-virtio|s390-ccw-virtio-2.6|s390-ccw-virtio-2.9]
  -mem-path ...
  -mem-prealloc
  -mon ...
  -monitor ...
  -msg ...
  -name ...
  -net [bridge|l2tpv3|nic|none|tap|user] ... (for model= only e1000, rtl8139,
                                             and virtio are supported)
  -netdev [bridge|tap|user] ...
  -no-acpi
  -nodefaults
  -nodefconfig
  -no-frame
  -nographic
  -no-hpet
  -no-quit
  -no-reboot
  -no-shutdown
  -no-user-config
  -object ...
  -parallel ...
  -pidfile ...
  -qmp ...
  -qmp-pretty ...
  -readconfig ...
  -realtime ...
  -rtc ...
  -runas ...
  -s
  -S
  -sandbox ...
  -sdl
  -serial ...
  -show-cursor
  -smp ...
  -spice
  -trace ...
  -uuid ..
  -version
  -vga [none|qxl|std]
  -virtfs ...
  -virtioconsole ...
  -vnc ...
  -watchdog ...
  -watchdog-action ...
  -writeconfig ...

- The following monitor commands are supported:
  ?
  balloon target ...
  block_resize ...
  boot_set ...
  c
  change device ...
  chardev-add ...
  chardev-remove ...
  client_migrate_info ...
  closefd ...
  cont
  cpu ...
  cpu-add ...
  delvm ...
  device_add ...
  device_del ...
  drive_add ...
  drive_backup ...
  drive_del ...
  dump_guest_memory ...
  eject ...
  gdbserver ...
  help
  i ...
  info ...
  loadvm ...
  logfile ...
  logitem ...
  mce ...
  memsave ...
  migrate ...
  migrate_cancel
  migrate_incoming
  migrate_set_cache_size ...
  migrate_set_capability ...
  migrate_set_downtime ...
  migrate_set_parameter ...
  migrate_set_speed ...
  migrate_start_post_copy
  mouse_button ...
  mouse_move ...
  mouse_set ...
  nmi ...
  o ...
  object_add ...
  object_del ...
  p ...
  pci_add ...
  pci_del...
  pmemsave ...
  print ...
  q
  qemu-io ...
  qom-list
  qom-set
  ringbuf_read ...
  ringbuf_write ...
  savevm ...
  screendump ...
  sendkey ...
  snapshot_blkdev_internal ...
  snapshot_delete_blkdev_internal ...
  stop
  sum ...
  system_powerdown
  system_reset
  system_wakeup
  trace-event ...
  watchdog_action ...
  x ...
  xp ...

- The following QMP commands are supported:
  add_client
  add-fd
  balloon
  blockdev-add
  blockdev-backup
  blockdev-change-medium
  blockdev-close-tray
  blockdev-del
  blockdev-mirror
  blockdev-open-tray
  blockdev-snapshot
  blockdev-snapshot-delete-internal-sync
  blockdev-snapshot-internal-sync
  blockdev-snapshot-sync
  block-commit
  block-dirty-bitmap-add
  block-dirty-bitmap-clear
  block-dirty-bitmap-remove
  block_passwd
  block_resize
  block_set_io_throttle
  block-set-write-threshold
  block_stream
  change
  change-vnc-password
  chardev-add
  chardev-remove
  client_migrate_info
  closefd
  cont
  cpu
  cpu-add
  device_add
  device_del
  device-list-properties
  dump-guest-memory
  eject
  expire_password
  getfd
  human-monitor-command
  inject-nmi
  input-send-event
  memsave
  migrate
  migrate_cancel
  migrate-incoming
  migrate-set-cache-size
  migrate-set-capabilities
  migrate_set_downtime
  migrate_set_speed
  migrate-set-parameters
  migrate-start-postcopy
  object-add
  object-del
  pmemsave
  qmp_capabilities
  qom-get
  qom-list
  qom-list-types
  qom-set
  query-acpi-ospm-status
  query-balloon
  query-block
  query-block-jobs
  query-blockstats
  query-chardev
  query-chardev-backends
  query-command-line-options
  query-commands
  query-cpu-definitions
  query-cpu-model-baseline
  query-cpu-model-comparison
  query-cpu-model-expansion
  query-cpus
  query-dump
  query-dump-guest-memory-capability
  query-events
  query-fdsets
  query-gic-capabilities
  query-hotpluggable-cpus
  query-iothreads
  query-kvm
  query-machines
  query-memdev
  query-memory-devices
  query-mice
  query-migrate
  query-migrate-cache-size
  query-migrate-capabilities
  query-migrate-parameters
  query-name
  query-named-block-nodes
  query-pci
  query-qmp-schema
  query-rocker
  query-rocker-of-dpa-flows
  query-rocker-of-dpa-groups
  query-rocker-ports
  query-rx-filter
  query-spice
  query-status
  query-target
  query-uuid
  query-version
  query-vnc
  query-vnc-servers
  quit
  remove-fd
  ringbuf-read
  ringbuf-write
  rtc-reset-reinjection
  screendump
  send-key
  set_link
  set_password
  stop
  system_powerdown
  system_reset
  system_wakeup
  trace-event-get-state
  trace-event-set-state
  transaction

-  The following command line options are unsupported:
  -acpitable ...
  -bt ...
  -chroot ...
  -cpu ... (all except host)
  -curses
  -device [AC97|adlib|allwinner-ahci|am53c974|amd-iommu|AMDVI-PCI|
          ccid-card-emulated|ccid-card-passthrough|cirrus-vga|cs4231a|dc390|
          diag288|e1000-82544gc|e1000-82545em|edu|ES1370|floppy|generic-sdhci|
          generic-sdhci|gus|hda-duplex|hda-micro|hda-output|hyperv-testdev|
          *-i386-cpu|i8042|i82550|i82551|i82557a|i82557b|i82557c|i82558a|
          i82558b|i82559a|i82559b|i82559c|i82562|i82801|i82801b11-bridge|ib700|
          ich9-intel-hda|ich9-usb-ehci1|ich9-usb-ehci2|ich9-usb-uhci1|
          ich9-usb-uhci2|ich9-usb-uhci3|ich9-usb-uhci4|ich9-usb-uhci5|
          ich9-usb-uhci6|ide-cd|ide-drive|ide-hd|igd-passthrough-isa-bridge|
          intel-hda|intel_iommu|ioh3420|ipmi-bmc-extern|ipmi-bmc-sim|ipoctal232|
          isa-applesmc|isa-cirrus-vga|isa-debugcon|isa-debug-exit|isa-fdc|
          isa-ide|isa-ipmi-bt|isa-ipmi-kcs|isa-parallel|isa-serial|isa-vga|
          kvm-pci-assign|lsi53c810|lsi53c810a|megasas|mptsas1068|ne2k_isa|
          ne2k_pci|nec-usb-xhci|nvdimm|nvme|pc-dimm|pci-testdev|pcnet|
          pc-testdev|piix3-ide|piix3-ide|piix3-ide-xen|piix4-usb-uhci|pvscsi|
          pxb|pxb-host|pxb-pcie|qemu,register|qemuregister|qemu-s390-cpu|rocker|
          s390-flic|s390-flic-qemu|s390-ipl|s390-pcihost|
          s390-sclp-event-facility|s390-skeys-qemu|sb16|sclp|sclpconsole|
          sclp-cpu-hotplug|sclplmconsole|sclp-memory-hotplug-dev|sclpquiesce|
          sd-card|sdhci-pci|sdhci-pci|secondary-vga|sga|smbus-eeprom|tpci200|
          unimplemented-device|usb-audio|usb-bot|usb-bt-dongle|usb-ccid|usb-mtp|
          usb-uas|vfio-amd-xgbe|vfio-calxeda-xgmac|vfio-pci|vhost-scsi-ccw|
          vhost-vsock-ccw|virtio-9p-device|virtio-balloon-ccw|virtio-blk-ccw|
          virtio-crypto-ccw|virtio-mmio|virtio-net-ccw|virtio-rng-ccw|
          virtio-scsi-ccw|virtio-serial-ccw|virtio-vga|virtual-css-bridge|
          vmware-svga|vmxnet3|vt82c686b-usb-uhci|x3130-upstream|*-x86_64-cpu|
          xen-backend|xen-pci-passthrough|xen-platform|xen-pvdevice|xen-sysdev|
          xio3130-downstream|z10BC.2-base-s390-cpu|z10BC.2-s390-cpu|
          z10BC-base-s390-cpu|z10BC-s390-cpu|z10EC.2-base-s390-cpu|
          z10EC.2-s390-cpu|z10EC.3-base-s390-cpu|z10EC.3-s390-cpu|
          z10EC-base-s390-cpu|z10EC-s390-cpu|z114-base-s390-cpu|z114-s390-cpu|
          z13.2-base-s390-cpu|z13.2-s390-cpu|z13-base-s390-cpu|z13-s390-cpu|
          z13s-base-s390-cpu|z13s-s390-cpu|z196.2-base-s390-cpu|z196.2-s390-cpu|
          z196-base-s390-cpu|z196-s390-cpu|z800-base-s390-cpu|z800-s390-cpu|
          z890.2-base-s390-cpu|z890.2-s390-cpu|z890.3-base-s390-cpu|
          z890.3-s390-cpu|z890-base-s390-cpu|z890-s390-cpu|z900.2-base-s390-cpu|
          z900.2-s390-cpu|z900.3-base-s390-cpu|z900.3-s390-cpu|
          z900-base-s390-cpu|z900-s390-cpu|z990.2-base-s390-cpu|z990.2-s390-cpu|
          z990.3-base-s390-cpu|z990.3-s390-cpu|z990.4-base-s390-cpu|
          z990.4-s390-cpu|z990.5-base-s390-cpu|z990.5-s390-cpu|
          z990-base-s390-cpu|z990-s390-cpu|z9BC.2-base-s390-cpu|z9BC.2-s390-cpu|
          z9BC-base-s390-cpu|z9BC-s390-cpu|z9EC.2-base-s390-cpu|z9EC.2-s390-cpu|
          z9EC.3-base-s390-cpu|z9EC.3-s390-cpu|z9EC-base-s390-cpu|z9EC-s390-cpu|
          zBC12-base-s390-cpu|zBC12-s390-cpu|zEC12.2-base-s390-cpu|
          zEC12.2-s390-cpu|zEC12-base-s390-cpu|zEC12-s390-cpu|zpci]
          (the following are aliases of these unsupported devices: lsi|
          virtio-input-host|virtio-keyboard|virtio-mouse|virtio-tablet|
          virtio-gpu|virtio-9p|pci-assign|ahci|e1000-82540em)
          (note that some of these device names represent supported devices and
          are used internally, but are not specifyable via -device even though
          they appear in the list of devices)
  -drive ,if=[scsi|mtd|pflash], snapshot=on, format=[anything besides qcow2, qed
         or raw]
  -dtb file
  -fda/-fdb ...
  -g ...
  -hda/-hdb/-hdc/-hdd ...
  -icount ...
  -L ...
  -M [s390-ccw-virtio-2.4|s390-ccw-virtio-2.5|s390-ccw-virtio-2.7|
     s390-ccw-virtio-2.8]
  -machine [s390-ccw-virtio-2.4|s390-ccw-virtio-2.5|s390-ccw-virtio-2.7|
           s390-ccw-virtio-2.8]
  -mtdblock file
  -net [dump|socket|vde] ...
  -netdev [dump|hubport|l2tpv3|socket|vde] ...
  -no-fd-bootchk
  -no-kvm
  -no-kvm-irqchip
  -no-kvm-pit
  -no-kvm-pit-reinjection
  -numa ...
  -option-rom ...
  -pflash file
  -portrait
  -prom-env ...
  -qtest ...
  -qtest-log ...
  -rotate
  -sd file
  -set ...
  -singlestep
  -smbios ...
  -snapshot
  -soundhw ...
  -tb-size ...
  -tdf
  -tpmdev passthrough ...
  -usb
  -usbdevice [braile|disk|host|mouse|net|serial|tablet]
  -vga [cg3|tcx|virtio|cirrus|xenfb|vmware]
  -win2k-hack
  -xen-attach ...
  -xen-create
  -xen-domid ...

- The following monitor commands are unsupported:
  acl_add ...
  acl_policy ...
  acl_remove ...
  acl_reset ...
  acl_show ...
  block_job_cancel ...
  block_job_complete ...
  block_job_pause ...
  block_job_resume ...
  block_job_set_speed ...
  block_passwd ...
  commit ...
  drive_mirror ...
  expire_password ...
  hostfwd_add ...
  hostfwd_remove ...
  host_net_add ...
  host_net_remove ...
  nbd_server_add ...
  nbd server_start ...
  nbd_server_stop ...
  netdev_add
  netdev_del ...
  pcie_aer_inject_error ...
  set_link ...
  set_password ...
  singlestep ...
  snapshot_blkdev ...
  stopcapture ...
  usb_add ...
  usb_del ...
  wavcapture ...
  x_colo_lost_heartbeat

- The following QMP commands are unsupported:
  block-job-cancel
  block-job-complete
  block-job-pause
  block-job-resume
  block-job-set-speed
  change-backing-file
  drive-backup
  drive-mirror
  dump-skeys
  netdev_add
  netdev_del
  nbd-server-add
  nbd-server-start
  nbd-server-stop
  query-tpm
  query-tpm-models
  query-tpm-types
  x-blockdev-change
  x-blockdev-insert-medium
  x-blockdev-remove-medium
  x-colo-lost-heartbeat
  xen-load-devices-state
  xen-save-devices-state
  xen-set-global-dirty-log