History

Louie KWAN a680e323b8 Introspective Instance Monitoring through QEMU Guest Agent https://blueprints.launchpad.net/masakari/+spec/introspective-instance-monitoring Currently, Masakari instance monitoring is strictly black-box type monitoring through qemu and libvirt. There are however a number of internal instance/VM faults, that if monitored and detected by Masakari, could be recovered by existing Masakari auto-recovery mechanisms; increasing the overall availability of the instance/VM. This blueprint introduces the capability of performing introspective instance monitoring of VMs, in order to detect, report and optionally recover VMs from internal VM faults. Specifically, VM Heartbeating Monitoring via the QEMU Guest Agent. Change-Id: I9efc6afc8d476003d3aa7fee8c31bcaa65438674 Implements: blueprint introspective-instance-monitoring		2018-07-12 14:02:34 +00:00
..
README.rst	Introspective Instance Monitoring through QEMU Guest Agent	2018-07-12 14:02:34 +00:00
__init__.py	Introspective Instance Monitoring through QEMU Guest Agent	2018-07-12 14:02:34 +00:00
cache.py	Introspective Instance Monitoring through QEMU Guest Agent	2018-07-12 14:02:34 +00:00
instance.py	Introspective Instance Monitoring through QEMU Guest Agent	2018-07-12 14:02:34 +00:00
qemu_utils.py	Introspective Instance Monitoring through QEMU Guest Agent	2018-07-12 14:02:34 +00:00
scheduler.py	Introspective Instance Monitoring through QEMU Guest Agent	2018-07-12 14:02:34 +00:00

README.rst

masakarimonitors-introspectiveinstancemonitor

Introspective instance monitor for Masakari ----------------------------------------- masakarimonitors-introspectiveinstancemonitor, provides Virtual Machine High Availability (VMHA) service for OpenStack clouds by automatically detecting the system-level failure events via QEMU Guest Agent. If it detects VM heartbeat failure events, it sends notifications to the masakari-api.

Based on the QEMU Guest Agent, masakarimonitors-introspectiveinstancemonitor aims to provide access to a system-level agent via standard qemu-ga protocol

How does it work?

libvirt and QEMU Guest Agent are used as the underlying protocol for messaging to and from VM.
- The host-side qemu-agent sockets are used to detemine whether VMs are configured with QEMU Guest Agent.
- qemu-guest-ping is used as the monitoring heartbeat.
For the future release, we can pass through arbitrary guest agent commands to check the health of the applications inside a VM.

QEMU Guest Agent Installation notes

Set image property: hw_qemu_guest_agent=yes.
- This tells NOVA to setup the virtual serial interface thru QEMU to VM
- e.g.
  
  $ openstack image create --public --disk-format qcow2 --container-format bare --file ~ubuntu/xenial-server-cloudimg-amd64-disk1.img --public --property hw_qemu_guest_agent=yes xenial-server-cloudimg

Inside VM:

$ sudo apt-get install qemu-guest-agent
$ sudo systemctl  start qemu-guest-agent
$ ubuntu@test:~$ ps -ef | fgrep qemu
$ ...  /usr/sbin/qemu-ga --daemonize -m virtio-serial -p /dev/virtio-ports/org.qemu.guest_agent.0
$ ubuntu@test:~$ ls /dev/virtio-ports/
$ org.qemu.guest_agent.0

Configure masakarimonitors-introspectiveinstancemonitor ----------------------------------------------#. Clone masakari using:

$ git clone https://github.com/openstack/masakari-monitors.git

Create masakarimonitors directory in /etc/.
Run setup.py from masakari-monitors:
```
$ sudo python setup.py install
```
Copy masakarimonitors.conf and process_list.yaml files from masakari-monitors/etc/ to /etc/masakarimonitors folder and make necessary changes to the masakarimonitors.conf and process_list.yaml files. To generate the sample masakarimonitors.conf file, run the following command from the top level of the masakari-monitors directory:
```
$ tox -egenconfig
```
To run masakari-introspectiveinstancemonitor simply use following binary:
```
$ masakari-introspectiveinstancemonitor
```