Libvirt virtio Networking

Devling deeper into Libvirt, has my trying to find ways to improve the previous build through lab testing.

The latest testing is virtio networking with an isolated network in order to mitigate libvirt not being able to snapshot guests unless the volumes they use are all qcow2.

With this limitation in mind, I employed NFS to a common datastore for guests that require access to the datastore, however the path taken in the current configuration is suboptimal and takes the path of the hosts management interface.

The virtio model provides much better throughput while at the same time allowing guests to communicate with the host, but not outside the host.

In my testing with a virtio model I was able to achieve over 10Gbps with no tuning whatsoever as follows;

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  16.3 GBytes  14.0 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  16.3 GBytes  14.0 Gbits/sec                  receiver

The current path which uses the suboptimal path is not only limited to the hardware NIC/switch, but we can also observe quite a lot of retries indicating TCP retransmits are likely also occuring which would be introducing latency with NFS.

[ ID] Interval           Transfer     Bitrate         Retr

[  5]   0.00-10.00  sec  1.10 GBytes   942 Mbits/sec  315             sender

[  5]   0.00-10.00  sec  1.09 GBytes   939 Mbits/sec                  receiver

I now have yet amother defined improvement concept ready for implementation on the new server build.

Libvirt pool storage management

I was really looking forward to improving on my previous homelab by building a new server, defining succinct and well thought out pools that leverages and manages LVM, mounts etc in order to abstract away some of the sysadmin tasks.

In my limit testing, I've found that libvirt storage management is flexible yet limited insofar as the fact that I could have potentially done away with the complexities of mdadm, manual definition of a PV and/or VG and LVs, formatting, creating mountpoints and then adding the mounted filesystem(s) to libvirt or let libvirt mount it for me, but since I'm using crypto in order to mitigate potential data breaches during hard drive disposal, it means that I can't leverage RAID functionality within LVM itself as I require a simplified encryption with a single key on a single volume or in my case, an md array.

If I didn't require crypto, I may have been able to skip the manual mdadm RAID configuration and carved out nicer storage management, however this is unfortunately not the case.

It seems as though you can't easily carve up an LV as if it where a PV from libvirt's perspective when defining a pool (that is without the headaches that comes with partitioning LVs or overcomplicating the solution with pools defined from libvirt volumes). Libvirt pools also seem flat in nature and I can't figure out how to define a volume under a directory without defining seperate volumes (such as dir-based) to overcome this.

So for now my solution is to handle most of the storage manually with one single mount point based on a single md and crypto device along with a single LVM PV, VG and LV with dir-based pools defined to manage volumes.

It doesn't seem ideal nor efficient, but right now I need a solution to move the project forward to completion.

I will further test and refine (and possibly even automate) the solution on the new hypervisor host at some point. Who knows, there may be better tools or newly discovered ways of doing this in the future.

The next step in the overall solution is to test a virtiofs shared for and/or virtio high-speed (10Gbps) isolated SAN solution.