Install vm-bhyve:

pkg install vm-bhyve

Add to /etc/rc.conf:

vm_enable="YES"
vm_dir="zfs:zroot/vms"
vm_list=""
vm_delay="5"

Create /vms:

zfs create -o mountpoint=/vms zroot/vms

Change the ZFS recordsize property from its default value of 128K to 64K:

zfs set recordsize=64K zroot/vms

https://klarasystems.com/articles/from-0-to-bhyve-on-freebsd-13-1/

Initialize vm:

vm init

crete the switch:

vm switch create -i em0 -n 20 public

I am using interface em0 and vlan 20

Create ubuntu template: /vms/.templates/ubuntu.conf:

loader="uefi"
cpu=2
memory=2G
network0_type="virtio-net"
network0_switch="public"
disk0_name="disk0"
disk0_dev="sparse-zvol"
disk0_type="virtio-blk"
graphics="yes"

Download ubuntu:

vm iso https://releases.ubuntu.com/22.04.1/ubuntu-22.04.1-live-server-amd64.iso

Create VM using template ubuntu:

vm create -t ubuntu -s 40G m1

Install VM using the iso

vm install m1 ubuntu-22.04.1-live-server-amd64.iso

Use VNC client :5900

To stop/start:

vm stop m1
vm start m1

or using bhyvectl:

bhyvectl --destroy --vm=m1

To enable on boot the VM:

sysrc vm_list="m1"

If you have multiple VM's:

sysrc vm_list="m1 m2 m3"

To get a list of VM's:

vm list

To create more VM's using an existing one, create an image with:

vm image create -d debian <name of existing VM>

use -u for not compressing the image

Create the VM:

vm image provision ced71c04-a971-11ed-b840-649d99b18002 <name for new VM>

You can check the size of volume with:

zfs get volsize tank/vms/<name of vm>/disk0

And resize if required with:

zfs set volsize=100G tank/vms/<name of vm>/disk0

Configure values for new created VM:

vm configure <name of VM>

🔗Booting debian

If can't boot after the first reboot exit the SHELL:

  1. Enter exit get out of the UEFI Shell (you will come to BHYVE panel)
  2. Choose Boot Maintenance Manager
  3. (In the Boot Maintenance Manager ) Choose Boot From File
  4. (In File Explorer) NO VOLUME LABEL, [PciRoot(0x0)/ Pci()] press Enter
  5. (Still in the File Explorer) Choose
  6. (Still in the File Explorer) Choose debian
  7. (Still in the File Explorer) Choose grubx64.efi
  8. Then you will get into the system

Then:

# cd /boot/efi
# mkdir -p EFI/BOOT
# cp EFI/debian/grubx64.efi EFI/BOOT/bootx64.efi

https://record99.blogspot.com/2021/12/bdsdex-failed-to-load-boot0001-uefi-bhyve-sata-disk.html

🔗Add datastore

To add a new datastore, fore example a pool of SATA disks:

vm datastore add sata zfs:bhyve/vms

Then use it like this:

vm create -d sata my-new-vm

🔗console

To access the VM, if VNC is not available, you can either use cu or vm console for example:

cu -l /dev/nmdm-poudriere.1B

or

vm console poudriere

Type ~ + Ctrl-D to exit back to your host

To get more info about the console type do:

cat /vms/<name of vm>/console
com1=/dev/nmdm-<name of vm>.1B
vnc=0.0.0.0:5902

I am using /vms/ because of the `vm_dir="zfs:tank/vms"