Most of you have probably always faced the question of which solution to use for your virtualization needs in order to build a personal lab for playing around. The two most common ones seen so far would most probably end up being Cisco VIRL (cloud and offline/your server versions) and Unetlab/EVE-NG . There are other variants related more or less to Eve-NG that also seem to do partially the job ( GNS3 ) but we won’t be talking about them here. Purpose is to show how to modify the templates of Unetlab in order to add the Juniper VMX phase 2, to show how one can debug Unetlab errors when trying to start your virtual devices (in order to figure out what the right parameters to put in the template are) and also add new options in the HTML interface (VMX with dual RE for example).

Getting VMX Phase2 into Unetlab/Eve-NG

Unpacking the archive

tar -xzvf vmx-bundle-17.3R1.10.tgz

From the resulting files what interests us is:

vmx/images/junos-vmx-x86-64-17.3R1.10.qcow2 vmx/images/metadata-usb-re.img vmx/images/vmxhdd.img vmx/images/vFPC-20170810.img

Check all files we are interested in and their formats

qemu-img info junos-vmx-x86-64-17.3R1.10.qcow2

image: junos-vmx-x86-64-17.3R1.10.qcow2
file format: qcow2
virtual size: 25G (26844594176 bytes)
disk size: 1.1G
cluster_size: 65536
Format specific information:
    compat: 0.10
    refcount bits: 16

qemu-img info vmxhdd.img

image: vmxhdd.img
file format: qcow2
virtual size: 6.0G (6440426496 bytes)
disk size: 104M
cluster_size: 65536
Format specific information:
    compat: 0.10
    refcount bits: 16

qemu-img info metadata-usb-re.img

image: metadata-usb-re.img
file format: raw
virtual size: 16M (16777216 bytes)
disk size: 16M

qemu-img info metadata-usb-fpc0.img

image: metadata-usb-fpc0.img
file format: raw
virtual size: 16M (16777216 bytes)
disk size: 16M

Convert all of them to qcow2 (those that are raw)

Command (sample) is:

qemu-img convert -p -f raw vFPC-20170810.img -O qcow2 hda.qcow2

Copy them to the respective Unetlab directory

vmxvfp-17-3-R1 & vmxvcp-17-3-R1

Rename them

junos-vmx-x86-64-17.3R1.10.qcow2 -> hda.qcow2 metadata-usb-re.qcow2 -> metadata.qcow2 (I assume you converted it from raw & img extension into qcow2 as said before) vmxhdd.qcow2 (converted from img to qcow2)

Modify the templates

/opt/unetlab/html/templates/vmxvcp.php

$p[‘qemu_nic’] = ‘virtio-net-pci’; $p[‘qemu_options’] = ‘-machine type=pc-0.13,accel=kvm -serial mon:stdio -nographic -cpu IvyBridge,+hypervisor,+osxsave,+pcid,+vmx,+ss,+ds -smbios type=0,vendor=Juniper -smbios type=1,manufacturer=VMX,product=VM-vcp_vmx1-161-re-0,version=0.1.0 -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -hdb vmxhdd.qcow2 -no-user-config -nodefaults -drive file=metadata.qcow2,format=qcow2,if=none,id=drive-usb-disk0,cache=directsync -device usb-storage,drive=drive-usb-disk0,id=usb-disk0,removable=off’;

/opt/unetlab/html/templates/vmxvfp.php

$p[‘qemu_nic’] = ‘virtio-net-pci’; $p[‘qemu_options’] = ‘-machine type=pc-i440fx-trusty,accel=kvm -serial mon:stdio -nographic -nodefaults -cpu IvyBridge,+hypervisor,+osxsave,+pcid,+vmx,+ss,+ds -no-user-config -nodefaults’;

There is a chance (can be big:) ) that the VMs -> RE & vFPC do not start, in which case one has to debug this step by step by going into the directory with the .qcow images for each and trying to start them manually with the qemu parameters (this will be explained later in the Debug section of the page). Normally, from what I’ve seen it is the machine type=XX that changes and has to be a adjusted a bit.

How to debug this

Dual Routing Engine?

Juniper allows the VMX to also have a dual routing engine setup but in this case you will need some more modifications:

  • Add a new element in the web php of Eve-NG/Unetlab The old / existing VMX VCP can’t be reused as we need a way to differentiate which VMX is RE0, which RE1, etc. The vPFC/PFE remains the same. /opt/unetlab/html/includes/__node.php

    case ‘vmxvcp-dual-re0’: for ($i = 0; $i < $this -> ethernet; $i++) { if (isset($old_ethernets[$i])) { // Previous interface found, copy from old one $this -> ethernets[$i] = $old_ethernets[$i]; } else { if ($i == 0) { $n = ’em0 / fxp0’; // Interface name } else if ($i == 1) { $n = ’em1 / int’; // Interface name } else { $n = ’em’.$i.’ / ge-0/0/’.($i - 2); } try { $this -> ethernets[$i] = new Interfc(Array(’name’ => $n, ’type’ => ’ethernet’), $i); } catch (Exception $e) { error_log(date(‘M d H:i:s ‘).‘ERROR: ‘.$GLOBALS[‘messages’][40020]); error_log(date(‘M d H:i:s ‘).(string) $e); return 40020; } } // Setting CMD flags (virtual device and map to TAP device) if ($i == 0) { $this -> flags_eth .= ’ -device e1000,netdev=net’.$i.’,mac=50:’.sprintf(’%02x’, $this -> tenant).’:’.sprintf(’%02x’, $this -> id / 512).’:’.sprintf(’%02x’, $this -> id % 512).’:00:’.sprintf(’%02x’, $i); } else { $this -> flags_eth .= ’ -device %NICDRIVER%,netdev=net’.$i.’,mac=50:’.sprintf(’%02x’, $this -> tenant).’:’.sprintf(’%02x’, $this -> id / 512).’:’.sprintf(’%02x’, $this -> id % 512).’:00:’.sprintf(’%02x’, $i); } $this -> flags_eth .= ’ -netdev tap,id=net’.$i.’,ifname=vunl’.$this -> tenant.’’.$this -> id.’’.$i.’,script=no’; }

    case ‘vmxvcp-dual-re1’: for ($i = 0; $i < $this -> ethernet; $i++) { if (isset($old_ethernets[$i])) { // Previous interface found, copy from old one $this -> ethernets[$i] = $old_ethernets[$i]; } else { if ($i == 0) { $n = ’em0 / fxp0’; // Interface name } else if ($i == 1) { $n = ’em1 / int’; // Interface name } else { $n = ’em’.$i.’ / ge-0/0/’.($i - 2); } try { $this -> ethernets[$i] = new Interfc(Array(’name’ => $n, ’type’ => ’ethernet’), $i); } catch (Exception $e) { error_log(date(‘M d H:i:s ‘).‘ERROR: ‘.$GLOBALS[‘messages’][40020]); error_log(date(‘M d H:i:s ‘).(string) $e); return 40020; } } // Setting CMD flags (virtual device and map to TAP device) if ($i == 0) { $this -> flags_eth .= ’ -device e1000,netdev=net’.$i.’,mac=50:’.sprintf(’%02x’, $this -> tenant).’:’.sprintf(’%02x’, $this -> id / 512).’:’.sprintf(’%02x’, $this -> id % 512).’:00:’.sprintf(’%02x’, $i); } else { $this -> flags_eth .= ’ -device %NICDRIVER%,netdev=net’.$i.’,mac=50:’.sprintf(’%02x’, $this -> tenant).’:’.sprintf(’%02x’, $this -> id / 512).’:’.sprintf(’%02x’, $this -> id % 512).’:00:’.sprintf(’%02x’, $i); } $this -> flags_eth .= ’ -netdev tap,id=net’.$i.’,ifname=vunl’.$this -> tenant.’’.$this -> id.’’.$i.’,script=no’; } break;

  • Make new templates for this case

    • /opt/unetlab/html/templates/vmxvcp-dual-re0.php

    $p[‘qemu_options’] = ‘-machine type=pc-0.13,accel=kvm -serial mon:stdio -nographic -cpu host -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -smbios type=0,vendor=Juniper -smbios type=1,manufacturer=VMX,product=VM-vcp_vmx1-161-re-0,version=0.1.0 -hdb vmxhdd.qcow2 -no-user-config -nodefaults -drive file=metadata.qcow2,format=qcow2,if=none,id=drive-usb-disk0,cache=directsync -device usb-storage,drive=drive-usb-disk0,id=usb-disk0,removable=off’;

    • /opt/unetlab/html/templates/vmxvcp-dual-re1.php

    $p[‘qemu_options’] = ‘-machine type=pc-0.13,accel=kvm -serial mon:stdio -nographic -cpu host -smbios type=0,vendor=Juniper -smbios type=1,manufacturer=VMX,product=VM-vcp_vmx1-161-re-1,version=0.1.0 -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -hdb vmxhdd.qcow2 -no-user-config -nodefaults -drive file=metadata.qcow2,format=qcow2,if=none,id=drive-usb-disk0,cache=directsync -device usb-storage,drive=drive-usb-disk0,id=usb-disk0,removable=off’;

  • Obey the new directory format

    root@eve-ng:/opt/unetlab/addons/qemu# ls -la | grep dual drwxr-xr-x 2 root root 4096 Apr 18 2017 vmxvcp-dual-re0-16-2 drwxr-xr-x 2 root root 4096 May 10 2017 vmxvcp-dual-re0-17-1 drwxr-xr-x 2 root root 4096 Apr 18 2017 vmxvcp-dual-re1-16-2 drwxr-xr-x 2 root root 4096 May 10 2017 vmxvcp-dual-re1-17-1

  • Be on the safe side - Fix permissions

    /opt/unetlab/wrappers/unl_wrapper -a fixpermissions

Debug

Logs

/opt/unetlab/data/Logs From here you can look into: /opt/unetlab/data/Logs/unl_wrapper.txt

Example:

starting /opt/unetlab/wrappers/qemu_wrapper -T 0 -D 39 -t “vMX-VFP” -F /opt/qemu/bin/qemu-system-x86_64 -d 0 – -nographic -device virtio-net-pci,netdev=net0,mac=50:00:00:27:00:00,bus=pci.0,addr=0x3 -netdev tap,id=net0,ifname=vunl0_39_0,script=no -device virtio-net-pci,netdev=net1,mac=50:00:00:27:00:01,bus=pci.0,addr=0x4 -netdev tap,id=net1,ifname=vunl0_39_1,script=no -device virtio-net-pci,netdev=net2,mac=50:00:00:27:00:02,bus=pci.0,addr=0x5 -netdev tap,id=net2,ifname=vunl0_39_2,script=no -device virtio-net-pci,netdev=net3,mac=50:00:00:27:00:03,bus=pci.0,addr=0x6 -netdev tap,id=net3,ifname=vunl0_39_3,script=no -device virtio-net-pci,netdev=net4,mac=50:00:00:27:00:04,bus=pci.0,addr=0x7 -netdev tap,id=net4,ifname=vunl0_39_4,script=no -device virtio-net-pci,netdev=net5,mac=50:00:00:27:00:05,bus=pci.0,addr=0x8 -netdev tap,id=net5,ifname=vunl0_39_5,script=no -device virtio-net-pci,netdev=net6,mac=50:00:00:27:00:06,bus=pci.0,addr=0x9 -netdev tap,id=net6,ifname=vunl0_39_6,script=no -device virtio-net-pci,netdev=net7,mac=50:00:00:27:00:07,bus=pci.0,addr=0xa -netdev tap,id=net7,ifname=vunl0_39_7,script=no -device virtio-net-pci,netdev=net8,mac=50:00:00:27:00:08,bus=pci.0,addr=0xb -netdev tap,id=net8,ifname=vunl0_39_8,script=no -device virtio-net-pci,netdev=net9,mac=50:00:00:27:00:09,bus=pci.0,addr=0xc -netdev tap,id=net9,ifname=vunl0_39_9,script=no -device virtio-net-pci,netdev=net10,mac=50:00:00:27:00:0a,bus=pci.0,addr=0xd -netdev tap,id=net10,ifname=vunl0_39_10,script=no -device virtio-net-pci,netdev=net11,mac=50:00:00:27:00:0b,bus=pci.0,addr=0xe -netdev tap,id=net11,ifname=vunl0_39_11,script=no -smp 4 -m 6144 -name vMX-VFP -uuid 28bc47d5-19d7-4bda-bbb4-1a75fc047d4d -hda hda.qcow2 -machine type=pc-i440fx-trusty,accel=kvm -serial mon:stdio -nographic -nodefaults -cpu IvyBridge,+hypervisor,+osxsave,+pcid,+vmx,+ss,+ds -no-user-config -nodefaults > /opt/unetlab/tmp/0/09f95132-1711-44ca-9e48-e0e844a840d9/39/wrapper.txt 2>&1 &

Take the string after “-F” and then go to the /opt/unetlab/addons/qemu/vmxvfp-your-version directory for example and try running that command by hand. This way you can see what errors are being reported or why qemu does not start the appliance. Most usual is that some parameter used in the template changed available values (in which case it is easy to change and adjust to something compatible).

One can also look directly in the startup log of the appliance which is at the end of the log output from below: /opt/unetlab/tmp/0/09f95132-1711-44ca-9e48-e0e844a840d9/39/wrapper.txt

Extras

Make changes to a device, then convert its image to be the new default/template one

Let’s say you configured your Router in some way (maybe the root password, maybe something else) and you need to always have that configuration used whenever a new element gets deployed. First, as before, look in unl_wrapper.txt. Take from the output of the qemu_wrapper command the last part "/opt/unetlab/tmp/0/09f95132-1711-44ca-9e48-e0e844a840d9/39/". Go into that directory. From here on, you can check to which backing store (full image) this delta of your particular VFP image is connected:

qemu-img info –backing-chain hda.qcow2 image: hda.qcow2 file format: qcow2 virtual size: 2.2G (2313158656 bytes) disk size: 48M cluster_size: 65536 backing file: /opt/unetlab/addons/qemu/vmxvfp-16-2/hda.qcow2 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false image: /opt/unetlab/addons/qemu/vmxvfp-16-2/hda.qcow2 file format: raw virtual size: 2.2G (2313158656 bytes) disk size: 2.2G

You want now that your delta becomes the new backing store / full image to be used in the future for each new VFP instance to be spawned.

qemu-img convert -p -f qcow2 hda.qcow2 -O qcow2 test.qcow2

Copy / Move the test.qcow2 in the place where the original VFP image was:

mv test.qcow2 /opt/unetlab/addons/qemu/vmxvfp-16-2/hda.qcow2 /opt/unetlab/wrappers/unl_wrapper -a fixpermissions