CML2 - Some Thoughts and Comparison to EVE-ng

Since I'm back in study mode I thought I'd get a hold of a Cisco Moddeling Labs (CML)² licence so that I can try and gain some efficiency and therefore more focus in LABs at hand rather than troubleshooting and working around various kinks and nuances of the LAB environment, which I found I was doing a lot of in EVE-ng (prior to 6.0.x).

Installation

Once the purchase over at lerningnetworkstore was complete, Installation was quite straight forward except I had an extra step, which I will explain later.

Here's the high-level steps I took to acomplish the task;

1. Downloaded the OVA and the refplat bundle from the Cisco software center
2. Copied the OVA to the hypervisor
3. Converted the vmdk from the OVA to qcow2
4. Imported the qcow2 image into the hypervisor

Step 3 was only required for me since I'm using QEMU/KVM+libvrt as my hypervisor, but a quick search online guided me to the solution which allowed me to almost seamlessly import it.

I imported the qcow2 image and started the VM, but it would not boot properly, but it seems as though UEFI is required. Easy fix.

Initial Setup

Initial setup required access to the VM console, but it was very straight forward, it offered to use DHCP, expand the disk to its biggest possible extent and set the passwords. All quite lacklustre, painless and somewhat anticlimactic.

Once booted the console informs you that you can log into the CML application and also the Cockpit web interface for sysadmin tasks (queue the sysadmin credentials).

Licence and Registration

At first login to CML the application setup continues a bit more, you then can register the instance by inputting the licence key (providing you're not distracted by the setup option to navigate away from the wizard).

LAB Time

Once CML is configured I seem to remember it immediately creating a new lab and leaving me in the driving seat at that point. I found it trivial to navigate, add nodes access console, create links etc.

So far so good. Less than a couple of hours to set up as opposed to EVE-ng, which took me over a few hours to install from scratch (ISO) in a VM and that's not including the time it took to copy and set up each image (convert the few Cisco qcow2 images that were actually raw), test and then start labbing and figure out all the strange and weird behaviours like interface state configuration not being saved in exported configs etc.

I have no idea yet how to add custom (non-Cisco) nodes to CML. I know that qcow2 images (only) can be added, but they demand a lot of hypervisor options for node definitions, which I also don't want to have to worry myself over. And then theres the quirk of how node default node definitions are read-only. I want to edit them but CML scares you into not doing so warning you that it could break LABs and there doesn't seem to be an option to revert them back to default.

Another quirk I found is that CML (out of the box) doesn't give you the same sort of naming construct as EVE-ng does with bulk node numbering. While you can prefix nodes, it puts a '-' and then a number which is starts from 0, not 1. So I end up with R-0, R-1, R-2 and so on. Not a big deal as renaming is fairly straight-forward, but renaming 5-10 devices isn't something I want to have to spend time doing.

The last thing I'd like to mention is that that I'm noticing a lot of EVE-ng like similarities with regards to LAB IDs, exported configs (or as CML puts it Fetch config, which I discovered needs to be done individually on each node in order to include the config in the exported LAB YAML).

Final Opinion

CML is very polished. It's a breath of fresh air for Cisco-centric stuff out-of-the-box. Where it is lacking though is the system performance in the bottom bar is quite distracting. When starting a lab and while its settling or whenever a router reloads or just decides it needs more CPU, you the user see it. This is distraction and if there isn't an option to toggle it off or hide it completely, there should be. I want to focus on the LAB at hand not sysadmin tasks.

I've never been a fan of the EVE-ng UNL file format for UNet Labs, but the ability to easily export and import labs in a standard file format (YAML) is fantastic.

Licensing is something that I don't like. While CML does come with an eval licence, you still have to purchase it to get access to download it. CML Personal could still be free/accessible for personal/eval use and could include a perpetual licence or just require it to be registered to get a free licence would be better. Cisco are definitely bringing in a revenue stream across the entire CML product-line, but that goes without saying that they probably pumped a lot of resources into developing the KVM+Cockpit-based hypervisor and WebGUI into quite a polished product which also has a rich API for automation which can be leveraged for things like CI/CD.

The disk capacity of the OVA seems rather small, so I'm going to consider using libvirts guestfs tools to expand the qcow2 image and then figure out how to expand the PV/LV within the OS/Cockpit.

CML is now my go-to Network Modelling LAB tool for my next CCNP ENARSI exam since it offers less quirks and more polish to allow me to more easily create, manage and operate my labs and focus on what matters. Learning.

HomeLab Mk.3 - Project Closeout

From a project methodology-standpoint, I'm missing some udates since the last post, but this is because I had since entered a redundancy, had immediate funding as a result, not to mention, limitted time to kick-off, execute and deploy before securing new employment.

The whole project is now complete with a 4RU AMD Ryzen-based custom-built server runnig Debian GNU/Linux.

Some of the improvemnts that have been made so far are as follows (in no particular order);

1. Employed cryptsetup on top of software RAID
2. Purchased and installed the 4RU system into an 18RU rack
3. Installed Cockpit for easier host/hypervisor management
4. Migrated the VMs from the previous HomeLab hypervisor to the new one
5. Built a functioning eve-ng instance as a VM using nested virtualisation for network moddeling

One key compromise, was that I decided to reduce costs with memory so the hypervisor host is outfited with 64Gb instead of the maximum 192Gb of RAM. This was due to the higher than expected motherboard cost not to mention my requirements are farily low at this stage so the cost of that sort of outlay isn't justified.

In addition to the above, I've also embarked on a more secured and virtualised infrastructure by using OPNSense for PROD, DEV, (NET)LAB and DMZ networks which pretty much just stiches together and firewalls multiple isolated virtual networks inside of libvirt and peers with the multi-layer switch over a P2P L3 interface via a dot1q trunk while also advertising a summary route and accepts a default route only from upstream.

I think its a failry elegant design given my constraints and requirements but more importantly, it is a much more manageble setup now which reduces some technical debt for me. Now theres very few improvements to make even in the next iteration of the HomeLab in future, which will mostly be a hardware refresh - That and re-racking everything since the racks mounting rails needs adjusting to accomidate the 4RU server depth which was unfortunately not able to be done in time.

While I would love to share the overall design itself, it unfortunately has far too much information that is now considered somewhat confidential, but those who I trust and those who know me are always welcome to take a read (preferably onscreen) as I'm not in a position to re-write it for public consumption.

Debugging Cisco Access Lists

I want to share something specific I learned, that seems to be outside the official CCNP curriculum.

Despite the fact that I've done some (L2) traffic seperation for untrusted devices, there's still, unfortunately some that need to be on my internal L3 network for now (Google-based devices like a Google TV-based TV and an old Google Home - Nest products don't interest me) so I decided to do something about this to restrict vertical traffic and potential attacks from old, unsupported or not-so-trusted hosts.

While I could seperate the traffic at L2 and forward it to a virtual firewall or my FGT internet firewall appliance, that in my opinion, causes sub-optimal traffic flows due to network limitations/design since the budget won't allow better gear for my needs (like VXLAN/VPNV4 overlay with route leaking etc).

So, all I have to work with is an old unsupported Cisco IOS v15 (Classic) Multilayer central switch in my home network.

I thought, this would be pretty easy. Just allow host services like DHCP/netboot, intra-VLAN traffic etc., block RFC1918 and allow everything else. Ez Pz. Except netboot to my netboot.xyz server didn't work initially and I couldn't easilly figure out why.

ip access-list extended RESTRICTED_ACCESS
 remark NETWORK_SERVICES
 permit udp any eq bootpc any eq bootps
 permit udp any any eq domain
 remark ALLOW_PING
 permit icmp any any echo
 permit icmp any any echo-reply
 remark ALLOW_PXE_SERVER
 permit udp any host 192.168.56.3 eq tftp
 permit tcp any host 192.168.56.3 eq www

 remark PERMIT_INTRA-VLAN
 permit ip 192.168.0.0 0.0.0.255 192.168.0.0 0.0.0.255 log
 remark DENY_RFC1918
 deny   ip any 10.0.0.0 0.255.255.255
 deny   ip any 172.16.0.0 0.15.255.255
 deny   ip any 192.168.0.0 0.0.255.255

 remark ALLOW_EVERYTHING_ELSE
 permit ip any any log

I needed some visibility on the ports and protocols like a firewall log... Cisco conditional debugging to the rescue!

The specific Cisco debug I used was `debug ip packet detail`

Unfortunately, the detail was overwhelming and showed far too much information for any human to interpret and nearly brought down the switch, so I had to contrain or filter the output with a debug condition similar to the following:

`debug condition ip 192.168.0.4`

This produced the information I required and allowed me to pinpoint the missing port and protocol required!

21w3d: IP: s=192.168.0.1 (local), d=192.168.0.4 (Vlan666), len 56, sending

21w3d:     ICMP type=3, code=13
21w3d: IP: s=192.168.0.1 (local), d=192.168.0.4 (Vlan666), len 56, output feature
21w3d:     ICMP type=3, code=13, Check hwidb(88), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
21w3d: IP: s=192.168.0.1 (local), d=192.168.0.4 (Vlan666), len 56, sending full packet
21w3d:     ICMP type=3, code=13pak 599DB6C consumed in input feature , packet consumed, Access List(31), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
21w3d: IP: s=192.168.0.4 (Vlan666), d=192.168.56.3, len 32, access denied
21w3d:     UDP src=62557, dst=30002
21w3d: FIBipv4-packet-proc: route packet from Vlan666 src 192.168.0.4 dst 192.168.56.3
21w3d: FIBfwd-proc: packet routed by adj to Vlan56 192.168.56.3
21w3d: FIBipv4-packet-proc: packet routing succeeded
21w3d: IP: s=192.168.0.1 (local), d=192.168.0.4, len 56, local feature
21w3d:     ICMP type=3, code=13, CASA(4), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
21w3d: IP: s=192.168.0.1 (local), d=192.168.0.4, len 56, local feature

As you can see in the above output, UDP port 3002 was blocked (due to the implicit deny any rule), so adding that in before the deny RFC1918 entry resolved this for me. Happy days.

So here's the final ACL that worked a treat.

ip access-list extended RESTRICTED_ACCESS
 remark NETWORK_SERVICES
 permit udp any eq bootpc any eq bootps
 permit udp any any eq domain
 remark ALLOW_PING
 permit icmp any any echo
 permit icmp any any echo-reply
 remark ALLOW_PXE_SERVER
 permit udp any host 192.168.56.3 eq tftp
 permit udp any host 192.168.56.3 eq 30002
 permit tcp any host 192.168.56.3 eq www
 remark PERMIT_INTRA-VLAN
 permit ip 192.168.0.0 0.0.0.255 192.168.0.0 0.0.0.255 log
 remark DENY_RFC1918
 deny   ip any 10.0.0.0 0.255.255.255
 deny   ip any 172.16.0.0 0.15.255.255
 deny   ip any 192.168.0.0 0.0.255.255
 remark ALLOW_EVERYTHING_ELSE

 permit ip any any log

Yes, I know I can (and probably will) tighten it some more and make DNS more specific (or remove it entirely to enforce quad9 DNS and prevent poisoning), but I wanted an ACL that is as simple as possible so I can easlily model and apply to other interfaces and SVI's which I might add is being done and so far it is working well.

Git

I'm not sure why git is called 'the stupid content tracker' (according to the man page that is), but I've discovered that - despite many tutorials overcomplicating the setup by adding the creation of a git user account and SSH key-based authentication - it is stupidly trivial to set up a remote repository.

By stupid I mean that git does not reference any of the object files in a way that you would expect or as you are used to working with them in your locally checked-out repository or IDE.

This method of file storage threw me off and caught me off guard but I eventually managed to get the initial comit added to the remote.

I also learned that git appears to work locally, meaning you can clone on the same system that's hosting the repository using directory paths without a transport protocol!

I'm now armed with information on how private git repo hosting works, which is especially useful for interim SCM or when private hosting is required for whatever reason.

Libvirt virtio Networking

Devling deeper into Libvirt, has my trying to find ways to improve the previous build through lab testing.

The latest testing is virtio networking with an isolated network in order to mitigate libvirt not being able to snapshot guests unless the volumes they use are all qcow2.

With this limitation in mind, I employed NFS to a common datastore for guests that require access to the datastore, however the path taken in the current configuration is suboptimal and takes the path of the hosts management interface.

The virtio model provides much better throughput while at the same time allowing guests to communicate with the host, but not outside the host.

In my testing with a virtio model I was able to achieve over 10Gbps with no tuning whatsoever as follows;

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  16.3 GBytes  14.0 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  16.3 GBytes  14.0 Gbits/sec                  receiver

The current path which uses the suboptimal path is not only limited to the hardware NIC/switch, but we can also observe quite a lot of retries indicating TCP retransmits are likely also occuring which would be introducing latency with NFS.

[ ID] Interval           Transfer     Bitrate         Retr

[  5]   0.00-10.00  sec  1.10 GBytes   942 Mbits/sec  315             sender

[  5]   0.00-10.00  sec  1.09 GBytes   939 Mbits/sec                  receiver

I now have yet amother defined improvement concept ready for implementation on the new server build.

Libvirt pool storage management

I was really looking forward to improving on my previous homelab by building a new server, defining succinct and well thought out pools that leverages and manages LVM, mounts etc in order to abstract away some of the sysadmin tasks.

In my limit testing, I've found that libvirt storage management is flexible yet limited insofar as the fact that I could have potentially done away with the complexities of mdadm, manual definition of a PV and/or VG and LVs, formatting, creating mountpoints and then adding the mounted filesystem(s) to libvirt or let libvirt mount it for me, but since I'm using crypto in order to mitigate potential data breaches during hard drive disposal, it means that I can't leverage RAID functionality within LVM itself as I require a simplified encryption with a single key on a single volume or in my case, an md array.

If I didn't require crypto, I may have been able to skip the manual mdadm RAID configuration and carved out nicer storage management, however this is unfortunately not the case.

It seems as though you can't easily carve up an LV as if it where a PV from libvirt's perspective when defining a pool (that is without the headaches that comes with partitioning LVs or overcomplicating the solution with pools defined from libvirt volumes). Libvirt pools also seem flat in nature and I can't figure out how to define a volume under a directory without defining seperate volumes (such as dir-based) to overcome this.

So for now my solution is to handle most of the storage manually with one single mount point based on a single md and crypto device along with a single LVM PV, VG and LV with dir-based pools defined to manage volumes.

It doesn't seem ideal nor efficient, but right now I need a solution to move the project forward to completion.

I will further test and refine (and possibly even automate) the solution on the new hypervisor host at some point. Who knows, there may be better tools or newly discovered ways of doing this in the future.

The next step in the overall solution is to test a virtiofs shared for and/or virtio high-speed (10Gbps) isolated SAN solution.

Regular Expressions - Examples and Use Cases

Background

This post should serve as a repository of selected use-case reqular expressions, sorted by utility/name. It is predominantly centered around Linux and user-space utilies (with a certain amount of Cisco IOS-based examples as well in its heading and subheadings). It will hopefully be continually updated as I intent to keep adding to it as I see buld more regular expression use cases.

MDADM

The following was useful to gather mdadm information when I had an issue with a missing block device in a RAID array (which turned out to be SATA cables that where accidently swapped when performing maintenance/cleaning causing device unexpected device renaming which ultimately bumped a device off the array - sdb in my case). The examples here uses simple patterns to show the linux block devices in an array and looking for log entries

user@host:~$ sudo mdadm --detail /dev/md0 | egrep '\/dev\/sd?'
       3       8       64        0      active sync   /dev/sde
       1       8       32        1      active sync   /dev/sdc
       4       8       48        2      active sync   /dev/sdd
user@host:~$ cat /etc/mdadm/mdadm.conf | egrep '\/dev\/sd?'
DEVICE /dev/sdb /dev/sdc /dev/sdd /dev/sde
user@host:~$
user@host:~$ sudo dmesg | grep md0
[    2.701684] md/raid:md0: device sdc operational as raid disk 1
[    2.701686] md/raid:md0: device sdd operational as raid disk 2
[    2.701687] md/raid:md0: device sde operational as raid disk 0
[    2.702549] md/raid:md0: raid level 5 active with 3 out of 3 devices, algorithm 2
[    2.702574] md0: detected capacity change from 0 to 8001304920064
user@host:~$ 

HDPARM

For similar reasons to the MDADM, I initially suspected that a disk was faulty and wanted to extract the serial numbers of each for warranty lookup. This is how I acheived that outcome (sans actual serial numbers).

user@host:~$ sudo hdparm -I /dev/sd? | egrep '(\/dev\/sd?|Serial\ Number)'
/dev/sda:
        Serial Number:      *** REDACTED ***
/dev/sdb:
        Serial Number:      *** REDACTED ***
/dev/sdc:
        Serial Number:      *** REDACTED ***
/dev/sdd:
        Serial Number:      *** REDACTED ***
/dev/sde:
        Serial Number:      *** REDACTED ***
user@host:~$

SCREEN

So, sometimes a screen is killed or exited (often accidently) and rather than opening up the local user screenrc file, looking for the screen/entry/command and then executing the screen command manually to restore it, with the help of grep, I simply execute it dirrectly with bash substitution. Here are a couple of examples:

$(grep virsh ~/.screenrc)
$(grep /var/log/messages ~/.screenrc)
$(grep virt_snapshot ~/.screenrc)

LVM

At some point, we might need to review LVM volumes to see where we can scale and resize etc. The following allowed me to quickly see everything at a glance in order to formulate a plan for resizing.

user@host:~$ sudo lvdisplay | egrep "LV (Name|Size)"

[sudo] password for user:
  LV Name                video
  LV Size                <4.02 TiB
  LV Name                audio
  LV Size                750.00 GiB
  LV Name                hdimg
  LV Size                <2.51 TiB
  LV Name                swap
  LV Size                16.00 GiB
  LV Name                var-tmp
  LV Size                8.00 GiB
user@host:~$

Cisco IOS

A collection of various Cisco IOS commands and the very limited IOS regular expression engine on an IOS device (or IOS-XE's IOSD).

show version

Show a consolidated view of uptime, firmware and software version & reason for reload (minus all the Cisco copyright and releng information):

SWITCH#show ver | incl Cisco IOS Software|(ROM|BOOTLDR)|uptime|System (returned|restarted|image)

Cisco IOS Software, C3750 Software (C3750-IPSERVICESK9-M), Version 15.0(2)SE11, RELEASE SOFTWARE (fc3)

ROM: Bootstrap program is C3750 boot loader

BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.2(44)SE5, RELEASE SOFTWARE (fc1)

SWITCH uptime is 1 week, 3 days, 22 hours, 29 minutes

System returned to ROM by power-on

System restarted at 12:28:16 WST Sun Sep 17 2023

System image file is "flash:/c3750-ipservicesk9-mz.150-2.SE11.bin"

SWITCH#

show etherchannel

Show portchannel member state times - This is particularly useful in correlating events for possible cause without having to rely on syslog:

SWITCH#show etherchannel 1 detail | incl ^(Port: |Age of the port)

Port: Gi1/0/15

Age of the port in the current state: 10d:22h:41m:32s

Port: Gi1/0/16

Age of the port in the current state: 10d:22h:41m:31s

Port: Gi1/0/17

Age of the port in the current state: 10d:22h:41m:30s

Port: Gi1/0/18

Age of the port in the current state: 10d:22h:41m:30s

SWITCH#

Cisco IOS IPv6 observations

I wanted to delve deeper into some of the intricacies of IPv6, specifically Neighbour discovery and Directly attached Static routes as well as OSPFv3 using the legacy configuration. I recently discovered two odd Cisco behaviours with these following topics, possibly related to virtual lab devices, so not tested on real equipment.

1. IPv6 Directly Attached Static Routes
2. OSPF IPv6

IPv6 Directly Attached Static Route

This doesn't seem to work as described (at least not in a lab). Only fully-specified or a net-hop static route works. This could be due to either;

• No MAC address to IPv6 neighbor binding - since IPv6 doesn't use the concept of ARP like IPv4 does, it instead relies on Neighbor discovery, which doesn't seem to work - more testing/research is required.
• Limitation with the way Layer 2 forwarding is handled in an Emulated/Virtual environment.

OSPF IPv6

The protocol, according to some old Cisco Press article I dug up[1]. It appears to "leverage" the OSPFv3 engine, however it can be configured/started using the legacy IPv6 OPSF configuration similar to IPv4 as per the following:

ipv6 ospf process-id

Now, if there's an existing, legacy OSPF IPv4 configuration using the same process-id, it appears to silently fail when entering the configuration (except perhaps if you enable debugging). No neighbours will establish at all, despite documetation claiming that it migrates it to the OSPFv3 configuration (it most likely does this internally though as I observed that the configuration stays pretty much as you entered it in both running and start-up configuration).

The lesson I learned here, is to identify if multiple OSPF address-families share the same process in legacy configuration mode and either;

1. Update all your configuration so that one of the "confliting" addresss families is unique or
2. You migrate the conflicting processes/address families to the new OSPFv3 configuration as a consolidation of address families under the one process.

Further to the above, when removing an OSPF process with no ipv6 ospf process-id, any interface-specific IPv6 process/area configuration is also removewd without warning.

[1] https://www.ciscopress.com/articles/article.asp?p=2294214&seqNum=4

HomeLab Mk.3 - Planning Phase

Background

I kicked off my homelab refresh project not long ago, dubbed "HomeLab mk.3" as its the third iteration since circa 2010. I'm now well into the planning phase but I've found that I'm also overlapping into the procurement phase (as described herein).

So far, I've decided to replace my pre-Ryzen AMD-based full-tower hyperconverged system with another hyperconverged system, but this time it will be housed in an 18RU rack for providing a small amount of noise management, but also neaten up the office a little, which will have the added benefit of assisting in home improvement (flooring) later.

Key requirements;

1. Costs must be kept as low as possible
2. Software RAID (due to #1)
3. Hyperconverged system (due to item #1 and power constraints)
4. Nested virtulisation for EVE-NG/network modelling

Therefore based on requirements, the system (excluding the rack) will comprise of the following;

• One SSD for the hypervisor stack/host OS
• Up to six (6) 8Tb CMR disks for the storage of guests etc.
• 4RU ATX rackmount case (including rails of course) ✅
• As much memory as the mainboard allows which relates to key requirement #4

Challenges

The current challenges surrounding the build are;

1. Choice of Hypervisor (oVirt, libvirt, OpenStack, EVE-NG)
2. Choice of CPU architecture (due to key requirement #4 and likely challenge #1)
3. Possible Network re-architecture required to support the system including possible infrastructure Re-IP addressing.

Choice of Hypervisor

For item #1 the choices don't look that great, and I will probably stick with libvirt and the various virt toolsets, only because;

• oVirt appears to no longer be supported downstream by RedHat which means contributions to the upstream project (oVirt) will likely and eventually kill the project
• OpenStack is a pain to set up, even the all-in-one "packstack" which also means that could impact scalability in future if required
• EVE-NG appears to be an inappropriate choice. While it supports KVM/QEMU/qcow2 images, I'm not sure I want this as the underlying HomeLab hypervisor (unless certain constraints can be overcome - these are considered not in scope of this post).

Choice of CPU architecture

For item #2 the CPU architecture is important only because network vendor (QEMU/qcow2) images highlight strict CPU architecture requirements as being Intel-based, and AFAIK nested virtulisation requires that the guest architecture matches that of the host.

Possible Network re-architecture

Item #3 is not insurmountable, but it is still a challenge nonetheless as I'm not sure about whether I will change the Hypervisor guest networks (dev, prod, lab etc) to connect back upstream at L2 or L3.

Procurement

As I mentioned already, the project planning phase is somewhat overlapping with the procurement phase, the reason for this is so that I can not only procure certain less tech-depreciating items over time to allow project budget flexibility, but also allow a certain level of reduced risk in operation of the system:

Case in point: HDD's - I never risk buying them from the same batch in case of multiple catastrophic failures.

I've already purchased 3 HDD's, the 4RU rackmount case and rails and an 18RU rack to house the new gear along with the existing kit (switch, router and UPS).

I'll continue to procure the HDD's until I have enough to build the system then all that left is to purchase the key parts for the rack mount case/system (CPU, mainboard, memory & PSU) once CPU architecture/hypervisor testing (see Hardware selection below) and the design is complete.

Hardware selection (CPU architecture)

In order to determine whether the new system will be Intel or AMD will depend on the testing performed on my AMD Ryzen-based desktop. If EVE-NG and the required images work in nested virtualisation (and/or bare-metal) with said CPU architecture, then I will be in a good position to stick with AMD for this iteration (and likely future iterations) of the HomeLab. After all, AMD-based systems appear to have a good pricepoint which relates back to key requirement #1

EVE-NG and IOL copy run unix:

Lately, I've found myslelf working more on EVE-NG than the Cisco Learning Labs (CLL) which has allowed me to go beyond the constraints of the traditional learnings and key topics and allows me to tinker more than I probably should.

A long time ago I thought that EVE (possibly pre-NG) allowed the user to litterally download the text file of the running config to file instead of having to rely on term len 0, show run and screen-scraping the contents and then offloading the resulting clipboard to a file and saving it *yawn*

Today I discovered that you can save a config straight to a file in EVE-NG on the linux filesystem (at least you can with IOL).

The way to do this is simply use the copy command with unix:file as the destination, replacing file with the name of the file;

R1#copy start unix:r1.txt 

Destination filename [r1.txt]? 

1683 bytes copied in 0.011 secs (153000 bytes/sec)

R1#

It is litterally that simple.

You can then find the file under the EVE-NG staging area, which you can then work on as a plain-text file;

root@eve-ng:~# ls -alh /opt/unetlab/tmp/1/e6eadfea-e000-41d7-abe9-98f8004bb23f/1 | egrep "r.\.txt$"

-rw-rw-r-- 1 unl1 unl 1.7K Sep 19 16:40 r1.txt

root@eve-ng:~#

I can only imagine how useful this could be in reverse by merging config snips straight from the emulated nodes off the host filesystem and perhaps even generating templates for labs etc.

More testing is required.

X11 Forwarding woes

Here is a very quick post. It's for a very common/annoying Linux X11/Forwarding quirk which I previously forgot what the solution was and decided that I would add it into my blog as a reference

Background:

When working on my Development laptop remotely with a LiveUSB (for either ripping audio or video content or using it as a dev environmenmt for my recent study), I found that having to setup/install packages locally with tge laptop keyboard/trackpad was anoying when having to constantly switch (physically) from main desktop to the laptop. Thanks to SSH most setup and interactions with the machine can be done from another machine.

The Problem:

Having SSH access is nice, but having access the locally installed browser and other apps displayed locally (via X11Forwarding) is even better (purely so I can avoid having to transfer downloaded content).

Some applications wil just work without changing anything, others (read: Firefox) require a little more persuaion to work.

This is kind of what it looks like out of the box on the host (with X11 Forwarding enabled in sshd and configured on the client SSH application):

kubuntu@kubuntu:~$ firefox &

[2] 17790 

kubuntu@kubuntu:~$ PuTTY X11 proxy: Unsupported authorisation protocol

Unable to init server: Broadway display type not supported: localhost:10.0

Error: cannot open display: localhost:10.0

[2]+  Exit 1                  firefox

kubuntu@kubuntu:~$

The solution:

For me it was quite simple (not sure if this will work for others though so YMMV).

Update the $DISPLAY environment variable:

export DISPLAY={{ x11_server }}:0.0

Where {{ x11_server }} is the hostname or IP address of the X11 Server (XLauch or whatever).

There, now I shouldn't forget this ever again :-D

Additional information:

1. This solution also seems to work inside individual screen sessions.
2. Source of the solutiuon (#2 in the list) was rediscovered at https://stackoverflow.com/questions/61221498/x11-forwarding-works-on-ubuntu-using-windows-10-cmd-line-ssh-only-after-first-us

Adventures in Open Source (Part 3)

The (somewhat) popular Adventures in Open Source series is back and even better than before

SYSLINUX
Along time ago and, I was able to force a Toshiba Satellite A10 to boot a Ghost Boot Wizard created disc (ISO) thanks to syslinux.
This seemed kind of tricky to begin with but looking back on it, it is pretty trivial.

First of all I mounted the ghost iso file as loopback and copied the file contents to a temporary directory ($pathspec can be any empty temporary directory such as /tmp/ghost).

sudo mount -o loop /usr/temp/ghost.iso /mnt/loop0
cp -R /mnt/loop0/* $pathspec

Next up I copied the necessary syslinux files to the temporary directory

cp /usr/lib/syslinux/isolinux.bin $pathspec
cp /usr/lib/syslinux/memdisk /$pathspec

I then had to create an isolinux configuration file called isolinux.cfg in the temporary ($pathspec) directory with the following contents.

cat > $pathspec/isolinux.cfg
default ghost
timeout 150
prompt 1
label ghost
  kernel memdisk
  append initrd=osboot.img
^Z

Lastly, I moved up one directory and created the iso with mkisofs/genisoimage

cd ..
mkisofs -v -J -V $volid -N -A '' -sysid '' -o $filename -b isolinux.bin -c boot.cat \
-no-emul-boot --boot-load-size 4 -boot-info-table $pathspec

That's all!

NOTE: Due to the varying nature of Linux distributions, I have purposefully used variables (named in accordance with mkisofs/genisoimage documentation) so as to aid in making this procedure as dynamic as possible.


ntfsclone(8)
Since I still help people with Windows (only close friends and relatives now), and I decided to give this tool another try (last time I used it, I used the "special" image format, which cannot be loopback mounted).

Bellow is the output (proof) of a successful ntfsclone (and ntfs-3g loopback mount).

localhost ~ # ntfsclone -o /u1/S3A1378D001-ntfsclone.img /dev/sde1ntfsclone v2.0.0 (libntfs 10:0:0)NTFS volume version: 3.1Cluster size       : 4096 bytesCurrent volume size: 39999500288 bytes (40000 MB)Current device size: 39999504384 bytes (40000 MB)Scanning volume ...100.00 percent completedAccounting clusters ...Space in use       : 13676 MB (34.2%)Cloning NTFS ...100.00 percent completedSyncing ...

This is me loopback mounting a standard ntfsclone (not special image format) image:

ntfs-3g -o loop /u1/S3A1378D001a-ntfsclone2.img /mnt/loop0mount | grep fuse/u1/S3A1378D001a-ntfsclone2.img on /mnt/loop0 type fuse (rw,noatime,allow_other)

[Cisco] Static Route Leaking

So, I had specific use case for allowing access to loopbacks on the same VRF across a point to point link without the need for a routing protocol.

None of the resources I found online gave a specific example of how to do this, however after some trial and error and piecing together various parts of specific solutions for use cases with much more elaborate designs, I came up with the following solution.

R2#show run int lo 1

Building configuration...

Current configuration : 156 bytes

!

interface Loopback1

description Management

vrf forwarding MGMT

ip address 192.168.10.2 255.255.255.255

end

R2#show run | incl ip route

ip route 192.168.10.2 255.255.255.255 Loopback1 name R2_LO1

ip route vrf MGMT 192.168.10.1 255.255.255.255 192.168.1.1 global name R1_LO1

R2#ping vrf MGMT 192.168.10.1

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 192.168.10.1, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

R2#ping vrf MGMT 192.168.10.1

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 192.168.10.1, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

R2#ssh -vrf MGMT -l admin 192.168.10.1

Password:

R1#show run int lo 1

Building configuration...

Current configuration : 144 bytes

!

interface Loopback1

description Management

vrf forwarding MGMT

ip address 192.168.10.1 255.255.255.255

end

R1#show run | incl ip route

ip route 192.168.10.1 255.255.255.255 Loopback1 name R1_LO1

ip route vrf MGMT 192.168.10.2 255.255.255.255 192.168.1.2 global name R2_LO1

R1#ping vrf MGMT 192.168.10.2

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 192.168.10.2, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

R1#

So based on the above, you not only have to inform the global routing table (underlay) how to get to it's own loopback, but you also have to inform the VRF to use the GRT to transport traffic for the destination IP, in this case the host route.

Adventures in docker and portainer

Around 2007 I was gifted some old hardware which entailed an ASUS motherboard, 8Gb or RAM and an AMD CPU.

It wasn't until a few years later that I decided to build it into a home server.

There was no hardware virtualisation and either I didn't know how to do or didn't want to do software virtualisation and software RAID, instead all my services, DHCP, DNS, SAMBA, FTP etc. and I think even Plex as well (or maybe that came later) was running co-resident on a bare-metal JBOD server.

Since it was simple design, it was relatively simple to operate and maintain. I even managed to successfully P2V the server when I did a hardware refresh and it continued operating for the most part.

One day I upgraded the system and a Python-based application which catastrophically broke and I since abandoned picking it back up until recently because I discovered Docker containers.

Fast-forward to today, now I have a big proponent of my services and apps hosted in a dedicated docker-engine VM and maintenance has never been easier.

I've even learned how to share the same network namespace as other containers such as stacking a container with a VPN container and all it took was using the following in the compose file under the containers service definition:

network_mode: "container:<container>"

Which, I leaned and adapted the above from the following YouTube video:

How to route any docker container through a VPN container

Further to this my Docker engine VM is exclusively managed now using portainer.io where I can easily create and delete (and in the process upgrade) containers with ease, which means everything stays fresh and all I have to be concerned about is backing up the persistent storage!

Armed with knowledge of how docker works, I've written up slide deck on docker to help demystify docker containers and hopefully improve overall understanding for the potentially emerging DevOps capability.

Adventures in Automation: Part 1

So ever since I heard about automation (and then orchestration) I have finally taken some measures to not only learn but implement some of my own.

Since I realised that I have a reasonable amount of unwanted technical debt building up while having to maintain my 'homelab' in its current state, some the daily hastles can be automated away.

Two such tasks I have just undertaken and completed are"

1. Ansible automation to update, upgrade and clean orphaned/unused packages
2. Script to cleanup snapshots taken prior to ansible playbooks being run

I have a very long way to go before I can get other things automated, but its a start.

I'm also in the process if defining a docker environment where most of my services will operate from within one or more virtual machines.

I will also have to look into github repository of my code and a complete cloud backup solution with onsite encryption (encryped using my own privately owned and stored keys).

Free Range Routing

Since I discovered Docker, I have been busy designing my homelab to be as Cloud Native as possible, but in doing so, I realised that the default docker network (aka bridge) and the other networks defined by other containers from docker-compose bridge type networks isn't known by the upstream network collapsed core access layer network.

In the past I have been adding static routes upstream and a default route on the docker host, but this was not ideal (read scalable) given the dynamic nature of docker networks created with docker-compose.

I quickly realised that since I've developed significant experience in BGP (in service provider environments), I planned to just peer the docker host to the upstream access layer, but until now didn't know how to do this with Linux.

I have always known about Quagga, but been a bit concerned about the learning curve required to get it working, but remembered about its fork called Free Range Routing. So I decided to make the a leap of faith and I have no regrets whatsoever.

I was supprised at how easy it was to install and configure and to get it working which comprised of the following;

On the Docker host;

1. Setup the REPO
2. Installed FRR and configred services for vtysh as per the official documentation
3. Connected to the vtysh interface and
4. Configured BGP and redistributed only conneted routes using route-map/prefix-list

On the upstream switch/access layer;

1. Configured peering to FRR and squelched all but a summary route prefix-list/route-map.

Not only did I define the policies perfectly (woot!), but the neighbor came up without any issues at all!!!

My switch can now route traffic to any docker/docker-compose containers/services that are created with far less effort required to define routes to the containers as well as deleting them if no longer in use.

I am very supprised to learn that not only is it extremely easy to get FRR up and running but it is very easy to configure routing daemons/config for them especially if you have a grounding in Cisco as its vtysh is very closely matched with that of Cisco CLI - there are some slight and obvious differences to Cisco's not to mention the fact that you need to connect to the vtysh interface kind of like the root user does on a Juniper platform (cli).

Next up I need to figure out how to leverage Linux VRF namespaces using FRR vtysh, then I can migrate my infrastructure to MPLS!

juniper-nuances-part1

Once day I was relaiming an interface on my newly aquired Juniper SRX300 (yes, I managed to get one of these nice little units!) and when trying to commit, I came accross the below error, which wile it seems a little misleading, it actually gives a very good clue at the issue (which may seem cryptic to the inexperienced user).

root@srx300# set interfaces ge-0/0/1 unit 0 family ethernet-switching interface-mode trunk
[edit]
root@srx300# set interfaces ge-0/0/1 unit 0 family ethernet-switching vlan members all
[edit]
root@srx300# commit check
[edit routing-instances emetric interface]
  'ge-0/0/1.0'
    Interface with 'interface-mode' is allowed only in a virtual-switch
error: configuration check-out failed: (statements constraint check failed)

In the above example, the system claims that 'the interface-mode is allowed only in a virtual-switch' however upon removing the associated interface from the routing-instance (in this case I just deleted the routing instance) and making sure that the following was in place, then commit was successful and no further errors.

[edit]
root@srx300# delete routing-instances emetric
[edit]
root@srx300# commit check
configuration check succeeds
[edit]
root@srx300# show interfaces ge-0/0/1 | display set
set interfaces ge-0/0/1 unit 0 family ethernet-switching interface-mode trunk
set interfaces ge-0/0/1 unit 0 family ethernet-switching vlan members all

So, you see the lesson here is READ the output carfeully for the clue as to why the commit is throwing an error. In my case the interface was assigned in a routing-instance and therefore mutually exclusive when the interface is a L2 trunk (highlighted text).

NBN: The Somewhat Expected Journey

In the beginning

About three and a half years ago I was planning on purchasing a house with my then-girlfriend and choose a location that was earmarked to get Fibre to the Premise (FttP) National Broadband Network (NBN)​ with a three year build date. Within a few months of settlement an election was called and my location was wiped off the NBN map completely.

Three years on and I receive an email from my ISP advising that HFC is going to be available. Then junkmail in the mailbox with offers from one random unknown and a couple of known much disliked Retail Service Providers (RSP) - in the NBN world, they are now referred to as Retail Service Providers (RSP) and formally known as Internet Service Provider (ISP). Yes there is a difference.

The only thing that I think our government may have managed to achieve with their three words slogan for NBN during their campaigning is simply "cheaper" They appear to failed on the "faster" and "sooner" parts.

Back on track with the story, I jumped at the RSP email and decided to 'pre-order' my NBN (which turned out to be a mistake) as I learned from an NBN representative that that a pre-order is actually a new order and would have created a whole new account not linked or related to my existing, long tenure one, which I want to keep at least until the day I decide to switch to a new RSP. But that's not all I learned, just not from my RSP.

Curse of the cabling

Not only did I not realise that there was an existing Telstra cable/Foxtel box out the front of the house from a previous owner/residents installation, but it was in an inconvenient location and I needed to remediate the coax cabling before the NBN appointment (to install the Network Termination Device (NTD) - otherwise known as an NBN "Connection Box".

So in preparation for the NBN appointment, I decide to cancel the NBN order (also under advise by the RSP), except I wasn't quick enough to cancel the router that was immediately sent out. No biggie. It only cost $10 to have it sent to me.

So that brings me to the next part. The cabling. I forgot about the existing Telstra cable box at the front of the house outside, but since there was also foxtel dish on the roof (I actually thought the foxtel dish was the source of the cabling point), so I decided to remove the foxtel dish, but not before a trip to Bunnings to get two blank wall plates so that I could make it appear as though nothing was cabled or points inside the house, whereby upon removing said dish, I discovered that the dishes cable didn't actually go to the wall point I expected, but I continued to remove it anyway, putting aside the dish, mounting bracket and cable once removed.

I then located the actual coax cable drop (after realising/remembering about the cable box out front. Derp) and proceeded to pull that across to the (new) location where I wanted it. At this point I realised I had to go to Bunnings again, this time for an electrical snake, electrical tape, 16mm masonry drill bit and a new coax f-type wall plate.

Thinking that I had everything I needed, I tidied up the wall plates from the previous coax cable points and started to painstaking pull the existing cable through the roof eves trying to keep minimal turns to maximise length only to find out that the cable still fell short of approximately three metres. Being the engineer that I am, I saved myself I some money by reusing the cable from the foxtel dish I removed earlier and joined the coax with a connector salvaged from a removed f-type wall plate. Brilliant!

So after wrapping the join with a generous amount of electrical tape, there was more than enough length for the cable to reach the new wall point, for which was not as straight forward as just drilling a forty five degree hole to the wall cavity as I managed to loose the coax connector after I managed to get the snake and cable wedged in the bend of the newly drilled hole thinking I could stupidly un-wedge it by pulling harder on the snake then on the actual cable (minus the connector) and then on the snake again - finally dislodging the snake (minus the connector - which now lives somewhere inside the cavity walls). All this only because I mistakenly taped the coax cable and it's connector to the top of the snake instead of the bottom to allow it to go around the the bend of the newly drilled hole for the wall plate. Lesson learned.

Once I had the coax pulled through, I went off to Jaycar this time to get the replacement F-type connector (including a spare in case I borked it), but quickly realised that they were for a coax cable with a smaller core, so after yet another trip to Bunnings for the correct F-type connector, I got the wall plates finished and titles put back on the roof. Lucky for me I didn't break any tiles during the walking about on the roof!

Order in the court!

After all the saga of cabling was completed, I contacted my RSP and requested a new NBN order, this time under my existing account. So far so good except I receive another email telling me a router is being delivered. Again.

The next day I contact my RSP via reply email for the hardware order asking to cancel it. No response. I call them the following day to cancel it. They tell me there is no evidence of any charge but by that time I already have an email telling me it's been dispatched and on its way (It arrived the next day shoved into the mailbox, whereas the previous one I had to sign for at the local post office).

At this point the NBN appointment to install the NTD had also been brought forward but in the meantime I decided to break open the HG659b since I had two of them, so one of them was doomed to become my sacrificial test unit (read on to find out how/why).

./hack

Hacking the sacrificial unit involved soldering header pins to the empty holes where a serial port has been identified thanks to an openwrt hardware wiki article on it. De-soldering the factory solder points to put the header pins took far longer than actually soldering on the header pins though.

Having access to the serial port it was easy to get into the Common Firmware Environment (CFE) by interrupting the boot sequence so that I could flash spark (NZ) firmware onto this unit as it was more likely to include the download configuration file exploit not available in the crippled TPG firmware.

After doing the configuration file download exploit on the sacrificial unit with alternative firmware, I decrypted the configuration file with the help of a whirlpool knowledge article, and then the root password from the modified unit, I was then able to successfully use those credentials to log into the unmodified unit with normal, unadulterated, uncripled access.

With noob access defeated, I quickly discovered that not only does the unit only allow static routes, on the WAN interfaces but it is very picky about its LAN management IP address and subnet mask. According to the admin UI, apparently 192.168.2.1/24 is invalid and the Command Line Interface (CLI) over telnet is cripled and you cannot get a shell whatsoever. By this stage you can probably imagine the eyetwitch starting.

By now, it was blindingly clear that the HG659b is complete rubbish (ESPECIALLY WITH TPGs CRIPLED FIRMWARE!) - even more so for a network engineer such as myself.

In between all this, the NBN contractors had attended the premises (at the very end of the appointment window no less) to install the NTD (yes, it took two of them) and found the existing Telstra cable to have no signal! Thinking that it was my fault, I simply explained that I "had the the wall point moved". They then opened up the telstra cable box out front and not only found two cables coming from it but one of them was not connected - which was obviously the one I had remediated. They switched the cabling and the line had signal and they where able to activate it. By now one of the NBN contractors - what looked like the junior of the two - had left. The NBN contractor then used the excuse that his phone battery was flat so that he didn't have to wait around for the half an hour for confirmation of the service being activated. I wonder where the other cable leads to then?

MOAR HARDWARE!

Knowing that the HG659 is completely useless to me for my requirements, I dug around for an OpenWRT compatible router (not an AP and ADSL and everything all-in-one-gateway-that-everyone-calls-a-damn-router!) and settled on a MikroTik RouterBoard RB750GL from WISP - these guys have a questionable website security wise (certificate is fine but pages may include a form(s) with a non-secure "action" attribute.) but they shipped this thing in record time!

With the new router in hand I took a quick look at the very ugly UI for RouterOS and promptly installed OpenWRT on it using a combination OpenWRT installation methods from both the device page and the general common procedures (the latter of which mentions the missing initrd).

Before I got it off the default IP address (and adding a static summary route for all my subnets), I realised was getting annoyed at this point having to deal with equipment which ships with IP addresses which conflict with my own network, so I decided to change my Local Area Network (LAN) subnet to something more sane, which took about three hours of solid uninterrupted work to migrate configuration to a previously unused Cisco 48 port PoE switch (some things are outstanding but I can defer those for another time).

Lastly, I added a new PPPoE connection to the the port already tagging on VLAN ID 2 and viola! I'm connected to the NBN with a device which I have much more trust and configuration options. Even the default firewall rules where reasonable.

Summary

I have learned a few things these last few weeks: how to save some money, how not chase a cable through a wall and not to necessarily jump onto something no matter how exiting it seems and could help with working from home and studying in the future but even more recently, how NBN HFC is delivered and more recently, how important security is with the NBN (more on this in a new post hopefully).

It has been an interesting journey and the thing I enjoyed the most about this was the hardware hacking (however futile it was) and the handyman type work (drilling holes chasing cables etc). What I least enjoyed was the level of service from my RSP and the fact that they managed to bork the username on the account, but all-in-all the actual throughput/bandwidth of the service seems reasonable for now and by the time you read this, I will probably have shut down my ADSL service.

The funniest thing about all this all is the fact that the wife hasn't noticed the improvement in speeds, since I told her (and she agreed) to the experiment of not telling her when it was actually done!

Oh and NBN sent a survey asking about how likely I was to recommend NBN, based on the "excellent" service I received with a scale of 1-10, only to tell me that I had to choose 4 or higher. Go figure.


If you have read this far, thanks for reading and feel free to share your experience with me by posting in the comments or via Google+ (link to post is best).