Emulated/Virtual Test Network

Today I finally managed to get the foundations for my test network working at L2 within a virtual environment.

The purpose of what I'm trying to achieve allows me to simulate various aspects of my home network and hyper-converged homelab within the homelab itself!

Over on LinkedIn, I posted that I got L2 port-channelling/bonding working, but I as you can see in the snip below, Po2 doesn't show LACP as it's protocol. This is because I cheated with the config and used 'channel-group 2 mode on` instead of `channel-group 2 mode active` which brought the Port-channel interface up on the switch, but the bond on the Debian GNU/Linux host would still not form.

This post serves as a correction to that article/post.

The cause for the behaviours I was experiencing was because the libvirt VirtIO-based network adapters don't seem to report the speed to the guest however, I believe they operate at 10Gbps by default, which would make the bond interfaces incompatible with the IOS-based peer's port-channel interfaces, which are limited to 1Gbps (and LACP in general). 

Changing the speed and duplex with nmcli solved this for me ^[1].

for i in 3 4 5 6; do sudo nmcli conn mod ens$i 802-3-ethernet.speed 1000 802-3-ethernet.duplex full; done

As soon as the speed and duplex was applied, the port-channel came up straight away. Marvellous.

Switch#show etherchan 2 summ | beg Port-

Group  Port-channel  Protocol    Ports

------+-------------+-----------+-----------------------------------------------

2      Po2(SU)         LACP      Gi1/0(P)    Gi1/1(P)    Gi1/2(P)

                                 Gi1/3(P)

Switch#

Now, I can proceed to further network-related components similar to my 'production' network.

^[1]https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html/configuring_and_managing_networking/configuring-802-3-link-settings_configuring-and-managing-networking#proc_configuring-802-3-link-settings-using-the-nmcli-utility_configuring-802-3-link-settings

 

Migrating away from BGP default-information originate

Background

I recently had yet another nbn unplanned outage. Now I have a GL-MT300N-V2 which I have basic config and a floating, static route on my central/downstream multilayer switch as a backup route with a worse metric than BGP, so that I can share my mobile phone's Mobile Broadband in the event that my Fortigate (FGT) can't forward default route traffic, but for some reason it was not working as expected/intended.

Problem #1 - IPTABLES default reject on FORWARD table

I did not capture the issue in detail, but it turned out that the GL-MT300N-V2 was blocking traffic in the forwarding table, changing this setting is what allowed forward traffic to pass to the MBB tether.

Problem #2 - default-information originate

The upstream BGP default route from my FGT persisted even in the event of an outage, when it should have disappeared so that the floating static route comes takes over internet forwarding (the Fortigate article linked herein explains this however this is normal BGP behaviour, but it was initially overlooked at the time of implementation. whoops!), but this was because I was using the Fortinet option 'set capability default-information-originate` in the BGP configuration, so I ended up tuning the BGP configuration and made the default route more dynamic as follows:

The solution

1. Created a DEFAULT route prefix list
2. Created a Route-map that uses the prefix list
3. Redistributed static routes into the BGP table using the route-map

It now looks something like this:

config router prefix-list

    edit "PL_DEFAULT"

        config rule

            edit 1

                set prefix 0.0.0.0 0.0.0.0

                unset ge

                unset le

            next

        end

    next

end

config router route-map

    edit "RM_DEFAULT"

        config rule

            edit 1

                set match-ip-address "PL_DEFAULT"

            next

        end

    next

end

config redistribute "static"

    set status enable

    set route-map "RM_DEFAULT"

end

I then disconnected the nbn and enabled `debug ip routing` on my switch to test the solution.

During testing and while the nbn was offline, the floating static was in place, exactly as expected:

SWITCH#show ip route | invl 0\.0\.0\.0\/0

S*    0.0.0.0/0 [254/0] via 192.168.81.1

SWITCH#

Once the nbn service was back and the upstream FGT inserted a static default, it wasn't long before I saw this the resulting debug message:

1w0d: RT: updating bgp 0.0.0.0/0 (0x0):

    via 10.8.18.1

1w0d: RT: closer admin distance for 0.0.0.0, flushing 1 routes

1w0d: RT: add 0.0.0.0/0 via 10.8.18.1, bgp metric [20/0]

1w0d: RT: default path is now 0.0.0.0 via 10.8.18.1

I followed this up with a check on the routing table, and here is the dynamic default route from an upstream ppp(oe) link in all its glory.

SWITCH#show ip route bgp | incl 0\.0\.0\.0\/0

B*    0.0.0.0/0 [20/0] via 10.8.18.1, 00:27:41

SWITCH#

Conclusion

This method provides a more elegant solution so that the backup internet solution can be leveraged with almost no touch.

In case your wondering why I use a floating static route, this is because the GL-MT300N-V2 is extremely limited in flash storage making it difficult to install and operate Quagga/FRR and I am tired of resetting the device as it has a tendency to fall over after a while which I suspect is due to lack of space.

The only possible improvement I could do right now is improving security through policy by putting the GL-MT300N-V2 behind the firewall itself, but that is a project for another day (not to mention it runs OpenWRT under the hood and has its own IPTABLES firewall anyway). I also plan to swap out the FGT for a dedicated, OPNSense appliance hosted on an SBC.

I hope this has been informative and I'd like to thank you for reading!

Stay tuned for more...

Reflections on Cisco ENARSI Study

One LAB to rule them all

While studying for my Cisco Certified Network Processional Enterprise Advance Routing (ENARSI) specialisation, I eventually figured out a strategy to help me focus on learning and less on constantly creating LABs.

I decided to build a reusable (flexible) lab by simply specifying the L2 VLAN at the router with a sub-interface so that I could potential attach any router to any other router in a P2MP broadcast setup.

It is impractical for a production network as each router shares the bandwidth of a single link for all VLANs on the trunk, but it does provide a very simple, elegant and flexible solution in its design allowing for focusing on less lab building and more hands on in a variety of scenarios.

Interestingly, both IOSvL2 switches refuse to provision certain VLAN ranges in the device config I built, but that's probably a bug with either CML or the image itself.

EXAM

So the exam to date is nothing short of frustrating. Just when I think I've nailed a lot of the concepts in EIGRP, OSPF, redistribution and many other L3 topics and begin feeling more confident, I find that the exams pool of questions are completely disjointed. For example, in my last exam - just before the time of writing this - I had less than 50% of actual advanced routing questions and more around services, device access and extremely low-level and corner-case things like really nuanced MPLS. It's not helping me to overcome imposter syndrome and to me makes it feel like I'm just part of Cisco's additional revenue stream and business strategy rather than learning and valuable certification process.

It also frustrates me that I seem to have to retain and recall an insane and almost inhuman amount of low-level information on EVERYTHING no mater how relevant or related to advanced routing it may seem.

I may just pivot across to other vendors and technology because it seems as though my brain is incompatible with rote learning.

Dealing with old Cisco gear and SSH

I've spent enough time dealing with old Cisco get to know that the old outdated Ciphers and Key exchanges can be tricky to deal with. Unfortunately, we can't just run the latest and greatest in a LAB and it generally considered isolated, so we have to live with this to a certain degree even if insecure.

I'm documenting the process of how to use an SSH client (Linux) to force it to use the right KEX and cypher etc. so people (including myself) don't have to piece the solution together from different sources every single time.

First off, the answers to the command-line parameters required lie in debugging in the client application itself.

ssh -vvv $host

This spits out a lot of information, which I could not seem to be able to filter through egrep, nonetheless, key items are listed here for reference

debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: KEX algorithms: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa

With that information gleaned, I was able to construct the parameters required to successfully connect to an SSH session in a lab.

ssh -oStrictHostKeyChecking=no -oKexAlgorithms=+diffie-hellman-group1-sha1,diffie-hellman-group14-sha1 -oCiphers=aes128-ctr -oHostkeyAlgorithms=+ssh-rsa $host

HomeLab Mk.3 - Planning Phase

Background

I kicked off my homelab refresh project not long ago, dubbed "HomeLab mk.3" as its the third iteration since circa 2010. I'm now well into the planning phase but I've found that I'm also overlapping into the procurement phase (as described herein).

So far, I've decided to replace my pre-Ryzen AMD-based full-tower hyperconverged system with another hyperconverged system, but this time it will be housed in an 18RU rack for providing a small amount of noise management, but also neaten up the office a little, which will have the added benefit of assisting in home improvement (flooring) later.

Key requirements;

1. Costs must be kept as low as possible
2. Software RAID (due to #1)
3. Hyperconverged system (due to item #1 and power constraints)
4. Nested virtulisation for EVE-NG/network modelling

Therefore based on requirements, the system (excluding the rack) will comprise of the following;

• One SSD for the hypervisor stack/host OS
• Up to six (6) 8Tb CMR disks for the storage of guests etc.
• 4RU ATX rackmount case (including rails of course) ✅
• As much memory as the mainboard allows which relates to key requirement #4

Challenges

The current challenges surrounding the build are;

1. Choice of Hypervisor (oVirt, libvirt, OpenStack, EVE-NG)
2. Choice of CPU architecture (due to key requirement #4 and likely challenge #1)
3. Possible Network re-architecture required to support the system including possible infrastructure Re-IP addressing.

Choice of Hypervisor

For item #1 the choices don't look that great, and I will probably stick with libvirt and the various virt toolsets, only because;

• oVirt appears to no longer be supported downstream by RedHat which means contributions to the upstream project (oVirt) will likely and eventually kill the project
• OpenStack is a pain to set up, even the all-in-one "packstack" which also means that could impact scalability in future if required
• EVE-NG appears to be an inappropriate choice. While it supports KVM/QEMU/qcow2 images, I'm not sure I want this as the underlying HomeLab hypervisor (unless certain constraints can be overcome - these are considered not in scope of this post).

Choice of CPU architecture

For item #2 the CPU architecture is important only because network vendor (QEMU/qcow2) images highlight strict CPU architecture requirements as being Intel-based, and AFAIK nested virtulisation requires that the guest architecture matches that of the host.

Possible Network re-architecture

Item #3 is not insurmountable, but it is still a challenge nonetheless as I'm not sure about whether I will change the Hypervisor guest networks (dev, prod, lab etc) to connect back upstream at L2 or L3.

Procurement

As I mentioned already, the project planning phase is somewhat overlapping with the procurement phase, the reason for this is so that I can not only procure certain less tech-depreciating items over time to allow project budget flexibility, but also allow a certain level of reduced risk in operation of the system:

Case in point: HDD's - I never risk buying them from the same batch in case of multiple catastrophic failures.

I've already purchased 3 HDD's, the 4RU rackmount case and rails and an 18RU rack to house the new gear along with the existing kit (switch, router and UPS).

I'll continue to procure the HDD's until I have enough to build the system then all that left is to purchase the key parts for the rack mount case/system (CPU, mainboard, memory & PSU) once CPU architecture/hypervisor testing (see Hardware selection below) and the design is complete.

Hardware selection (CPU architecture)

In order to determine whether the new system will be Intel or AMD will depend on the testing performed on my AMD Ryzen-based desktop. If EVE-NG and the required images work in nested virtualisation (and/or bare-metal) with said CPU architecture, then I will be in a good position to stick with AMD for this iteration (and likely future iterations) of the HomeLab. After all, AMD-based systems appear to have a good pricepoint which relates back to key requirement #1

Free Range Routing

Since I discovered Docker, I have been busy designing my homelab to be as Cloud Native as possible, but in doing so, I realised that the default docker network (aka bridge) and the other networks defined by other containers from docker-compose bridge type networks isn't known by the upstream network collapsed core access layer network.

In the past I have been adding static routes upstream and a default route on the docker host, but this was not ideal (read scalable) given the dynamic nature of docker networks created with docker-compose.

I quickly realised that since I've developed significant experience in BGP (in service provider environments), I planned to just peer the docker host to the upstream access layer, but until now didn't know how to do this with Linux.

I have always known about Quagga, but been a bit concerned about the learning curve required to get it working, but remembered about its fork called Free Range Routing. So I decided to make the a leap of faith and I have no regrets whatsoever.

I was supprised at how easy it was to install and configure and to get it working which comprised of the following;

On the Docker host;

1. Setup the REPO
2. Installed FRR and configred services for vtysh as per the official documentation
3. Connected to the vtysh interface and
4. Configured BGP and redistributed only conneted routes using route-map/prefix-list

On the upstream switch/access layer;

1. Configured peering to FRR and squelched all but a summary route prefix-list/route-map.

Not only did I define the policies perfectly (woot!), but the neighbor came up without any issues at all!!!

My switch can now route traffic to any docker/docker-compose containers/services that are created with far less effort required to define routes to the containers as well as deleting them if no longer in use.

I am very supprised to learn that not only is it extremely easy to get FRR up and running but it is very easy to configure routing daemons/config for them especially if you have a grounding in Cisco as its vtysh is very closely matched with that of Cisco CLI - there are some slight and obvious differences to Cisco's not to mention the fact that you need to connect to the vtysh interface kind of like the root user does on a Juniper platform (cli).

Next up I need to figure out how to leverage Linux VRF namespaces using FRR vtysh, then I can migrate my infrastructure to MPLS!

juniper-nuances-part1

Once day I was relaiming an interface on my newly aquired Juniper SRX300 (yes, I managed to get one of these nice little units!) and when trying to commit, I came accross the below error, which wile it seems a little misleading, it actually gives a very good clue at the issue (which may seem cryptic to the inexperienced user).

root@srx300# set interfaces ge-0/0/1 unit 0 family ethernet-switching interface-mode trunk
[edit]
root@srx300# set interfaces ge-0/0/1 unit 0 family ethernet-switching vlan members all
[edit]
root@srx300# commit check
[edit routing-instances emetric interface]
  'ge-0/0/1.0'
    Interface with 'interface-mode' is allowed only in a virtual-switch
error: configuration check-out failed: (statements constraint check failed)

In the above example, the system claims that 'the interface-mode is allowed only in a virtual-switch' however upon removing the associated interface from the routing-instance (in this case I just deleted the routing instance) and making sure that the following was in place, then commit was successful and no further errors.

[edit]
root@srx300# delete routing-instances emetric
[edit]
root@srx300# commit check
configuration check succeeds
[edit]
root@srx300# show interfaces ge-0/0/1 | display set
set interfaces ge-0/0/1 unit 0 family ethernet-switching interface-mode trunk
set interfaces ge-0/0/1 unit 0 family ethernet-switching vlan members all

So, you see the lesson here is READ the output carfeully for the clue as to why the commit is throwing an error. In my case the interface was assigned in a routing-instance and therefore mutually exclusive when the interface is a L2 trunk (highlighted text).

NBN: The Somewhat Expected Journey

In the beginning

About three and a half years ago I was planning on purchasing a house with my then-girlfriend and choose a location that was earmarked to get Fibre to the Premise (FttP) National Broadband Network (NBN)​ with a three year build date. Within a few months of settlement an election was called and my location was wiped off the NBN map completely.

Three years on and I receive an email from my ISP advising that HFC is going to be available. Then junkmail in the mailbox with offers from one random unknown and a couple of known much disliked Retail Service Providers (RSP) - in the NBN world, they are now referred to as Retail Service Providers (RSP) and formally known as Internet Service Provider (ISP). Yes there is a difference.

The only thing that I think our government may have managed to achieve with their three words slogan for NBN during their campaigning is simply "cheaper" They appear to failed on the "faster" and "sooner" parts.

Back on track with the story, I jumped at the RSP email and decided to 'pre-order' my NBN (which turned out to be a mistake) as I learned from an NBN representative that that a pre-order is actually a new order and would have created a whole new account not linked or related to my existing, long tenure one, which I want to keep at least until the day I decide to switch to a new RSP. But that's not all I learned, just not from my RSP.

Curse of the cabling

Not only did I not realise that there was an existing Telstra cable/Foxtel box out the front of the house from a previous owner/residents installation, but it was in an inconvenient location and I needed to remediate the coax cabling before the NBN appointment (to install the Network Termination Device (NTD) - otherwise known as an NBN "Connection Box".

So in preparation for the NBN appointment, I decide to cancel the NBN order (also under advise by the RSP), except I wasn't quick enough to cancel the router that was immediately sent out. No biggie. It only cost $10 to have it sent to me.

So that brings me to the next part. The cabling. I forgot about the existing Telstra cable box at the front of the house outside, but since there was also foxtel dish on the roof (I actually thought the foxtel dish was the source of the cabling point), so I decided to remove the foxtel dish, but not before a trip to Bunnings to get two blank wall plates so that I could make it appear as though nothing was cabled or points inside the house, whereby upon removing said dish, I discovered that the dishes cable didn't actually go to the wall point I expected, but I continued to remove it anyway, putting aside the dish, mounting bracket and cable once removed.

I then located the actual coax cable drop (after realising/remembering about the cable box out front. Derp) and proceeded to pull that across to the (new) location where I wanted it. At this point I realised I had to go to Bunnings again, this time for an electrical snake, electrical tape, 16mm masonry drill bit and a new coax f-type wall plate.

Thinking that I had everything I needed, I tidied up the wall plates from the previous coax cable points and started to painstaking pull the existing cable through the roof eves trying to keep minimal turns to maximise length only to find out that the cable still fell short of approximately three metres. Being the engineer that I am, I saved myself I some money by reusing the cable from the foxtel dish I removed earlier and joined the coax with a connector salvaged from a removed f-type wall plate. Brilliant!

So after wrapping the join with a generous amount of electrical tape, there was more than enough length for the cable to reach the new wall point, for which was not as straight forward as just drilling a forty five degree hole to the wall cavity as I managed to loose the coax connector after I managed to get the snake and cable wedged in the bend of the newly drilled hole thinking I could stupidly un-wedge it by pulling harder on the snake then on the actual cable (minus the connector) and then on the snake again - finally dislodging the snake (minus the connector - which now lives somewhere inside the cavity walls). All this only because I mistakenly taped the coax cable and it's connector to the top of the snake instead of the bottom to allow it to go around the the bend of the newly drilled hole for the wall plate. Lesson learned.

Once I had the coax pulled through, I went off to Jaycar this time to get the replacement F-type connector (including a spare in case I borked it), but quickly realised that they were for a coax cable with a smaller core, so after yet another trip to Bunnings for the correct F-type connector, I got the wall plates finished and titles put back on the roof. Lucky for me I didn't break any tiles during the walking about on the roof!

Order in the court!

After all the saga of cabling was completed, I contacted my RSP and requested a new NBN order, this time under my existing account. So far so good except I receive another email telling me a router is being delivered. Again.

The next day I contact my RSP via reply email for the hardware order asking to cancel it. No response. I call them the following day to cancel it. They tell me there is no evidence of any charge but by that time I already have an email telling me it's been dispatched and on its way (It arrived the next day shoved into the mailbox, whereas the previous one I had to sign for at the local post office).

At this point the NBN appointment to install the NTD had also been brought forward but in the meantime I decided to break open the HG659b since I had two of them, so one of them was doomed to become my sacrificial test unit (read on to find out how/why).

./hack

Hacking the sacrificial unit involved soldering header pins to the empty holes where a serial port has been identified thanks to an openwrt hardware wiki article on it. De-soldering the factory solder points to put the header pins took far longer than actually soldering on the header pins though.

Having access to the serial port it was easy to get into the Common Firmware Environment (CFE) by interrupting the boot sequence so that I could flash spark (NZ) firmware onto this unit as it was more likely to include the download configuration file exploit not available in the crippled TPG firmware.

After doing the configuration file download exploit on the sacrificial unit with alternative firmware, I decrypted the configuration file with the help of a whirlpool knowledge article, and then the root password from the modified unit, I was then able to successfully use those credentials to log into the unmodified unit with normal, unadulterated, uncripled access.

With noob access defeated, I quickly discovered that not only does the unit only allow static routes, on the WAN interfaces but it is very picky about its LAN management IP address and subnet mask. According to the admin UI, apparently 192.168.2.1/24 is invalid and the Command Line Interface (CLI) over telnet is cripled and you cannot get a shell whatsoever. By this stage you can probably imagine the eyetwitch starting.

By now, it was blindingly clear that the HG659b is complete rubbish (ESPECIALLY WITH TPGs CRIPLED FIRMWARE!) - even more so for a network engineer such as myself.

In between all this, the NBN contractors had attended the premises (at the very end of the appointment window no less) to install the NTD (yes, it took two of them) and found the existing Telstra cable to have no signal! Thinking that it was my fault, I simply explained that I "had the the wall point moved". They then opened up the telstra cable box out front and not only found two cables coming from it but one of them was not connected - which was obviously the one I had remediated. They switched the cabling and the line had signal and they where able to activate it. By now one of the NBN contractors - what looked like the junior of the two - had left. The NBN contractor then used the excuse that his phone battery was flat so that he didn't have to wait around for the half an hour for confirmation of the service being activated. I wonder where the other cable leads to then?

MOAR HARDWARE!

Knowing that the HG659 is completely useless to me for my requirements, I dug around for an OpenWRT compatible router (not an AP and ADSL and everything all-in-one-gateway-that-everyone-calls-a-damn-router!) and settled on a MikroTik RouterBoard RB750GL from WISP - these guys have a questionable website security wise (certificate is fine but pages may include a form(s) with a non-secure "action" attribute.) but they shipped this thing in record time!

With the new router in hand I took a quick look at the very ugly UI for RouterOS and promptly installed OpenWRT on it using a combination OpenWRT installation methods from both the device page and the general common procedures (the latter of which mentions the missing initrd).

Before I got it off the default IP address (and adding a static summary route for all my subnets), I realised was getting annoyed at this point having to deal with equipment which ships with IP addresses which conflict with my own network, so I decided to change my Local Area Network (LAN) subnet to something more sane, which took about three hours of solid uninterrupted work to migrate configuration to a previously unused Cisco 48 port PoE switch (some things are outstanding but I can defer those for another time).

Lastly, I added a new PPPoE connection to the the port already tagging on VLAN ID 2 and viola! I'm connected to the NBN with a device which I have much more trust and configuration options. Even the default firewall rules where reasonable.

Summary

I have learned a few things these last few weeks: how to save some money, how not chase a cable through a wall and not to necessarily jump onto something no matter how exiting it seems and could help with working from home and studying in the future but even more recently, how NBN HFC is delivered and more recently, how important security is with the NBN (more on this in a new post hopefully).

It has been an interesting journey and the thing I enjoyed the most about this was the hardware hacking (however futile it was) and the handyman type work (drilling holes chasing cables etc). What I least enjoyed was the level of service from my RSP and the fact that they managed to bork the username on the account, but all-in-all the actual throughput/bandwidth of the service seems reasonable for now and by the time you read this, I will probably have shut down my ADSL service.

The funniest thing about all this all is the fact that the wife hasn't noticed the improvement in speeds, since I told her (and she agreed) to the experiment of not telling her when it was actually done!

Oh and NBN sent a survey asking about how likely I was to recommend NBN, based on the "excellent" service I received with a scale of 1-10, only to tell me that I had to choose 4 or higher. Go figure.


If you have read this far, thanks for reading and feel free to share your experience with me by posting in the comments or via Google+ (link to post is best).

BIND (named) server remidiation [part 2]

Following up from my previous post (BIND (named) server remidiation), I spent a good couple hours further developing and testing the configuration but failing to get a bind9 reverse lookup zone to load only to find out that I had a slight typo in the reverse lookup zone definition

named-checkzone was returning OK, but named itself was failing to load the zone file with the error:

zone X.X.X.in.addr.arpa/IN: has 0 SOA records
zone X.X.X.in.addr.arpa/IN: has no NS records
zone X.X.X.in.addr.arpa/IN: not loaded due to errors.

It wasn't until I had a friend take a closer look at then the problem became clear:

I defined the zone as .in.addr.arpa instead of .in-addr.arpa in the named.conf include file which references the zone file.

Some things I have learned is:

• Check the logs (in my case, on a default debian/bind9 install this was /var/log/syslog) when things don't work.
• Always check your config with the bind DNS tools before reloading
• Always check your zones files with the bind DNS tools before reloading
• Keep zone files neat and group together similar resource record types.

Now that I have the dev domain DNS working, I just need to look at setting up DHCP and testing dynamic DNS.

I also considered moving different resource records for each zone into a separate file, but this is not necessary, due to the (current) size of the network.

Once this is all done, tested and implemented in 'production', I will also consider keeping a similar configuration in dev as a slave for all zones from the primary DNS or just as it is and just for testing.

BIND (named) server remidiation

Since I virtualised my old failing physical server into a VM, I have found it less and less easy to administer and maintain (read: configuration files).

So, I am looking and spinning up new Debian servers for more specific tasks, network services, games servers, file services etc.

The fist, and most important thing I need to migrate is DNS. That way I can have it simply running in parallel with the old, ready to essentially, stop the service (after making sure DHCP serves out this DNS IP address as well of course!).

Now, here comes the "clever" part or the goals of this approach (or so I thought):

1. Install named.
2. Configure it to be a slave for the existing zones
3. re-configure it to be a master (complete with zone files)

Pretty simple right? Not so much. Well, thanks be to the 'Debian' way of doing things, it was very quick and easy to have a the zones slaved, but when I went to look at the files I was expecting, they where still empty, since I had created empty zone files to begin with.

Some poking around later and I discover that it is transferring the zones fine, but there was an issue with permissions for the zone files, or more specifically, the directory where they lived. A quick chmod -R 0777 /zone/file/directory later and a restart of the service, voila! Except.... something was not right...

The zone files seemed to be in a binary format as file would have me believe they were of type: data

I could have converted them back to plain text using the bind-tool named-compilezone(8) but, I couldn't commit my time to learning how to get the syntax correct for one small job, besides I learned that it is a crazy default in order to get a performance increase, however minuscule that would be given such a small DNS server implementation (for now).

So as per the article "Bind 9.9 – Binary DNS Slave file format" (linked above) or more authoratively as per the Chapter 6. BIND 9 Configuration Reference section of the BIND 9.9 Administrator Reference Manual (ARM) which states (incorrectly):

masterfile-format

Specifies the file format of zone files (see the section called “Additional File Formats”). The default value is text, which is the standard textual representation, except for slave zones, in which the default value is raw. Files in other formats than text are typically expected to be generated by the named-compilezone tool, or dumped by named.

So, knowing this I edited /etc/bind/named.conf.options to include the following:

masterfile-format text;

Perfect. (Just like me ;-) I now have a duplicate of the zones served on the master server, which can, and will soon be decommissioned, not to mention the new servers zones getting a makeover with many many more zones as well as a dynamic-update zone - more to come on this soon.

Great Success

Finally after weeks, no months of agonising failure though trial and error, I finally managed to get the outcome I desired with my Raspberry Pi 2!

History

A few years back I acquired a Cisco 3560 and quickly realised the potential of vlans and separate subnets for the purposes of testing among other valid reasons, and came to find that the nodes on most of the vlans could not communicate with the outside world (read: internet). It was then that I realised that something was wrong...

Long story short: the Netgear DGND4000 that I own does not route/NAT anything other than its resident subnet and I sure as heck was not going to implement double NAT!




Thanks be to LIbVirt's NAT networking which gave me an interim workaround and helped confirm this.

Getting the necessary bits

NAT issues aside, I began by purchasing a second-hand Netgear DM111P v2 from some random guy on Gumtree. The ADSL Modem in itself wasn't enough because it too, seemed to suffer from the same issue as the DGND4000 did, although admittedly, I didn't put much effort into testing that theory as I wanted a solution not more testing.

I then purchased a Raspberry Pi 2 along with a bunch of accessories. In the meantime (while I was waiting the excessively long shipping time). I did some research on the distributions that are capable of running on the bcm2709-based board and decided with OpenWRT. Yes, I know that I could have used Raspbian but OpenWRT seemed the most logical choice given the fact that it is essentially an internet router anyway, just without the wireless and ADSL modem.

Turns out I made the right choice despite the fact that OpenWRT is still in trunk (RC3 at the time of writing this).

Lastly (after destroying the extremely cheap Rpi2 case) I managed to get an image booted (helps when you use the bcm2709 not the bcm2708 barrier breaker version, thats for the Raspberry Model B!).

Configuration

First of all, this would have gone a lot smother had I have just tested with the USB network adapter I bought along with the Pi, but it didn't get here in time with partial shipping.

I configured the switch with a trunk port with two vlans, one for the LAN side of things (internal link) and another for the WAN or pppoe (public/external/internets) and set the mode appropriately.

NOTE: VLANS and IP addresses have been altered so as to protect the actual configuration used in my network infrastructure. Call me paranoid.

Cisco 3650 partial configuration

!
vlan 20
vlan 69
!
interface Vlan69
 description DMZ/LAN
 ip address 192.168.69.1 255.255.255.248
 no shutdown
!
! no interface defined for WAN because we do not want any L3 traffic
!

interface GigabitEthernet0/2
 description Trunk port for Rpi2 VLAN's: 20, 69
 switchport access vlan 69
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 20,69
 switchport mode trunk
 no shutdown
!
interface GigabitEthernet0/1
 description Link to DM111Pv2 modem (bridged) for PPPoE/L2 traffic
 switchport access vlan 20

 no shutdown

exit
!
ip route 0.0.0.0 0.0.0.0 192.168.69.66
!
end

OpenWRT network configuration

root@OpenWRT# vi /etc/config/network

config interface 'loopback'
        option ifname 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config interface 'lan'
        option proto 'static'
        option delegate '0'
        option _orig_ifname 'eth0'
        option _orig_bridge 'false'
        option ifname 'eth0.69'
        option ipaddr '192.168.69.66'
        option netmask '255.255.255.248'

config route
        option interface 'lan'
        option target '192.168.0.0'
        option netmask '255.255.0.0'
        option gateway '192.168.69.1'

config interface 'WAN'
        option proto 'pppoe'
        option ifname 'eth0.20'
        option delegate '0'
        option username 'myusername'
        option password 'mY$eCr3tP4sSw0rD'

Caveats/Adendums/Extra information

By now you may be wondering, "Why is there no IP addresses or switch virtual interface for vlan 20"? There is no need for it! That, and the fact one might only want traffic to go via one vlan and then the other (remember, this is essentially a router on a stick implementation and we want to separate the vlan's into L3 traffic for one and L2 for the other per requirements).

If you were thinking: "The netmask and destination network IP for the LAN route is wrong!", you would be incorrect. This is a perfectly legitimate summary route. It allows for much easier (read: slack) administration so one does not have to manage multiple static routes for subnets added or removed from the network (short of running a routing protocol) and it has the added benefit of consuming less memory and is a much more flexible approach for this design. Neat huh? I thought so too :-)

Conclusion

Let it be said that although this configuration is very simple, there where many hurdles accompanied by many choice words along the way. The one single most important thing that I kept getting wrong was routing. I had to remember to change the 'gateway of last resort' (Cisco's way of saying default route) on the switch so that all the subnets will route to the internet and the static (summarised) route for traffic to get back into the network from whence they came. That and trying to test this when the internet is depended upon so much by the two people in this household, was frustrating as my change windows where often short and had to be rolled back constantly.

Lastly, I must say that "out-of-the-box" pppoe/nat/routing on OpenWRT worked with like a charm with minimal configuration, however I will need to develop the scenario a little further so I can secure the connection by way of its firewall (read: iptables), but that itself is a beast I have yet to conquer.

Rasbpberri Pi Internet Connectivity Lab

I have successfully built a lab for testing internet connectivity to the Raspberry Pi 2, by using my phone in a USB tethering configuration.

I followed the majority of the configuration listed in the OpenWRT wiki, except I used the LuCI web configuration instead of the final manual step of using uci to use usb0 as the WAN connection

This will now allow me to test various scenarios including multiple default routes with different metrics as well as testing firewall configurations using OpenWRT running on the Rpi2.

The one gotcha is that I forgot to set the rout back to the internal network for which was previously miss-configured.

I am getting one step closer to having much more control of my internet as well as being able to NET/Route all of my subnets!

failure to focus

I have confirmed that I can get the Raspberry Pi to connect to the ISP using PPPoE through a VLAN, however, I cannot (or rather my brain cannot) get the OpenWrt to accept traffic other then ICMP to/from the device itself (I probably need to understand iptables or I am overlooking something very simple).

I'm finding it extremely hard to focus and get the networking part of this lab working right now when I don't actually have a lab to do it on and when others in the household rely on internet so much including myself, when I need to refer to something while trying to troubleshoot and find a solution to this 'router-on-a-stick' model of networking to overcome the shortfall of the existing router.

I've also lost my 4Gb micro SD for which I was planning using for building a Bluetooth (A2DP) Audio receiver from the Raspberri Pi which is making me a little less than happy considering they are not as easy to come by due to the size and I will have to spend another $10 (effectively $20 now) in order to get one.

For now, I'm going to go watch something and try again later (including looking for the SD card).

isp faithfulness

I just discovered today that I was being charged by my ISP for a broadband account in a place I used to live in...

I applied for broadband back in 2006 when I was living in Sorrento, but broadband was not available in that area until about 6 months after I left that suburb and cancelled my dial-up. My ISP has been charging me for it ever since!

Luckily for me, my ISP not only cancelled the old unused account, but have also refunded the last 6 months. At least my ISP seems to reward faithful customers...

xbox360 and WMP11 fail

After about two years of procrastination, I finally got an xbox360 to fulfil my Burnout gaming needs. Before I got the (MW2) bundle, I installed Windows Media Player 11 (using the validation bypass trick). Now sitting here relieved that I finally got media sharing to work, I thought I'd blog about the one simple thing wrong with media sharing:

1. UNC network paths.

It took almost 2 weeks to find an answer to why my library was not showing up on the xbox360 and the answer lies here.

In summary (for those that don't want to follow that URL for some reason):

"If you want to share files that are not located locally on your machine, Microsoft has chosen to prevent you from sharing content located on network locations (UNC paths) like \\MACHINENAME\sharename"

Luckilly there is an undocumented registry "hack" that will fix this:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MediaPlayer\Preferences\HME]
“EnableRemoteContentSharing”=dword:00000001

Huge thanks go out to GibStorm for documenting this so well...

10th Annual System Administrator Appreciation Day

Today marks the 10th Annual System Administrator Appreciation Day.

Treat some lonely, unforgiven and/or unloved sysadmin with a gift and/or note of appreciation today and show how much you appreciate the hard work and effort that they do (myself included).

They are usually the same people that make your internets work! so show us some love. Please.





Happy Sysadmin Day!

/respect

internets anew

Thanks to this article I was able to transform my crappy dg632 router into a dumb modem so that I could get better control (including better security) on my internet link.

The only problems I encountered was that I found it difficult to set the router into bridged mode, but finally found the answer here. The documentation on the gentoo wiki differed slightly too, in that the iptables exported variable for the WAN interface should be ppp0 instead of eth1.

Other than that I can now enjoy a properly firewalled, dyndns capable and port-forwarding capable setup at no extra cost.

Now all I need to do is get bind, ldap and openvpn working... having all this free time without a job does have it's benefits...

Be afraid. Be VERY afraid

Thanks to Senator Conroy, Australia may have to suffer not only internet censorship but very slow internet speeds, due to an old yet unfeasable policy that the government seems determined to force into our homes (and probably businesses too).

I now direct you to http://nocleanfeed.com/ which explains (better than I can at least) what this is, how it affects you and most importantly, what you can do to help prevent it.

No Clean Feed - Stop Internet Censorship in Australia

Home Network Redesign

I think I may have the motivation to redesign my network for the following reasons:

• Increased Security: Better Firewall solution
  
• Increased Security: Segmented network design, allowing for More control with untrusted hosts such as:
  
  
  
  • internet (DMZ)
    
  • Wireless and
    
  • temp hosts (ie. LAN party ^_^).
    
  
  
• Manageability: Implementing a secure routing protocol (RIP? maybe) may assist with network growth/changes
  
• Gloating: Show off my 1337 net skillz to my friends :P
  

The most important aspect of this is the first point (increased security). It should also allow me to easily create a vpn endpoint onto my server for remote access (from outside the network) and for internet access from the wireless network... *hmmm* radius anyone?

This should help me un-lazy myself and develop (internal) dns zone(s) too...

Hopefully I can have the design done and implemented by the end of next UNI break!

2Tb of goodness

So I finally shelled out (about ~$400+) for some more storage capacity (2 x 1Tb WD HDD). I was planning on buying a brand new fileserver ($2.5K worth) and doing software raid across them, but due to the cost of recovering from the car incident, I decided to do this on the cheap (JBOB). So now I have about 2Tb of storage at my disposal until I can afford the 10Tb fileserver (RAID6) *grin*