LXD/LXC networking a static crossover directly connected NIC

 For the life of me I couldn't find this out there on the net.  I looked at countless videos and read blogs but to no avail.

The task seemed easy, I have two physical hosts with dual-port 25GbE NICs.  I only have a 10GbE switch but want to take advantage of the faster link speed so why not use a DAC and just static IP the interfaces?!

Here is the setup:

srv1 & srv2 hosts are set up identically:

eth0_10gbe (NIC) --> br0_10gbe (host bridge) 10.10.0.0/24

eth1_25gbe (NIC) --> br1_25gbe (host bridge) 10.25.0.0/24

I then setup LXD to use the 10GbE bridge for networking using the unmanaged (by lxd) host bridge by using the lxd bridge nic.  This allows the DHCP request to be passed to my LAN DHCP server (Ubiquity UDMPro) for that particular segment (vlan tagged network).  This was relatively easy and there are a bunch of tutorials and things since this is rather standard.  

In Ubuntu I used netplan to add the VLANs to the host bridge and I use the subsequent "sub interface" in the container.  Said differently, UDMPro sends untagged traffic to the port, the NIC puts that in the host bridge which then breaks out the VLANs, then LXC containers are attached to the VLAN interfaces.

The NIC direct connected between hosts was super simple to configure for the host.  I was scping and stuff at 25GbE in mins.  Then trying to get LXC containers to do the same thing proved a little more challenging.

I eventually settled on creating a bridge (as stated above) and adding that to the container as eth1 using LXC's bridging nictype:

lxc config device add <container_name> eth1 nic nictype=bridged parent=br1_25gbe

I then need to go into the container OS and manually configure an IP.

Once of the LXD folks wrote a blog about some of the other kind of networking and includes code for doing manual static IP addressing from an LXC profile which is likely what I'll do next... all of my LXC containers are provisioned via terraform and then configured via ansible so maybe I'll go a different route (pun intended) but not sure yet.

But why though? This is all to allow for some of the containerized services to use the 25GbE network as a backhaul connection for their cluster traffic.

LXD (now Canonical's) a journey into the simplistically complex

The history of LXD is not part of this post...

LXC (Linux Containers) were not the goal.  I have experience with OpenStack and wanted to go that route but the hardware requirements and overhead for my particular project were so small (just a few different services) that someone on my team said "I'm just gonna install them all on my mac and be done with it.

I couldn't let that go, I needed some CI/CD and config management, I needed repeatable code based infrastructure!

I first found Canonical's MicroCloud and was kinda sold!  But their hardware requirement was min of three like servers and we just had two like servers.  So I did flirt with setting up LXD then putting a multi node MicroCloud cluster in it on VMs but that seemed a little crazy.

So I started with one host, configured LXD and started building containers with terraform and ansible.  It worked so well I was convinced that this was the right path forward.

The hardware are some Ryzen 5 boxes with max RAM and a few SSDs and a dual port 25GbE NIC.  One port from the NICs are direct attached to each other (no switching) and the other is into a 10GbE switch since that is what we have.

I configured each LXC server as a remote of the other and added the remote using the 25GbE network so copy/move operations would be max speed.

I had some trouble getting that part working since each time I would follow the steps to add the remote using the trust cert I would get an error.  No amount of googling found the answer so I tried to post on the LXD forums at Canonical... that was a waste of time since I couldn't even post and there was a disclaimer about "not for tech support" bla bla... so Reddit for the win!

I created a post "LXD to LXD host on one NIC, everything else on another?" in the LXD subreddit and within mins I was off and going!

I can run lxc commands from my laptop connected wifi and see both remotes and issue commands, it's wicked simple once you get it all setup and memorize all the CLI commands ;) 

I opted to have the 10GbE connected to the switch getting all the untagged VLAN traffic then use a linux bridge to pass the VLANs as networks into the containers.  I kinda followed the OpenStack pattern from the old days but also found Trevor Sullivan's YouTube Tutorials super helpful as well as the OG PM of LXD Stéphane Graber's YouTube and Site very helpful!

I might add more details in the future, but my hope is if anyone else is looking for some LXD/LXC servers that are not clustered and hit some of these road blocks maybe this will shorten their search a little.