Networking is an important component of containers and VMs. Most of the complexity that trip users up has to do with networking and a basic understanding will help users feel more confident about using containers and VMs.
In the article we will look at connecting containers across hosts with BGP. We have previously used Vxlan and Wireguard and you will find the links below and in the guides section.
We have already done a guide on container networking that is essential to get an understanding of basic container networks.
Normally containers are in a private NAT network on the host. They can reach the outside world but cannot be reached from outside.
This is the usually the default setup and you can design the network so containers are not in a private network. This would involve bridging the hosts network and having control of the router that manages the hosts so containers are in the same network as the host. This is often not possible, for instance on on the cloud. In these cases overlay networking and technologies like Vxlan, Wireguard and BGP become useful.
Take two servers in the same subnet connected to each other, both hosting container networks within them. While the hosts can talk to each other the containers cannot. One easy way to fix is to add a route on each of the hosts. While this works for a few hosts it can quickly become unwieldy as host count increases.
What if we could have an application that takes care of adding and maintaining all these routes across hosts. This is where BGP becomes useful. BGP actually runs the Internet. It's a layer 3 protocol that major providers use to peer with each other and share routes.
In our case we can use BGP to peer container hosts with each other and announce their respective container subnets.
There are a number of Linux applications that support BGP but we found Quagga the easiest to use and configure. Quagga is also available on most distributions.
Quagga is an open source routing software suite. Quagga is quite simple to use and is available on most distributions. Just install Quagga, configure a single file to peer with other hosts and share networks. That's all there is to it.
Flockport supports connecting networks with Quagga but in this guide we will look at setting it up manually so users understand how it works.
We will use 3 servers for this example. Server A with IP 192.168.50.10, Server B with 192.168.50.11 and Server C with IP 192.168.50.1. The servers host container networks 10.0.3.0/24, 10.0.4.0/24 and 10.0.5.0/24 respectively.
The first step is to install Quagga on all the servers and you can use your distributions package manager for this. For this guide we will use Ubuntu Xenial.
apt-get install quagga
Once Quagga is installed the Quagga configuration files are in '/etc/quagga'. Quagga supports a number of routing protocols. We are going to use BGP.
Edit the /etc/quagga/daemons file and enable BGP and Zebra protocols. Your /etc/quagga/daemons file should look like below.
zebra=yes bgpd=yes ospfd=no ospf6d=no ripd=no ripngd=no isisd=no babeld=no
Next we will create two default configuration files for bgp and zebra services enabled. This will /etc/quagga/bgp.conf and /etc/quagga/zebra.conf. You can find default configurations in /usr/share/doc/quagga/examples but for now lets go with the samples below.
This is how the zebra.conf file should look.
! password zebra enable password zebra log file /var/log/quagga/zebra.log !
This is how the bgp.conf file should look. Exclamation marks are used for comments.
! password zebra enable password zebra log file /var/log/quagga/bgpd.log router bgp 64512 !
Let's make sure the config files have the right owner and permissions. The quagga user is usually automatically created at the time of installation.
chown quagga:quagga bgp.conf zebra.conf chmod 640 bgp.conf zebra.conf
The steps above need to be done on all 3 servers. Now we are ready to start the quagga service on all 3 servers.
service quagga start
You should get no errors. Incase you do please retrace the steps above and ensure you have followed the instructions.
Now that the quagga daemon is running on all 3 servers hosts the next step is to peer them.
We will peer Server A with Server B and C. Server B and C will not be directly peered but simply peer with Server A. Server A will be responsible for populating the routes.
To peer with servers we need to add their IPs in the bgp.conf file.
password zebra enable password zebra log file /var/log/quagga/bgpd.log router bgp 64512 bgp router-id 192.168.50.10 neighbor 192.168.50.11 remote-as 64512 neighbor 192.168.50.12 remote-as 64512
The router bgp 64512 defines the BGP ASN - think of it as a network ID. Servers with the same ASN or autonomous system number are considered to be part of the same network group. The BGP protocol uses the AS number to detect whether the BGP connection is internal or external.
The bgp-router-id is the host server's IP over which the BGP network will work. Peers are added with the neighbor keyword. Neighbors define the IPs of the hosts we want to peer with. You can have multiple neighbor lines to peer with multiple servers.
The remote-as 64512 bit at the end of the neighbor line defines the peered server as part of the same internal BGP network. Servers in different ASNs peer over eBGP while servers in the same AS peer over iBGP.
To share networks we add them with the network keyword. In this case 10.0.4.0/24 is the container subnet Server A is sharing in its bgp.conf below. Similarly Server B and Server C will share their respective container networks in their own bgp.conf. You can share multiple networks.
password zebra enable password zebra log file /var/log/quagga/bgpd.log router bgp 64512 bgp router-id 192.168.50.10 neighbor 192.168.50.11 remote-as 64512 neighbor 192.168.50.12 remote-as 64512 network 10.0.4.0/24
Since we will not peer Server B with Server C and are relying on Server A to communicate the routes we need one more step.
BGP has a concept called route-reflectors for servers that are in the same network group. This is mainly designed so all servers do not have to peer with each other. In our case Server A is the route reflector and Server B and C will be route-reflector clients. Let's add that to the configuration.
! password zebra enable password zebra log file /var/log/quagga/bgpd.log ! router bgp 64512 bgp router-id 192.168.50.10 neighbor 192.168.50.11 remote-as 64512 neighbor 192.168.50.12 remote-as 64512 network 10.0.4.0/24 neighbor 192.168.50.11 route-reflector-client neighbor 192.168.50.12 route-reflector-client !
After we add that this is how Server A's bgp.conf will look. To configure a route reflector we simply define the neighbor as a route-reflector-client.
For reference this is how Server B's bgp.conf will look.
! password zebra enable password zebra log file /var/log/quagga/bgpd.log ! router bgp 64512 bgp router-id 192.168.50.11 neighbor 192.168.50.10 remote-as 64512 network 10.0.5.0/24 !
As we can see above Server B is peering with Server A. Its bgp-router-id is its own IP and it is sharing the 10.0.5.0/24 subnet.
Once this is done restart the quagga service on all 3 servers.
If the configuration is correct the servers will have successfully peered with each other and shared their respective networks.
You can try to ping the container networks on others servers to check this is working. Running the ip route command on your server will show you all routes. You will notice new routes to the peered servers container networks. These have been added by Quagga.
You can add more subnets to the network by adding them to the bgp.conf file. You can peer new servers by adding them as neighbours in Server A's bgp.conf file and peering with Server A in their bgp.conf file. The new server will automatically get the shared routes. Compared to other networking options BGP is relatively simple to use. Layer 3 networks are more scalable and robust.
Flockport once installed can provision servers and deploy containers across servers. You can then then connect these container subnets across servers with BGP. Here is a screencast of Flockport setting up a BGP network.
Flockport also lets you build container networks with Vxlan and Wireguard.
Quagga provides a shell to interact with the quagga daemon. It's called vytsh.
You can use the vtysh command to invoke the shell. There are a number of commands to interact with the shell. We won't go into detail right now but there are a couple of useful commands.
'show ip bgp' gives you an overview of current peers and shared routes. 'show ip route' gives you an overview of the routing table. 'show ip bgp neighbor' gives you detailed information on peers and connnections.
You can either run vtysh and then type the commands or run it directly from your shell with vtysh -c 'command'.
vytsh -c 'show ip bgp'
Similarly show ip route gives you a summary of the hosts routing table
If you do not see your BGP peers in the show ip bgp command you need to recheck your configuration and restart the quagga service.