When running tools like like docker, podman, kubernetes, etc. internal networks are set up that could collide with routed networks within the University of Waterloo’s infrastructure. If this collision happens, then any traffic that is supposed to route to those will stay on the given system. An example for this, if you are on VPN with the IPv4 address 172.25.10.1, and you start a container or compose project, if docker allocates a network route 172.25.0.0/20, your connection to the server will drop, as any traffic that was destined for you now gets directed to the docker network.

HINT: to see your routes; Windows: route print, Linux: ip route , MacOS netstat -rn

To prevent these issues from happening, campus Network Operations Centre (NOC) has reserved a subnet that will not be routed. This can be used for containers. The network block that has been allocated is:

10.255.0.0/16

This network contains 65,535 addresses that can be used for container networks. The various tools below use different default subnets and are configured differently. Known configurations are listed below. If there are any others that should be added, FAST members can edit this page, or a request can be made to document the specific tooling.

Configurations

1 Docker
2 Docker Swarm
3 Podman
4 Kubernetes
- 4.1 RKE2
5 LXC

Docker

/etc/docker/daemon.json

{
....
  "default-address-pools": [
    {"base":"10.255.0.0/16","size":25}
  ]
}

size specifies the default network size for each docker network. /25 contains 128 addresses, so adjust based on your needs.

If you are setting this on a host that has existing containers, networks will need to be recreated.

WIP: Instructions for doing this on Linux

#WARN: removing networks can break containers and require rebuilding!
#change to root user
sudo -i
<<EOF cat > /etc/docker/daemon.json
{ "default-address-pools": [ {"base":"10.255.0.0/16","size":25} ] }
EOF
systemctl stop docker.service
systemctl stop docker.socket
#remove all the networks that containers created.
#containers will need to rebuild their networks in the new pool
docker network purge
#remove the docker bridge
ip link delete docker0
### TODO: how to rebuild the networks?

Docker Swarm

In addition to the above docker configuration, you must also do the following

Default address pool must be set on creation and cannot be changed

docker swarm init --default-addr-pool 10.255.0.0/16 --default-addr-pool-mask-length 25

default-addr-pool-mask-length specifies the default network size for each docker network. /25 contains 128 addresses, so adjust based on your needs.

Podman

See common/docs/containers.conf.5.md at main · containers/common for configuration locations

[network]
default_subnet="10.255.1.0/24"
default_subnet_pools = [
  {"base" = "10.255.1.0/24", "size" = 25},
  {"base" = "10.255.2.0/23", "size" = 25},
  {"base" = "10.255.4.0/22", "size" = 25},
  {"base" = "10.255.8.0/21", "size" = 25},
  {"base" = "10.255.16.0/20", "size" = 25},
  {"base" = "10.255.32.0/19", "size" = 25},
  {"base" = "10.255.64.0/18", "size" = 25},
  {"base" = "10.255.128.0/17", "size" = 25},
]

size specifies the default network size for each docker network. /25 contains 128 addresses, so adjust based on your needs.

Kubernetes

This is highly dependent on which provider you are using. k3s uses docker, so use the Docker instructions.

RKE2

This should be done when creating the cluster, it isn’t well supported to change this after creation

/etc/rancher/rke2/config.yaml

cluster-cidr: 10.255.0.0/17
service-cidr: 10.255.128.0/17

LXC

/etc/default/lxc-net

LXC_BRIDGE="lxcbr0"
LXC_ADDR="10.255.0.1"
LXC_NETMASK="255.255.0.0"
LXC_NETWORK="10.255.0.0/16"
LXC_DHCP_RANGE="10.255.0.2,10.255.255.254"
LXC_DHCP_MAX="65533"

You could do a smaller range if you wanted to as well

$ sudo vi /etc/default/lxc-net
LXC_BRIDGE="lxcbr0"
LXC_ADDR="10.255.0.1"
LXC_NETMASK="255.255.255.0"
LXC_NETWORK="10.255.0.0/24"
LXC_DHCP_RANGE="10.255.0.2,10.255.0.254"
LXC_DHCP_MAX="253"

FAST Knowledgebase

Preventing Network Collisions Between Campus Infrastructure and Docker, Podman, Kubernetes, LXC, etc