When running tools like like docker, podman, kubernetes, etc. Internal networks are setup that could collide with routed networks within the University of Waterloo’s infrastructure. If this collision happens, then any traffic that is supposed to route to those will stay on the given system. An example for this, if you are on VPN with the ip address 172.25.10.1
, and you start a container or compose project, if docker allocates 172.25.0.0/20
, your connection to the server will drop, as any traffic that was destined for you now gets directed to the docker network.
To prevent these issues from happening, a subnet has been reserved so that it will not be routed. This can be used for containers. The network block that has been allocated is:
10.255.0.0/16
Which contains 65,535
addresses that can be used for container networks. The various tools use different default subnets and are configured differently. Known configurations are listed below, if there are any others that should be added, FAST members can edit this page, or a request can be made to document the specific tooling.
Configurations
Docker
/etc/docker/daemon.json
{ .... "default-address-pools": [ {"base":"10.255.0.0/16","size":25} ] }
size
specifies the default network size for each docker network. /25
contains 128 addresses, so adjust based on your needs.
If you are setting this on a host that has existing containers, networks will need to be recreated.
TODO: Fill in instructions for doing this
Docker Swarm
Default address pool must be set on creation and cannot be changed
docker swarm init --default-addr-pool 10.255.0.0/16 --default-addr-pool-mask-length 25
default-addr-pool-mask-length
specifies the default network size for each docker network. /25
contains 128 addresses, so adjust based on your needs.
Podman
See https://github.com/containers/common/blob/main/docs/containers.conf.5.md for configuration locations
[network] default_subnet="10.255.0.0/24" default_subnet_pools = [ {"base" = "10.255.1.0/24", "size": 25}, {"base" = "10.255.2.0/23", "size": 25}, {"base" = "10.255.4.0/22", "size": 25}, {"base" = "10.255.8.0/21", "size": 25}, {"base" = "10.255.16.0/20", "size": 25}, {"base" = "10.255.32.0/19", "size": 25}, {"base" = "10.255.64.0/18", "size": 25}, {"base" = "10.255.128.0/17", "size": 25}, ]
size
specifies the default network size for each docker network. /25
contains 128 addresses, so adjust based on your needs.
Kubernetes
This is highly dependent on which provider you are using. k3s uses docker, so use the Docker instructions.
RKE2
This should be done when creating the cluster, it isn’t well supported to change this after creation
/etc/rancher/rke2/config.yaml
cluster-cidr: 10.255.0.0/17 service-cidr: 10.255.128.0/17
LXC
/etc/default/lxc-net
LXC_BRIDGE="lxcbr0" LXC_ADDR="10.255.0.1" LXC_NETMASK="255.255.0.0" LXC_NETWORK="10.255.0.0/16" LXC_DHCP_RANGE="10.255.0.2,10.255.255.254" LXC_DHCP_MAX="65533"
You could do a smaller range if you wanted to as well
$ sudo vi /etc/default/lxc-net LXC_BRIDGE="lxcbr0" LXC_ADDR="10.255.0.1" LXC_NETMASK="255.255.255.0" LXC_NETWORK="10.255.0.0/24" LXC_DHCP_RANGE="10.255.0.2,10.255.0.254" LXC_DHCP_MAX="253"