systemd-networkd Migration and Benchmarks: Fast and Furious

Introduction

Some days ago Tom Gundersen (systemd developer) says that networkd works really fast with DHCP:

A couple of weeks ago I started profiling networkd’s DHCP client library, and found that we compared relatively favorably to the ‘competition’, but were still adding way too much to boot-time to be acceptable in containers. Acquiring a DHCP lease from the same host (so no network latency) took about 500ms.

Quite a bit of low-hanging fruit later we were down to 50ms, but with one big bottle-neck remaining. Today, with lots of help from +Kay Sievers and a crucial suggestion from Daniel Borkmann, I finally killed off the last obvious bottle-neck and we are now able to acquire a lease in about 750 micro seconds (so almost 1000x improvement :)).

Time to check it now with benchmarks. But after I need to tell how to work with systemd-networkd.

How to switch on systemd-networkd

Many years the Top Linux Distribution use the shell scripts and ISC Tools for network configuration. systemd-networkd is a part of systemd and can replace many standart tools. systemd-networkd now support DHCP (client and server), networks with static IP, bridges, tunnels, VLANs, Wireless (with wpa_supplicant) and others.

For example, I try to migrate from default network configuration in Fedora (/etc/rc.d/init.d/network initscript) to systemd-networkd. This article also can be used for others Linux Distribution.

systemd-networkd use configuration files in directory /etc/systemd/network. Three types of configs are available:

  • .link – describe physical parameters of each interface: MAC, name, MTU and others
  • .network – network parameters: IP, route, gateway, DNS and others
  • .netdev – describe virtual interfaces like a bridge

We have a machine with most typical network configuration: two interfaces with static IPs in local network and WAN.

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 04:01:40:23:1f:01 brd ff:ff:ff:ff:ff:ff
    inet 188.166.46.238/18 brd 188.166.63.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 2a03:b0c0:2:d0::69:7001/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::601:40ff:fe23:1f01/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 04:01:40:23:1f:02 brd ff:ff:ff:ff:ff:ff
    inet 10.133.248.54/16 brd 10.133.255.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::601:40ff:fe23:1f02/64 scope link
       valid_lft forever preferred_lft forever

Now we need to get the default system settings (in Deb-based Distros – /etc/network/interfaces, please check the documentation for your OS)

# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE='eth0'
TYPE=Ethernet
BOOTPROTO=none
ONBOOT='yes'
HWADDR=04:01:40:23:1f:01
IPADDR=188.166.46.238
NETMASK=255.255.192.0
GATEWAY=188.166.0.1
NM_CONTROLLED='yes'
IPV6INIT=yes
IPV6ADDR=2A03:B0C0:0002:00D0:0000:0000:0069:7001/64
IPV6_DEFAULTGW=2A03:B0C0:0002:00D0:0000:0000:0000:0001
IPV6_AUTOCONF=no
DNS1=2001:4860:4860::8844
DNS2=2001:4860:4860::8888
DNS3=8.8.8.8

This configuration is used in the next sheme: eth1 interface – for local network (LAN), eth0 – for global internet (WAN). We need to create *.link files for every interface and configure the MAC Addresses and names:

cat /etc/systemd/network/90-external.link
[Match]
MACAddress=04:01:40:23:1f:01
[Link]
Name=eth-external
cat /etc/systemd/network/90-internal.link
[Match]
MACAddress=04:01:40:23:1f:02
[Link]
Name=eth-inner

Easy, right? We need also the two *.link configuration files with network settings:

cat eth-external.network
[Match]
Name= eth-outer
[Network]
DHCP=no
Adress=188.166.46.238/18
Adress=2A03:B0C0:0002:00D0:0000:0000:0000:0069:7001/64
Gateway=188.166.0.1
Gateway= 2A03:B0C0:0002:00D0:0000:0000:0000:0000:0001
DNS=2001:4860:4860:8844
DNS=2001:4860:4860:8888
DNS=8.8.8.8
cat eth-internal.network
[Match]
Name=eth-inner
[Network]
Address=10.133.248.54/16

Some additional information about [Match] section – in .link configs we associate a name of interface with MAC Address, in .network – with name of .link config. Configuration is complete and we can run systemd-networkd:

// disable Network Manager and legacy network services
# systemctl stop network
# systemctl stop NetworkManager

# systemctl start systemd-networkd

It works!

$ networkctl
IDX LINK             TYPE               OPERATIONAL SETUP
  1 lo               loopback           n/a         n/a
  2 eth-outer        ether              routable    configured
  3 eth-inner        ether              routable    configured

I also recommend to use systemd-resolved as resolver with DNS cache.

Others types of networks

DHCP

In this configuration we use IPv4 & IPv6; you can exclude IPv6 if you need.

/etc/systemd/network/wired-dhcp.network
[Match]
Name=en*

[Network]
DHCP=ipv4
DHCP=ipv6
Bridge

First we need a virtual interface:

/etc/systemd/network/bridge.netdev
[NetDev]
Name=br0
Kind=bridge
/etc/systemd/network/bridge.network
[Match]
Name=br0

[Network]
DHCP=ipv4

And setup the connected interface:

/etc/systemd/network/wired.network
[Match]
Name=en*

[Network]
Bridge=br0

Benchmarks

For systemd-networkd I use the network configuration with DHCP, Network manager and dnsmasq is disabled. Software:

  • Fedora Rawhide (24)
  • kernel-4.4.0-0.rc0.git2.1
  • systemd 207
  • ISC DHCP client daemon and dhclient-script 4.3.3
  • GCC 5.1.1
Part 1 – interface is down

systemd-networkd

# systemctl start systemd-networkd

# journalctl -u systemd-networkd.service
Nov 05 13:07:35 localhost systemd[1]: Started Network Service.
Nov 05 13:07:35 localhost systemd-networkd[4113]: enp2s0: Gained carrier
Nov 05 13:07:35 localhost systemd-networkd[4113]: enp2s0: Lost carrier
Nov 05 13:07:37 localhost systemd-networkd[4113]: enp2s0: Gained carrier
Nov 05 13:07:38 localhost systemd-networkd[4113]: enp2s0: DHCPv4 address 192.168.1.114/24 via 192.168.1.1
Nov 05 13:07:38 localhost systemd-networkd[4113]: enp2s0: Configured

~3s for systemd-networkd

ISC DHCP

# time dhclient -v enp2s0
Internet Systems Consortium DHCP Client 4.3.3
Copyright 2004-2015 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/enp2s0/94:de:80:1a:da:af
Sending on   LPF/enp2s0/94:de:80:1a:da:af
Sending on   Socket/fallback
Created duid \000\001\000\001\035\316YG\224\336\200\032\332\257.
DHCPDISCOVER on enp2s0 to 255.255.255.255 port 67 interval 5 (xid=0x64d33c3a)
DHCPDISCOVER on enp2s0 to 255.255.255.255 port 67 interval 12 (xid=0x64d33c3a)
DHCPREQUEST on enp2s0 to 255.255.255.255 port 67 (xid=0x64d33c3a)
DHCPOFFER from 192.168.1.1
DHCPACK from 192.168.1.1 (xid=0x64d33c3a)
bound to 192.168.1.115 -- renewal in 17670 seconds.

real        0m10.222s
user        0m0.049s
sys        0m0.318s

10.222s! Next try: 7.326s, 7.356s, 9.512s.

Part 2 – interface is up

systemd-networkd

# systemctl start systemd-networkd

# journalctl -u systemd-networkd.service
Nov 05 13:04:41 localhost systemd[1]: Starting Network Service...
Nov 05 13:04:41 localhost systemd-networkd[4085]: Enumeration completed
Nov 05 13:04:41 localhost systemd[1]: Started Network Service.
Nov 05 13:04:41 localhost systemd-networkd[4085]: enp2s0: DHCPv4 address 192.168.1.114/24 via 192.168.1.1
Nov 05 13:04:41 localhost systemd-networkd[4085]: enp2s0: Configured

Less than 1s.

ISC DHCP

# time dhclient -v enp2s0
Interface up - dhclient
sortium DHCP Client 4.3.3
Copyright 2004-2015 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/enp2s0/94:de:80:1a:da:af
Sending on   LPF/enp2s0/94:de:80:1a:da:af
Sending on   Socket/fallback
DHCPREQUEST on enp2s0 to 255.255.255.255 port 67 (xid=0x5b763f4d)
DHCPACK from 192.168.1.1 (xid=0x5b763f4d)
bound to 192.168.1.115 -- renewal in 20662 seconds.

real        0m2.243s
user        0m0.042s
sys        0m0.216s

Resume: systemd-networkd is really faster than ISC DHCP and really impressive in Part 2 – less than 1s is a very good result. In SSD era the 10s waiting is very strange when OS boot since 3-4s.

Read more:

Tor Messenger Review – Just Another Messenger or Light at The End of Privacy Tunnel?

Ubuntu, Fedora, Mint, Elementary and Others in One USB Flash – How To Create MultiUSB Drive

Programming with GNOME Builder: Review and Coding Practice

19 Years of KDE History: Step by Step