Introducing 'innernet'

For the past year, all of the infrastructure at tonari — our installations, our laptops, our tools — have run on a single WireGuard network that's organized by an opinionated network manager we've been writing called innernet. Today we're happy to be open sourcing it.

In the beginning, we had a shared manually-edited WireGuard config file and many sighs were heard whenever we needed to add a new peer to the network.

In the middle ages, there were bash scripts and a weird Vault backend with questionable-at-best maintainability that got new machines on the network and coordinated things like IP allocation. Many groans could be heard whenever these flimsy scripts broke for any reason.

In the end, we decided to sit down, sigh one long and hopefully final time, and write innernet.

Think of innernet as an opinionated configuration system on top of WireGuard that comes with some added features to make life easy, and is friendly with various sizes of networks: one for your organization, one for your project, one for your social circle to create an idealistic alternate internet universe — your imagination's the limit.

We had some simple goals:

• Conveniences a typical WireGuard user wants: peer names, auto-updating peer lists, groups based on IP blocks, and automatic NAT holepunching.
• Free, open source, and made to be self-hosted. We think it's especially important for such a vital and low-level piece of our infrastructure to not be dependent on the livelihood of a company one has no control over.
• Straightforward architecture — no Raft consensus here. It's a simple SQLite server/client model.

Below, we'll explain how it works, then build a quick example network. The source is available as of today, so tinker away. We're pretty much a pure Rust house at this point, and this project is no exception!

The basics

innernet establishes three main primitives in defining your network:

• Peers: machines on the network,
• CIDRs: peer groups based on IP blocks, and
• Associations: connections between CIDRs for access control.

Peers: In this land, you are your IP

In a innernet network, peers are added or disabled by the creator of the network, or by other peers with administrative capabilities. Peers also always have only one assigned IP address, and that address is permanently associated with them. That is, a peer's IP address on the innernet network is permanent, unique, and immutable.

Unlike other network interfaces, in the WireGuard world, traffic from peers on your WireGuard interface are cryptographically authenticated and can be used to guarantee that traffic from a specific source IP is indeed from that peer.

However, it's important to note that this property alone does not make an IP alone a solid authentication method — there are a lot of critical security caveats in a world where various applications on your computer can make arbitrary network requests, and arbitrary source IPs can be forged via other network interfaces. We'll be working towards creating a set of recommendations for running services inside an innernet to document these caveats.

CIDRs: In an orchard of ripe IPs, make a CIDR

CIDRs are the "groups" in innernet. A CIDR can either have other CIDRs as children, or peers. Every peer belongs to a CIDR.

CIDR notation is very useful because it makes it much easier to create a tree of these subnet groups that don't have overlapping addresses.

For example, here is the rough CIDR layout of tonari's innernet:

10.80.0.0/15 tonari
    10.80.1.0/24 infrastructure
    10.80.64.0/18 humans
        10.80.64.0/23 engineering
        10.80.66.0/23 non-engineering
		10.80.128.0/20 tooling
    10.81.0.0/16 installations
        10.81.0.0/24 straylight
        10.81.1.0/24 frontier
        10.81.2.0/24 mars

Why is this powerful? Well, what if you want a web service that gives access to everyone in your organization? Allow traffic from 10.80.64.0/18 using your web server or your OS's firewall. Want to give only engineers access to the administrative panel? Check if their IP is in the 10.80.64.0/23 range - if it is, they're an engineer.

Associations: who can talk to who

By default, peers can only see other peers in their same CIDR, as well as the server peer on its "infrastructure" CIDR.

To allow peers in one CIDR to communicate with peers in another, you can create associations between CIDRs. For example, we would create an association between the "engineering" CIDR and the "installations" CIDR in order to give engineers the ability to reach all the tonari installations. From there, of course, we can use tools like iptables on the peers to control the security in a more granular way.

So, with this concept, we have a fairly straightforward method of access control.

Getting started

Let's say we are starting a new company called Kermpany. We sell educational cakes and have raised one billion dollars in venture capital and now it's time to get serious. And to get serious, we'll need an innernet.

Innernet consists of two binaries: innernet-server, the coordinating server, and innernet (also aliased as inn for easy access), the client that peers run to access the network.

First, let's install innernet-server on a server we have sitting around, create a new network, and start it up:

sudo innernet-server new
# Network name: kermpany
# Network CIDR: 10.42.0.0/16
# External endpoint: [Enter]
# Listen port: 51820

sudo systemctl enable --now innernet-server@kermpany

Our network CIDR is 10.42.0.0/16, giving us 64k IP addresses to play with! It can be much bigger than this of course, and it can be IPv6 as well.

Next, let's add a CIDR for all the lovely humans we work with.

sudo innernet-server add-cidr kermpany
# Parent CIDR: kermpany
# Name: humans
# CIDR: 10.42.128.0/17
# Create CIDR "humans"? yes

We could now make further sub-CIDRs within humans like bakers or bosses, but we won't do that here. Here at Kermpany, every human is a baker boss in our eyes.

Instead, we'll get straight to creating our first peer, Ryo.

sudo innernet-server add-peer kermpany
# CIDR: humans (10.42.128.0/17)
# IP: [Enter]
# Name: ryo
# Make ryo an admin? yes
# Create peer ryo? yes

Then we can send the generated invite, ryo.toml, to Ryo using a convenient tool like Magic Wormhole. Invites are redeemed by the first person who uses it, so make sure the invite is only sent to the person you intend to join the network!

Now, Ryo has received their invite and can redeem it on their computer to join the network:

sudo inn install ./ryo.toml
# Interface name: [Enter]

sudo systemctl enable --now innernet@kermpany

Since they're an admin, Ryo can now invite peers with sudo inn add-peer kermpany!

All peers within the same CIDR can see each other, and Ryo could make another CIDR and associate it with "humans" to make them visible to each other with:

sudo inn add-association kermpany

Other peers can connect with Ryo by either using his IP or ryo.kermpany.wg.

And like that, the network was born. Kermpany ended up adding a ton more CIDRs and peers and became a huge success, baking the worlds most educational cakes ever.

Security, though

As a security-minded reader you are, by this point, inner-screaming "well that server now looks like a pretty nice target to compromise." In order to appease you, dear reader, we have decided to try to make the server as unattractive to attackers as possible without losing too much simplicity.

1. The server's only internet-exposed port is its WireGuard listening port. The juicy HTTP API only listens on its internal IP address. Thus, the attacker would need to 1) find a way on to the WireGuard network by compromising an existing peer, or 2) find a vulnerability with WireGuard itself.
2. Peers cache and pin the public key-IP pairs of other peers as they are seen, since (IP, public key) tuples for peers are defined to be unique and immutable on innernet. Thus, if a compromised server swapped out public keys for a peer, existing peers won't take kindly to it.
3. New peers join the network via invitations that contain a temporary WireGuard keypair generated for them by a peer with admin rights. This keypair gives them the ability to communicate with the server API, and invitees are then required to submit a new static keypair's public key to redeem the invite. Thus, the server does not know any of the private keys of peers.
4. The server only shows a given peer the connection information of peers that are in the same CIDR or associated CIDRs. Thus, a compromised peer would not be able to attack the entire network unless they compromised the server or an admin peer first.

To us, that felt like a reasonable amount of thus's to justify our architectural choices, given our specific threat model. We're looking forward to hearing how others intend to use innernet and how we can adapt it to suit a wider range of situations.

Innernet compared to...

Tailscale

Tailscale is much more polished, VC-funded, and requires talking to a closed-source backend that you can't run yourself.

Currently, all connected peers are assigned IPs somewhere in the 100.64.0.0/10 range, and ACL is based off of assignable tags rather than IP ranges, so special awareness of Tailscale features would need to be baked into your applications if you needed more granular awareness for access control or identity.

It uses WireGuard as its underlying VPN protocol (with a packaged userspace implementation, not the kernel module, though), which is great, and we love the usability work they're doing too — we just don't want to be dependent on an external proprietary service for our internal networking.

Nebula

Nebula is Slack's (open-source!) answer to this problem, and it uses a userspace client that implement a custom tunnel protocol based on Noise (which WireGuard is also based on). Implementing the tunnels in userspace adds a speed and latency hit compared to the WireGuard kernel module.

Nebula's ACL system is based off of tags, similar to Tailscale. Because of that, it requires much more firewall/security-type code to exist in their codebase and configuration files, rather than utilizing existing controls for your OS's networking stack.

Hey, thanks

Thanks for reading along! We hope you had a good time here and have a great time on the innernet. Please feel free to say hey or even just hi at hey@tonari.no — we've loved your feedback from previous posts so I don't even feel afraid this time to post this email address again.

Note

This project is not affiliated with the WireGuard project. WireGuard is a registered trademark of Jason A. Donenfeld.

Shout out to cbonsai for the beautiful bonsai cherry blossoms in the cover :).

Errata

1. An earlier revision of this blog post claimed that innernet IP addresses can be used in lieu of other authentication methods in internal services, since IP addresses over WireGuard interfaces are cryptographically authenticated. This is flat-out a bad idea without additional care and authentication, and I'd like to apologize for making that incorrect claim and thank those that brought up the issue up so kindly.
2. It turns out that Tailscale doesn't in fact take advantage of the WireGuard kernel module even if it exists, and always uses a userspace WireGuard implementation.