How the internet really works: IP transit, peering and provider tiers

The internet is made of many Internet Service Providers (ISPs), and each one has two ways to connect to the rest: IP transit, which means buying access from an upstream provider, and peering, a direct connection with another ISP. Most ISPs combine the two to balance cost and performance.

Peering

In a peering relationship, two ISPs exchange routes and traffic so that each can reach the other’s customers. They exchange only their customers’ prefixes, never the prefixes they receive from their own upstream providers.

The traffic itself is not charged, because neither side has to pay an upstream provider to reach the other’s customers. This is called settlement-free peering. The ISPs pay only for their port on the exchange fabric at a public peering point (an IXP) or, for private peering, share the cost of the circuit. Every bit exchanged this way is a bit less of paid transit.

IP transit

IP transit is a paid service that connects a customer’s network, or a downstream ISP, to the whole internet. The transit provider carries the customer’s traffic across its network to every reachable destination. Unlike peering, it is billed on bandwidth, usually with the 95th percentile method.

Transit is delivered over the Border Gateway Protocol (BGP), and customers usually run their own Autonomous System (AS). They can then receive the full internet routing table, which includes:

• the prefixes of the upstream provider’s other customers;
• the prefixes advertised by the upstream provider’s peers;
• the prefixes the upstream provider receives from its own upstream providers.

In the other direction, the transit provider advertises all its customers’ prefixes to its peers and its own upstream providers.

Tier-1, Tier-2 and Tier-3 providers

ISPs are usually described in three tiers.

Tier-1 providers form the backbone of the internet. They do not pay anyone for transit: they peer with every other Tier-1 provider for free, and charge lower-tier ISPs for using their networks.

Tier-2 providers have large networks and a broad, often global presence. They peer with each other to save on transit, but still pay Tier-1 providers to reach part of the internet.

Tier-3 providers are local or national. They usually buy transit from Tier-2 providers, which costs less than Tier-1, and may peer locally. They typically do not sell transit to other networks.

Network topologies

Single-homed

A single-homed customer has one connection to one upstream provider. It is the simplest and cheapest option, but if that connection or its router fails, the customer loses internet access entirely. The dispute between Cogent and Telia, which cut traffic between their networks, showed how exposed single-homed customers are. This is why many companies connect to more than one provider.

Multi-homed

A multi-homed customer has connections to more than one upstream provider. If one connection fails, the other takes over. With a single customer router, though, that router remains a single point of failure.

Connecting to two routers on the provider side protects against the failure of one of them, but the customer’s own router is still a single point of failure. Adding a second customer router removes it.

The ideal topology

The ideal is at least two upstream providers and at least two routers, with one provider per router. If a provider or a router has a problem, the other path carries the traffic and the connection stays up.

This setup is more complex and more expensive than the others, but it protects against the most common outages, and in my view the cost is worth it.

Going further

Choosing how to connect to the internet means understanding transit, peering and these topologies. BGP is a very solid protocol, but its complexity calls for careful, professional planning and execution. Done well, it gives a stable and reliable connection without wasting resources.

The North American Network Operators’ Group (NANOG) has a wealth of material if you want to learn more.