How the Internet Works

I have worked for on a number of enterprises’ internet perimeters and there is a number of different ways these have been handled. There is lots of good practise out there, but I see more and more bad practice. So, I would like to layout how internet routing works compared to enterprise routing and how these differences effect how you might operate your Autonomous System (AS).

The Basics

The internet is a collection of thousands or separate AS that connect together and share paths between each other. This then allows the creation of a path from one host to another host that allows them to communicate with each other. An AS is a single administration unit that advertises its networks within its domain to other AS’s. These domains connect to each other and share routes using as AS to AS routing protocol called Border Gateway Protocol (BGP). Each AS has its own polies and advertises the routes it learns to other connected AS’s dependant on those polices. Each domain must be a single continuous network where everything inside of that domain can be reached independent of where the traffic arrives from. As each system has its own policies that might be very different from your own there is no guarantee that they will route traffic back to you on the same path you routed the traffic to them. Asymmetric routing is common on the internet and unlike an enterprise network where you can fix asymmetric routing, you do not have control to fix it on the internet. There are ways to influence this and get more control discussed further down.

Autonomous Systems

This is a single independent network that is operated by an organisation and most importantly is connected together. I have seen organisation running two separate networks with a single AS number advertising two different network out of either one with no connection between them. In theory this should be alright, but they see odd behaviour sometime. As internet traffic should follow the shortest prefix in a routing table and not a path to an AS. I know that Internet Carriers do have AS filters and will assume an AS is contiguous then this should cause issues. Remember all AS’s have their own policies and may choose to route traffic using these policies, like preferring a longer route to use a provider they prefer. If I learn a route from a customer or a peer then I might prefer that route over a better transit route which might cost more to use. So if you connect to the internet in two places it is not uncommon for you to prefer to send the traffic out one link and for the recipient to prefer your other link, so you get asymmetric traffic flow, you need to allow for this that’s why your AS need to be connected. I would recommend that the AS is connected together with a routing layer to allow the traffic to flow as it wishes and then within your AS you can then direct the traffic as you require. This also shows the internet less about how your systems/services are setup and the less you show the internet the better.

Types of Autonomous System

There are different types of AS, as in the way they function and what they are trying to achieve. It also depends on what you are doing how they look to you. This is a very relative thing, I would classify them like this, but it’s a bit of a carrier view of the internet:

Customer AS	An end user system, a combination of internet users and content hosting, basically an enterprise organisation. The main factor is that they have to buy all their internet access from one or more Internet Service Providers (ISP) via remote connections to the ISP. They advertise their own networks to the ISP and learn a default route or the internet routing table from the ISP(s).
Internet Service Provider AS	Sell internet access to Customer AS’s and local end users. Normally covers a geographic area, a country or region. They will peer with other ISPs and content providers but also buys transit from transit providers to get full internet access. Normally have a presence in one or more Co-Location Centre where there is an Internet Exchange Point(IXP) and a member of at least one peering network. They advertise their customer, peer and transit routes to their customers and advertise their routes and their customer routes to the transits and learn the rest of the internet routing table form their transits.
Transit AS	These are normally larger global networks with equipment in multiple Co-Location Centres and connect to many Internet Exchanges across multiple countries and regions. They will be members of multiple peering networks. Thay will have their own networks to carry data between these locations. They advertise all their routes to their customers and advertise their routes and their customer routes to their peer who are the other transit providers.

Where the complexity lies is that no AS actually fits in each of the categories perfectly. Big content providers should be Customer AS’s, but they partner with ISPs and Transits to get their content closer to the users. Content is normally king in this game. Some customers peer with other networks that are important to them, and some may have presence in an IXs. Basically, there are no hard as fast rules to any of this. Also, some AS’s have been treaded differently due to historic reasons. It’s in the interests of Teir1 Carriers to prevent new carriers joining their club.

It important to understand the types of routes

Your Routes	These are the networks allocated to you. Best practice is to advertise your entire allocated netblock in the most concise way you can into BGP and then out to the internet. On your external BGP routers there should network summary statements expressing your entire netblock and an associated null route that matches that net block to keep it active. This says to the internet these are my networks and send all the traffic bound for them to me. The null route means that any traffic sent to any of the space that you are not using and is not routed elsewhere will then be dropped with minimal processing from your router. There should then be more specific routes to your services for the addresses you are using ideally host routes or small bocks to other location. The less you can show to the internet the better. The more stable your routes are the better the transits will treat them. Flapping routes get suppressed. This will also reduce your attack surface and hopefully reduce the number of attacks on your network. If you do want to influence where traffic come back to, then advertising subnet can be used to do this, but remember all subnet MUST be reachable from all of your internet connection or traffic can be lost. This is best done with a contiguous routed internet facing layer with no firewalls or other devices that look for symmetric traffic flows. These routes are yours, so you advertise them to everyone you peer to as you are the source of these networks.
Customer Routes	The customer needs to treat their routes as above. You will advertise all of the routes you learn, the whole internet routing table or the default route. They will advertise all of their routes to you.
Peer Routes	These are route you learn at peering points like Internet Exchanges Points or Co-Location Centres. You would peer with other companies you do business with, or you buy applications from like Software as a Service (SaaS) providers, or some other mutual benefit. This is a way to start controlling and troubleshooting your internet traffic. If you peer with another AS, you will have a peering agreement and a contact for they Network Operation Centre (NOC) so if you have any issues, you have a contact and a contract and a Service Level Agreement (SLA) to get it fixed. Normally you would advertise all your own routes and your customer’s routes to a peer, and they will do the same to you. If you advertise to them your transit routes they might use, you as a free transit. Like wise if you advertise your peer routes to your transit, they might sent you their traffic again free transit.
Transit Routes	Thses are the routes to the rest of the internet. Normally you will have to pay per Megabit to sent and receive traffic to your transit. If you have more than one transit provider, then you might want the full routing table from each or just the default route. It is worth considering getting the full internet routing table and the default route. Then you could apply an AS filter to the routes you learn. If you allow anything that is one AS hop away, these will be your providers routes and if you allow routes that is two AS hops away, these your providers direct peers, then if there is anything further away then you don’t really care. Same with networks using AS prepend. This way if your destination is in one of your providers networks or is a customer of one of the providers then you will send the traffic directly to the best provider. You can then not have the overhead of holding the whole internet routing table on your internet routers.

Summary

There can be a lot to this, but the best thing is to try and keep things simple. Try to show as little as possible outside your AS. As you become more and more reliant on the internet for doing your business then the more control and reliability you can add the better. As you organisation gets bigger consider peering directly with your Cloud Computing either directly or through your ISP.