The Internet is down…. It was DNS, again

Photo by Adrian — Peacock skylight at Burgh Island, Devon

One of the common ways web services fail is problems with DNS. I’ve talked about how to harden systems against DNS related outages before but I don’t have my thoughts written down in one place. Today there was a large AWS outage related to DNS, so it seems timely to get this done.

the main errors we are seeing are Could not resolve host: dynamodb.us-east-1.amazonaws.com

The Domain Name System (DNS) is a simple lookup that translates the written name of a web service to it’s IP address. Something like aws.amazon.com to a routable internet address.

% nslookup aws.amazon.com
Server:  192.168.178.1
Address: 192.168.178.1#53

Non-authoritative answer:
aws.amazon.com canonical name = tp.8e49140c2-frontier.amazon.com.
tp.8e49140c2-frontier.amazon.com canonical name = dr49lng3n1n2s.cloudfront.net.
Name: dr49lng3n1n2s.cloudfront.net
Address: 3.166.65.11
Name: dr49lng3n1n2s.cloudfront.net
Address: 3.166.65.58
Name: dr49lng3n1n2s.cloudfront.net
Address: 3.166.65.78
Name: dr49lng3n1n2s.cloudfront.net
Address: 3.166.65.124

In this case looking up the address first went to my local Wifi router on port 53 and that looked it up for me via a DNS server and translated it into four of the AWS cloudfront service addresses. Four to provide redundancy in case one of them has an issue.

However when DNS breaks, there is no way to lookup the address, so the requests fail, and the service itself sees no traffic arriving. From the point of view of the people operating the service, it’s still up and running.

In this most recent failure it was the DNS endpoint for DynamoDB in the AWS us-east-1 region that wasn’t working. From the current status page:

Oct 20 2:01 AM PDT We have identified a potential root cause for error rates for the DynamoDB APIs in the US-EAST-1 Region. Based on our investigation, the issue appears to be related to DNS resolution of the DynamoDB API endpoint in US-EAST-1. We are working on multiple parallel paths to accelerate recovery. This issue also affects other AWS Services in the US-EAST-1 Region. Global services or features that rely on US-EAST-1 endpoints such as IAM updates and DynamoDB Global tables may also be experiencing issues. During this time, customers may be unable to create or update Support Cases.

The problem with this particular failure is that lots of internal AWS services use DynamoDB to store their information, so many additional services broke, as well as several customers. Amazon retail, Amazon Alexa, Lyft, Snapchat, Signal etc. Even customers that are not hosted in us-east-1 would be affected indirectly as they could not create support cases, or change configurations of Identity and Access Management (IAM).

There are several common causes of DNS outages, the most common is a configuration change that breaks the endpoint. It could occur during manual configuration, or as part of an automated update that didn’t work properly or didn’t propagate to all the places it was supposed to go. Although in today’s case it appears to just be a single endpoint, the entire DNS service itself can go down, there have been cases where DNS providers have failed. There are many companies that provide DNS services, and its usually where the domain is registered. The other cause of DNS outages is failure to maintain and renew the registration. There was a case where a SaaS company forgot to renew, and their entire service went offline for a few days while they tried to recover, with internal email and everything else down, the CEO was tweeting apologies to their customers.

So what can we do about it to survive DNS failures? They occur relatively infrequently so it’s not common to take the measures I’m going to describe, but I think it’s certainly possible to survive many DNS outages with careful planning.

1. Use more distinct domains. Don’t run your entire company on a single domain, for your email, your internal company services and your externally facing product. Easy if you think about it before you deploy your first customer facing product…
2. Support more than one DNS provider in your operations automation, and test your ability to migrate configurations between providers, and keep them synchronized. For example you could configure company.com and company.net to work interchangeably and register them at different providers, with different account expiration dates.
3. Support multiple domains in the code that looks up your services, via APIs or web pages, so that if the code fails to reach company.com it automatically tries company.net for example.
4. Don’t use DNS if you don’t really need to. The Netflix internal Ribbon client library would look up services and get the IP address from Eureka as well as the full URL. The IP address for a microservice will never change before its replaced, so it was called directly using the IP rather than the name, bypassing any possible DNS issues.

It’s difficult to keep DNS configurations automated and in sync across multiple providers. Many years ago when I was worrying about this problem at Netflix, I worked with Adrian Cole and others to build an open source Java library called Denominator that could maintain a DNS configuration and which had pluggable API support for several of the major DNS services and a simulator for testing. It’s fallen out of use nowadays, but something similar would be needed to automate DNS updates reliably. This still leaves the operational issues of changing configurations incorrectly, but by staging across more than one provider with a delay to verify the change, it could be a lot more reliable.

I hope people find these ideas useful as they figure out more resilient approaches to DNS.