Entropy Reducers Amalgamated

So last week at this time I had a hard, nasty thing happen to a client of mine: due to some classic incompetence at Network Solutions, they vanished off the internet for about 20 hours. In order to understand exactly what happened, I need to delve a little bit into how domain name registration and the DNS (domain name system) works.

In this day and age, when you want to register a domain name (say, www.jbaltz.com), there is actually a two step process that goes on:

1. You register a domain name with a registrar, like GoDaddy or 1and1 or Network Solutions (10 years ago NetSol was the only game in town, but that is another story.) and they verify that no one else has that domain name, and they reserve it for you.

2. At the same time, they notify the TLD name server for your TLD with a list of the authoritative name servers for your newly-formed domain.

What? Wait? Come again? What’s all that? Let’s define a few terms:

• A registrar is just the organization that registers your name and enforces global uniqueness—there can be no other “jbaltz.com” sites out there but this one. It may also hold “whois” information about the name of the responsible person or company are behind a domain, but nowadays many registrars will allow you to obscure your whois information to prevent onslaughts of UCE (spam).
• A TLD (Top Level Domain) is the last part of your domain name: typically “.com” or “.org” or such, or even a country-specific domain like “.uk” (British sites like www.amazon.co.uk) or “.il” (Israeli sites, like www.huji.ac.il, the Hebrew University in Jerusalem).
• An authoritative name server is a site that agrees to answer questions of the type: “Where do I find the IP (numerical) address of site www.example.com?” and “Who receives mail for www.whoever.com?” (It is at this point that discussions usually go into things like “SOA” and “glue records” and most peoples’ eyes glaze over, but this is actually an important contribution to the discussion.)
• The TLD name servers are a group of systems that hold all the names in a particular TLD, and a list of who the authoritative name servers are.

To wit, for jbaltz.com, the records that the .com TLD name servers hold is:

jbaltz.com. 172800 IN NS ns27.1and1.com. 
jbaltz.com. 172800 IN NS ns28.1and1.com. 

which means that the internet hosts “ns27.1and1.com” and “ns28.1and1.com” will be able to answer the “who” and “where” questions about jbaltz.com. (The other numbers and codes are somewhat irrelevant to this discussion, although they are important.)

(Digression: A long time ago, there was actually semantic difference between “.com”, “.org” and “.net”, but nowadays the difference appears to be entirely nominal: people just scoop up the “.org” name or the “.net” name if the “.com” name is taken. There are a few TLDs that do maintain an entry-barrier other than money: “.edu” requires that you actually prove to them that you’re an educational institution, and I believe “.museum” has a similar requirement. Also, I believe other country-wide TLDs require proof of residency or something to register a website with them, with notable exceptions being Tuvalu “.tv” and Western Samoa “.ws” )

If you’re a typical website hosting with your provider (like 1and1, which is the hosting provider for this site), your hosting provider may act as your registrar (holding your name in the global namespace of .com and telling the TLD nameservers who is the nameserver for your domain) and act as the authoritative name server for the domain, but they do not have to do so. jbaltz.com is registered through MelbourneIT (neé www.registerfree.com) but has its domain name service provided through 1and1. Many many other sites do that.

My client’s site was one of them.

He had registered his site through Network Solutions, but another site (his hosting provider) was the authoritative DNS for his domain. He was moving from one hosting provider to another, and in the interim it made sense to make Network Solutions his authoritative DNS, right? I mean, they already have his registration, and they have an easy web-based interface to set up the DNS entries that were needed. It seemed like the easiest way to have a smooth transition from one place to another.

Now, Network Solutions, oddly enough, does not make moving back to them for name service easy. You cannot set up all your various and sundry domain names (www.this.com, www2.this.com, mail directions) beforehand and then tell them “OK, we want NetSol to be the authoritative DNS for us, in addition to being our registrar.” Instead, you have to do it in two steps:

1. Move your DNS back to NetSol
2. Set up your DNS and all its addresses in high-speed.

Going on behind the scenes several things are going on: NetSol is setting up its own servers to be equipped to answer questions about the new domain, and NetSol is informing the TLD nameservers that it is going to be authoritative for the new domain. The former process is generally pretty quick, and the latter process can be time-consuming. (You are typically told that it takes 24-48 hours, although in reality 6 hours is about how fast it works for .com.)

What has happened now? We moved the DNS back and NetSol did the following: it notified the TLD nameservers that it was now authoritative, but it did not actually configure its own name servers to answer questions!

I think you can see where this is headed.

Now, after the move, it turns out the TLD nameservers were updated, mirabile dictu, almost immediately. NetSol’s own nameservers, however, were not updated. Which means the following things happened:

• A user out on The Vast Internet tried to find “www.jerrysclient.com”

• The user’s ISP’s nameserver asked the global nameserver who was responsible for www.jerrysclient.com. The global TLD nameserver replied: “NetSol is”

;; ANSWER SECTION:
jerrysclient.com.  3699    IN      NS      NS15.WORLDNIC.com.
jerrysclient.com.  3699    IN      NS      NS16.WORLDNIC.com.

• NetSol, of course, denied knowing anything about this domain, and said, in return, “go ask the root”.
• The root said “go ask NetSol”, and we get a nice little infinite loop.
• Eventually, the name query would time out, and no one could find my client’s site, and poof they have vanished off the internet!

Calling up Network Solutions technical support (“For a painful experience, press 1. To be on interminable wait, press 2”—I’m sure that Scott Adams had this in mind when coming up with Dogbert’s tech support.) was less than useful: they tried at great length to convince me that I simply had to wait for this information to propagate through the internet. I replied that it, indeed, had propagated, and the ball was now in Network Solutions’s court, and could I pretty please speak to someone in their DNS services group (I thought about posting something inquisitive to NANOG but decided later that it would be more efficacious to just wait.) and of course, I was told, I could not, but that he could enter a ticket for me, and the problem, being NetSol’s, should “clear up in 2-3 hours, tops”. The president of the client firm spent several fruitless hours, getting escalated up a never-ending chain of bureaucrats until he finally just got fed up. After about 20 hours, NetSol finally got their act together, and the site finally came “back to Earth”.

And there was much rejoicing.