So last week at this time I had a hard, nasty thing happen to a client of mine: due to some classic incompetence at
Network Solutions, they vanished off the internet for about 20 hours. In order to understand exactly what happened, I need to delve a little bit into how domain name registration and the DNS (domain name system) works.
In this day and age, when you want to
register a domain name (say, www.jbaltz.com),
there is actually a two step process that goes on:
You register a domain name with a
registrar, like GoDaddy or
1and1 or Network
Solutions (10 years ago NetSol was the only game in town, but
that is another story.) and they verify that no one else has that
domain name, and they reserve it for you.
At the same time, they notify the
TLD name server for your TLD with a list of the authoritative name
servers for your newly-formed domain.
What? Wait? Come again? What’s
all that? Let’s define a few terms:
- A registrar is just the
organization that registers your name and enforces global
uniqueness—there can be no other “jbaltz.com”
sites out there but this one. It may also hold “whois”
information about the name of the responsible person or company are
behind a domain, but nowadays many registrars will allow you to
obscure your whois information to prevent onslaughts of UCE (spam).
- A TLD (Top Level
Domain) is the last part of your domain name: typically
“.com” or “.org” or such, or even a
country-specific domain like “.uk” (British sites like
www.amazon.co.uk) or “.il”
(Israeli sites, like www.huji.ac.il,
the Hebrew University in Jerusalem).
- An authoritative name server
is a site that agrees to answer questions of the type: “Where
do I find the IP (numerical) address of site www.example.com?”
and “Who receives mail for www.whoever.com?”
(It is at this point that discussions usually go into things like
“SOA” and “glue records” and most peoples’
eyes glaze over, but this is actually an important contribution to
the discussion.)
- The TLD name servers are a
group of systems that hold all the names in a
particular TLD, and a list of who the authoritative name servers
are.
To wit, for jbaltz.com, the
records that the .com TLD name servers hold is:
jbaltz.com. 172800 IN NS ns27.1and1.com.
jbaltz.com. 172800 IN NS ns28.1and1.com.
which means that the internet hosts “ns27.1and1.com” and
“ns28.1and1.com” will be able to answer the “who”
and “where” questions about jbaltz.com.
(The other numbers and codes are somewhat irrelevant to this
discussion, although they are important.)
(Digression: A long time ago, there was actually
semantic difference between “.com”, “.org”
and “.net”, but nowadays the difference appears to be
entirely nominal: people just scoop up the “.org” name or
the “.net” name if the “.com” name is taken.
There are a few TLDs that do maintain an entry-barrier other than
money: “.edu” requires that you actually prove to
them that you’re an educational institution, and I believe
“.museum” has a similar requirement. Also, I believe
other country-wide TLDs require proof of residency or something to
register a website with them, with notable exceptions being Tuvalu
“.tv” and Western Samoa “.ws” )
If you’re a typical website hosting with your provider (like
1and1, which is the hosting provider for this site), your hosting
provider may act as your registrar (holding your name in the global
namespace of .com and telling the TLD nameservers who is the
nameserver for your domain) and act as the authoritative name
server for the domain, but they do not have to do so.
jbaltz.com is registered through MelbourneIT (neé
www.registerfree.com) but
has its domain name service provided through 1and1. Many many other
sites do that.
My client’s site was one of them.
He had registered his site through Network Solutions, but another
site (his hosting provider) was the authoritative DNS for his domain.
He was moving from one hosting provider to another, and in the
interim it made sense to make Network Solutions his authoritative
DNS, right? I mean, they already have his registration, and they have
an easy web-based interface to set up the DNS entries that were
needed. It seemed like the easiest way to have a smooth transition
from one place to another.
Now, Network Solutions, oddly enough, does not make moving back
to them for name service easy. You cannot set up all your various and
sundry domain names (www.this.com,
www2.this.com, mail directions) beforehand and then tell them “OK,
we want NetSol to be the authoritative DNS for us, in addition to
being our registrar.” Instead, you have to do it in two steps:
- Move your DNS back to NetSol
- Set up your DNS and all its addresses in high-speed.
Going on behind the scenes several
things are going on: NetSol is setting up its own servers to be
equipped to answer questions about the new domain, and NetSol is
informing the TLD nameservers that it is going to be authoritative
for the new domain. The former process is generally pretty quick, and
the latter process can be time-consuming. (You are typically told
that it takes 24-48 hours, although in reality 6 hours is about how
fast it works for .com.)
What has happened now? We moved the
DNS back and NetSol did the following: it notified the TLD
nameservers that it was now authoritative, but it did not
actually configure its own name servers to answer questions!
I think you can see where this is
headed.
Now, after the move, it turns out the
TLD nameservers were updated, mirabile dictu, almost
immediately. NetSol’s own nameservers, however, were not
updated. Which means the following things happened:
A user out on The Vast Internet tried to find
“www.jerrysclient.com”
The user’s ISP’s nameserver asked the global
nameserver who was responsible for www.jerrysclient.com.
The global TLD nameserver replied: “NetSol is”
;; ANSWER SECTION:
jerrysclient.com. 3699 IN NS NS15.WORLDNIC.com.
jerrysclient.com. 3699 IN NS NS16.WORLDNIC.com.
- NetSol, of course, denied knowing
anything about this domain, and said, in return, “go ask the
root”.
- The root said “go ask
NetSol”, and we get a nice little infinite
loop.
- Eventually, the name query would time out, and no one could
find my client’s site, and poof
they have vanished off the internet!
Calling up Network Solutions technical
support (“For a painful experience, press 1. To be on
interminable wait, press 2”—I’m sure that Scott
Adams had this in mind when coming up with Dogbert’s
tech support.) was less than useful: they tried at great length
to convince me that I simply had to wait for this information to
propagate through the internet. I replied that it, indeed, had
propagated, and the ball was now in Network Solutions’s court,
and could I pretty please speak to someone in their DNS
services group (I thought about posting something inquisitive to
NANOG but decided later that it
would be more efficacious to just wait.) and of course, I was told, I
could not, but that he could enter a ticket for me, and the problem,
being NetSol’s, should “clear up in 2-3 hours, tops”.
The president of the client firm spent several fruitless hours,
getting escalated up a never-ending chain of bureaucrats until he
finally just got fed up. After about 20 hours, NetSol finally got
their act together, and the site finally came “back to Earth”.
And
there was much rejoicing.
