Jeff Roberts
RHCE #804006066322833
Vim-Fu is now iPhone and Android friendly

Check out the Vim-Fu Store!

Vim-Fu

A Scalable DNS scheme for Amazon's EC2 Cloud

One of the fundamental issues to deal with while building out my company’s cloud deployment was the need to assign our own hostnames to our instances.  To accomplish this, we needed some sort of internal dns solution.  One complication we encountered was that the instances where located in different EC2 regions (EU, US and soon, a possible second US region).  Each Amazon region has it’s own private class A ip range, to which the instances are NAT’d, with public class B’s.  In essence, for a box in the US to talk to a box in the EU, it  would have to use the Public IP assigned by Amazon (and vice versa), but to talk to another box in it’s own region it would need the Private IP. We had to come up with a scheme to make this work.inflatable tent

First Attempt

Our first attempt utilized a single zone file which we quickly realized was going to be awkward.  The plan was to include internal and external IP’s in the same zone, differentiated by appending the extension “eu.ext” or “us.ext”.

For example:


prd.web01                  CNAME prd.web01.ee1a
prd.web01.ee1a                   IN A  10.x.x.x
prd.web01.ee1a.us.ext            IN A 172.x.x.x
prd.web02                  CNAME prd.web02.ez1a
prd.web02.ez1a                   IN A  10.x.x.x
prd.web02.ez1a.eu.ext            IN A 172.x.x.x

This meant that from any machine, you could get to another machine in the same region by using either the CNAME  as a shortname, or the full hostname, which included the Availability Zone.  To reach a machine that was not in the same region, you had to append “xx.ext” to the longname.  It did work.  You could get from any machine to any other machine regardless of where you were originating from DEPENDING on whether you applied the suffix, but I was never really satisfied with it.

Currently

We are gearing up for the “next phase” release of our EC2 deployment and I’ve taken the opportunity to re-think the scheme based on my experiences over the last 8 or so months of using EC2.  I have come up with something that is FAR better in my opinion.  It utilizes a separate zone file for each of the EC2 Regions and puts the logic to know which zone file to use on the instance itself.

The DNS Server

At OpenX, we keep records of all EC2 instances in a db so we can write scripts that do all sorts of cool things. For the new scheme, I wrote a perl script that pulls out the hostnames and ip’s and generates four include files named using abbreviations for the different regions, ee for us-east-1 and ez for eu-west-1.  These are arbitrary but I think this makes then names a little cleaner, you can use the actual region names if you like.  We also use this abbreviation in the hostnames to identify, at a glance, the physical location of the instance ie. ee1a would mean the ‘a’ data center in the us-east-1 region.

  • ee.private.include
  • ee.public.include
  • ez.private.include
  • ez.public.include

Next we construct 2 zone files called:

  • ee.domain.int
  • ez.domain.int

Associate the include with the zone files:

  • ee.domain.int

ee.private.include
ez.public.include

  • ez.domain.int

ee.public.include
ez.private.include

This means that the ee.domain.int zone file has records that point to private ip’s for any instance in the us-east-1 region and public ip’s for any instance in the eu-west-1 region.  Conversely, the ez.domain.int zone has public ip’s for the us-east-1 instances and private ip’s for eu-west-1 instances.

example:

ee.domain.int


prd.web01             CNAME    prd.web01.ee1a
prd.web01.ee1a                 IN A  10.x.x.x
prd.web02-eu          CNAME prd.web02-eu.ez1a
prd.web02-eu.ez1a              IN A 172.x.x.x

ez.domain.int


prd.web01             CNAME    prd.web01.ee1a
prd.web01.ee1a                 IN A 172.x.x.x
prd.web02-eu          CNAME prd.web02-eu.ez1a
prd.web02-eu.ez1a              IN A  10.x.x.x

The Instance

For the instance to query the correct zone file, we need to set the searchdomain and nameserver values in the /etc/reslov.conf file at startup based on the region in which the instance was started. You can use something like:

ec2-metadata -z

or pass the region to the instance at launch time using the user-metadata and retrieve it with

ec2-metadata -d

There are other methods to get this information from Amazon, like curl which is actually what ec2-metadata uses.  At OpenX, we use a custom deploy api that, among other things, appends the Availability Zone to the hostname  which makes for easy identification of the physical location of an instance just by looking at the hostname.  If you use this method, you can parse the hostname to determine which zone to query.

Whatever method you choose, write some logic that is executed during the startup sequence which uses the returned region value to re-write the /etc/resolv.conf file.  It should point to the proper zone file for the region the instance lives in and the local DNS server for that region.

/etc/resolv.conf


searchdomain ee.domain.int
nameserver a.a.a.a
nameserver b.b.b.b

– or –


searchdomain ez.domain.int
nameserver c.c.c.c
nameserver d.d.d.d

Where a.a.a.a and b.b.b.b are name servers in the us-east-1 region and c.c.c.c and d.d.d.d are in the eu-west-1 region.

So now, the instance has determined it’s region and set it’s /etc/resolv.conf to point to the correct zone file on the proper DNS server.  If it requests a machine in it’s own region, it will get a local address. If it queries a machine in another region, it will get a public address.

Scalability

The real beauty of this method, is it’s scalability.  If Amazon decides to add another Region or Regions, you simply create another zone file and two more include files, add the previous includes to the new zone file and the new includes to the existing zone files.  Your legacy instances will resolve the new regions from the same zone file it was using before only it now has the new region include file attached, so your new instances will know about the older Regions as well.

This might look something like:

Includes Generated

  • ee.private.include
  • ee.public.include
  • ez.private.include
  • ez.public.include
  • ew.private.include
  • ew.public.include

Zones

  • ee.domain.int
  • ez.domain.int
  • ew.domain.int

Zone Files

  • ee.domain.int

ee.private.include
ez.public.include
ew.public.include


prd.web01                CNAME prd.web01.ee1a
prd.web01.ee1a                 IN A  10.x.x.x
prd.web02                CNAME prd.web02.ez1a
prd.web02.ez1a                 IN A 172.x.x.x
prd.web03                CNAME prd.web03.ew1a
prd.web03.ew1a                 IN A 172.x.x.x
  • ez.domain.int

ee.public.include
ez.private.include
ew.public.include


prd.web01                CNAME prd.web01.ee1a
prd.web01.ee1a                 IN A 172.x.x.x
prd.web02                CNAME prd.web02.ez1a
prd.web02.ez1a                 IN A  10.x.x.x
prd.web03                CNAME prd.web03.ew1a
prd.web03.ew1a                 IN A 172.x.x.x
  • ew.domain.int

ee.public.iclude
ez.public.include
ew.private.include


prd.web01                CNAME prd.web01.ee1a
prd.web01.ee1a                 IN A 172.x.x.x
prd.web02                CNAME prd.web02.ez1a
prd.web02.ez1a                 IN A 172.x.x.x
prd.web03                CNAME prd.web03.ew1a
prd.web03.ew1a                 IN A  10.x.x.x

This will provide you with a very clean, scalable DNS scheme that is also very easy to work with. Enjoy!

UPDATE:

I recently added the below zone entries into the named.conf (on my DNS slaves) file to allow resolution of Amazon assigned public and private names:

zone “amazonaws.com” IN {
type forward;
forward only;
forwarders { EC2_DNS_SERVERS };
};

zone “amazonaws.com” IN {
type forward;
forward only;
forwarders { EC2_DNS_SERVERS };
};

zone “internal” IN {
type forward;
forward only;
forwarders { EC2_DNS_SERVERS };
};

You need to replace ‘EC2_DNS_SERVER’ with the ip(s) of an Amzon DNS server.  This came from the Amazon forum which, if you have not checked out, you definitely should.

BE AWARE: Any DNS queries coming from inside EC2 are returned PRIVATE ip’s by Amazon’s DNS servers no matter which address, public or private that you query.  It’s not a problem if you expect it, but it got me.  My script that builds my include files, got tricked and put private ip’s into the public include files.  This is why I put the above entries on the slaves and not on the DNS master where the include builder script runs.

9 comments to A Scalable DNS scheme for Amazon’s EC2 Cloud

Leave a Reply

 

 

 

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>