Updated on April 15, 2017 in #ansible, #deployment, #docker, #lets-encrypt

Debugging Let's Encrypt Errors, Sometimes It's Not Your Fault

How DNS may cause temporary errors when trying to register an SSL certificate with Let's Encrypt, or anything really.

A few days ago I launched a new Docker course called Dive Into Docker and it’s hosted on its own domain name, which is diveintodocker.com.

Both this website and Dive Into Docker are static websites that have been generated with Jekyll. They are also fronted by nginx and are being hosted on DigitalOcean.

It would be a shame not to use SSL nowadays, especially with awesome services like Let’s Encrypt available. I have been using Let’s Encrypt for quite some time now on this site, and it’s been a great experience. Things are set up to be completely hands free. The SSL certificates renew automatically.

Naturally I wanted to use Let’s Encrypt again for diveintodocker.com.

# Deploying a New Site With Ansible

I have everything being deployed with Ansible, so adding a new site is very straight forward. For example, this is all I have to do to launch a completely new site on its own domain (or sub-domain):

nginx:
  nickjanetakis:
    # ...properties associated to this site.

  diveintodocker:
    domains: ['diveintodocker.com', 'www.diveintodocker.com']

letsencrypt_domains:
  - 'nickjanetakis.com'
  - 'www.nickjanetakis.com'
  - 'diveintodocker.com'
  - 'www.diveintodocker.com'

Then I just run a single Ansible command and about a minute later I have a new website deployed with fully automated SSL renewals, sweet!

I know for sure my Ansible roles for Let’s Encrypt and nginx are working well because I’ve used them for a number of things but alas, when Let’s Encrypt tried to generate a new certificate that included both domains, it threw this error:

Note: I replaced a few tokens with “foo” and “bar” for clarity.

Parsing account key...
Parsing CSR...
Registering account...
Registered!
Verifying www.nickjanetakis.com...
www.nickjanetakis.com verified!
Verifying www.diveintodocker.com...
www.diveintodocker.com verified!
Verifying diveintodocker.com...

ValueError: diveintodocker.com challenge did not pass:
{
    u'status': u'invalid',
    u'validationRecord': [
      {
        u'url': u'http://diveintodocker.com/.well-known/acme-challenge/foo',
        u'hostname': u'diveintodocker.com',
        u'addressUsed': u'',
        u'port': u'80',
        u'addressesResolved': []
      }
    ],
    u'keyAuthorization': u'foo',
    u'uri': u'https://acme-v01.api.letsencrypt.org/acme/challenge/foo/bar',
    u'token': u'foo',
    u'error': {
      u'status': 400,
      u'type': u'urn:acme:error:unknownHost',
      u'detail': u'No valid IP addresses found for diveintodocker.com'
    },
    u'type': u'http-01'
}

The line to notice there is No valid IP addresses found for diveintodocker.com. Well, that’s interesting because I have both nickjanetakis.com and diveintodocker.com being hosted on the same DigitalOcean droplet (same IP address). If no IP can be found, why does my main site work?

Of Course, DNS (Duh!)

At the domain registrar level, both domains have an A record that points to the same IP address but I had only added it to the diveintodocker.com domain a few hours prior to running the deploy script.

Now, I’m not completely crazy. Let’s Encrypt has pretty strict temporary restrictions for certain actions. For example if you try to renew the same certificate more than 5 times in 1 week, you will be temporary unable to issue new requests.

There are exceptions and things that increase that amount, but my point is, you don’t want to keep spamming their servers until it works. I also didn’t want to use their staging server because that doesn’t generate “real” certificates and it would have caused my main nickjanetakis.com domain to serve insecure certificates.

Sure, I could have separated them but I didn’t want to spend the energy on that because my thought process was, by the time I do that, things would likely work on their own.

# Verify Your DNS a Records in a Number of Ways

Let’s look at a few tools we can use:

curl

Usage: curl http://example.com

You can always curl your domain name and see if it reports back what it should, but that’s only going to show you the results of your view of the world when it comes to DNS. In other words, that’s going to use your DNS server, which may be drastically different than the rest of the world.

You might also not have a web server set up yet.

host

Usage: host example.com

Output:

example.com has address 93.184.216.34
example.com has IPv6 address 2606:2800:220:1:248:1893:25c8:1946

My DNS server includes example.com because it reported back an IP address that is associated to it (93.184.216.34). If my DNS server wasn’t up to date, I wouldn’t have seen the IP address line at all.

Also it’s worth noting that when I say “my DNS server”, I actually mean my ISP’s DNS server. In your case, it’s probably the same situation unless you configured your system to use a third party DNS server, such as Google’s (8.8.8.8).

Using host is good, but your view of the world cannot be fully trusted. In my case I was able to see my domain’s IP address but Let’s Encrypt’s certificate issuing server couldn’t.

Then I went AFK for an hour, and instead of trying again I spot checked the world’s DNS servers by running a slight variant to the above command.

Using a Custom DNS Server With Host

Usage: host example.com 8.8.8.8

Output:

Using domain server:
Name: 8.8.8.8
Address: 8.8.8.8#53
Aliases:

example.com has address 93.184.216.34
example.com has IPv6 address 2606:2800:220:1:248:1893:25c8:1946

That does the same thing as the first time we ran it, but this time it uses Google’s DNS server (8.8.8.8) instead of ours. That’s pretty handy because now you can start popping in DNS servers from all over of the place and check to see if your domain name is linked to the IP you specified in your A records.

Lucky for us a huge list exists at http://public-dns.info/.

You can also check a few very popular DNS servers:

You can spot check a dozen or so DNS servers from around the world and if all of them can see your domain <-> IP address, then chances are everyone can see your domain.

# It’s Not Always Your Fault

So to wrap things up. Sometimes errors aren’t your fault. Once in a while you need to wait for the rest of the world to catch up. Now you know how to test that case with host (dig would also work btw).

In this specific case, all I did was spot check a few DNS servers successfully and waited an extra hour. Then Let’s Encrypt happily created the certificate without me making any changes.

Have you ever been bit by DNS resolution issues? Let me know in the comments below.