Encrypt the InTeRnEt!

This post is all about a very nice thing that revolutionized encryption on the internet. In 2015, the famous organisation Let’s Encrypt made the decision to throw out TLS certificates for free to everyone who needed them. In September 2015, the first certificate was issued, and in May 2018, Let’s Encrypt took the next step, and started supporting wildcard certificates.

Let’s talk about getting certificates from Let’s Encrypt.

Why this is important for us

Back in January last year we had the dream to support Let’s Encrypt as a standard certificate for our customers on our WhiteLabel platform. We created our first system to issue certificates for shop domains using Let’s Encrypt’s certbot script, http-01 challenge, and some templating magic to create configurations for our loadbalancer.

Technical summary

First off a word about HTTP-01 challenge defined in the ACME protocol. In general, these challenges are designed to check if the requesting instance is valid for creating a certificate for the requested domain. The HTTP-01 challenge assumes that the webroot of the webserver a domain points to is under your control. You have to put a challenge into your webroot folder <path/to/webroot>/.well-known/acme-challenges/... to prove you are trustworthy to get a certificate for your domain. Let’s Encrypt checks this challenge using the domain, the standard path .well-known/acme-challenges/, as well as the generated challenge to validate your request.

But there were some steps to go to automate the whole issuing process for a shared platform running in a big cluster with lots of loadbalancing in the background. Our scenario is a bit different from the standard case. But sometimes pictures say more than words:

First we needed a list of domains we wanted to issue. This list comes from our ePages software, and includes all domains with their certification status.

Our Network LoadBalancer (NLB) (the orange thing) is configured to proxy requests going to /.well-known, and then to a system where our certbot puts the challenges in place (the blue service system). Now we were able to issue certificates for any shopdomain on the platform, because every such domain points now to our loadbalancer IP address (if not … nobody can buy something from the ePages online shops 😮). Now you may ask why not just do the whole Let’s Encrypt stuff on the loadbalancer. Yes, you can do that. But our loadbalancers are rented from a service provider, so we don’t have this option.

Next, we have to create nginx configuration files from a template using the new certificate, since the loadbalancer using Nginx terminates the SSL/TLS. The templates often look similar to this:

server {
        server_name $domain_www $domain;
        client_max_body_size 0;
        listen $ip:443;
        ssl on;
        ssl_certificate     /path/to/letsencrypt/fullchain.pem;
        ssl_certificate_key /path/to/letsencrypt/privkey.pem;
        include <headers, defaults, params etc.>;
        location / {
                proxy_pass ...;
                proxy_set_header ...;
                proxy_hide_header ...;
        }
}

With these templates you can do basically all the magic you want to do for your software stack, such as proxy stuff, setting additional headers or including special configurations. We have several templates designed for our use cases depending on what a shop our customers have.

The certificate and configurations get rsynced to the NLB. Before the configurations go into production, everything gets tested and integrated after having checked successfully that nothing is broken. Here is a simplified picture that shows what happens:

Now we are able to automatically create all certificates for our customers’ online shops. This was a huge step forward, because our new ePages Now product was also designed by using protected internet communication, and therefore requires a valid certificate by default.

Into the wild…card

In May 2018, Let’s Encrypt started supporting wildcard domains using DNS-01 challenge and we thought … NICE! As an internet company, we need many wildcard certificates, especially for all newly created shops. These shops perhaps don’t have an own domain at the beginning. They get a standard domain that looks like shopalias.some.domain, where some.domain is one of our domains. In the past we bought all these *.some.domain certificates. But now we can automate the issuing process for wildcards. There is also no longer the need to get your hands dirty by changing old expired certificates. Here is how it works!

Technical summary

Using DNS-01 challenge of the ACME protocol requires you to create, update, (and delete) TXT records for your domain zone. For example, if you want a wildcard certificate for *.example.com, you have to create a specific subdomain with a TXT record. This subdomain for our example would be _acme-challenge.example.com, and the TXT record would contain a validation string created within the issuing process. (Btw, it doesn’t matter if there are “some” more TXT records. Let’s Encrypt loops over all of them to check … at least until bad things happen). You can also issue normal certificates this way.

Here we faced our first challenge:

We have many such domain zones we control for our wildcard domains, but there’s no neat way to automate the TXT creation process. This happens if your domain control is just a webinterface, and no API is implemented. (Creating creapy curl -X POST requests to fake a user using the webinterface to create TXT records was not really an option …, but anyway we did that as a first try 😁).

If your setup is simpler by using some kind of domain provider having a standard API implementation for Let’s Encrypt, you don’t have to do such workaround, and should be able to create the TXT record directly.

To overcome our TXT creation problem there is a common practice called CNAMES that is supported by the ACME protocol. Let’s assume we want a wildcard certificate for *.ourfancy.domain. We create the _acme-challenge.ourfancy.domain, and CNAME it to _acme-challeenge-cname.some.other.domain where we have better options to manage TXT records. In our case the some.other.domain zone is managed by a DNS server that we have under our control.

We CNAME the _acme-challenge domains to domains managed by our DNS server, and create TXT records using nsupdate magic (we are using BIND here for the DNS part). All you have to do is to set up one initial CNAME, and the rest is done by bash magic.

This is how to these TXT records using nsupdate:

#!/bin/bash

DNS_SERVER="dns.some.other.domain"
DYNZONE="some.other.domain"
HOST="_acme-challenge-cname"

if [ -z "$CERTBOT_VALIDATION" ]
then
echo "EMPTY VALIDATION"
exit -1
fi

/usr/bin/nsupdate -k /path/to/some/key.private << EOM
server ${DNS_SERVER}
zone ${DYNZONE}
update add ${HOST}.${DYNZONE} 300 TXT "${CERTBOT_VALIDATION}"
send
EOM
# give DNS some time to update
sleep 60

This script is used by certbot itself using the --manual-auth-hook option, which sets the $CERTBOT_VALIDATION variable. Also it makes sense to clean up the records after using them. This can be done using the --manual-cleanup-hook option. We use the following script:

#!/bin/bash

DNS_SERVER="dns.some.other.domain"
DYNZONE="some.other.domain"
HOST="_acme-challenge-cname"

/usr/bin/nsupdate -k /path/to/some/key.private << EOM
server ${DNS_SERVER}
zone ${DYNZONE}
update delete ${HOST}.${DYNZONE}
update add ${HOST}.${DYNZONE} 300 TXT ".empty"
send

Setting an .empty TXT record here prevents some caching problems.

To use nsupdate on a BIND DNS server, you need to add a key and an update-policy. This may look like this:

zone "some.other.domain" {
        type master;
        file "/etc/bind/db.some.other.domain";
        ...
        update-policy { grant my.nsupdate.key zonesub TXT; };
}

...

key "my.nsupdate.key" {
        algorithm hmac-sha512
        secret "<some weird characters come up here>"
}

Important notice: For this setup you only have to setup the CNAME from _acme-challenge.ourfancy.domain to _acme-challenge-cname.some.other.domain once. The renew will go the same way as the issue as long as the CNAME exists. You may have also noticed that the my.nsupdate.key is only allowed to update TXT records, and nothing more, to prevent breaking other record types.

The following picture describes the complete process a bit more abstract:

Now, for the not so tech-savvy readers who really made it all the way through this post … all you have to know is:

You’re safe now, everything is encrypted ! 😁

That’s it for now, folks. I hope you enjoyed our special ride through the Let’s Encrypt world, and you have an incentive to make the internet more secure by encrypting your own services.

Stay tuned!

About the author

Carsten Seeger is an Application Engineer, Sysadmin, Automation Developer, and Golang code nerd. He keeps stuff running.