Let’s Encrypt, Dehydrated, Curl and redirects

March 15th, 2018 by

We use Let’s Encrypt for SSL certificates, and our preferred client for obtaining certificates is the simple but effective dehydrated shell script, not least because it’s packaged for Debian.

On Sunday, we started getting some alerts relating to a failure to automatically re-issue Let’s Encrypt certificates. A quick bit of digging yielded this error:

+ Creating fullchain.pem…
  + ERROR: An error occurred while sending get-request to http://cert.int-x3.letsencrypt.org/ (Status 301)

Let’s Encrypt have started including an HTTP redirect as part of the certificate issue process and dehydrated doesn’t pass the necessary option to curl to follow the redirect. This can be fixed by patching dehydrated (and a packaged fix for Debian Stretch is now available via Debian backports), but it can also be solved with a simple config change:

echo 'CURL_OPTS="-L"' > /etc/dehydrated/conf.d/curl.sh

Naturally, customers of our managed hosting services and customers using the free HTTPS option on our hosting accounts need not worry about this issue. Our managed hosting includes monitoring all HTTPS websites for certificates nearing expiry, so we become aware of any issues well before your users do.

Chrome to brand non-HTTPS sites as “insecure” – time to click the button

February 12th, 2018 by

As reported by The Register, sites which do not use HTTPS will soon be actively labelled as “insecure” by the Chrome browser. HTTPS is the secure form of HTTP that makes the little green padlock appear in browsers.

Ultimately, sites which use HTTP are going to be labelled like this:

Example of HTTP site labelled as "not secure"

Not subtle, eh?

The Reg article suggests that initial changes will be deployed July 2018, and will be a little more subtle, but with Chrome having 55-60% market share, it really is time to switch your website to HTTPS.

Fortunately, if you’re hosted with Mythic Beasts this is really easy.  All of our hosting accounts include free SSL (aka TLS) certificates (provided by Let’s Encrypt), and you can enable HTTPS hosting by just clicking a button in the control panel.  Here’s how:

Enabling HTTPS for your Mythic Beasts-hosted website

First, log in to our customer control panel, click on “Hosting and shell accounts”, and click through to the hosting account for your site.  Now find your site in the list, and click on “web settings”:

If you have both a “www” prefixed and bare version, as above, you’ll want to do both. 

On the web settings page, scroll down to the “security” section:

Screen shot of security settingsYou almost certainly want the third option: this will enable HTTPS hosting, and ensure that users see the secure version of the site by default.  (Once you’re happy that your HTTPS site is working exactly as you want it, you could consider switching to the fourth option).

Click, hit “save changes”:

Screenshot of "changes saved" messageWe’ve got plans to make this faster, but for the moment, you’ll need to wait a few minutes.  We’ll go and obtain a certificate for your site, and once installed update your site so that it redirects to the HTTPS by default.

Screenshot of HTTPS location bar

Bingo!

If you haven’t got a working HTTPS site within 10 minutes, email us – we’re here to help.

Any gotchas?

The instructions here will only work if the HTTP version of your site is hosted by Mythic Beasts.   If you’re configuring a new site with Mythic Beasts, make sure that you can access your site via HTTP before enabling HTTPS.

If you’re transferring a site to us that is already using HTTPS, please see our transfer in instructions for how to do this with an interruption to service.

Managed hosting

We’ve been deploying HTTPS as the default for customers of our managed services for some time. We’re going to be doing an audit of all managed sites to warn customers of this upcoming change, but in the meantime, if you’re a managed customer with an http site, just email us and we’ll sort it out.

Domain price reductions

February 9th, 2018 by

Fortunately we don’t buy domains in Bitcoin

Most of our domains are billed to us in US Dollars, so our pricing is at the mercy of the GBP/USD exchange rate.  The pound has strengthened significantly against the dollar since we last reviewed our pricing, so we’ve just rolled out reductions on many of our domain registration prices.  For example, .com domains drop by £2 to £11 + VAT, and .cymru (which, confusingly, we buy in dollars) drops by £3.50 to £22.

We aim to offer straightforward, no-nonsense pricing with no unsustainable introductory discounts that punish customer loyalty with inflated prices in subsequent years.  We price our domains at a level that allows us to properly support our customers.

 

Sender Rewriting Scheme

October 30th, 2017 by

tl;dr: SRS changes the sender address when you forward email so it doesn’t get filed as spam.

We’ve just deployed an update to our hosting accounts that allows you to enable Sender Rewriting Scheme when forwarding mail for your domain.

We’ve previously mentioned how we’re seeing increased adoption of Sender Policy Framework (SPF), a system for ensuring that mail from a domain only comes from authorised servers. Whilst this may or may not reduce spam, it does very reliably break email forwarding.

If someone at sender.com sends you an email to you at yourdomain.com and you forward it on to your address at youremailprovider.com, the email that arrives at your final address will come from the mail server hosting yourdomain.com which almost certainly isn’t listed as a valid sender in the SPF record for sender.com.  Your email provider may reject the mail, or flag it as “untrusted”.

To fix this, we need a different TLA: SRS, or Sender Rewriting Scheme. As the name suggests, this rewrites the sender address of a forwarded email, from one in a domain that you don’t control (sender.com) to one that you do (yourdomain.com).

In the example above, the actual rewritten address would be something like:

SRS0-9oge=B5=sender.com=them@yourdomain.com

This includes an encoded version of the original address, and any email sent to it will be routed back to the sender.  This means that any bounces messages will end up in the right place.

The sender and recipient in these examples refer to the “envelope” sender and receiver.  The addresses that are normally visible to users are the “from” and “to” headers, which may be different and are unaffected by sender rewriting.  Applying SRS should be invisible to the end users.

SRS is now available as an option whenever you create or edit a forwarder using the customer control panel for email accounts hosted on our main hosting servers.  If your account is hosted on sphinx, we need to do a little extra magic to enable it, so please email support.

CAA records

September 1st, 2017 by

A handful of the hundreds of different organisations, all of whom must be trustworthy.

Everybody knows that SSL is a good idea. It secures communications. At the heart of SSL is a list of certificate authorities. These are organisations that the confirm the identity of the SSL certificate. For example, if GeoTrust says that Raspberry Pi is Raspberry Pi we know that we’re talking to the right site and our communications aren’t being sniffed.

However, the list of certificate authorities is large and growing and as it stands, you’ve got to trust all of them to only issue certificates to the right people. Of course, through incompetence or malice, they can make mistakes.

CAA records are a relatively new mechanism that aims to stop this happening, making it harder to impersonate secure organisations, execute bank robberies and steal peoples’ identities.



CAA records enable you to list in your domain’s DNS the certificate authorities that are allowed to issue certificates for your domain. So, Google has a record stating that only Google and Symantec are allowed to issue certificates for google.com. If someone manages to persuade Comodo they are Google and should be issued a google.com certificate, Comodo will be obliged to reject the request based on the CAA records.

Of course, in order to be of any use, you need to be able to trust the DNS records. Fortunately, these days we have DNSSEC (dns security).

How does it work?

A typical CAA record looks something like this:

example.com. IN CAA 3600 0 issue "letsencrypt.org"

This states that only Let’s Encrypt may issue certificates for example.com or its subdomains, such as www.example.com.

Going through each part in turn:

  • example.com – the name of the hostname to which the record apply. In our DNS interface, you can use a hostname of “@” to refer to your domain.
  • IN CAA – the record type.
  • 3600 – the “time to live” (TTL). The amount of time, in seconds, for which this record may be cached.
  • 0 – any CAA flags
  • issue– the type of property defined by this record (see below)
  • "letsencrypt.org" – the value of the property

At present, there are three defined property types:

  • issue – specifies which authorities may issue certificates of any type for this hostname
  • issuewild – specifies which authorities may issue wildcard certificates for this hostname
  • iodef – provides a URL for authorities to contact in the event of an attempt to issue an unauthorised certificate

CAA records can be added using the new section at the bottom of the DNS management page in our control panel:

The @ in the first field denotes a record that applies to the domain itself.

At Mythic Beasts, we’re a bit skeptical about the value of CAA records. In order to protect against the incompetence of CAs, they rely on CAs competently checking the CAA records before issuing certificates. That said, they do provide a straightforward check that CAs can build into their automated processes to detect and reject unauthorised requests, so publishing CAA records will raise the bar somewhat for anyone looking to fraudulently obtain a certificate for your domain.

One click HTTPS + HSTS

March 27th, 2017 by

Last year we rolled out one-click HTTPS hosting for our hosting accounts using free Let’s Encrypt certificates.  We’ve been making some further improvements to our control panel so that once you have enabled and tested HTTPS hosting, it’s also easy to redirect all HTTP traffic to your HTTPS site.

We’ve also added an option to enable HTTP Strict Transport Security (HSTS).  This allows you to use HTTPS on your website and commit that you’re not going to stop using it any time soon (we use 14 days by default).  Once a user has visited your site their browser will cache the redirect from HTTP to HTTPS and will automatically redirect any future requests without even visiting the HTTP version of your site.

HSTS makes it harder for an attacker to impersonate your site as even if they can intercept your traffic, they won’t be able to present an non-HTTPS version of your site to any user that has visited your site within the last 14 days.

HTTPS and HSTS control panel settings

We believe that the web should be secure by default, and hope that these latest changes will make it that little bit easier to secure your website.  These features are available on all of our web and email hosting accounts.  We’ll also happily enable this as part of the service for customer of our managed server hosting.

 

PHP7 on Pi 3 in the cloud (take 2)

March 24th, 2017 by

On Wednesday, we showed you how to get PHP7 up and running on one of our Pi 3 servers. Since then, we’ve implemented something that’s been on our to do list for a little while: OS selection. You can now have Ubuntu 16.04 and the click of a button, so getting up and running with PHP7 just got easier:

1. Get yourself a Pi 3 in our cloud.

2. Hit the “Reinstall” button:

3. Select Ubuntu 16.04:

4. Upload your SSH key (more details), turn the server on, SSH in and run:

apt-get install apache2 php7.0 php7.0-curl php7.0-gd php7.0-json \
    php7.0-mcrypt php7.0-mysql php7.0-opcache libapache2-mod-php7.0
echo "<?=phpinfo()? >" >/var/www/html/info.php

Browse to http://www.yourservername.hostedpi.com/info.php and you’re running PHP7:

Hosting a website on an IPv6 Pi part 2: PROXY protocol

March 10th, 2017 by

In our previous post, we configured an SSL website on an IPv6-only Raspberry Pi server, using our IPv4 to IPv6 reverse proxy service.

The one problem with this is that our Pi would see HTTP and HTTPS requests coming from the proxy servers, rather than the actual clients requesting them.

Historically, the solution to this problem is to have the proxy add X-Forwarded-For headers to the HTTP request, but this only works if the request is unencrypted HTTP, or an HTTPS connection that is decrypted by the proxy. One of the nice features of our proxy is that it passes encrypted HTTPS straight to your server: we don’t need your private keys on the proxy server, and we can’t see or interfere with your traffic.

Of course, this means that we can’t add X-Forwarded-For headers to pass on the client IP address. Enter PROXY protocol. With this enabled, our proxies add an extra header before the HTTP or HTTPS request, with details of the real client. This is easy to enable in our control panel:

You also need to configure Apache to understand and make use of the PROXY protocol header. This is a little more involved, as the necessary module isn’t currently packaged as part of the standard Apache distribution (although this is changing), so we need to download and build it ourselves. First some extra packages are needed:

apt-get install apache2-dev git

This will install a good number of packages, and take a few minutes to complete. Once done, you can download, install and build mod_proxy_protocol

git clone https://github.com/roadrunner2/mod-proxy-protocol.git
cd mod-proxy-protocol
make

At this point you should just be able to type make install but at time of writing, there seems to be some problem with the packaging. So instead do this:

cp .libs/mod_proxy_protocol.so /usr/lib/apache2/modules/

Now you can load the module:

echo "LoadModule proxy_protocol_module /usr/lib/apache2/modules/mod_proxy_protocol.so" > /etc/apache2/mods-available/proxy_protocol.load
a2enmod proxy_protocol

You also need to configure Apache to use it. To do this, edit /etc/apache2/sites-enabled/000-default.conf and replace each line that contains CustomLog with the following two lines:

	ProxyProtocol On
	CustomLog ${APACHE_LOG_DIR}/access.log "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\""

This tells Apache to use Proxy Protocol, and to use the supplied IP address in its log files. Now restart Apache:

systemctl reload apache2

Visit you website, and if all is working well, you should start seeing actual client IP addresses in the log file, /var/log/apache2/access_log:

93.93.130.44 - - [24/Feb/2017:20:13:25 +0000] "GET / HTTP/1.1" 200 10701 "-" "curl/7.26.0"

Trusting your log files

With the above configuration, we’ve told Apache to use the client IP address supplied by our proxy servers. What we haven’t done is told it that it can’t trust any random server that pitches up talking PROXY protocol. This means that it’s trivial to falsify IP addresses in our log files. To prevent this, let’s set up a firewall, so that only our proxy servers are allowed to connect on the HTTP and HTTPS ports. We use the iptables-persistent package to ensure that our firewall is configured when the server is rebooted.

apt-get install iptables-persistent

ip6tables -A INPUT -s proxy.mythic-beasts.com -p tcp -m tcp --dport 80 -j ACCEPT
ip6tables -A INPUT -s proxy.mythic-beasts.com -p tcp -m tcp --dport 443 -j ACCEPT
ip6tables -A INPUT -p tcp --dport 80 -j REJECT
ip6tables -A INPUT -p tcp --dport 443 -j REJECT

ip6tables-save

And we’re done! Our IPv6-only Raspberry Pi3 is now hosting an HTTPS website, and despite being behind a proxy server, we’re tracking real client IP addresses in our logs.

One-click SPF

March 9th, 2017 by

Sender Policy Framework (SPF) has been around for a while, but recently we’ve seen email providers getting much more active in using it to filter mail. Most notably, Gmail appearing to be flagging mail from all domains without an SPF record as untrusted.

In a nutshell, SPF allows you to publish a DNS record that declares a list of all of the mail servers that may legitimately send mail from your domain. It’s not perfect, but it’s a useful tool in reducing email with a forged sender address.

Getting SPF records right can be a bit tricky, but for domains hosted with Mythic Beasts that send mail exclusively via our mail servers, you can now add the correct SPF record with a single click.

One-click SPF enablement

The SPF settings are available on the domain pages in our control panel.

We’d love to make it even easier and just add the record for you, but we can’t be sure that customers are only using our mail servers to send mail, and if not, adding the record will make things worse, although we are planning to add this record by default for newly hosted domains.

It’s worth noting that SPF does not cause problems when sending mail via mailing lists as all decent mailing list software will use its own sender address rather than yours. You may be aware of a change made by Yahoo! that caused considerable problems for mailing lists, but this was related to another system, DMARC, which builds on top of SPF. SPF on its own works just fine with mailing lists.

Hosting a website on a Raspberry Pi with IPv6 and SSL (part 1)

March 2nd, 2017 by

Our hosted Raspberry Pi 3 servers make a great platform for learning how to run a server. They’re particularly interesting as they only have IPv6 connectivity, yet they can still be used very easily to host a website that’s visible to the whole Internet. This guide walks through the process of setting up a website on one of our hosted Pis, including hosting your own domain name, setting up an SSL certificate from Let’s Encrypt, automating certificate renewal, and using our IPv4 to IPv6 HTTP reverse proxy.

Get a Raspberry Pi

First, get yourself a hosted Raspberry Pi server. You can order these from our website, and be up and running in two minutes:

Click on the link to configure your server and you’ll be shown details of your server, and prompted to configure an SSH key:

We use SSH keys rather than passwords. Click on the link, and you’ll be asked to paste in an SSH public key.  If you don’t have an SSH public key, you’ll need to generate one.  On Unix you can use ssh-keygen and on Windows you can use PuTTYgen. Details of exactly how to do this are beyond this guide, but Google will throw up plenty of other guides.

Connect to your server

Once done, you’re ready to SSH to your server. If you’ve got an IPv6 connection, you can connect directly. The Pi used for this walkthrough is called “mywebsite”, so where you see that in these instructions, use whatever name you chose for your server. To SSH directly, connect directly to mywebsite.hostedpi.com. Sadly, the majority of users currently only have IPv4 connectivity, which means you’ll need to use our gateway box. Your server page will give you details of the port you need to connect to. In my case, it’s 5125:

$ ssh -p 5125 root@ssh.mywebsite.hostedpi.com
The authenticity of host '[ssh.mywebsite.hostedpi.com]:5125 ([93.93.134.53]:5125)' can't be established.
ECDSA key fingerprint is SHA256:Hf/WDZdAn9n1gpdWQBtjRyd8zykceU1EfqaQmvUGiVY.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[ssh.mywebsite.hostedpi.com]:5125,[93.93.134.53]:5125' (ECDSA) to the list of known hosts.

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Fri Nov  4 14:49:20 2016 from 2a02:390:748e:3:82cd:6992:3629:2f50
root@raspberrypi:~#

You’re in!

Install a web server

We’re going to use the Apache web server, which you can install with the following commands:

apt-get update
apt-get install apache2

And upload some content:

scp -P 5125 * root@ssh.mywebsite.hostedpi.com:/var/www/html/

Now, visit http://www.yourserver.hostedpi.com in your browser, and you should see something like this:

Another computer on the web serving cat pictures!
 

Host your own domain name

Magically, this site on your IPv6-only Raspberry Pi 3 is accessible even to IPv4-only users. To understand how that magic works, we’ll now host a different domain name on the Pi. We going to use the name mywebsite.uid0.com.

First, we need to set up the DNS for this hostname, but rather than pointing it directly at our server, we going to direct it at our IPv4 to IPv6 HTTP proxy, by creating a CNAME to proxy.mythic-beasts.com:

If you’re using a hostname that already has other records, such as a bare domain name that already has MX and NS records, you can use an ANAME pseudo-record.

Our proxy server listens for HTTP and HTTPS requests on both IPv4 and IPv6 addresses, and then uses information in the request header to determine which server to direct it to. This allows us to share one IPv4 address between many IPv6-only servers (actually, it’s two IPv4 addresses as we’ve got a pair of proxy servers in different data centres).

We need to tell the proxy server where to send requests for our hostname. To do this, visit the IPv4 to IPv6 Proxy page the control panel.  The endpoint address is the IP address of your server, which you can find on the details page for your server, as shown above.

For the moment, leave PROXY protocol disabled – we’ll explain that shortly.  After adding the proxy configuration, wait a few minutes, and after no more than five, you should be able to access the website using the hostname set above.

Enable HTTPS

We’re firmly of the view that secure connections should be the norm for websites, and now that Let’s Encrypt provide free SSL certificates, there’s really no excuse not to.

We’re going to use the dehydrated client, as it’s packaged for the Debian operating system that Raspbian is based on. Unfortunately, it’s not yet in the standard Raspian distribution, so in order to get it, you’ll need to use the “backports” repository.

To do this, first add the backports package repository to your apt configuration:

echo 'deb http://httpredir.debian.org/debian jessie-backports main contrib non-free' > /etc/apt/sources.list.d/jessie-backports.list

Then add the keys that these packages are signed with:

apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 7638D0442B90D010
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 8B48AD6246925553

Now update your local package list, and install dehydrated:

apt-get update
apt-get install dehydrated-apache2

We need to configure dehydrated to tell it which hostnames we want certificates for, which we do by putting the names in /etc/dehydrated/domains.txt:

echo "mywebsite.uid0.com" > /etc/dehydrated/domains.txt

It’s also worth setting the email address in the certificate so that you get an email if the automatic renewal that we’re going to setup fails for any reason, and the certificate is close to expiry:

echo "CONTACT_EMAIL=devnull@example.com" > /etc/dehydrated/conf.d/mail.sh

Now we’re ready to issue a certificate, which we do by running dehydrated -c. This will generate the necessary private key for the server, and then ask Let’s Encrypt to issue a certificate. Let’s Encrypt will issue us with a challenge: a file that we have to put on our website that Let’s Encrypt can then check for. dehydrated automates this all for us:

root@raspberrypi:~# dehydrated -c
# INFO: Using main config file /etc/dehydrated/config
# INFO: Using additional config file /etc/dehydrated/conf.d/mail.sh
Processing mywebsite.uid0.com
 + Signing domains...
 + Generating private key...
 + Generating signing request...
 + Requesting challenge for mywebsite.uid0.com...
 + Responding to challenge for mywebsite.uid0.com...
 + Challenge is valid!
 + Requesting certificate...
 + Checking certificate...
 + Done!
 + Creating fullchain.pem...
 + Done!

We now need to configure Apache for HTTPS hosting, and tell it about our certificates. First, enable the SSL module:

a2enmod ssl

Now add a section for an SSL enabled server running on port 443. You’ll need to amend the certificate paths to match your hostname. You can copy and paste the block below straight into your terminal, or you can edit the 000-default.conf file using your preferred text editor.

cat >> /etc/apache2/sites-enabled/000-default.conf <<EOF
<VirtualHost *:443>
	ServerAdmin webmaster@mywebsite.hostedpi.com
	DocumentRoot /var/www/html

	ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined

        SSLEngine On
        SSLCertificateFile /var/lib/dehydrated/certs/mywebsite.uid0.com/fullchain.pem
        SSLCertificateKeyFile /var/lib/dehydrated/certs/mywebsite.uid0.com/privkey.pem

</VirtualHost>
EOF

Now restart Apache:

systemctl reload apache2

and you should have an HTTPS website running on your Pi:

Automating certificate renewal

Let’s Encrypt certificates are only valid for three months. This isn’t really a problem, because we can easily automate renewal by running dehydrated in a cron job. To do this, we simply create a file in the directory /etc/cron.daily/:

cat > /etc/cron.daily/dehydrated <<EOF
#!/bin/sh

exec /usr/bin/dehydrated -c >/var/log/dehydrated-cron.log 2>&1
EOF
chmod 0755 /etc/cron.daily/dehydrated

dehydrated will check the age of the certificate daily, and if it’s within 30 days of expiry, will request a new one, logging to /var/log/dehydrated-cron.log.

Rotate your log files!

When setting up a log file, it’s always good practice to also set up log rotation, so that it can’t grow indefinitely (failure to do this has cost one of our founders a number of beers due to servers running out of diskspace). To do this, we drop a file into /etc/logrotate.d/:

cat > /etc/logrotate.d/dehydrated <<EOF
/var/log/dehydrated-cron.log
{
        rotate 12
        monthly
        missingok
        notifempty
        delaycompress
        compress
}
EOF

Client IP addresses

If you look at your web server log files, you’ll see one disadvantage of using our proxy to expose your site to the IPv4 world: all requests appear to come from our proxy servers, rather than the actual clients. This is obviously a bit annoying for log file analysis, but is a big problem for any kind of IP-based access controls or rate limiting. Fortunately, there’s a solution, which we’ll look at in the next post.