More bandwidth

October 19th, 2016 by
We've added 476892 kitten pictures per second of capacity.

We’ve added 476892 kitten pictures per second of capacity.

We’ve brought up some new connectivity today; we’ve added a new 10Gbps transit link out of our Sovereign House data centre. This gives not only more capacity but also some improved DDoS protection options with distance-based blackholing.

We also added a 1Gbps private peering connection to IDNet. We’ve used IDNet for ADSL connections for a long time, not least for their native IPv6 support. A quick inspection shows 17% of traffic over this private link as native IPv6.

ANAME records

October 7th, 2016 by
Company policy requires that blog posts have a picture.

Company policy requires that all blog posts have a picture.

We’ve just added support to our control panel and DNS API for “ANAME” records. ANAME records, also known as ALIAS records, aren’t real DNS records, but are a handy way of simulating CNAME records in places where you can’t use a real CNAME.

It works like this:

You’ve got DNS for your domain managed with Mythic Beasts, and you want to host your website with some 3rd party service provider. They’ll tell you to point DNS for your website at their server. You create a CNAME record for and point it at So far so good.

You also want requests for your bare domain, e.g. to be served by your provider, so you try to create a CNAME for and get told you can’t. This is because you will already have MX, NS and SOA records for your bare domain, and CNAMEs aren’t allowed to co-exist with other records for the same name.

The usual fall back is to create A or AAAA records that point directly to the IP address of, but this sucks because their IP is now hard coded into your zone, and if they ever want to change the IP of that server they’ve got to try and get all of their customers to update their DNS.

The nice solution would be SRV records, standardised DNS records that allow you to point different protocols at different servers. Unfortunately, they’re not supported for HTTP or HTTPS.

This is where ANAME records come in. You can create an ANAME just like a CNAME, but without the restrictions on co-existing with other records. We resolve the ANAME and substitute the corresponding IP addresses into records into your zone. We then regularly check for any changes, and update your changes accordingly.

Naturally, our ANAME implementation fully supports IPv6: if the hostname you point the ANAME at returns AAAA records, we’ll include those in addition to any A records returned.

Sneak preview from Mythic Labs, Raspberry Pi netboot

August 5th, 2016 by

We don’t like to pre-announce things that aren’t ready for public consumption. It’s no secret that we’d love to offer hosted Raspberry Pis in the data centre, and in our view the blocker for this being possible is the unreliability of SD cards which require physical attention when they fail. So we’ve provided some assistance to Gordon to help with getting netboot working for the Raspberry Pi. We built a sensible looking netboot setup and spent a fair amount of time debugging and reading packets to try and help work out why the netboot was occasionally stopping.

This isn’t yet a production service and you can’t buy a hosted Raspberry Pi server. Yet. But if you’d be interested, we’d love to hear from you at

This is a standard Raspberry Pi 3 with a Power over Ethernet (PoE) adapter. You have to boot the Pi once from a magic SD card which enables netboot. Then you remove the SD card and plug it in to the powered network port. PoE means we can power cycle it using the managed switch. At boot, it talks to a standard tftpd server and isc-dhcp-server, this then delivers the kernel which runs from an NFS root. It’s a minimal Raspbian Jessie from debootstrap plus sshd and occupies a mere 381M versus the 1.3G for a standard Raspbian install. The switch is reporting the Pi 3 consuming 2W.

The Raspberry Pi topple is just for fun.


June 17th, 2016 by

cloud-cpuWe’ve upgraded our VPS Lite service to bring it onto the same hardware platform as our standard VPS offerings. Our base level virtual server is now the VPS 1 and has 1GB RAM and 20GB disk. It’s fractionally more expensive than the old VPS Lite, but offers double the disk and RAM. We’ve also introduced an SSD option, in addition to traditional spinning disks, for faster IO performance.

Customers of the existing VPS Lite service will be migrated onto the new hardware platform, and given the option of keeping their current spec on faster hardware, or upgrading to the new VPS 1 spec.

Naturally, the new VPS 1 comes with IPv6 as standard: IPv4 is an option, although our IPv4 to IPv6 reverse proxy, and the availability of NAT64 for outbound traffic means that most websites can be hosted easily on a IPv6-only server.

Full details of the current specifications can be found on the Virtual Servers page.

I know that I know nothing

May 13th, 2016 by
Over a thousand people put together 43000 packages which forms the universal operating system.

Over a thousand people put together 43000 packages which forms the universal operating system.

One of the hazards of going to the pub in Cambridge is that very smart people will occasionally ask you difficult questions. Steve McIntyre, a former Debian Project leader asked our advice as to how Debian should specify a new central build server. Did we think that they’d be best off with lots of RAM or fast SSD, was PCI-E attached SSD better than SATA SSD or even sticking with cheap, slow but very large spinning hard disks.

On the unusual occasions we build software it completes very quickly, and for any big complicated package we’d just install the binary package from Debian. Advice that is almost always spot on, unless you are Debian attempting to make the binary packages in the first place.

We thought about this for a short time and proclaimed confidently that we didn’t know the answer.

However in Mythic Beasts we have a very strong science background. We suggested the right plan was to test it, take a big machine with multiple types of storage, disk, SATA attached SSD and PCI-E flash and try it out. Shortly afterwards our brain kicked in and realised that this looked just like the new VM hosts we were commissioning and with only a slight rearrangement to our plans we could lend one to Debian. Some weeks of work later and the answer is that an SSD makes a huge difference for the working filesystem, otherwise it doesn’t matter.

DNSSEC now in use by Raspberry Pi

May 12th, 2016 by

Over the past twelve months we’ve implemented Domain Name Security Extensions, initially by allowing the necessary records to be set with the domain registries, and then in the form of a managed service which sets the records, signs the zone files, and takes care of regular key rotation

Our beta program has been very successful, lots of domains now have DNSSEC and we’ve seen very few issues. We thought that we should do some wider testing with a larger number of users than our own website, so we asked some friends of ours with a busy website if they felt brave enough to give it a go

Eben Upton> I think this would be worth doing.
Ben Nuttall> I'll go ahead and click the green button for each domain.
-- time passes --
Ben Nuttall> Done - for all that use HTTPS.

So now we have this lovely graph that indicates we’ve secured DNS all the way down the chain for every request. Mail servers know for definite they have the correct address to deliver mail to, Web requests know they’re at the correct webserver.

The only remaining task is to remove the beta label in our control panel.

Raspberry Pi DNSSEC visualisation, click for interactive version

Raspberry Pi DNSSEC visualisation, click for interactive version

PROXY protocol + nginx = broken header

May 9th, 2016 by

We recently announced support for PROXY protocol in our IPv4 to IPv6 reverse proxy, and happily linked to the instructions for making it work with NGINX. One of our customers has pointed out that they didn’t actually work, and we’ve now got to the bottom of why not.

NGINX version

First issue: you need NGINX >= 1.9.10, as there was a bug with using proxy_protocol on IPv6 listeners. If you’re on Debian Jessie, you can get a suitable version from Jessie backports.

PROXY protocol version

Second issue: NGINX only speaks PROXY protocol v1 and our proxy was attempting to speak v2.

v1 is a human readable plain text protocol, whereas v2 is binary. If you see something like this in the error log:

2016/05/09 11:11:30 [error] 6058#6058: *1 broken header: "

 ]Y??.????PGET / HTTP/1.1

Then that’s a good sign that you’ve got a v2 reverse proxy talking to you.

We’ve now changed our proxy to only speak PROXY protocol v1 by default. We will look into making this a configurable option in the future. The Apache module seems happy speaking either version.

Whilst we’re here, here are some other failure modes you might see. This in the access log, is v2 PROXY protocol being spoken to NGINX which is not configured for PROXY protocol at all.

2a00:1098:0:82:1000:3b:1:1 - - [09/May/2016:11:08:55 +0100] "\x00" 400 172 "-" "-"

And this is v1 PROXY protocol being spoken to NGINX which is not configured for it:

2a00:1098:0:82:1000:3b:1:1 - - [09/May/2016:11:39:30 +0100] "PROXY TCP4 64221 80" 400 173 "-" "-"

PROXY protocol support for our, err, proxy

April 29th, 2016 by

We’re increasingly using our IPv4 to IPv6 reverse proxy to host websites on IPv6-only virtual machines. One of the downsides of proxying is that your server doesn’t get to see the client’s real IP address. For non-SSL connections, the proxy can insert an “X-Forwarded-For” header, but SSL is increasingly becoming the norm, and one of the nice things about an SNI-aware reverse proxy is that it doesn’t need to do SSL off load: we don’t need your certificates on our proxy and your traffic stays encrypted until it hits your server. Of course, this means that we can’t go inserting any headers into your connection either.

Fortunately, there is a solution: PROXY protocol. This is a protocol-agnostic mechanism for passing information from a reverse proxy to a server, including the client IP address.

We’ve just added support for PROXY protocol to our reverse proxy:


Turning this on allows your server to get the client IP address, but as it’s an additional protocol, not part of HTTP, your server must be expecting it: turning this on and pointing it at a standard HTTP server will result in a broken website.

Most web servers have support for this. NGINX has support built in, and just needs “proxy_protocol” adding after the listen directive:

server {
    listen 80   proxy_protocol;
    listen 443  ssl proxy_protocol;

You will probably also want some additional configuration to actually set the IP address that gets used for logs etc., and also to ensure that you only trust proxy information from the real proxy servers.

For Apache, support is provided by mod_proxy_protocol, which needs to be installed manually. Once done, configuration is easy:

<VirtualHost *:443>
  ProxyProtocol On

  CustomLog ${APACHE_LOG_DIR}/access.log "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\""

The CustomLog line instructs Apache to use the real client IP rather than the proxy. You should now see v4 addresses being happily logged on your IPv6 server:

root@vm1:~# tail -n 1 /var/log/apache2/access.log - - [29/Apr/2016:14:05:32 +0100] "GET / HTTP/1.1" 200 321 "-" "curl/7.26.0"

Unfortunately the module doesn’t currently provide a way to restrict enablement to trusted proxies only. As such, you’ll probably want to install a firewall to restrict HTTP/HTTPS traffic to only come from our proxies, as otherwise clients could easily fake their IP address.

One thing to watch out for is that although this is applied within a VirtualHost configuration, it’ll actually apply to all virtual hosts on the same IP address and port. This is an unavoidable side effect of the fact that the proxy information is sent before we start talking HTTP. Of course, with IPv6, throwing another IP address at the problem isn’t an issue.

IPv6 only hosting

April 27th, 2016 by

Last week at the UK Network Operators Forum Pete gave a talk about our IPv6 only hosting, progress we’ve made and barriers we’ve overcome.

It’s now available to view online

Let’s Encrypt IPv6-only

April 18th, 2016 by

Let’s Encrypt on a v6-only host

One of the much requested features for Let’s Encrypt free SSL certificates is support for IPv6-only hosts. Whilst this is promised in the very near future we’re happy to say that IPv6-only hosts behind our NAT64 & Proxy services work out of the box with Let’s Encrypt.

To test it we took the traditional dogfood approach, this website is run on an IPv6-only VM and we’ve just enabled Lets Enrypt SSL support on our own blog. As soon as Let’s Encrypt offer SSL certificates for IPV6-only hosts with no proxy and no NAT64 we’ll give that a try too.

DNS-based domain validation (dns-01)

An alternative approach would be to use dns-01 validation using our DNS API. Our API speaks native v6, so that should work just fine on a truly single-stack IPv6 host.