Let’s Encrypt, Dehydrated, Curl and redirects

March 15th, 2018 by

We use Let’s Encrypt for SSL certificates, and our preferred client for obtaining certificates is the simple but effective dehydrated shell script, not least because it’s packaged for Debian.

On Sunday, we started getting some alerts relating to a failure to automatically re-issue Let’s Encrypt certificates. A quick bit of digging yielded this error:

+ Creating fullchain.pem…
  + ERROR: An error occurred while sending get-request to http://cert.int-x3.letsencrypt.org/ (Status 301)

Let’s Encrypt have started including an HTTP redirect as part of the certificate issue process and dehydrated doesn’t pass the necessary option to curl to follow the redirect. This can be fixed by patching dehydrated (and a packaged fix for Debian Stretch is now available via Debian backports), but it can also be solved with a simple config change:

echo 'CURL_OPTS="-L"' > /etc/dehydrated/conf.d/curl.sh

Naturally, customers of our managed hosting services and customers using the free HTTPS option on our hosting accounts need not worry about this issue. Our managed hosting includes monitoring all HTTPS websites for certificates nearing expiry, so we become aware of any issues well before your users do.

Chrome to brand non-HTTPS sites as “insecure” – time to click the button

February 12th, 2018 by

As reported by The Register, sites which do not use HTTPS will soon be actively labelled as “insecure” by the Chrome browser. HTTPS is the secure form of HTTP that makes the little green padlock appear in browsers.

Ultimately, sites which use HTTP are going to be labelled like this:

Example of HTTP site labelled as "not secure"

Not subtle, eh?

The Reg article suggests that initial changes will be deployed July 2018, and will be a little more subtle, but with Chrome having 55-60% market share, it really is time to switch your website to HTTPS.

Fortunately, if you’re hosted with Mythic Beasts this is really easy.  All of our hosting accounts include free SSL (aka TLS) certificates (provided by Let’s Encrypt), and you can enable HTTPS hosting by just clicking a button in the control panel.  Here’s how:

Enabling HTTPS for your Mythic Beasts-hosted website

First, log in to our customer control panel, click on “Hosting and shell accounts”, and click through to the hosting account for your site.  Now find your site in the list, and click on “web settings”:

If you have both a “www” prefixed and bare version, as above, you’ll want to do both. 

On the web settings page, scroll down to the “security” section:

Screen shot of security settingsYou almost certainly want the third option: this will enable HTTPS hosting, and ensure that users see the secure version of the site by default.  (Once you’re happy that your HTTPS site is working exactly as you want it, you could consider switching to the fourth option).

Click, hit “save changes”:

Screenshot of "changes saved" messageWe’ve got plans to make this faster, but for the moment, you’ll need to wait a few minutes.  We’ll go and obtain a certificate for your site, and once installed update your site so that it redirects to the HTTPS by default.

Screenshot of HTTPS location bar

Bingo!

If you haven’t got a working HTTPS site within 10 minutes, email us – we’re here to help.

Any gotchas?

The instructions here will only work if the HTTP version of your site is hosted by Mythic Beasts.   If you’re configuring a new site with Mythic Beasts, make sure that you can access your site via HTTP before enabling HTTPS.

If you’re transferring a site to us that is already using HTTPS, please see our transfer in instructions for how to do this with an interruption to service.

Managed hosting

We’ve been deploying HTTPS as the default for customers of our managed services for some time. We’re going to be doing an audit of all managed sites to warn customers of this upcoming change, but in the meantime, if you’re a managed customer with an http site, just email us and we’ll sort it out.

Meltdown and Spectre

January 17th, 2018 by

A rack of Pi 3s… possibly the only cloud computers immune to Spectre?

There’s been a lot of activity in the news regarding two new security issues called Meltdown and Spectre.

The security issues are newsworthy because they’re different to any security issues we’ve seen before. They’re not an issue in software, but in your computer itself. As a result the vulnerabilities cross multiple operating systems – Windows/Linux/OSX and multiple devices – Laptop/Desktop/Server/Phone, and they’re also a lot harder to fix.  Meltdown affects only Intel processors.  Spectre also affects AMD, Power, RISC and some ARM CPUs.

If you’d like to know how the vulnerabilities work, Eben Upton wrote up a clear explanation for Raspberry Pi – the only common functional computer that isn’t affected.

At present the fixes for Meltdown are effective but can cause significant slowdowns. Fixes for Spectre are incomplete and we have had reports that they can cause instability in Haswell and Broadwell families of Intel CPUs (which we own). Spectre is difficult and slow to exploit because it relies on reading memory one bit at a time.  At 1500bytes/second a full memory dump of one of our virtual server hosts (256GB RAM) would take around six years to complete.

Impact

Both issues allow information leakage so that lower priority processes on a server can read secret data from higher priority processes on the same CPU.  Any computer that accepts instructions from an untrustworthy source is at risk.  We’ve reviewed the impact across all of our services, and have applied or will be applying patches as required.  The impact on live hosting platforms is as follows:

Shared hosting servers

Our web hosting and shell account hosting platforms may have untrustworthy users on them. These servers have already been fully patched against Meltdown and fixes for Spectre will be applied as they become available.

Virtual server hosts

Our Virtual Server Cloud uses KVM with hardware virtualisation which is not vulnerable to Meltdown. Spectre patches are being worked on for the kernel which require new microcode for the CPU. KVM will also need to be updated to fully patch. When these updates are available and have been demonstrated to be stable we will be applying them to our host servers.

This will require a restart of our VM hosts and all guest VMs. Customers will be notified in advance of requiring a restart and each of our datacentres will be restarted at a different time to minimise disruption to customers with split site services.

Virtual server guests

Whilst the use of KVM with hardware virtualisation ensures that Meltdown cannot be used to break the isolation between virtual server guests, virtual servers themselves are potentially vulnerable to both Meltdown and Spectre.   Customers should ensure that their servers are patched and rebooted if they have untrusted users or execute untrusted code.

Dedicated servers

Dedicated servers are at no significant risk unless you allow untrusted third parties to upload and execute code. If that’s the case managed customers can contact support@mythic-beasts.com and we’ll apply the Meltdown and Spectre fixes and reboot as a mutually convenient time.

Raspberry Pi 3 servers

As mentioned above, the Raspberry Pi is not affected and no action is needed.

All other systems

We have reviewed the risk to all other systems and are applying patches as required.  This has included patching, as a high priority, all staff desktops and laptops; websites are allowed to execute javascript which can be used to execute a successful Meltdown attack.

Education, and the teacher becomes the student.

October 6th, 2017 by

Learn more about XSS with Google

For a long time we’ve sponsored Gwiddle, a project that outgrew its hosting on Microsoft Azure, providing free hosting accounts for students. They’ve now become a fully fledged charity, The Gwiddle Foundation, and we’ve had to upgrade the servers we donated to accommodate their ever expanding user base.

Part of their security team is the very talented Aaron Esau (15), who recently applied his penetration testing skills to our website and picked up a difficult to exploit bug.

On our page that allows you to search for domain names, our code embedded the search terms in the results page without appropriately escaping the content. This is a classic cross site scripting bug. Exploiting this bug was far from trivial, as the search term had to be short and from a restricted character set.

Aaron managed to craft an exploit using an ingeniously short payload to extract a session cookie and has posted a full write-up of the vulnerability and exploit.

If you had recently logged into our control panel, not logged out, and then visited a malicious page with this exploit, then the attacker could steal a cookie which would, in theory, give the attacker access to your control panel pages. However, we practise defence in depth, and our cookies are tied to an IP address so simply stealing the session cookie doesn’t give you access unless you also share a source IP address. This is an example where NAT and IPv4 is less secure than having IPv6.

Once Aaron brought the bug to our attention we swiftly fixed the page, thanked him for notifying us and sent him an Amazon voucher to thank him for his time and responsible disclosure.

We should emphasise that we do not believe that anyone has ever attempted to exploit this bug, and that the IP restrictions on session cookies mean that the consequences were fully mitigated.

Nonetheless, it’s embarrassing for us to have such a stupid bug in our code and we’ve been investigating how it occurred. It seems that the reason it crept in is because the domain ordering pages use a different form framework from everything else. Most of our pages have HTML generated by a template, and wherever dynamic data is included, it’s run through a filter to escape any HTML characters. The domain ordering pages use a different approach with much of the HTML being generated by a form module which we then include verbatim into our output. Obviously the HTML in this data mustn’t be escaped, as it would break the form; the form module is responsible for escaping any user input. Unfortunately, there are some other parts of the page which don’t come from the form module, and so do need to be escaped. It’s not very clear from the template code which is which, leading to the bug of not escaping some fields.

CAA records

September 1st, 2017 by

A handful of the hundreds of different organisations, all of whom must be trustworthy.

Everybody knows that SSL is a good idea. It secures communications. At the heart of SSL is a list of certificate authorities. These are organisations that the confirm the identity of the SSL certificate. For example, if GeoTrust says that Raspberry Pi is Raspberry Pi we know that we’re talking to the right site and our communications aren’t being sniffed.

However, the list of certificate authorities is large and growing and as it stands, you’ve got to trust all of them to only issue certificates to the right people. Of course, through incompetence or malice, they can make mistakes.

CAA records are a relatively new mechanism that aims to stop this happening, making it harder to impersonate secure organisations, execute bank robberies and steal peoples’ identities.



CAA records enable you to list in your domain’s DNS the certificate authorities that are allowed to issue certificates for your domain. So, Google has a record stating that only Google and Symantec are allowed to issue certificates for google.com. If someone manages to persuade Comodo they are Google and should be issued a google.com certificate, Comodo will be obliged to reject the request based on the CAA records.

Of course, in order to be of any use, you need to be able to trust the DNS records. Fortunately, these days we have DNSSEC (dns security).

How does it work?

A typical CAA record looks something like this:

example.com. IN CAA 3600 0 issue "letsencrypt.org"

This states that only Let’s Encrypt may issue certificates for example.com or its subdomains, such as www.example.com.

Going through each part in turn:

  • example.com – the name of the hostname to which the record apply. In our DNS interface, you can use a hostname of “@” to refer to your domain.
  • IN CAA – the record type.
  • 3600 – the “time to live” (TTL). The amount of time, in seconds, for which this record may be cached.
  • 0 – any CAA flags
  • issue– the type of property defined by this record (see below)
  • "letsencrypt.org" – the value of the property

At present, there are three defined property types:

  • issue – specifies which authorities may issue certificates of any type for this hostname
  • issuewild – specifies which authorities may issue wildcard certificates for this hostname
  • iodef – provides a URL for authorities to contact in the event of an attempt to issue an unauthorised certificate

CAA records can be added using the new section at the bottom of the DNS management page in our control panel:

The @ in the first field denotes a record that applies to the domain itself.

At Mythic Beasts, we’re a bit skeptical about the value of CAA records. In order to protect against the incompetence of CAs, they rely on CAs competently checking the CAA records before issuing certificates. That said, they do provide a straightforward check that CAs can build into their automated processes to detect and reject unauthorised requests, so publishing CAA records will raise the bar somewhat for anyone looking to fraudulently obtain a certificate for your domain.

Encryption is vital

June 7th, 2017 by


We refuse to bid for government IT work because we can’t handle the incompetence.

At Mythic Beasts we make use of free secure encryption all the time. Like all powerful tools such as roads, trains, aeroplanes, GPS navigation, computers, kitchen knives, vans and Casio watches; things that are very useful for day to day life are also useful for criminals and terrorists. It’s very popular for our politicians and the Home Office, especially our current prime minister King Canute Theresa May and leader of The Thick Party, to suggest that fully secure encryption should be banned and replaced with a weaker version that will reveal all of your secrets but only to the UK security services.

We disagree and think this is a terrible idea. There’s the basic technical objection that a backdoor is a backdoor and keeping knowledge of the backdoor secret is essentially impossible. There’s a recent practical demonstration of this: the NSA knew of an accidental backdoor in Windows and kept it secret.  It was leaked, resulting in the thankfully not very effective WannaCry virus which disabled substantial fractions of the NHS. The government is very good at scope creep: the Food Standards Agency refused to disclose why it needs the power to demand your entire internet history. We think it fundamentally wrong that MPs excluded themselves from the Investigatory Powers act. Then there’s simple commercial objections: it’s a slight commercial disadvantage if every UK product has an ‘Insecure By Order of The Home Office’ sticker on the front when your foreign competitors products don’t.

However, Mathematics does not care what our politicians wish and refuses to change according to their desires. Strong cryptography is free, available on every computer, and can be given away on the front of a magazine. Taking away secure cryptography is going to involve dragging a playstation out of your teenagers’ hands, quite literally stealing from children. Of course secure communications will still be available to any criminal who can illegally access some dice and a pencil.

It’s a good job you can’t build encryption machines with childrens toys.

At Mythic Beasts we make extensive use of open source free cryptography. OpenSSH protects our administrative access to the servers we run and the customers we manage. OpenSSL protects all our secure web downloads, including last month’s million or so copies of Raspbian ensuring that children with a Raspberry Pi don’t have their computer compromised. We make extensive use of free certificates through Let’s Encrypt and we’ve deployed tens of thousands of upgraded packages to customers which are securely verified by GnuPG.

Without these projects, vast quantities of the internet would be insecure. So we’ve made donations to OpenSSH, GnuPG and Let’s Encrypt to support their ongoing work. We’d like to donate to OpenSSL but we can’t see easily how to pay from a UK bank account.

Cambridge Beer Festival, Raspberry Pi powered Apps for Beer

May 22nd, 2017 by

We drew the architecture diagram for the beer festival on a beer mat.

Today marks the first day of the Cambridge Beer Festival, the longest running CAMRA beer festival, one of the largest beer festivals in the UK and in our obviously correct opinion, by far the best. Not only have we run the web back-end for many Cambridge CAMRA websites for many years, this year we’ve been involved with Cambridge App Solutions who run the iPhone Beer Festival App. They’d been having some trouble with their existing hosting provider for the back-end. In frustration they moved it to their Cloud Raspberry Pi which worked rather better. They then suggested that we keep the production service on the Raspberry Pi, despite it being a beta service.

Preparing for production

We’ve set up all our management services for the hosted Pi in question, including 24/7 monitoring and performance graphing. We then met up with Craig, their director in the pub to discuss the app prior to launch. The Pi 3 is fronted by CloudFlare who provide SSL. However, the connection to the Pi3 from Cloudflare was initially unencrypted. We took Craig through our SSL on a Raspberry Pi hosting guide and about a minute later we had a free Let’s Encrypt certificate to enable full end-to-end data security.

 

 

The iPhone app that runs the Cambridge Beer Festival (also found at Belfast and Leestock)

 

The iPhone Beer Festival App tracks which beers are available and the ratings for how good they are. Availability is officially provided, ratings are crowd sourced.  The app is continuously talking to the back end to keep the in app data up to date. All this data is stored and served from the Raspberry Pi 3 in the cloud.

Proximity

The festival also has some Estimote beacons for proximity sensing which use Bluetooth Low Energy to provide precise location data to the phone. On entry to the beer festival the app wakes up and sends a hello message.

One click HTTPS + HSTS

March 27th, 2017 by

Last year we rolled out one-click HTTPS hosting for our hosting accounts using free Let’s Encrypt certificates.  We’ve been making some further improvements to our control panel so that once you have enabled and tested HTTPS hosting, it’s also easy to redirect all HTTP traffic to your HTTPS site.

We’ve also added an option to enable HTTP Strict Transport Security (HSTS).  This allows you to use HTTPS on your website and commit that you’re not going to stop using it any time soon (we use 14 days by default).  Once a user has visited your site their browser will cache the redirect from HTTP to HTTPS and will automatically redirect any future requests without even visiting the HTTP version of your site.

HSTS makes it harder for an attacker to impersonate your site as even if they can intercept your traffic, they won’t be able to present an non-HTTPS version of your site to any user that has visited your site within the last 14 days.

HTTPS and HSTS control panel settings

We believe that the web should be secure by default, and hope that these latest changes will make it that little bit easier to secure your website.  These features are available on all of our web and email hosting accounts.  We’ll also happily enable this as part of the service for customer of our managed server hosting.

 

DNSSEC now in use by Raspberry Pi

May 12th, 2016 by

Over the past twelve months we’ve implemented Domain Name Security Extensions, initially by allowing the necessary records to be set with the domain registries, and then in the form of a managed service which sets the records, signs the zone files, and takes care of regular key rotation

Our beta program has been very successful, lots of domains now have DNSSEC and we’ve seen very few issues. We thought that we should do some wider testing with a larger number of users than our own website, so we asked some friends of ours with a busy website if they felt brave enough to give it a go

Eben Upton> I think this would be worth doing.
Ben Nuttall> I'll go ahead and click the green button for each domain.
-- time passes --
Ben Nuttall> Done - for all that use HTTPS.

So now we have this lovely graph that indicates we’ve secured DNS all the way down the chain for every request. Mail servers know for definite they have the correct address to deliver mail to, Web requests know they’re at the correct webserver.

The only remaining task is to remove the beta label in our control panel.

Raspberry Pi DNSSEC visualisation, click for interactive version

Raspberry Pi DNSSEC visualisation, click for interactive version

Let’s Encrypt IPv6-only

April 18th, 2016 by

Let’s Encrypt on a v6-only host

One of the much requested features for Let’s Encrypt free SSL certificates is support for IPv6-only hosts. Whilst this is promised in the very near future we’re happy to say that IPv6-only hosts behind our NAT64 & Proxy services work out of the box with Let’s Encrypt.

To test it we took the traditional dogfood approach, this website is run on an IPv6-only VM and we’ve just enabled Lets Enrypt SSL support on our own blog. As soon as Let’s Encrypt offer SSL certificates for IPV6-only hosts with no proxy and no NAT64 we’ll give that a try too.

DNS-based domain validation (dns-01)

An alternative approach would be to use dns-01 validation using our DNS API. Our API speaks native v6, so that should work just fine on a truly single-stack IPv6 host.