Shell Shock 2: The AfterShock

September 26th, 2014 by

As has been widely reported, a very major vulnerability in the bash shell was announced a couple of days ago (the event has been dubbed “Shell Shock” by the media). Sadly, the first set of updates released were insufficient to close the hole completely (“AfterShock” is the catchy name). Further updates were released late last night. These have been applied to all Mythic Beasts internal servers, and all managed customer servers.

Customers with unmanaged servers are urged to apply this second set of updates as quickly as possible.

So far, Apple have not released any updates for OS X. The version of bash distributed with OS X is demonstrably vulnerable to the security bug, so no doubt updates will be forthcoming. In the mean time, one possibility is to build bash from source; instructions for doing so can be found on the Internet.

Security issue in bash

September 24th, 2014 by

We’ve just become aware of a potentially very serious security hole in bash which is potentially remotely executable.

https://securityblog.redhat.com/2014/09/24/bash-specially-crafted-environment-variables-code-injection-attack/

Whilst we don’t have enough details yet to evaluate the seriousness of this we’ve already applied the fixes to our administration servers and VPN gateways, and are now looking at rolling out the updates to all affected managed customers. Managed customers should expect an email shortly with further details.

Ticket escalation

September 24th, 2014 by

Managed server customers receive as standard 24/7 monitoring of their servers, we check that the machines are up, that ssh is running, that the web-server is delivering the correct content amongst other checks. In the event a check fails our staff are alerted via SMS/pager to investigate the issue.

We’ve now enhanced this service for managed server customers, in the unlikely event you have a service affecting issue that the automated monitoring hasn’t caught, you can file an urgent ticket through our control panel which will create a new support ticket and alert our staff via SMS to deal with your issue.

This was a feature request from a customer in a meeting last Thursday and went into production as a service enhancement on Tuesday, we’re always receptive to suggestions from customers to make our offering better.

Explain To Us: an explanation

September 17th, 2014 by

This post is written by Ben Howe who’s with us for his gap year. He took some holiday to go to Young Rewired State and he’s written up what he did.

During the summer I participated in the Young Rewired State Festival of Code, where I wrote a website called “Explain To Us” – designed to make the list of bills currently before parliament less confusing. For example, the “summary” listed on the website is the “long title” from the legal text, as opposed to being clearer more detailed. Explain To Us is a platform which allows members of parliament to upload a short video “pitch” to explain what their bill is about and why it should be given debating time in parliament.

Young Rewired State is an independent global network of young people aged 18 and under who have taught themselves to code. They bring together like-minded peers at events around the world where they use open data to make websites, apps and other solutions to real world challenges. The Festival of Code is an annual celebration of programming held over a week in multiple centres, culminating in a huge showcase weekend where everyone presents their solutions (this year in Plymouth, hosted by Plymouth University).

A Python script interprets the RSS feed for all bills before parliament. It then visits the <link> for each bill, scraping the sponsor information (as this is not included in the RSS feed). HTML pages are then generated, which are served by nginx. In future, I’m planning to automatically email members  of parliament (using contact details from TheyWorkForYou) when they submit a new bill asking them to upload a video.

After my initial presentation, I was nominated for “Best In Show” – although sadly I was knocked out by “YouDraw” (who later won Best In Show). Having gained valuable feedback from presenting to the Speaker’s Commission on Digital Democracy and Tracy Green (head of online services at parliament), I’m now intending on building a second beta with more functionality.

If you’d like to work for Mythic Beasts in your gap year, you need to demonstrate how you can use Google to find our jobs page and work the rest out from there.

SSL certificates: SHA-1 deprecation

September 16th, 2014 by

We’ve been asked a few times recently about the announcement from the developers of the  Chrome browser concerning SHA-1 deprecation. This post gives some background, and answers the most common questions. If you’re in a hurry: don’t panic! Mythic Beasts has got you covered.

What’s it all about? For an SSL certificate to be trusted by browsers around the world, it needs to be digitally signed by a well-known and trusted body, called a Certification Authority or CA. (For the SSL certificates that we sell, the CA is usually GeoTrust.) When a certificate is issued, the CA takes the details of your certificate (most notably the address of your website, and the public half of your crypto key) and digitally signs them. When a user browses to your site, they receive your signed certificate and they check the CA’s signature of it to be certain that they are talking securely to the right site.

Except… the CA doesn’t actually sign all the data in the certificate. It first passes it through a cryptographic hash function, which securely reduces the data to a small fingerprint, and then it signs that. Currently, the hash function most commonly used in SSL certificates is SHA-1. This is going to change over the next couple of years.

What’s wrong with SHA-1? Cryptographic hash functions become weaker over time, as Moore’s law makes computers ever faster, and cryptologists discover flaws in the algorithm. A significant flaw in SHA-1 was first published back in 2005, and it is now believed that it may be feasible to find a SHA-1 collision by 2018 or sooner.

Is there an alternative? Yes! SHA-2 (also known as SHA-256) is already implemented in all major browsers. However, because of the nature of the problem, every SSL certificate needs to be upgraded, so we can tell browsers to stop accepting SHA-1 signed certificates. Note that SHA-1 has not yet been “broken”, and while weakened, it is still strong enough for the time being. However, for a smooth transition to SHA-2, we need to start now.

What’s the Google announcement? Although this has generated a lot of discussion, it doesn’t actually say very much that’s new. Microsoft announced in November 2013 that they will not accept SHA-1 signed certificates after 2016. The Chrome developers have recently confirmed that they will do the same, and have filled in some details for a hopefully smooth transition.

Rather than having a single cut-off date, Chrome will be gradually ratcheting up the level of warnings. SHA-1 certificates with an expiry date in 2017 or later will start to receive the “yellow triangle” warning in the browser address later this year. Of course, the connection is still encrypted, but this is a clear indication to users, and hopefully the site administrators too, that something is amiss.

yellow-triangle

By the middle of next year, if those sites are still running with a SHA-1 certificate that lasts into 2017, the warning will be upgraded to a red cross, making it crystal clear to all concerned that action is needed.

red-cross

For SHA-1 certificates that have an expiry date in 2016, the situation is not so serious. Sites with these certificates will start to receive yellow triangle warnings next year, but the warning won’t be escalated beyond that. In both cases, only the security icon will be changed; there will be no click-through warnings. This move, which is also supported by Mozilla and Opera, is forcing the hand of the CAs, and much misinformation has been spread about it.

Is my certificate OK? Yes. Almost all* certificates issued by Mythic Beasts have only a 1 year expiry, so your current certificate will not solicit any warnings from Chrome, or any other browser. *Very occasionally we do issue certificates for longer than 1 year; we’ve checked the issue dates for all these, and are in contact with all affected customers to reissue their certificates.

And then…? We already have a process in place for issuing SHA-2 certificates and renewals. It’s currently a bit more fiddly than we’d like (in fact, it involves obtaining a SHA-1 certificate and then reissuing it as SHA-2!), but we will be trying to make this slicker over the next few weeks. In any case, after October 2014, we will only be issuing SHA-2 certificates.

bzip2

September 15th, 2014 by

bzip2 is one of the great unix tools. It compresses and uncompresses data, and it does it very well. We’ve been using it within Mythic Beasts for years and it’s operated absolutely flawlessly.

We’re happy to report that we’re now hosting the main distribution site for bzip2

Get your site ready for IPv6: a step-by-step guide

September 15th, 2014 by

We were asked on Twitter to add a page describing exactly how to get 10/10 on our IPv6 domain readiness checker, so here it is.

1. Add an IPv6 address to your web server

The first step is to get your web server listening on an IPv6 address, as well as an IPv4 address. How you achieve this will depend on how your web server is managed. If you’re on a shared hosting account, you’ll be dependent on your hosting provider. If you run your own server, you’ll need to obtain an IPv6 address from your hosting provider (assuming they support IPv6), configure your server to use it and then ensure that your web server (e.g. Apache is listening on this address).

If you’re using a standard Mythic Beasts hosting account, this step is already done for you. If you’re using a sphinx shell account, please contact support.

2. Add an AAAA record for your website

AAAA records are the IPv6 equivalent of A records, which resolve hostnames to IP addresses.  In order for users to find your website over IPv6, you will need to add an AAAA record for www.yourdomain.com pointing to the IPv6 address configured above.  You can check that this is in place using the dig command:

$ dig +short A www.mythic-beasts.com
93.93.131.39

$ dig +short AAAA www.mythic-beasts.com
2a00:1098:0:86:1000::15

It’s possible that your existing “www” record will be a CNAME for another hostname, in which case you should add the AAAA record to that hostname, rather than the “www” record.

If you’re using a Mythic Beasts hosting account, then provided that you are using the standard template for your account in the DNS configuration for your domain, then this step will already be done for you.

Our health checker will skip this test if your domain doesn’t have an A record for “www”.

3. Add an AAAA record for your bare domain

Most websites are configured to work if the user omits the “www” prefix from the name, for example http://mythic-beasts.com

In order for this to work, you will need an A record for your domain name itself, and to be IPv6-enabled, you’ll also need a corresponding AAAA record.

Once again, our checker will skip this test if the bare domain doesn’t have an A record.

4. Ensure your DNS servers have IPv6 addresses

The steps above make it possible to access your website over IPv6, but unless your DNS servers are accessible over IPv6, users (or more specifically, their DNS resolvers) will still need to use IPv4 in order to find your site in the first place. To avoid this, you need to ensure that your nameservers have IPv6 addresses.

You can find the nameservers for your domain using “whois”, and you can check whether the servers have IPv6 addresses using dig, as before:

$ whois mythic-beasts.com
[ ... ]
Name Server: NS0.BEASTS.ORG
Name Server: NS3.MYTHIC-BEASTS.COM
Name Server: NS2.MYTHIC-BEASTS.COM
Name Server: NS0.MYTHIC-BEASTS.COM
Name Server: NS1.MYTHIC-BEASTS.COM
[ ... ]
$ dig +short AAAA ns1.mythic-beasts.com
2600:3c00::f03c:91ff:fe96:beac

If your nameservers do not have IPv6 addresses, then unless you run your own nameservers, you’ll either need to persuade your hosting provider to enable IPv6, or switch your DNS provider to a different provider.

Mythic Beasts customers should be using ns1 and ns2.mythic-beasts.com, and both of these nameservers have IPv6 addresses, and IPv6 glue (see below).

For a full pass, our health checker requires that at least two of your servers have IPv6 addresses.

5. Add IPv6 glue for your nameservers, if necessary

In order to find the address for your website, a DNS resolver will first need to find the address of your nameservers. If your nameservers are in your own domain, this creates a bootstrapping problem. For example, in order to find the address for ns1.mythic-beasts.com, you need to ask the nameservers for mythic-beasts.com, which includes ns1.mythic-beasts.com. The solution to this is a glue record, a record containing the address of your nameserver which is held by the nameserver for the next zone up. In this case, the next zone up is .com, so the .com nameservers would contain glue records for the ns*.mythic-beasts.com nameservers.

If a nameserver has an IPv6 address, then any glue records for it should also contain that IPv6 address.

Checking for glue records is a little bit involved. The quickest way to do it is to use “dig +trace” to find a nameserver for the next zone up:

$ dig +trace ns1.mythic-beasts.com
[...]
com.      172800  IN  NS  a.gtld-servers.net.
com.      172800  IN  NS  b.gtld-servers.net.
com.      172800  IN  NS  c.gtld-servers.net.
[...]

We can now ask any of those servers for the NS records for our domain. Any glue records that exist will be returned in the “additional” section of the response:

$ $ dig NS mythic-beasts.com @a.gtld-servers.net.
[...]
;; ADDITIONAL SECTION:
ns1.mythic-beasts.com.  172800  IN  AAAA  2600:3c00::f03c:91ff:fe96:beac
ns1.mythic-beasts.com.  172800  IN  A 69.56.173.190
ns2.mythic-beasts.com.  172800  IN  AAAA  2a00:1098:0:80:1000::10
ns2.mythic-beasts.com.  172800  IN  A 93.93.128.67
[...]

If your servers are missing glue records, you will need to get your domain registrar to add them.

It’s worth noting that even if you don’t directly require glue because your nameservers are in a different zone, at some point along the chain there will be a nameserver that does require glue.

For a full pass, our glue checker requires at least two nameservers to be discoverable by a single-stack IPv6 resolver at every step of the chain of delegation.

6. Add IPv6 addresses for your incoming mail servers

In order to receive mail over IPv6, at least some of the mail servers listed in the MX records for your domain must have IPv6 addresses. You can find the mail servers for your domain using dig:

$ dig +short MX mythic-beasts.com
10 mx1.mythic-beasts.com.
10 mx2.mythic-beasts.com.

You can then check that these servers have IPv6 address by using dig to resolve an AAAA record, as before.

Once again, if you’re using a Mythic Beasts hosting account using the standard DNS template for your account type, this will be done for you.

In order to pass this test, at least one of the servers listed in your MX records must have an IPv6 address.

7. Add reverse DNS for your mail servers’ IPv6 address

It is generally advisable to have working reverse DNS for any addresses from which you send outgoing mail. In the case of IPv6, this becomes pretty much essential, as one of the biggest mail providers in the world, Google, will reject mail over IPv6 unless the sending server has working reverse DNS for its IPv6 address.

Unless you run your own mail servers, adding support for IPv6 will be down to your mail provider.

Unfortunately, there is no reliable way to obtain the outgoing mail servers that are used for a particular domain, so instead our health check makes a bold assumption that your outgoing servers are the same as the incoming servers listed in your MX records, and checks those. This assumption is certainly not true of all domains, which is why a failure of this test is only treated as a warning.

8. Check your SPF records

SPF (Sender Policy Framework) is a mechanism for publicly listing your outgoing mail servers, so that receivers can detect spoofed email sent from other servers. If you enable your outgoing mail servers to start sending mail over IPv6, and you have an existing SPF record, it is important that you make sure that it includes the IPv6 addresses for your mailservers.

There are various ways of doing this. If your incoming and outgoing mail servers are the same, then you can use the “mx” mechanism in your SPF record. This means that any hosts listed in the MX records for your domains will be regarded as a legitimate source of mail for your domain, and this will automatically include any IPv6 addresses (assuming you’ve done step 6).

If you list IPv4 addresses or address ranges in your SPF record explicitly, then you will need to add corresponding IPv6 addresses for those servers.

The rules applied by our health checker aren’t entirely trivial, as it’s not uncommon for legitimate third party servers to be included in a domain’s SPF record, and there’s no way of pairing up IPv6 addresses with their IPv4 counter parts. Our health checker applies some very broad rules: if you use the “mx” mechanism, then the checker requires at least one IPv6 address for a server listed in the relevant MX records. If there are any explicit “ip4” addresses or ranges specified in the record, then the health checker expects to find at least one explicit “ip6” mechanism.

If your domain does not list an SPF record then this test will pass automatically, as this effectively defaults to “accept from all”.

These rules aren’t watertight, but have proven to be quite effective in identifying mail sources that either haven’t been enabled by IPv6, or which have but haven’t been added to the SPF record.

Feedback

Hopefully this gives some insight into exactly what our domain tester is looking for, and how to comply with it. Customers who are having trouble getting 10/10 should email support. We’re also always very happy to hear any feedback on the health check, particularly suggestions for additional tests that we should be running.

New improved IPv6 site tester

September 5th, 2014 by

We’ve just given our IPv6 Health Check a significant overhaul.

IPv6 is coming, and there’s a lot more to being ready for it than adding an AAAA record and enabling IPv6 on your web server. Google are often cited as an example of an IPv6-enabled website, but the truth is, in an IPv6-only world, nobody would ever find Google because none of their nameservers have IPv6 addresses:

Screen Shot 2014-09-05 at 10.02.00

 

Our IPv6 Health Check aims to reach the parts that other testers don’t reach to find out if your domain is really ready for IPv6.  Do your mail servers have IPv6 addresses?  Do they have reverse DNS for those addresses?  Do your SPF records include IPv6 addresses?

The latest version of our health check includes an experimental IPv6 nameserver delegation and glue test that checks that all nameservers the delegation chain between the root DNS servers and your zone have IPv6 addresses, and sufficient glue to allow you to find them in an IPv6-only world. This test has already uncovered some interesting anomalies which we’ll dig into further in a future post.

Ice Bucket

September 1st, 2014 by

Thanks to Jonathan Wright who runs a very big website, for a nomination.

I’ve nominated Matt Smith, Rob McQueen and Neil McGovern.

Thanks to Ben Howe, our gap year student who’s adequately demonstrated to his colleagues the definition of a career limiting move by dunking a bucked of ice over his boss, The Haymakers for kindly providing the location for the company meeting, the chilled water and the ice, and the rest of my Mythic Beasts colleagues for filming and laughing.