LINX now running at 2x10Gbps

November 29th, 2013 by

Today we’ve upgraded both of our connections to the London Internet Exchange (LINX) from 1Gbps to 10Gbps.

Over the past few weeks we’ve repeatedly broken the company bandwidth record. And since we’ve recently secured more peering agreements — including every major UK connectivity provider — a greater proportion of our traffic is now going out over LINX. So at peak times our bandwidth usage has been enough that in the unlikely event of a failure of one of the LINX LANs, we would have come close to running out of capacity on our other link. Clearly an upgrade was in order!

Our network engineers performed the upgrade this morning, with no disruption as traffic was automatically and transparently rerouted during the brief down time. After the upgrade, we have 10Gbps from our data centre in Telecity Sovereign House to LINX Juniper; and 10Gbps from our Harbour Exchange data centre to LINX Extreme.

In the event of the failure of either link or router, traffic will automatically reroute around our internal fibre ring to our other site and out to the peering exchange via our other connection. And, for the time being, we have plenty of capacity to spare.

Sender Verify vs Hotmail

November 26th, 2013 by

We aim to give our users the choice of a range of anti-spam measures. One of the options we provider is sender verify, a simple check whereby before you accept a mail, you check that the sender of that email exists, and would accept mail from you. You can argue about how effective this is as an anti-spam measure, but it seems a perfectly reasonable check to want to make, in the same way that many people choose to not answer their phone to those who withhold caller ID.

Unfortunately, some people object to you asking the question.

We recently had some complaints from users who said that they couldn’t receive mail from people with addresses hosted on Microsoft’s Hotmail servers, and sure enough, Hotmail have blacklisted one of our servers’ IPs for daring to enquire about whether particular sender addresses were valid. This affects not just hotmail.com, but various other Microsoft domains.

Sadly, Microsoft aren’t going to change their policy for us, so we needed to whitelist them. This isn’t entirely trivial as what matters is where the sender’s email address is hosted, which means looking up the MX records for that domain. Fortunately, Exim makes this easy enough, provided that you’re not offended by curly brackets. Adding the following condition to a sender verify ACL will disable the check for Hotmail hosted domains:

!condition = ${if forany{${lookup dnsdb{>: mxh=$sender_address_domain}{$value}fail}}{match {$item}{\Nmx.\.hotmail\.com\N}}}

I should note that for quite some time, we’ve used a dedicated IP address for performing our sender verify checks in order to minimise the impact of exactly this type of blacklisting. If we hadn’t done this, the blacklist would have made it impossible for any users to send mail to Hotmail-hosted addresses too. As it was, the problem only affected users who had elected to use sender verify on their domains.

IPv6 Reverse DNS

November 20th, 2013 by

You can now configure reverse DNS for IPv6 through our customer control panel. If you’ve previously been handling reverse DNS for your allocation through delegation and would prefer to use the control panel, then please get in touch.

If you’ve got a server with us and are interested in trying IPv6 and don’t already have an allocation then please email support and we’ll be happy to provide you with a block of addresses.

Tricky debugging

November 12th, 2013 by

After cloning a server for a customer we noticed that something was a little bit odd:

# md5sum /etc/sudoers

worked fine but:

# sudo -l

responded with:

sudo: unable to stat /etc/sudoers: Permission denied

How odd we thought. More odd was:

# su - username
Cannot execute /bin/bash: Permission denied

A bit of time with Google and strace revealed that we’d managed to set the permissions on / wrongly:

drwx------  27 root root  4096 Jun  4 11:48 ..

rather than:

drwxr-xr-x  27 root root  4096 Jun  4 11:48 ..

What amazed us was not that the machine didn’t work properly but that we could log in at all.

If this is the sort of problem you’d be able to fix, you should look at our jobs page. If you’d like someone else to fix it for you then our Managed hosting is probably of a lot more interest.

Migrating the Science Media Centre

November 12th, 2013 by

Over the past week or so we’ve given the Science Media Centre a hand in moving their WordPress site into a virtual machine hosted by Mythic Beasts. They’re a charity who work with journalists, scientists and engineers to try and improve the quality of science reporting and removing the misleading rubbish that otherwise gets written. Mythic Beasts is a company founded by science graduates who are very easily angered by terrible science articles in the papers. We’re hoping the saving on destroyed laptops and monitors will easily cover all the management and consultancy services we’ve donated.

If we have fewer idiotic articles proving that Coffee cures cancer* and Coffee causes cancer* and rather more articles that our talented university friends pioneer new cancer treatments we’ll consider the time and effort we’ve put in to helping them well spent.


* Actual links removed in the name of good taste. Here’s something more interesting to read, and if you’re still curious, you can look up coffee in the index.

Enabling IPv6 on your mail servers? Don’t forget SPF

November 8th, 2013 by

Our network has supported IPv6 for a while, but recently we’ve been making a concerted effort to enable IPv6 on more of our servers. What we’ve learned (mostly the hard way) is that the challenge in doing this is not so much in enabling specific services, such as making your webserver speak IPv6, but in the less obvious side effects of bringing up an IPv6 address on the server in question. Once you do this, the server will start making outgoing connections over IPv6 where possible, and that’s when you find out all the places that you’ve got IP-based access controls squirreled away.

One that caught us out recently when we brought up IPv6 addresses on our mail servers was an SPF record that listed our outgoing servers by their IP (v4) addresses. In hindsight, including IP addresses in an SPF record was never a great idea. It would be much better to use the “mx” or “a” SPF terms, referring to mail servers by name rather than address.

To help others avoid making the same mistake, we’ve added SPF record checking to our IPv6 Health Check. The rules on this are necessarily a bit arbitrary: if you have an explicit reference to an IPv4 address, it expects you to have at least one reference to an IPv6 address. In addition, any time that you use an MX term, it expects that MX to have both IPv4 and IPv6 addresses.

For an example of this, compare the results for twitter.com with the results for google.com. We fail twitter.com because of the “mx:one.textdrive.com” term. There are other parts of Twitter’s SPF that don’t appear to have IPv6 equivalents (e.g. “_netblocks.zdsys.com”) but there’s no easy way to determine which IPv6 address block corresponds to each IPv4 address block. Suggestions for better ways to categorise these test results gratefully received.

Sphinx aka Trigger’s Broom

November 7th, 2013 by

Last night we quietly upgraded the disks in our Sphinx shell server to a pair of SSD drives. Sphinx has been suffering under heavy I/O load for a while now, and it’s safe to say that the SSDs have resolved that problem for the foreseeable future.

The upgrade was without downtime, using the magic of LVM’s pvmove command.

It’s been upgraded with a pair of fiendishly expensive server-grade SSDs. We’re not normally ones to pay too much attention to whether kit is designated as “server-grade” but in the case of SSDs it really matters due to the limited number of write cycles on SSDs. The new disks are good for 8TB of writes per day for 5 years, whereas the equivalent consumer grade version is only rated for 20GB/day, which wouldn’t last very long in Sphinx.

Sphinx has a special place in our hearts as it’s the machine on which the company was founded nearly 14 years ago, and it’s been in pretty much continuous service ever since. Of course, the current hardware has absolutely nothing in common with the dual Celeron BP6 that we deposited in a Fulham datacentre back in 2000, and it now lives in Docklands, but it’s still the same machine (right?) which is why it still says:

[pdw@sphinx ~]$ rpm -q redhat-release
redhat-release-6.1-1

(don’t worry, that’s probably the only package from RH 6.1 that we’re still using…)

Rocket Science

November 5th, 2013 by

Today is Guy Fawkes Night, when traditionally in the UK we launch fireworks to commemorate not blowing up King James I, the House of Lords and a large chunk of central London. Most modern firework displays use rather less gunpowder than the original plot in 1605.

At Mythic Beasts we think with the advent of four hundred years additional technology we should be aiming a bit higher. That’s why instead of buying fireworks we’ve donated some hosting to Southampton University Spaceflight. Certainly I (Pete) was incredibly inspired as a child by space flight and I remember watching the first untethered spacewalk as a child which motivated me strongly to learn all about rockets and space ships and from there to reverse engineer 8 bit computer games ultimately leading to running a hosting company.

This is of course the second university space program we’re helping out with – some of the tracking for Cambridge University Spaceflight is also done on servers hosted with us.

Filtering on received headers? Seriously?

November 5th, 2013 by

As if it’s not bad enough that we have to waste a huge amount of time, not to mention a non-trivial amount of hardware, bandwidth and electricity, trying to deal with spam, we also waste a fair amount of time dealing with the “ingenious” ways that the anti-spam brigade come up with to stop legitimate mail from getting through.

This week’s contribution was so special that it took three of us to confirm that they really were doing something as silly as it first appeared.

First some background: the IP address that you get given by your ISP for your broadband connection is usually dynamically allocated. This means that it may change every time you reconnect, and may be reallocated to other users when you’re not using it. This obviously makes it impossible to selectively blacklist the users of such addresses in response to spam complaints, so it is common practice for mail servers to block connections from all IPs that are known to be allocated on this basis, using something like the PBL. Users of such IP addresses are expected to use their ISP or hosting provider’s mail servers to send outgoing mail, and the administrators of those servers take responsibility for policing their customers (on pain of having their mail servers blacklisted).

Today, a customer complained that a legitimate email being sent via our server in this manner (using authenticated SMTP) was being blocked. On closer inspection, it turned out that the IP addresses that the receiving server was objecting to was not our server’s IP address, but the sender’s IP address on the basis of it having a “poor reputation”. Well duh – it’s a dynamically allocated IP: there’s a decent chance that at some point in its life it’s been allocated to an infected computer and used to send spam. They’s why you don’t accept mail from them directly, right?

The stupid thing is that the only way that the receiver knows the originating IP address is through a “Received” header that we add. That’s right, we could trivially defeat this anti-spam measure by configuring our mail server to not add the header.

Of course we’re not going to do that, as it would break the incredibly useful trace that is provided by the received headers, and is one of the few things that helps keep sane mail admins sane.

@Mythic_Beasts now on Twitter

November 2nd, 2013 by

If you want to hear what we’ve got to say, but can only cope with 140 characters at a time then you can now follow us on Twitter.

We may use Twitter to give updates on service status when we’ve got problems, but the official source for such information remains our status page.