libssh emergency update

October 17th, 2018 by

An attack so simple my cat could get root on your server.

Managed customers of Mythic Beasts with libssh installed will have just received a notification that we updated it without warning or testing.

This is obviously bad practice, so what were we thinking?

A security advisory for libssh has just come out which is very bad. To paraphrase,

libssh -> hello new user
user -> can I have a root shell
libssh -> can you authenticate?
user -> yes but I'm not going to
libssh -> okay, have a root shell

This is completely secure, unless the client is prepared to lie in order to exploit your system. In the late 1990s some of our founders might have once exploited an online quiz in exactly the same way to get perfect scores. Don’t trust the client.

In our risk analysis, the risk of breakage to a customer site though a botched patch is vastly lower than giving an attacker a root shell, which is why we pushed an emergency update within a few hours of updated packages being available.

If this is the first you’ve heard about the issue, we suggest you’d benefit from our Managed Services

Toby Goodwin (1968-2018)

October 5th, 2018 by

At Mythic Beasts we rotate staff members around different roles. This is to protect the company from the unlikely event that a staff member is abducted by aliens and someone else has to take over at short notice.

With great sadness we have to report that Toby Goodwin, our first full time employee was not abducted by aliens. Much worse, he had an undiagnosed asymptomatic heart problem and passed away unexpectedly and painlessly last week.

Back in 2010 Toby had been running a bookshop in Cambridge with a quirky and eclectic selection of books. That business had come to an end and Toby was wondering about dusting off his UNIX skills and looking for work. At the same time Mythic Beasts had grown too large for the two then-active founders to effectively keep up and after an interview over a beer in the Devonshire Arms, Toby joined Mythic Beasts.

We didn’t initially realise how lucky we were because Toby had the perfect blend of skills. An experienced UNIX hacker from his days at Cygwin, he quickly figured out most of the technical operations to keep Mythic running. Meanwhile his experience at the bookshop gave him incredible patience and empathy for confused customers. He took it on himself to continuously improve our operations introducing radical new ideas like helper scripts having consistent names to make them easy to find, continuous integration and automated testing of our control panel.

Toby implemented the bulk of our managed server update system. When he started, we had tens of managed customers and updating packages was starting to become time consuming. Gradually this became a highly reliable and flexible system which means we can audit and update thousands of servers quickly and efficiently, whilst correctly notifying every affected customer in a timely fashion. Toby was always modest about his achievements and never suffered from being defensive about his code. When our summer students discovered a significant security flaw in a piece of configuration, he congratulated them and worked with them to resolve it quickly.

After working with us for a few years in Cambridge, Toby met Heather and moved with her to her native Scotland where they married and brought into the world a highly reliable early morning alarm clock called Zachary. Toby would regularly work early in the morning before taking some time out to deliver Zachary to nursery or work with him on significant structural engineering projects.

.

In addition to being a skilled software developer, Toby was also a brilliant railway engineer in the face of feline opposition.

Goodnight Toby. We’ll miss you.

Mythic Beasts acquires BHost

July 1st, 2018 by

Having a hungry Wyvern in our logo makes eating other companies much easier to draw.

Hot on the heels of acquiring Retrosnub, we’ve also bought the customers and assets of BHost. BHost are a virtual server provider with services in London, Amsterdam and California based on OpenVZ and KVM.

We’re excited about this acquisition as it provides us with a great opportunity to expand our network using BHost’s Amsterdam infrastructure. At the same time, we’re confident that we can provide some immediate and longer term improvements to the BHost service, not least through our larger support team being able to offer more timely and helpful responses to customer queries.

Although handover officially happened today, BHost customers have had access to our control panel for several weeks, mostly so that we could start tackling EU VAT bureaucracy. BHost are a US-registered business. We’re a VAT-registered business in the UK. Thanks to VAT MESS, it’s actually much harder for us to sell to EU-based consumers than it was for BHost, as we’re required to collect an unreasonable amount of evidence of customer location.

The good news for BHost customers is that we’re matching BHost’s current pricing with our UK VAT-inclusive price. This means that EU VAT-registered businesses, and all non-EU customers will see a significant reduction in the price that they pay.

If you’re a BHost customer and you’ve not already done so, please log in to our customer control panel using your BHost username (email address) and password and confirm your contact details.

Network Expansion

BHost run a network from London/Amsterdam with multiple 10Gbps uplinks and some peering in each site. We will be moving the BHost London network behind our own so that BHost customers can take advantage of our larger capacity uplinks and significantly better peering arrangements, which includes transit-free connections to every major UK ISP.

We’re also taking the opportunity to significantly improve the connectivity to our Cambridge data centre. We currently have two uplinks via different London data centres. We will replace one of these links with a direct connection to Amsterdam, and bring both up to 10Gbps. Combined with BHost’s existing London/Amsterdam connection, this will create a 10Gbps ring around London, Cambridge and Amsterdam, complementing our 50Gbps ring around our three London sites. This will provide increased bandwidth and improved resiliency for our Cambridge customers, whilst also providing a second London/Amsterdam link to improve resilience within the BHost network.

BHost Amsterdam customers will gain direct UK connectivity through our extensive London peering. We will gain the Amsterdam Internet Exchange connection (AMSIX) from BHost, bringing improved European connectivity to all London customers. We expect to be able to substantially increase the number of AMSIX peers, improving EU connectivity for all customers.

Cloud expansion

BHost’s London presence is in the Meridian Gate (MER) data centre. We already have a significant footprint in MER, although it’s not currently available as a zone in our public cloud. We’re investing in new hardware to deploy in Meridian Gate which is both substantially faster and more power efficient than the current hosts. We’ll be deploying this into our existing suite in MER, and then migrating BHost servers into it. BHost customers will see a small window of scheduled downtime as we migrate each server, but should then seen significantly improved performance on the new hardware.

Our Amsterdam and US presences will give additional options to customers that need to be physically hosted within the (post-Brexit) EU or US. We expect this to become more relevant after Brexit when the UK and EU may have diverging regulatory requirements.

Additional services

All BHost customers can now take advantage of additional Mythic Beasts services including management services for virtual servers, domain registration and DNSSEC-enabled, API-backed DNS hosting.

Support

Mythic Beasts have a larger support team and we’re very well placed to provide significantly improved customer service to all of our new customers. Of course, we do expect the period immediately after the transition to be very busy as customers become familiar with the new billing arrangements, and we get to grips with supporting BHost’s services. We will have additional staff during this period, but please be patient if support responses are a little slower than usual.

Finance and Administrative Assistant

June 20th, 2018 by

Invoices, contracts, cheques and a free company mug.

We’re looking to expand our team by employing a part-time finance and administrative assistant for between 12 and 20 hours a week. The duties of the role will likely include dealing with invoicing queries from customers, reconciling bank transfers with invoices, uploading receipts to Receipt Bank and Xero, chasing up overdue invoices, dealing with some paper mail to the company, taking cheques to the bank, filing paperwork and keeping a calendar of administrative deadlines. Previous experience in a similar role is desirable, as is experience with Xero accounting software.

Mythic Beasts don’t have an office, so the job primarily involves working from home. We’ll provide you with a laptop and cover reasonable home-working expenses. Hours are flexible, but we would normally expect you to be available in Cambridge on Wednesday afternoons. You will be entitled to 30 days of holiday a year plus bank holidays, pro rata for the hours worked. Salary is subject to experience. You will be eligible for company health insurance and membership of our employee share scheme after the qualifying period.

If you’re interested or would like to know more, drop us an email.

Retrosnub Acquisition

June 4th, 2018 by

A Mythic Beast eating a Retrosnub (artists impression)

Just before Christmas we were approached by Malcolm Scott, director of Retrosnub, a small cloud hosting provider in Cambridge. His existing connectivity provider had run out of IPv4 addresses. They’d decided to deal with this issue by adding charges of £2 per IPv4 address per month to encourage existing customers to return unused IPv4 addresses to them. As a cloud hosting provider with a substantial number of virtual machines (VMs) on a small number of hosts this had the result of tripling the monthly colocation bill of Retrosnub.

Aware of my presentation on IPv6-only hosting at UKNOF, Malcolm knew that opportunities for significant expansion were severely limited due to the difficulty of obtaining large amounts of IPv4 address space. Retrosnub faced a future of bankruptcy or remaining a very niche provider. His connectivity providers seemed strongly in favour of Retrosnub going bust so they could reclaim and re-sell the IPv4 space for higher margin services.

There are no expansion opportunities for new cloud hosting providers.

As a larger provider with our own address space, we had sufficient spare capacity in our virtual machine cloud to absorb the entire customer base of Retrosnub with no additional expenditure. Our work in supporting IPv6-only virtual machines will also make it easier to significantly reduce the number of IPv4 addresses required to support Retrosnub services. We formed a deal and agreed to buy the customer base of Retrosnub.

Combining operations

Since agreeing the deal, we’ve been working hard to merge our operations with minimum disruption.

The top priority was the domain name services because domains expire if you don’t renew them. Doing a bulk transfer of domain names between registrars is something which Nominet, the body responsible for UK domains, makes extremely easy, as it just requires changing the “tag” on all the domains.

Unfortunately, just about all other TLDs follow a standard ICANN process, which requires that a domain be renewed for a year at the time of transfer, and that the owner of the domain approves the process. If you were designing a process to destroy competition in a market by making it hard for resellers to move between registrars, it would look quite like this.

We’ve now got the bulk of domains transferred, and the next steps will be to migrate the DNS records from Retrosnub to Mythic Beasts so that our control panel can be used to change the records.

At the same time, we rapidly formulated a plan to migrate all the virtual machines in to stem the financial losses. Moving the VMs required an unavoidable change in IP address, and we also wanted to get them migrated from their current platform (Citrix Xenserver with para-virtualisation) to our own platform (KVM with full hardware virtualisation).

In order to ease the transition, we arranged for a pair of servers to do IP forwarding: a server in our cloud that forwarded the new IP to the VM in the Retrosnub cloud until it was migrated in, and another in the Retrosnub cloud that forwarded the old IP after the server had been moved. By doing this we were able to give customers a one week window in which to complete their IP migration, rather than forcing it to be done at the time that we actually moved the VM.

In the process of this migration, all customers received a significant bandwidth upgrade and majority received disk, RAM and CPU upgrades too.

We completed this on schedule before the quarterly colocation bill arrived, so instead of paying the much increased bill, we cancelled the contract and removed the servers from the facility.

Next steps

Our next step will be to migrate all the web and email hosting customers into our standard shared hosting environment. This has some time pressure as Google have plans for Chrome to start marking all non-HTTPS websites as insecure. We offer one click HTTPS hosting using Let’s Encrypt on all of our hosting accounts.

Let’s Encrypt, Dehydrated, Curl and redirects

March 15th, 2018 by

We use Let’s Encrypt for SSL certificates, and our preferred client for obtaining certificates is the simple but effective dehydrated shell script, not least because it’s packaged for Debian.

On Sunday, we started getting some alerts relating to a failure to automatically re-issue Let’s Encrypt certificates. A quick bit of digging yielded this error:

+ Creating fullchain.pem…
  + ERROR: An error occurred while sending get-request to http://cert.int-x3.letsencrypt.org/ (Status 301)

Let’s Encrypt have started including an HTTP redirect as part of the certificate issue process and dehydrated doesn’t pass the necessary option to curl to follow the redirect. This can be fixed by patching dehydrated (and a packaged fix for Debian Stretch is now available via Debian backports), but it can also be solved with a simple config change:

echo 'CURL_OPTS="-L"' > /etc/dehydrated/conf.d/curl.sh

Naturally, customers of our managed hosting services and customers using the free HTTPS option on our hosting accounts need not worry about this issue. Our managed hosting includes monitoring all HTTPS websites for certificates nearing expiry, so we become aware of any issues well before your users do.

Raspberry Pi 3B+

March 14th, 2018 by

Today is Pi Day where we celebrate all things mathematical. Today is a super special Pi day, because a new Raspberry Pi has been released.

It takes the previously excellent Raspberry Pi 3 (or 3B, to give it its full name) and upgrades it with an extra 200Mhz of CPU speed and gigabit ethernet over USB 2. It fixes many of the netboot issues which Pete highlighted at the last big Pi Birthday Party and will soon have a new smaller and cheaper Power over Ethernet HAT. These new features are of particular interest for our Raspberry Pi Cloud service, as we use netbooted Pis, with network file storage and Power over Ethernet to enable remote powercycling.

Raspberry Pi 3B+.

We’ve had one to play with, and we’ve run our favourite benchmark – Raspberry Pi’s own website. We installed the full stack (MySQL, WordPress & PHP7) under Debian Stretch onto a Pi 3B and a Pi 3B+, and tried it out with 32 concurrent connections. We’re running near identical setups on the two servers: both have their files stored over the network on an NFS file server and it’s the same operating system and applications; only the kernel differs.

Model Pages/second
Raspberry Pi 3B 3.15
Raspberry Pi 3B+ 3.65

The new model is about 15% faster than the old one which is almost exactly as expected from the boost in clock speed; WordPress is CPU limited.

Checksumming the 681MB database file shows up the gigabit ethernet rather effectively. All our storage is over the network so reading files is a benchmark of the network speed.

Model Elapsed time Data rate
Raspberry Pi 3B 54.4s 11.25MB/s
Raspberry Pi 3B+ 28.1s 22.1MB/s

This is very nearly twice as fast as the previous model.

When is it coming to the Raspberry Pi Cloud?

The Raspberry Pi 3B+ is an obvious upgrade for our Raspberry Pi Cloud. We need to wait for the PoE HAT to become available. That will allow us better density and lower capital costs. However, the 3B+ consumes more power than the 3B so we need to do some thermal and airflow work before we can make it generally available.

Flatpak: pre-assembled furniture applications for Linux

February 23rd, 2018 by

Flatpack is furniture you build yourself. Flatpak is preassembled applications for Linux. This is apparently not at all confusing. (image thanks to https://www.flickr.com/photos/51pct/)

Flatpak provides Linux desktop applications in a secure sandbox which can be installed and run independently of the underlying Linux distribution. Application developers can produce one Flatpak and select the versions of libraries that their application is built, tested and run with so it’s easy for users on any Linux OS to get whatever was intended by the application developer.

Flathub is a distribution service to make sure that Flatpaks are available for popular Linux desktop applications, and at its heart is a private cloud running BuiltBot which builds popular Linux and free/open source desktop apps in Flatpak format. This lives in Mythic Beasts’ Cambridge data centre.

At Mythic Beasts we like this idea so much we offered them lots of free bandwidth (100TB) to help get them started. We’ve now upgraded this with a pair of virtual machines in our core Docklands sites to provide redundancy and more grunt for traffic serving.


Some of their users noticed and were appreciative immediately:

2017-02-23 16:30:00irc wow! Flathub is *so* much faster i’m getting like 10 MB/s compared to less than 1 this morning … and the search is now instant
2017-02-26 11:35PersiFlathub is _really_ fast now, great job to whoever is responsible
🙂

Chrome to brand non-HTTPS sites as “insecure” – time to click the button

February 12th, 2018 by

As reported by The Register, sites which do not use HTTPS will soon be actively labelled as “insecure” by the Chrome browser. HTTPS is the secure form of HTTP that makes the little green padlock appear in browsers.

Ultimately, sites which use HTTP are going to be labelled like this:

Example of HTTP site labelled as "not secure"

Not subtle, eh?

The Reg article suggests that initial changes will be deployed July 2018, and will be a little more subtle, but with Chrome having 55-60% market share, it really is time to switch your website to HTTPS.

Fortunately, if you’re hosted with Mythic Beasts this is really easy.  All of our hosting accounts include free SSL (aka TLS) certificates (provided by Let’s Encrypt), and you can enable HTTPS hosting by just clicking a button in the control panel.  Here’s how:

Enabling HTTPS for your Mythic Beasts-hosted website

First, log in to our customer control panel, click on “Hosting and shell accounts”, and click through to the hosting account for your site.  Now find your site in the list, and click on “web settings”:

If you have both a “www” prefixed and bare version, as above, you’ll want to do both. 

On the web settings page, scroll down to the “security” section:

Screen shot of security settingsYou almost certainly want the third option: this will enable HTTPS hosting, and ensure that users see the secure version of the site by default.  (Once you’re happy that your HTTPS site is working exactly as you want it, you could consider switching to the fourth option).

Click, hit “save changes”:

Screenshot of "changes saved" messageWe’ve got plans to make this faster, but for the moment, you’ll need to wait a few minutes.  We’ll go and obtain a certificate for your site, and once installed update your site so that it redirects to the HTTPS by default.

Screenshot of HTTPS location bar

Bingo!

If you haven’t got a working HTTPS site within 10 minutes, email us – we’re here to help.

Any gotchas?

The instructions here will only work if the HTTP version of your site is hosted by Mythic Beasts.   If you’re configuring a new site with Mythic Beasts, make sure that you can access your site via HTTP before enabling HTTPS.

If you’re transferring a site to us that is already using HTTPS, please see our transfer in instructions for how to do this with an interruption to service.

Managed hosting

We’ve been deploying HTTPS as the default for customers of our managed services for some time. We’re going to be doing an audit of all managed sites to warn customers of this upcoming change, but in the meantime, if you’re a managed customer with an http site, just email us and we’ll sort it out.

Domain price reductions

February 9th, 2018 by

Fortunately we don’t buy domains in Bitcoin

Most of our domains are billed to us in US Dollars, so our pricing is at the mercy of the GBP/USD exchange rate.  The pound has strengthened significantly against the dollar since we last reviewed our pricing, so we’ve just rolled out reductions on many of our domain registration prices.  For example, .com domains drop by £2 to £11 + VAT, and .cymru (which, confusingly, we buy in dollars) drops by £3.50 to £22.

We aim to offer straightforward, no-nonsense pricing with no unsustainable introductory discounts that punish customer loyalty with inflated prices in subsequent years.  We price our domains at a level that allows us to properly support our customers.