Mastodon security update

February 2nd, 2024 by

Yesterday, the following not-so-subtle notice appeared on the admin interface of all Mastodon instances:

The Mastodon team announced on Monday that this release was coming, so we were ready for it:

Details of the vulnerability are still limited, but from what we do know it sounds serious (“Remote account takeover“).

All our managed Mastodon instances were safely patched just over an hour after the new packages dropped. One instance gave us a bit of trouble, as the new version appeared to tickle a bug in Elasticsearch causing ES to consume all CPU on the server. After we eventually pinned down the cause, it was resolved by an upgrade of Elasticsearch. Turns out the ES upgrade didn’t fix it, and we’re still working with our customer to get this resolved.

Managed open source hosting

Open source software such as Mastodon, GitLab and Nextcloud can offer a great alternative to the lock-in associated with proprietary cloud equivalents, but the effort associated with hosting them can be significant: backups, monitoring, security patching, and the investigation and debugging required when a supposedly innocuous software upgrade leaves your CPU usage wedged at 100%.

Our managed open source hosting provides the best of both worlds: the convenience of a “cloud” solution, but without the lock-in. Your data is yours, and if you don’t like our service you can take your data and host it somewhere else (although we’re confident you won’t want to). And because there’s no lock-in, you get straightforward pricing based on the resources you’re using, rather than loss-leaders followed by price hikes once you’re hooked.

Read more about our managed hosting, or drop us an email at for more information.

Exim 0-day

October 4th, 2023 by
exim logo

We sponsor exim and provide a VM for their buildfarm.

Recently Trend Micro, through their Zero Day Initiative, published a critical flaw for the Exim mail server. It’s described as allowing remote attackers to execute arbitrary code on the Exim server without authentication. On the face of it, any server running Exim and listening on the internet can immediately be taken over by an attacker. What makes this worse is that they claim they reported this in June 2022, and the Exim team have ignored fixing it.

ZDI say ‘The only salient mitigation strategy is to restrict interaction with the application.’ and have allocated a scarily high severity score of 9.8/10.

Mythic Beasts make pretty heavy use of Exim in our mail infrastructure, and mitigating the security risk by turning off email is a pretty severe step while we wait for a fix. On top of that amongst servers we manage for ourselves and clients there’s nearly a thousand installed copies of Exim that will need to be updated.

The Exim team have a different view on the severity, as do other reputable security specialists. Watchtowr have a nice write-up explaining that, by default, none of the six issues can be exploited. Cross checking to Mythic Beasts mail infrastructure we can quickly confirm we’re not affected, and we believe that none of the managed customers should be either.

As this is now not especially time critical, we can wait for the supported operating systems to release updated packages which we can install.

Patching

The security issue is definitely significant enough to meet our 0-day policy of patching immediately as it’s network listening software with a risk of compromise. Debian released packages with the most important fixes on Monday 2nd October. Because this issue covers a very large number of affected machines, some of which are absolutely critical we decided to stage the rollout. First we did our staging servers, then one of our core mailhubs. We then paused for a short while to check no functionality was affected. Then we completed the full roll-out to all managed servers both customer and internal. The final step is our audit – recheck the Exim package on every managed server to make sure the update had applied everywhere. The full rollout and audit completed in around three hours.

We’re expecting updated packages from Ubuntu shortly, which will then be rolled out to all supported managed Ubuntu customers when available.

HEX-it

September 27th, 2023 by

Last year, we undertook a significant data centre migration, with the closure of Digital Realty’s Meridian Gate requiring us to move our entire presence there to Redcentric’s City Life Line. Having done it once, why not do it again?

Southern Serval, leaping

Our shared hosting server “serval” has already migrated to SOV. [ Photo by Wynand Uys]

This year, we’re planning a move out of Harbour Exchange (HEX), and starting a presence in Telehouse South. A lot of the things we learned during the previous move are making this move easier to manage, although it is still a prodigious effort, both physically and in terms of design and infrastructure.

One of the things we’ve been working on for some time is improved network infrastructure within our data centres. This introduces IP address portability so that IP addresses do not need to change when servers are moved between data centres, as well as significantly higher bandwidth uplinks for our virtual server hosts.

In the last year, we’ve live migrated over a thousand VMs across two data centres, with minimal interruption to service.

We’re about to start migrating all VMs out of our HEX data centre. We have two available London destinations, SOV and CLL. If you’re a customer with a VM in our HEX data centre, we’ll be emailing you over the next couple of weeks, to check if you have a preference (for instance because you have existing services in one of those data centres, and would prefer to be moved to the other to maintain fault-tolerance).

We will also soon be able to offer Telehouse South as a virtual server zone, in addition to SOV and CLL. This means we will continue to provide three London-based zones for our customers running distributed services. We’ll retain a small residual presence in HEX for connectivity.

PHP 8.2

September 25th, 2023 by

 

Last year we enhanced our web hosting service with the ability to choose your own PHP verison. You can choose a different PHP version for each website hosted with us, so you can upgrade your staging site and test before you upgrade the production one. With PHP 8.0 about to go end-of-life, the addition of PHP 8.2 provides more options for migrating production applications.

Screenshot of account control panel

Choose your PHP version in the control panel

Since the initial roll-out, we’ve added more PHP versions to help with moving and upgrading older applications. Not only is the newest version PHP 8.2 available, but you can also select the older 7.3 and 7.4 versions. We’re proud to sponsor Ondřej Surý who creates the debian packages we rely on.

Our hosting accounts still support unlimited websites, have free and automatic SSL through Let’s Encrypt to keep your sites secure, and include MariaDB databases.

.ie domains and reduced domain pricing

June 19th, 2023 by
Trinity College library Dublin

A 400 year old data warehouse at Trinity College Dublin, Ireland.

We’ve just rolled out a price reduction for domain registration for the vast majority of the TLDs that we offer, including .com, .net and .org. We pay for most of our domains in US dollars, and thanks to the increasing strength of the pound against the dollar, we’ve been able to reduce our prices for all such domains by an average of just over 10%.

.ie domains

We’re also pleased to announce that we’re now able to offer .ie domain registrations. Unfortunately, ID requirements mean that we’re only able to offer these to corporate registrants, and standard .ie residency requirements apply. .ie domains have been a frustrating gap in our available TLDs for many years, so we’re very happy that we’re now at least partially able to fill it.

No-nonsense pricing

Our full price list can be found on our domains page.

We don’t offer loss-leading promotional pricing — we charge the same for new and existing customers alike, don’t ramp up pricing on renewals, and never charge transfer-out fees.

We offer small multi-year discounts for registration or renewals in advance, and pride ourselves on offering a good service for a reasonable price.

Other domains

We’re also a JISC registrar, meaning that we can provide .ac.uk and .gov.uk domains. We can provide credit accounts (subject to checks), allowing organisations to pay for domains via PO and invoice, if required.

DNS, APIs, DNSSEC, IPv6

Domain registrations include DNS with API access as standard. We also support DNSSEC, and naturally, our nameservers are IPv6-enabled. If you’re migrating existing domains to us, you can import zone files directly, via our control panel or the API. We also provide a Domain management API.

Debian Bookworm released and fully supported by Mythic Beasts

June 16th, 2023 by
Bookworm in a damaged book

A bookworm, photo by Dominic Mason

 

On Saturday the Debian team released the latest version of Debian, Bookworm. We’re pleased to announce that this is now available on our virtual and dedicated servers.

Bookworm is a fully supported operating system for our managed hosting and we already have it running on some of our internal production servers. Our preferred open source server management system, Sympl, has also been updated to support Bookworm. Other feature enhancements include much more control over PHP versions and settings. Our virtual server cloud has pre-built images for standard Bookworm and Bookworm with Sympl pre-installed.

There are many improvements in Bookworm, with PHP 8.2 support being the most anticipated by our customers. We would like to thank the Debian team for all their hard work in making this release.

IPv4 to IPv6 Proxy API

April 21st, 2023 by

We’ve been offering IPv6-only hosting for eight years now, and have demonstrated that many websites can forego the expense of an IPv4 address pretty easily. You can read more about how we do this on this blog post from 2020. This blog post itself is being served from an IPv6-only server!

A key part of this is our IPv4-to-IPv6 proxy. This listens for incoming traffic on a shared IPv4 address and forwards it to your IPv6-only server. In order to use the proxy, you need to tell it which hostnames to listen for, and which server or servers to forward traffic to. This can be done using our control panel, and as of today, it can also be done via an API.

Having an API for proxy configuration makes it possible to automatically add or remove backend servers, allowing you to spin up additional servers, or take servers out of service for failover or maintenance.

You can also use the API to add and remove hostnames handled by the proxy, and so can be used to automate the provisioning of new services.

Fine-grained access controls

As for our DNS API and Domain API, the Proxy API provides fine-grained access control for API keys. For example, you can create an API key that only has access to a specified domain or hostname, or you can create a read-only API key if you only need to read the current configuration.

Getting started

Our IPv4-to-IPv6 proxy is available to all customers with a Mythic Beasts server, including virtual servers, Raspberry Pi servers, dedicated and colo. You can find more information on the proxy service, and the Proxy API on our support pages.

Finance and Administrative Assistant

November 25th, 2022 by

Invoices, contracts, cheques and a free company mug.

We’re looking to employ a part-time finance and administrative assistant for between 15 and 25 hours a week. The duties of the role will likely include:

  • dealing with invoicing and payment queries from customers;
  • reconciling bank transfers with invoices;
  • uploading receipts to Hubdoc and Xero;
  • chasing up overdue invoices;
  • dealing with some paper mail to the company;
  • banking cheques;
  • filing paperwork;
  • administering annual leave;
  • keeping a calendar of administrative deadlines; and
  • other administrative and accounting duties.

Previous experience in a similar role is desirable, as is experience with Xero accounting software.

Mythic Beasts don’t have an office, so the job primarily involves working from home. We’ll provide you with a laptop and cover reasonable home-working expenses. Hours are flexible, but we would normally expect you to be available in Cambridge on Wednesday afternoons. Holiday entitlement will be based pro rata on hours worked, from a full-time allowance of 30 days per year, plus bank holidays. Salary is subject to experience. You will be eligible for company health insurance and membership of our employee share scheme after the qualifying period.

If you’re interested or would like to know more, drop us an email.

The secret to great technical support? No support staff.

October 21st, 2022 by

Over the years, we’ve gained a reputation for providing support that is above average for the hosting industry. Obviously it helps that the average is really quite low, and simply providing helpful answers in a timely manner puts you some way above it, but we’re proud of this reputation and work hard to provide the best support that we possibly can.

So what do we do differently?

Perhaps the biggest thing is that we don’t have any dedicated support staff.

Our support rota

Our support queue is staffed by a rolling rota that includes all of our technical staff. The staff responsible for managing our routers, running our DNS servers, developing our control panel and maintaining all our other infrastructure, all take it in turns to do regular days on “first line support”. And, yes, this includes our founders & directors.

The most obvious benefit of this is that customers get straight through to someone who can actually deal with their issue — all tickets are effectively escalated to what might elsewhere be considered second, or more likely third, line support, but without the hassle of fighting your way past chat bots and scripted replies.

XKCD 806

There’s no need to say “Shibboleet” to our staff.

That’s obviously better for the customer, but conventional wisdom is that good technical staff are too expensive to put on first line support, and you won’t retain them if you do.

Our company trades on its reputation for good support, so cost cutting here would be a false economy, and you only have to look at the likes of Stack Overflow and Quora to see that many technical experts enjoy using their knowledge to help others.

It is true that our staff probably wouldn’t want to do support full-time, but mixing support with normal responsibilities actually provides some useful variety, and has a number of other benefits.

Direct customer feedback

One of the most valuable benefits of this arrangement is the direct contact between our technical staff and our customers. Our staff get to see directly what our customers want to do, and what parts of our website and systems our customers find confusing. They’ve also got a strong incentive to improve them so that they don’t find themselves answering the same simple questions again and again when on support, and because our “support staff” are also the people responsible for those systems, they’re in a position to actually fix them.

Perhaps one of the best measures of how well this works is that the average time to deal with a support ticket has gone up over the years. All the easy support tickets that we used to be able to clean up before the first coffee in the morning have gone, because the customer did it themselves the night before. The tickets in the support queue are getting harder, and this is good thing (and yet another lesson in the hazards of optimising for KPIs).

Why we prefer email support

Our rolling rota of support staff is one of the reasons why we insist on email for support. Having a written record of all communications on a ticket makes it much easier to hand tickets from one person to the next. Customers don’t have to spend time explaining an issue each time it’s passed to a different member of staff – although for more complicated tickets, we do quite often ask the person who first picked it up to carry on with it, even if they’re no longer on support.

How far will this approach scale?

We’ve operated this system for quite a few years and the amount of time we spend dealing with support queries has grown steadily with the company.

We’ve no plans to change this approach, but it’s quite possible that there will come a point where it makes sense to hire staff whose primary role is support. Like all things, the more you do, the better you get, and one of the costs of our approach is that using non-dedicated staff is inefficient — they’re more likely to have to look things up or check with colleagues when responding to tickets.

We have already taken the step of splitting out finance-related support tickets into a separate queue, which is dealt with by our finance staff.

If we do ever take that step of employing dedicated support staff we won’t compromise on the quality of support that we provide, and it’s likely to be in addition to, rather than instead of, our rolling rota, because of the benefits it provides to both us and our customers.

New data centre presence: City Lifeline

May 27th, 2022 by

The rest room has a nice view, proper coffee and our branded mugs


In June last year, Digital Realty informed us that they planned to close the Meridian Gate (MER) data centre in 2023. Meridian Gate is our largest presence, so initially this seemed like really bad news. Moving data centres is such a daunting – and expensive – prospect that we’d never really consider it on its own, even if there are long term cost savings or technical benefits. But, once you’re forced to do it, it becomes a rare opportunity to do the kind of upgrades, reorganisation and general tidying that’s so hard to do in racks full of live servers.

Since the announcement we’ve been working hard to figure out not only how to replace the space in MER, but also how to make the most of this chance to configure and kit out new space exactly as we want it.

A key part of the plan is taking on a presence in a new London data centre so that we retain three separate sites in London, and we’re very pleased to announce that our new suite in Redcentric’s City Lifeline (CLL) data centre in Shoreditch is now live, and that our migration out of MER is well underway.

Our CLL presence is connected back to our other two London data centres, Digital Realty’s Sovereign House (SOV) and Equinix LD8 (aka Harbour Exchange/HEX), via a lit fibre ring. The new space allows us to offer dual, redundant 10Gbps to servers, as well as dual redundant power feeds. As with all our data centre space, we have switched PDUs, enabling power to be remotely controlled via our control panel, and remotely accessible serial consoles, so that almost all server issues can be resolved remotely.

If you have services in MER and haven’t already heard from us we’ll be in touch soon to discuss migration plans. We’ve been working hard behind the scenes to minimise disruption to services from the migration out of MER. This includes network upgrades to enable IP portability between MER and CLL so that servers will not need to change IPs during the move and our team are doing a lot of late nights to reduce the impact of any unavoidable disruption.

If you’re interested in taking on new colocated or dedicated servers, please do get in touch as we’ve now got lots of capacity.