HEX-it

September 27th, 2023 by

Last year, we undertook a significant data centre migration, with the closure of Digital Realty’s Meridian Gate requiring us to move our entire presence there to Redcentric’s City Life Line. Having done it once, why not do it again?

Southern Serval, leaping

Our shared hosting server “serval” has already migrated to SOV. [ Photo by Wynand Uys]

This year, we’re planning a move out of Harbour Exchange (HEX), and starting a presence in Telehouse South. A lot of the things we learned during the previous move are making this move easier to manage, although it is still a prodigious effort, both physically and in terms of design and infrastructure.

One of the things we’ve been working on for some time is improved network infrastructure within our data centres. This introduces IP address portability so that IP addresses do not need to change when servers are moved between data centres, as well as significantly higher bandwidth uplinks for our virtual server hosts.

In the last year, we’ve live migrated over a thousand VMs across two data centres, with minimal interruption to service.

We’re about to start migrating all VMs out of our HEX data centre. We have two available London destinations, SOV and CLL. If you’re a customer with a VM in our HEX data centre, we’ll be emailing you over the next couple of weeks, to check if you have a preference (for instance because you have existing services in one of those data centres, and would prefer to be moved to the other to maintain fault-tolerance).

We will also soon be able to offer Telehouse South as a virtual server zone, in addition to SOV and CLL. This means we will continue to provide three London-based zones for our customers running distributed services. We’ll retain a small residual presence in HEX for connectivity.

Finance and Administrative Assistant

November 25th, 2022 by

Invoices, contracts, cheques and a free company mug.

We’re looking to employ a part-time finance and administrative assistant for between 15 and 25 hours a week. The duties of the role will likely include:

  • dealing with invoicing and payment queries from customers;
  • reconciling bank transfers with invoices;
  • uploading receipts to Hubdoc and Xero;
  • chasing up overdue invoices;
  • dealing with some paper mail to the company;
  • banking cheques;
  • filing paperwork;
  • administering annual leave;
  • keeping a calendar of administrative deadlines; and
  • other administrative and accounting duties.

Previous experience in a similar role is desirable, as is experience with Xero accounting software.

Mythic Beasts don’t have an office, so the job primarily involves working from home. We’ll provide you with a laptop and cover reasonable home-working expenses. Hours are flexible, but we would normally expect you to be available in Cambridge on Wednesday afternoons. Holiday entitlement will be based pro rata on hours worked, from a full-time allowance of 30 days per year, plus bank holidays. Salary is subject to experience. You will be eligible for company health insurance and membership of our employee share scheme after the qualifying period.

If you’re interested or would like to know more, drop us an email.

Testimonials

February 5th, 2021 by

We’ve had a variety of customer being very complimentary recently. Andy Steven runs a series of web cams in the Shetland Islands that stream live views of the northern lights. The cameras relay the stream via one of our virtual servers in our MER data centre and the current bandwidth record is several Gbps.

I am proud to say that our new ‘AuroraCam’ network just delivers and for the first time I no longer break out in a sweat watching the demand increase from that AuroraWatchUK alert or a celebrity weather personality sending out a Tweet.

— Andy Steven, Shetland Webcams (full article)

Beautiful shot of the northern lights captured by Shetland Webcams. Could be improved by adding a kitten though.

We provide 10Gbps fibre connectivity to the Cambridge office of DarkTrace. Darktrace uses machine learning to identify and neutralise security threats in real time.

You’ve been much more transparent & approachable than any provider I’ve dealt with previously. Very happy with the service so far.

— Harry Godwin, Head of Business Infrastructure. Darktrace

The Web hosting review and advice site Hosting Advice interviewed us and wrote a great article about the management and infrastructure services we provide.

Recognizing that there is no one-size-fits-all approach to managed hosting, Mythic Beasts can take on varying responsibility levels as needed. This range of services includes everything from ensuring that servers are up and running to providing the extensive monitoring, security, and assistance necessary to keep custom web applications functioning reliably.

— Hosting Advice (full article)

Lastly our strong stance about returning Nominet to its public benefit roots garnered entirely positive responses at Twitter.

 

 

Covid 19 update

March 27th, 2020 by

Microscopy image of the coronavirus.
(Image copyright NIAID; licensed under CC-BY 2.0)

Covid-19 has dramatically changed life in the UK, and the lock-down announced on Monday has lead to further changes to how we, and our customers, are operating.  This page provides an update to our previously announced Covid-19 plan.

We’re happy to report that our staff member who had some of the symptoms of covid-19 is now fully recovered and has returned to our team after a few days of rest.

Data centre access changes

Our data centre suppliers have altered their operation: 24/7 walk in access is now prohibited and every visit needs booking and justification. The “remote hands” service remains available, but at a reduced capacity as they’ve moved as many staff as possible to home working to minimise their risks.

Equinix, who supply two of our core facilities, have completely closed their facilities in Italy, Germany, France and Spain to customers. Changes are only possible via their remote hands service in those countries.  We have a significant amount of equipment in two Equinix facilities in London and Amsterdam (LD8 and AM5).

Since this announcement on Sunday, we have been anticipating a similar closure being applied to London.  This has  now been announced and will be disruptive for us, as our normal operating procedures rely on being able to move spare equipment easily between the London data centres where we have a presence.  We have been taking steps to increase the spare hardware that we have at each of our London sites in order to mitigate the impact of this when it comes into force on Tuesday 31st March.

The data centres have well-considered policies to reduce risk, and to handle a confirmed case within the facility with rapid quarantine and deep clean procedures. No access is allowed to customers showing symptoms, and all customers’ temperatures are measured before entry to the facilities. Their operations are also robust in the event they need to manage the facility remotely for a period of time.

We’ve also altered our own policy for data centre access. Customer access to our data-centre space is suspended until further notice, including for documentation-related audits (e.g. ISO27001 compliance). This should have a minimal impact; we only allowed accompanied access previously and visits have always been exceedingly infrequent.

Items shipped to us will be quarantined for 24 hours before opening. Cardboard will self-decontaminate in 24 hours.

Staff members may not meet in a data centre unless it is specifically for a piece of work that requires two people for safety reasons (typically very heavy server deployment), and only if is to maintain an essential service. Staff members may not visit multiple key data centres in a single visit to minimise the risk of transmission between key sites, and may not visit if they are showing symptoms. Data centre visits are being minimised to reduce infection risk. This may limit the range of dedicated servers we are able to provision, and we have decided to stop offering Mac Minis with OS X due to the difficulty of provisioning them remotely.

Customer support

Unsurprisingly, a wholesale shift of the UK to remote working has a significant impact on all kinds of online systems, which are now critical for day to day operations. We’re supporting existing customers to make this transition, as well as provisioning new orders for services that now need to be in the cloud.

We run a system for POhWER that is used by all their advisors to track their cases. This is a critical system; if it’s offline, hundreds of people are unable to work. We maintain this as an active/standby pair split across two of our data centres.

“You’re supporting us to enable our vital work with the most vulnerable people in society to continue in these very trying times and, through your swift upgrade actions, our new fully remote working model is delivering the information, advice and advocacy our clients depend on.”
Sandra Black, Head of Training, Risk and Quality at POhWER.

The shift to remote working means that usage on their system has approximately doubled in ten days and has started to see performance limitations. We identified the bottleneck and proposed a cost-effective upgrade combined with some configuration improvements. We then made staff available to apply the changes in an emergency late night maintenance window, restoring their site to full performance by the next working day.

Direct efforts

We were approached at the weekend by a small team comprising local IT experts and doctors who are building an information website to efficiently distribute information to NHS staff members about how to use and select the correct protective equipment for the environment they are working in. We’re providing the virtual server, security updates, backups and 24/7 monitoring service for this free of charge, which has allowed the volunteer IT experts to concentrate on building the site. We’re expecting go-live in the next day or so once the content is checked.

We’re keen to hear of any other efforts where we may be able to assist.

Adding capacity

We’ve ordered more servers to expand the three busiest VM clouds to support existing customers scaling up, and new customers with urgent needs. We want to avoid cloud full and thankfully our server supplier is fully able to continue to build and deliver servers whilst maintaining 2m spacing between employees.

Out standing in a field

May 24th, 2019 by

Mythic Beasts: out standing in a field

Last year the Cambridge Beer Festival tried accepting payments by contactless cards. This didn’t work very well. They built a wireless LAN around the bar so that their card payment machines could process transactions. This went to an uplink that was a Raspberry Pi with a 4G dongle attached, this wasn’t really reliable enough for a full payment system, but worked as a proof of concept.

To improve things for this year we had a conversation with some friends at the recently incorporated Light Blue Fibre Ltd and between us were able to arrange for Jesus Green to have a fibre and an interlink to Mythic Beasts. As this is a prototype, we’re running below optimum speeds so we’ve delivered a relatively leisurely 1Gbps to the festival. The access points will happily deliver 150Mbps symmetric at any point on the bar if you have a quick enough wifi card in your laptop. We’ve still got the 3G uplink enabled as a backup just in-case someone slices the fibre.

If my phone had an Ethernet socket we’d be ten times as fast.

This year the plan was to restrict things to the tills and the administration network. However, being techies in a beer festival there is a tiny chance we may have been slightly drunk and enabled public wifi with a 100Mbps rate limit. This works well around the bar but there’s nowhere near enough access points to cover the outdoors and the onsite router is limited to 500 devices. It’s not yet production ready for 5,000 beer-drinking visitors, but we have a beer mat and a pencil and we’re sketching out ideas for next year.

Mythic Beasts acquires VMHaus

November 26th, 2018 by

Our pet wyvern was hungry again.

We’re pleased to announce that Mythic Beasts has acquired VMHaus, a virtual server provider with facilities in London and Los Angeles. We will continue to run VMHaus as a separate brand selling low-cost, prepaid virtual servers, which we believe will complement our own virtual server products well. We’re also pleased to announce that VMHaus co-founder Basil Fillan has joined Mythic Beasts as a full time employee. Basil has been responsible for the development of the VMHaus technical infrastructure, and will be ideally placed to help us provide support to VMHaus customers.

In the short term, VMHaus customers will see no changes to their services. Payments and invoicing will continue to be through VMHaus Ltd, and we will continue to accept new orders for VMHaus products. In the medium term, we’re planning improvements to both the VMHaus platform and our own virtual server infrastructure, based on our combined experiences of developing the two systems.

On the VMHaus side we hope to be able to start selling virtual servers in Amsterdam early in the new year, and also be able to offer IPv6-only virtual servers at a discounted rate. VMHaus customers will also be able to take advantage of our other services such as domain registration and backups.

On the Mythic Beasts side, we expect to be able to offer service upgrades thanks to the economies of scale resulting from the acquisitions of VMHaus and of BHost this summer.

OpenWRT install to RAM – run iftop on a router with very limited flash

November 23rd, 2018 by

OpenWRT is awesome, as it allows you to run proper Linux tools on your home router. I’m currently using a very old, underspecced TP Link box, with 32MB of RAM, but just 4MB of flash storage. This is just enough to get what I need installed, but one thing I’ve always wanted to do is use iftop to quickly see what’s using all the bandwidth. Unfortunately iftop, with its dependencies on libpcap and libncurses, just won’t fit into a 4MB image.

I recently stumbled across opkg’s install-to-RAM option, allowing me to use the 32MB of RAM to install the package, with the minor and obvious downside that it gets uninstalled when the router gets rebooted. For something like iftop, which is used for ad-hoc diagnostics, this isn’t a big issue.

Installing to RAM puts the packages under /tmp, so a little effort is required to make sure that libraries and other resources can be found. I now have the following shell script which installs iftop if it isn’t already, sets some environment variables and invokes iftop:

#!/bin/sh

if [ ! -f /tmp/usr/bin/iftop ] ; then
  opkg update
  opkg install -d ram iftop
fi

export LD_LIBRARY_PATH=/tmp/usr/lib
export TERM=xterm
export TERMINFO=/tmp/usr/share/terminfo/

/tmp/usr/bin/iftop $@

Fortunately I do have enough free space on flash storage to store the above script.
Obviously a similar approach could be used with other packages that are only needed “on demand”.

libssh emergency update

October 17th, 2018 by

An attack so simple my cat could get root on your server.

Managed customers of Mythic Beasts with libssh installed will have just received a notification that we updated it without warning or testing.

This is obviously bad practice, so what were we thinking?

A security advisory for libssh has just come out which is very bad. To paraphrase,

libssh -> hello new user
user -> can I have a root shell
libssh -> can you authenticate?
user -> yes but I'm not going to
libssh -> okay, have a root shell

This is completely secure, unless the client is prepared to lie in order to exploit your system. In the late 1990s some of our founders might have once exploited an online quiz in exactly the same way to get perfect scores. Don’t trust the client.

In our risk analysis, the risk of breakage to a customer site though a botched patch is vastly lower than giving an attacker a root shell, which is why we pushed an emergency update within a few hours of updated packages being available.

If this is the first you’ve heard about the issue, we suggest you’d benefit from our Managed Services

Toby Goodwin (1968-2018)

October 5th, 2018 by

At Mythic Beasts we rotate staff members around different roles. This is to protect the company from the unlikely event that a staff member is abducted by aliens and someone else has to take over at short notice.

With great sadness we have to report that Toby Goodwin, our first full time employee was not abducted by aliens. Much worse, he had an undiagnosed asymptomatic heart problem and passed away unexpectedly and painlessly last week.

Back in 2010 Toby had been running a bookshop in Cambridge with a quirky and eclectic selection of books. That business had come to an end and Toby was wondering about dusting off his UNIX skills and looking for work. At the same time Mythic Beasts had grown too large for the two then-active founders to effectively keep up and after an interview over a beer in the Devonshire Arms, Toby joined Mythic Beasts.

We didn’t initially realise how lucky we were because Toby had the perfect blend of skills. An experienced UNIX hacker from his days at Cygwin, he quickly figured out most of the technical operations to keep Mythic running. Meanwhile his experience at the bookshop gave him incredible patience and empathy for confused customers. He took it on himself to continuously improve our operations introducing radical new ideas like helper scripts having consistent names to make them easy to find, continuous integration and automated testing of our control panel.

Toby implemented the bulk of our managed server update system. When he started, we had tens of managed customers and updating packages was starting to become time consuming. Gradually this became a highly reliable and flexible system which means we can audit and update thousands of servers quickly and efficiently, whilst correctly notifying every affected customer in a timely fashion. Toby was always modest about his achievements and never suffered from being defensive about his code. When our summer students discovered a significant security flaw in a piece of configuration, he congratulated them and worked with them to resolve it quickly.

After working with us for a few years in Cambridge, Toby met Heather and moved with her to her native Scotland where they married and brought into the world a highly reliable early morning alarm clock called Zachary. Toby would regularly work early in the morning before taking some time out to deliver Zachary to nursery or work with him on significant structural engineering projects.

.

In addition to being a skilled software developer, Toby was also a brilliant railway engineer in the face of feline opposition.

Goodnight Toby. We’ll miss you.

Mythic Beasts acquires BHost

July 1st, 2018 by

Having a hungry Wyvern in our logo makes eating other companies much easier to draw.

Hot on the heels of acquiring Retrosnub, we’ve also bought the customers and assets of BHost. BHost are a virtual server provider with services in London, Amsterdam and California based on OpenVZ and KVM.

We’re excited about this acquisition as it provides us with a great opportunity to expand our network using BHost’s Amsterdam infrastructure. At the same time, we’re confident that we can provide some immediate and longer term improvements to the BHost service, not least through our larger support team being able to offer more timely and helpful responses to customer queries.

Although handover officially happened today, BHost customers have had access to our control panel for several weeks, mostly so that we could start tackling EU VAT bureaucracy. BHost are a US-registered business. We’re a VAT-registered business in the UK. Thanks to VAT MESS, it’s actually much harder for us to sell to EU-based consumers than it was for BHost, as we’re required to collect an unreasonable amount of evidence of customer location.

The good news for BHost customers is that we’re matching BHost’s current pricing with our UK VAT-inclusive price. This means that EU VAT-registered businesses, and all non-EU customers will see a significant reduction in the price that they pay.

If you’re a BHost customer and you’ve not already done so, please log in to our customer control panel using your BHost username (email address) and password and confirm your contact details.

Network Expansion

BHost run a network from London/Amsterdam with multiple 10Gbps uplinks and some peering in each site. We will be moving the BHost London network behind our own so that BHost customers can take advantage of our larger capacity uplinks and significantly better peering arrangements, which includes transit-free connections to every major UK ISP.

We’re also taking the opportunity to significantly improve the connectivity to our Cambridge data centre. We currently have two uplinks via different London data centres. We will replace one of these links with a direct connection to Amsterdam, and bring both up to 10Gbps. Combined with BHost’s existing London/Amsterdam connection, this will create a 10Gbps ring around London, Cambridge and Amsterdam, complementing our 50Gbps ring around our three London sites. This will provide increased bandwidth and improved resiliency for our Cambridge customers, whilst also providing a second London/Amsterdam link to improve resilience within the BHost network.

BHost Amsterdam customers will gain direct UK connectivity through our extensive London peering. We will gain the Amsterdam Internet Exchange connection (AMSIX) from BHost, bringing improved European connectivity to all London customers. We expect to be able to substantially increase the number of AMSIX peers, improving EU connectivity for all customers.

Cloud expansion

BHost’s London presence is in the Meridian Gate (MER) data centre. We already have a significant footprint in MER, although it’s not currently available as a zone in our public cloud. We’re investing in new hardware to deploy in Meridian Gate which is both substantially faster and more power efficient than the current hosts. We’ll be deploying this into our existing suite in MER, and then migrating BHost servers into it. BHost customers will see a small window of scheduled downtime as we migrate each server, but should then seen significantly improved performance on the new hardware.

Our Amsterdam and US presences will give additional options to customers that need to be physically hosted within the (post-Brexit) EU or US. We expect this to become more relevant after Brexit when the UK and EU may have diverging regulatory requirements.

Additional services

All BHost customers can now take advantage of additional Mythic Beasts services including management services for virtual servers, domain registration and DNSSEC-enabled, API-backed DNS hosting.

Support

Mythic Beasts have a larger support team and we’re very well placed to provide significantly improved customer service to all of our new customers. Of course, we do expect the period immediately after the transition to be very busy as customers become familiar with the new billing arrangements, and we get to grips with supporting BHost’s services. We will have additional staff during this period, but please be patient if support responses are a little slower than usual.