HEX-it

September 27th, 2023 by

Last year, we undertook a significant data centre migration, with the closure of Digital Realty’s Meridian Gate requiring us to move our entire presence there to Redcentric’s City Life Line. Having done it once, why not do it again?

Southern Serval, leaping

Our shared hosting server “serval” has already migrated to SOV. [ Photo by Wynand Uys]

This year, we’re planning a move out of Harbour Exchange (HEX), and starting a presence in Telehouse South. A lot of the things we learned during the previous move are making this move easier to manage, although it is still a prodigious effort, both physically and in terms of design and infrastructure.

One of the things we’ve been working on for some time is improved network infrastructure within our data centres. This introduces IP address portability so that IP addresses do not need to change when servers are moved between data centres, as well as significantly higher bandwidth uplinks for our virtual server hosts.

In the last year, we’ve live migrated over a thousand VMs across two data centres, with minimal interruption to service.

We’re about to start migrating all VMs out of our HEX data centre. We have two available London destinations, SOV and CLL. If you’re a customer with a VM in our HEX data centre, we’ll be emailing you over the next couple of weeks, to check if you have a preference (for instance because you have existing services in one of those data centres, and would prefer to be moved to the other to maintain fault-tolerance).

We will also soon be able to offer Telehouse South as a virtual server zone, in addition to SOV and CLL. This means we will continue to provide three London-based zones for our customers running distributed services. We’ll retain a small residual presence in HEX for connectivity.

New data centre presence: City Lifeline

May 27th, 2022 by

The rest room has a nice view, proper coffee and our branded mugs


In June last year, Digital Realty informed us that they planned to close the Meridian Gate (MER) data centre in 2023. Meridian Gate is our largest presence, so initially this seemed like really bad news. Moving data centres is such a daunting – and expensive – prospect that we’d never really consider it on its own, even if there are long term cost savings or technical benefits. But, once you’re forced to do it, it becomes a rare opportunity to do the kind of upgrades, reorganisation and general tidying that’s so hard to do in racks full of live servers.

Since the announcement we’ve been working hard to figure out not only how to replace the space in MER, but also how to make the most of this chance to configure and kit out new space exactly as we want it.

A key part of the plan is taking on a presence in a new London data centre so that we retain three separate sites in London, and we’re very pleased to announce that our new suite in Redcentric’s City Lifeline (CLL) data centre in Shoreditch is now live, and that our migration out of MER is well underway.

Our CLL presence is connected back to our other two London data centres, Digital Realty’s Sovereign House (SOV) and Equinix LD8 (aka Harbour Exchange/HEX), via a lit fibre ring. The new space allows us to offer dual, redundant 10Gbps to servers, as well as dual redundant power feeds. As with all our data centre space, we have switched PDUs, enabling power to be remotely controlled via our control panel, and remotely accessible serial consoles, so that almost all server issues can be resolved remotely.

If you have services in MER and haven’t already heard from us we’ll be in touch soon to discuss migration plans. We’ve been working hard behind the scenes to minimise disruption to services from the migration out of MER. This includes network upgrades to enable IP portability between MER and CLL so that servers will not need to change IPs during the move and our team are doing a lot of late nights to reduce the impact of any unavoidable disruption.

If you’re interested in taking on new colocated or dedicated servers, please do get in touch as we’ve now got lots of capacity.

IPv6 Deployment World Leader

December 8th, 2021 by

Yesterday (7th December) we attended the virtual IPv6 forum annual meeting. We were delighted that our director Pete Stevens has been added to the IPv6 Hall Of Fame as an IPv6 Deployment World Leader.

Unlike most awards we turn down, you can’t win this one just by paying for a hugely expensive table at an awards ceremony.

We also got an update on how IPv6 deployment is going through the UK. Happy to hear from BT that they’re making excellent progress replacing all the old HomeHubs with new IPv6-capable consumer routers. Sky Italia has deployed a consumer broadband network that’s effectively IPv6 only – IPv4 is provided as a service on top with MAP-T. As this is a form of carrier NAT they’ve managed one IPv4 address per 16 subscribers. This compares with their initial dual stack rollout we reported on from the 2019 council meeting.

Lastly it was noted that the cost of an IPv4 address on the open market is now around $60; increasing numbers of server providers are following our lead and making an IPv4 address an additional and removable option on the order form.

Teaching our network some MANRS

April 30th, 2021 by


We’ve recently rolled out software upgrades to our networks that enable improved routing security and we have joined the MANRS (Mutually Agreed Norms for Routing Security) initiative.

Our MANRS certification for our EU and US networks confirms that we block spoofed traffic, drop incorrect routing information and work with others to maintain routing security.

This is beneficial for any customer using our transit and IP services, which includes all dedicated server and virtual server customers.

Resource Public Key Infrastructure (RPKI)

Amazingly, up until the advent of RPKI the entire internet worked on a trust relationship. When another network told us that they were responsible for a range of internet addresses we’d believe them. Border Gateway Protocol (BGP) is how networks communicate routing data to each other and it had no mechanism to confirm that the route and address space being advertised to you were genuine.

Incorrect advertisements result in network traffic being delivered to the wrong destination and incidents, both deliberate and accidental, are common and can cause real harm. For example, $17m in crypto currency was stolen in 2018 via an IP address hijack aimed at Amazon. Youtube has been taken offline as have large parts of the Cloudflare network.

RPKI seeks to address this by providing signed proof that a network operator (identified by their Autonomous System Number) is permitted to originate a specific range of IP addresses. Once a range of IP addresses is signed you know that any announcement of the address space from any other provider is invalid and should be dropped.

Our transit providers are also certified by MANRS for further protection.

An RPKI example

RIPE Labs have created a deliberately invalid routing announcement that can be used to demonstrate and test RPKI. RIPE Labs have published a Resource Origination Authorisation (ROA) that says only AS0 is permitted to announce the prefix 209.24.0.0/24. They then announce that prefix under AS15562.

With RPKI we see that the network listed in the ROA does not match the network announcing the route, so that route is considered invalid and rejected as being a hijack.

Ripe Labs have published a checker that runs in your browser and detects whether you can see this invalid route on your ISP’s network.

From our network, we now get the big smiley face:

Internet Resource Registry (IRR)

RPKI complements another approach to routing security: filtering based on Internet Resource Registry (IRR) data. RPKI allows us to verify if a network is a valid ultimate destination for a particular IP range. Most networks we don’t see directly, we go through another transit providing network. IRR allows us to verify that the network advertising a given route is authorised to originate or transit that route.

The Regional Internet Registries (RIR) allow network providers to register a link between their network and an IP block. Various tools exist (e.g. bgpq3) to create a list of all the internet addresses that a network can originate or transit from their downstream customers. This is be used to generate a filter list that restricts what routes we will accept from peers and downstream customers.

These lists can be very long and change frequently – the list for our network (AS-MYTHIC) is usually 5000 or so records with tens to hundreds of changes per day.

Best Common Practice 38 (BCP 38)

Another issue with insecure routing is “spoofing” — sending IP packets with a fake source address. This is widely used by attackers to cause denial of service attacks. An attacker sends packets with a sender IP address faked to be that of the target machine. The recipient of these packets will send replies to the target machine instead of the originator. This makes it very easy to create distributed denial of service attacks.

BCP38 is a Best Common Practice which requires that networks filter packets that aren’t either to or from an address within their network.

Part of MANRS is not only to implement BCP 38 but also to host an active spoofer. This means if we drop our BCP38 filtering our non-compliance will be published including regular mailings to network operator groups.

Having good MANRS

By combining all these methods routing security is significantly improved. RPKI provides dynamic checking that doesn’t rely on us adding static route lists to our routers. This also provides excellent protection against accidental hijacks from a “route optimiser” gone wrong. IRR forces accurate routing data to generate filters. BCP38 reduces risks to other networks from spoofed packets. Combining all of these means we have much better MANRs at the price of terrible acronyms.

RPKI filtering is now fully deployed on our US and European network and they both now pass Cloudflare’s “Is BGP Safe Yet” test.

IPv4/IPv6 transit in HE Fremont 2

September 18th, 2020 by

Back in 2018, we acquired BHost, a virtual hosting provider with a presence in the UK, the Netherlands and the US. Since the acquisition, we’ve been working steadily to upgrade the US site from a single transit provider with incomplete IPv6 networking and a mixture of container-based and full virtualisation to what we have now:

  • Dual redundant routers
  • Two upstream network providers (HE.net, CenturyLink)
  • A presence on two internet Exchanges (FCIX/SFMIX)
  • Full IPv6 routing
  • All customers on our own KVM-based virtualisation platform

With these improvements to our network, we’re now able to offer IPv4 and IPv6 transit connectivity to other customers in Hurricane Electric’s Fremont 2 data centre. We believe that standard services should have a standard price list, so here’s ours:

Transit Price List

Prices start at £60/month on a one month rolling contract, with discounts for longer commits. You can order online by hitting the big green button, we’ll send you a cross-connect location within one working day, and we’ll have your session up within one working day of the cross connect being completed. If we don’t hit this timescale, your first month is free.

We believe that ordering something as simple as IP transit should be this straightforward, but it seems that it’s not the norm. Here’s what it took for us to get our second 10G transit link in place:

  • 24th April – Contact sales representative recommended by another ISP.
  • 1st May – Contact different sales representative recommended by UKNOF as one of their sponsors.
  • 7th May – 1 hour video conference to discuss our requirements (a 10Gbps link).
  • 4th June – Chase for a formal quote.
  • 10th June – Provide additional details required for a formal quote.
  • 10th June – Receive quote.
  • 1st July – Clarify further details on quote, including commit.
  • 2nd July – Approve quote, place order by email.
  • 6th July – Answer clarifications, push for contract.
  • 7th July – Quote cancelled. Provider realises that Fremont is in the US and they have sent EU pricing. Receive and accept higher revised quote.
  • 10th July – Receive contract.
  • 14th July – Return signed contract. Ask for cross connect location.
  • 15th July – Reconfirm the delivery details from the signed contract.
  • 16th July – Send network plan details for setting up the network.
  • 27th July – Send IP space justification form. They remind us to provision a cross connect, we ask for details again.
  • 6th August – Chase for cross connect location.
  • 7th August – Delivery manager allocated who will process our order.
  • 11th August – Ask for a cross connect location.
  • 20th August – Ask for a cross connect location.
  • 21st August – Circuit is declared complete within the 35 day working setup period. Billing for the circuit starts.
  • 26th August – Receive a Letter Of Authorisation allowing us to arrange the cross connect. We immediately place order for cross connect.
  • 26th August – Data centre is unable to fulfil cross connect order because the cross connect location is already in use.
  • 28th August – Provide contact at data centre for our new provider to work out why this port is already in use.
  • 1st September – Receive holding mail confirming they’re working on sorting our cross connect issue.
  • 2nd September – Receive invoice for August + September. Refuse to pay it.
  • 3rd September – Cross connect location resolved, circuit plugged in, service starts functioning.

Shortly after this we put our order form live and improved our implementation, we received our first order on the 9th September and provisioned a few days later. Our third transit customer is up and live, order form to fully working was just under twelve hours; comfortably within our promise of two working days.

VMHaus services now available in Amsterdam

July 3rd, 2019 by

Integration can be hard work

Last year we had a busy time acquiring Retrosnub, BHost and VMHaus. We’ve been steadily making progress in the background integrating the services the companies provide to reduce costs and complexity of management. We can now also announce our first significant feature upgrade for VMHaus. We’ve deployed a new virtual server cluster to our Amsterdam location and VMHaus services are now available in Amsterdam. VMHaus is using Mythic Beasts for colocation and network and in Amsterdam they will gain access to our extensive set of peers at AMSIX, LINX and LoNAP. Per hour billed virtual servers are available from VMHaus with payment through Paypal.

As you’d expect, every VM comes with a /64 of IPv6 space.

In the background we’ve also been migrating former-BHost KVM-based services to Mythic Beasts VM services in Amsterdam. Shortly we’ll be starting to migrate former-BHost and VMHaus KVM-based services in London to new VM clusters in the Meridian Gate data centre.

Mythic Beasts gaan naar Nederland

February 20th, 2019 by

The art warehouses in Amsterdam look much prettier than the data warehouses.

Back in July 2018, Mythic Beasts acquired Bhost, giving us additional virtual machine (VM) clusters in London, Amsterdam and California.

Today we’re pleased to announce that we’ve deployed a substantial new VM cloud to Amsterdam, running our own VM platform. Virtual machines in Amsterdam are available to purchase immediately through our website in sizes from 1GB/1vCPU to 160GB/12vCPUs, and with both SSD and spinning rust disk options. Server management and backup options are also available.

Thanks to Brexit-related regulatory uncertainty, some of our existing clients informed us that they must be hosted outside of the UK before 29th March. Deploying capacity on our own platform in Amsterdam means that we can migrate virtual servers directly to the new location.

Once we’ve dealt with the immediate Brexit-driven server moves, we’ll be looking at migrating former-Bhost VMs into this new cloud, giving a significant performance boost in the process.

Deploying the Amsterdam VM cloud is a significant milestone in the integration of the Bhost infrastructure into our own. The integration provides improved performance and redundancy for both Mythic Beasts and Bhost customers whilst simultaneously cutting our operating costs. In preparation for this, we completed upgrades to our core network last October. The existing fibre ring around our three main London sites, which is currently lit at 50Gbps, is now complemented by a 10Gbps ring around London (HEX) ⟺ Cambridge ⟺ Amsterdam ⟺ London (MER). This replaces the old 2x1Gbps connectivity from Cambridge to London with diverse 10Gbps feeds to London and Amsterdam. Our network has gained an additional 10Gbps transit in Amsterdam (NTT) and we are also now connected on the Amsterdam Internet Exchange (AMS-IX).

On a trip to deploy new routers, Pete even managed a tour of the city on foot in just over three hours.



Primary reasons for choosing Amsterdam include being a flat country that’s easy to cycle around, a remarkably nice overnight ferry journey and superb boy bands asking us to stay. Secondary reasons are all boring such as a well developed market for data centres and internet transit, a world class internet exchange and remarkably few insane British politicians. We’re looking forward to the first Anglo-Dutch cricket match.

Mythic Beasts acquires BHost

July 1st, 2018 by

Having a hungry Wyvern in our logo makes eating other companies much easier to draw.

Hot on the heels of acquiring Retrosnub, we’ve also bought the customers and assets of BHost. BHost are a virtual server provider with services in London, Amsterdam and California based on OpenVZ and KVM.

We’re excited about this acquisition as it provides us with a great opportunity to expand our network using BHost’s Amsterdam infrastructure. At the same time, we’re confident that we can provide some immediate and longer term improvements to the BHost service, not least through our larger support team being able to offer more timely and helpful responses to customer queries.

Although handover officially happened today, BHost customers have had access to our control panel for several weeks, mostly so that we could start tackling EU VAT bureaucracy. BHost are a US-registered business. We’re a VAT-registered business in the UK. Thanks to VAT MESS, it’s actually much harder for us to sell to EU-based consumers than it was for BHost, as we’re required to collect an unreasonable amount of evidence of customer location.

The good news for BHost customers is that we’re matching BHost’s current pricing with our UK VAT-inclusive price. This means that EU VAT-registered businesses, and all non-EU customers will see a significant reduction in the price that they pay.

If you’re a BHost customer and you’ve not already done so, please log in to our customer control panel using your BHost username (email address) and password and confirm your contact details.

Network Expansion

BHost run a network from London/Amsterdam with multiple 10Gbps uplinks and some peering in each site. We will be moving the BHost London network behind our own so that BHost customers can take advantage of our larger capacity uplinks and significantly better peering arrangements, which includes transit-free connections to every major UK ISP.

We’re also taking the opportunity to significantly improve the connectivity to our Cambridge data centre. We currently have two uplinks via different London data centres. We will replace one of these links with a direct connection to Amsterdam, and bring both up to 10Gbps. Combined with BHost’s existing London/Amsterdam connection, this will create a 10Gbps ring around London, Cambridge and Amsterdam, complementing our 50Gbps ring around our three London sites. This will provide increased bandwidth and improved resiliency for our Cambridge customers, whilst also providing a second London/Amsterdam link to improve resilience within the BHost network.

BHost Amsterdam customers will gain direct UK connectivity through our extensive London peering. We will gain the Amsterdam Internet Exchange connection (AMSIX) from BHost, bringing improved European connectivity to all London customers. We expect to be able to substantially increase the number of AMSIX peers, improving EU connectivity for all customers.

Cloud expansion

BHost’s London presence is in the Meridian Gate (MER) data centre. We already have a significant footprint in MER, although it’s not currently available as a zone in our public cloud. We’re investing in new hardware to deploy in Meridian Gate which is both substantially faster and more power efficient than the current hosts. We’ll be deploying this into our existing suite in MER, and then migrating BHost servers into it. BHost customers will see a small window of scheduled downtime as we migrate each server, but should then seen significantly improved performance on the new hardware.

Our Amsterdam and US presences will give additional options to customers that need to be physically hosted within the (post-Brexit) EU or US. We expect this to become more relevant after Brexit when the UK and EU may have diverging regulatory requirements.

Additional services

All BHost customers can now take advantage of additional Mythic Beasts services including management services for virtual servers, domain registration and DNSSEC-enabled, API-backed DNS hosting.

Support

Mythic Beasts have a larger support team and we’re very well placed to provide significantly improved customer service to all of our new customers. Of course, we do expect the period immediately after the transition to be very busy as customers become familiar with the new billing arrangements, and we get to grips with supporting BHost’s services. We will have additional staff during this period, but please be patient if support responses are a little slower than usual.

Flatpak: pre-assembled furniture applications for Linux

February 23rd, 2018 by

Flatpack is furniture you build yourself. Flatpak is preassembled applications for Linux. This is apparently not at all confusing. (image thanks to https://www.flickr.com/photos/51pct/)

Flatpak provides Linux desktop applications in a secure sandbox which can be installed and run independently of the underlying Linux distribution. Application developers can produce one Flatpak and select the versions of libraries that their application is built, tested and run with so it’s easy for users on any Linux OS to get whatever was intended by the application developer.

Flathub is a distribution service to make sure that Flatpaks are available for popular Linux desktop applications, and at its heart is a private cloud running BuiltBot which builds popular Linux and free/open source desktop apps in Flatpak format. This lives in Mythic Beasts’ Cambridge data centre.

At Mythic Beasts we like this idea so much we offered them lots of free bandwidth (100TB) to help get them started. We’ve now upgraded this with a pair of virtual machines in our core Docklands sites to provide redundancy and more grunt for traffic serving.


Some of their users noticed and were appreciative immediately:

2017-02-23 16:30:00irc wow! Flathub is *so* much faster i’m getting like 10 MB/s compared to less than 1 this morning … and the search is now instant
2017-02-26 11:35PersiFlathub is _really_ fast now, great job to whoever is responsible
🙂

Capacity upgrades, cheaper bandwidth and new fibre

December 8th, 2017 by

We don’t need these Giant Scary Laser stickers yet.

We’ve recently upgraded both our LONAP connections to 10Gbps at our two London POPs bring our total external capacity to 62Gbps.

We’ve been a member of LONAP, the London Network Access Point, since we first started running our own network. LONAP is an internet exchange, mutually owned by several hundred members. Each member connects to LONAP’s switches and can arrange to exchange traffic directly with other members without passing through another internet provider. This makes our internet traffic more stable because we have more available routes, faster because our connections go direct between source and recipient with fewer hops and usually cheaper too.

Since we joined, both we and LONAP have grown. Initially we had two 1Gbps connections, one in each of our two core sites. If one failed the other could take over the traffic. Recently we’ve been running both connections near capacity much of the time and in the event of failure of either link we’d have to fall back to a less direct, slower and more expensive route. Time to upgrade.

The upgrade involved moving from a copper CAT5e connection to optic fibre. As a company run by physics graduates this is an excellent excuse to add yet more LASERs to our collection. Sadly the LASERs aren’t very exciting, being 1310nm they’re invisible to the naked eye and for safety reasons they’re very low powered (~1mW). Not only will they not set things on fire (bad) they also won’t blind you if you accidentally look down the fibre (good). This is not universally true for all optic fibre though; DWDM systems can have nearly 100 invisible laser beams in the same fibre at 100x the power output each. Do not look down optic fibre!

The first upgrade at Sovereign House went smoothly, bringing online the first 10Gbps LONAP link. In Harbour Exchange proved a little more problematic.  We initially had a problem with an incompatible optical transceiver. Once replaced, we then saw a further issue with the link being unstable which was resolved by changing the switch port and optical transceiver at LONAP’s end. We then had further low level bit errors resulting in packet loss for large packets. This was eventually traced to a marginal optical patch lead. Many thanks to Rob Lister of LONAP support for quickly resolving this for us.

With the upgrade completed, we now have two 10Gbps connections to LONAP, in addition to our two 10Gbps connections into the London Internet Exchange and multiple 10Gbps transit uplinks, as well as some 1Gbps private connections to some especially important peers.

To celebrate this we’re dropping our bandwidth excess pricing to 1p/GB for all London based services.  The upgrades leave us even better placed to offer very competitive quotes on high bandwidth servers, as well as IPv6 and IPv4 transit in Harbour Exchange, Meridian Gate and Sovereign House.  Please contact us at sales@mythic-beasts.com for more information.