Working with talented people.

February 14th, 2020 by

You can buy another copy in a bookshop if your cat refuses to return the one you already own.


We like working with talented people be they staff, customers or suppliers. That’s why we give discounts to people who can navigate our jobs challenge even if they don’t want to work for us.

Occasionally we’ve drafted in Gytha Lodge to help us copy write various articles and turn a jumble of thoughts into a coherent and interesting article.

Formerly an aspiring author, her full title is now Richard and Judy book club pick and Sunday Times bestselling author, Gytha Lodge.

We’re also pleased to report that she took our advice on her first book seriously and the new book starts with a murder being watched over a webcam.

OpenWRT install to RAM – run iftop on a router with very limited flash

November 23rd, 2018 by

OpenWRT is awesome, as it allows you to run proper Linux tools on your home router. I’m currently using a very old, underspecced TP Link box, with 32MB of RAM, but just 4MB of flash storage. This is just enough to get what I need installed, but one thing I’ve always wanted to do is use iftop to quickly see what’s using all the bandwidth. Unfortunately iftop, with its dependencies on libpcap and libncurses, just won’t fit into a 4MB image.

I recently stumbled across opkg’s install-to-RAM option, allowing me to use the 32MB of RAM to install the package, with the minor and obvious downside that it gets uninstalled when the router gets rebooted. For something like iftop, which is used for ad-hoc diagnostics, this isn’t a big issue.

Installing to RAM puts the packages under /tmp, so a little effort is required to make sure that libraries and other resources can be found. I now have the following shell script which installs iftop if it isn’t already, sets some environment variables and invokes iftop:

#!/bin/sh

if [ ! -f /tmp/usr/bin/iftop ] ; then
  opkg update
  opkg install -d ram iftop
fi

export LD_LIBRARY_PATH=/tmp/usr/lib
export TERM=xterm
export TERMINFO=/tmp/usr/share/terminfo/

/tmp/usr/bin/iftop $@

Fortunately I do have enough free space on flash storage to store the above script.
Obviously a similar approach could be used with other packages that are only needed “on demand”.

Ten years on, Chris Lightfoot looks more prescient than ever

February 13th, 2017 by

(gif from imgur via gify)

(title shamelessly stolen from Tom Steinberg, MySociety founder from his tribute to Chris, Mythic Beasts founder who died ten years ago).

Lots of people have been excited recently about this script, which allows you to remotely reinstall a Linux system with a different version of Linux by giving you a shell in a ramdisk and letting you reinstall the operating system from there.

Chris did this the hard way. Back in 2005 I remember being asked to code review ‘evil.c’, a script that allocated a lot of RAM (800MB!), compressed a filesystem into it, then uncompressed it back to the disk. On reboot it should come up with Debian instead of FreeBSD that it had earlier. It’s really very important not to swap during this process.

Amazingly it worked, and the first test was on a remote box and it saved us a data centre visit. Here’s the code in its full glory.

#include <sys/types.h>

#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <zlib.h>

#include <sys/mman.h>
#include <sys/reboot.h>

#define SIZE        ((size_t)896058269)

#define E(...)      fprintf(stderr, __VA_ARGS__)
#define die()       do { fprintf(stderr, "%s; aborted\n", strerror(errno)); exit(1); } while (0)
#define zdie()      do { fprintf(stderr, "%s; aborted\n", Z.msg); exit(1); } while (0)

int main(void) {
    unsigned char *buf, *outbuf, *p;
    int fd;
    FILE *fp;
    z_stream Z = {0};
    unsigned int nin = 0, nout = 0;

    E("size = %lu\n", (unsigned long)SIZE);

    E("open /dev/amrd0... ");
    if (-1 == (fd = open("/dev/amrd0", O_RDWR | O_DIRECT)))
        die();
    E("done\n");
    close(fd);

    E("allocate file buffer... ");
    if (!(buf = malloc(SIZE)))
        die();
    E("done\n");

    E("allocate write buffer... ");
    if (!(outbuf = malloc(1024 * 1024)))
        die();
    E("done\n");

    E("lock into memory... ");
    if (-1 == mlockall(MCL_CURRENT | MCL_FUTURE))
        die();
    E("done\n");

    E("open file... ");
    if (!(fp = fopen("/usr/bitter-first-2100M-of-sda.gz", "rb")))
        die();
    E("done\n");

    E("read file... ");
    p = buf;
    while (nin < SIZE) {
        size_t n;
        n = fread(p, 1, 262144, fp);
        if (n == 0)
            die();
        nin += n;
        p += n;
        E("\rread file... %.2f%% ", 100 * (float)nin / (float)SIZE);
    }
    E("done\n");

    fclose(fp);
    E("zlib version = \"%s\"\n", zlibVersion());

    /* Now we need to walk through the buffer decompressing it into the
     * write buffer, then writing the results to the device. */
    E("initialise inflate object... ");
    Z.next_in = buf;
    Z.avail_in = SIZE;
    if (Z_OK != inflateInit2(&Z, 15 + 32))
        zdie();
    E("done\n");

    while (nout < 2100) {
        int i;
        size_t N;

        Z.next_out = outbuf;
        Z.avail_out = 1024 * 1024;
        i = inflate(&Z, 0);
        if (i != Z_OK && i != Z_STREAM_END)
            zdie();
        if (Z.next_out != outbuf + 1024 * 1024) {
            fprintf(stderr, "\ndidn't get 1MB of output\n");
        }

        /* this is where we'd write the data */
        N = 0;
        p = outbuf;
        while (N < 1024 * 1024) {
            ssize_t n;
            do
                n = write(fd, p, 1024 * 1024 - N);
            while (n == -1 && errno == EINTR);
            if (n == -1)
                die();
            N += n;
            p += n;
        }

        ++nout;
        fprintf(stderr, "\r%d / 2100 MB", nout);
    }

    fprintf(stderr, "\n");

    /* this is where we reboot */
    reboot(RB_NOSYNC);

    E("we should have rebooted by now -- probably best to assume we're completely\n"
      "screwed at this point\n");

    return 0;
}

Server Castle

November 17th, 2014 by

So last week we built a fort from some old customer servers. Sometimes, though, it’s important to just try a little bit harder.

HipHop and WordPress: If you’re tired of tea then you’re tired of life…

November 14th, 2014 by

Hip Hop is not only a style of music, but also the name of a virtual machine written by Facebook which compiles PHP Just In Time to make it go quickly.

Now we receive lots of unsolicited advice about how to run a not very popular wordpress blog and cope with the volume of traffic. Usually this involves ripping and replacing the entire infrastructure from a standard Linux/Apache/MySQL/PHP stack to something different (Nginx/MariaDB/PostgreSQL) which may not even be able to run WordPress at all (e.g. node.js).

At Mythic Beasts we like to understand what we’re doing, rather than blindly installing Magic Go Faster Solution Number 7. So we set up a test 2GB dual core virtual machine, that runs WordPress and a selection of popular plugins ( WordPress SEO, Akismet, Safe Report Comments, Liveblog, Facebook, Yet Another Related Posts Plugin, WordPress Supercache and Jetpack, no endorsement implied). Then we benchmarked with siege and managed the following results.

Apache/mod_php : 5.10 trans/sec

and when you turn supercache on and serve cached pages you get

Apache/mod_php/supercache : 873.50 trans/sec

So this gives us two scenarios, pages which we have to generate content for which can easily cause load issues, and pages served from supercache in which our VM is fast enough for all practical purposes and will easily weather even very big traffic spikes from news websites or television adverts.

Now, it’s very popular to tell us to use Ngnix as it’s faster than Apache. Is it though?

Nginx/php-fpm: 5.70 trans/sec
Nginx/php-fpm/supercache: 2230.58 trans/sec

Wow! Nginx is three times quicker than Apache at serving cached pages. This is amazing, but not very helpful. It means when our webserver is serving pages really quickly, we serve pages at three times really quickly, but when we’re generating pages on demand, it’s about 10% quicker. That’s not very special and doesn’t justify a rip and replace of the whole installation for a 10% performance improvement.

A quick look at the VM during the testing tells us that the bottleneck is executing the PHP code which creates WordPress pages. The choice of webserver is basically irrelevant; almost all the server time is spent executing PHP and reading data from the database.

Enter HipHop Virtual Machine.


This is nothing to do with the HipHop Virtual Machine. But we like tea and Banging Tunes

It has one focus, to execute PHP quickly for Facebook. Facebook have a lot of servers and spend hundreds of millions to billions per year on servers and data centres. A 50% performance improvement in PHP saves them huge sums of money in data centres and servers alone, so it’s clearly worth them trying to optimise as much as possible.

Here’s what happens with Apache/Nginx running HHVM.

Apache/HHVM :           35.93 trans/sec
Apache/HHVM/supercache: 928.70 trans/sec
Nginx/HHVM :            33.78 trans/sec
Nginx/HHVM/supercache : 2137.67 trans/sec

This is a huge improvement for non cached pages – seven times faster. Cached pages are bottlenecked in the webserver so it makes minimal difference, but they were already so fast we weren’t worried about them. Again Apache/Nginx are still pretty much the same speed for generated pages, we’re still dominated by the code execution time but a seven fold performance improvement is worth seriously considering.

 Whilst we can reconfigure servers standing on our heads, we usually don't.

Whilst we can reconfigure servers standing on our heads, we usually don’t.
Photo credit: Mark Dolby, Flickr, CC-BY.

All I need to do now is see if I can find someone with a very busy WordPress site and a million complaining users who would like to test it to see if it’s really as good as the lab tests suggest it might be.


Very sorry to hear the news that Big Bank Hank who co-wrote the first ever hit Rap track Rappers Delight died earlier this week from kidney complications related to cancer.


You see, he was six foot one, and he was tons of fun

Explain To Us: an explanation

September 17th, 2014 by

This post is written by Ben Howe who’s with us for his gap year. He took some holiday to go to Young Rewired State and he’s written up what he did.

During the summer I participated in the Young Rewired State Festival of Code, where I wrote a website called “Explain To Us” – designed to make the list of bills currently before parliament less confusing. For example, the “summary” listed on the website is the “long title” from the legal text, as opposed to being clearer more detailed. Explain To Us is a platform which allows members of parliament to upload a short video “pitch” to explain what their bill is about and why it should be given debating time in parliament.

Young Rewired State is an independent global network of young people aged 18 and under who have taught themselves to code. They bring together like-minded peers at events around the world where they use open data to make websites, apps and other solutions to real world challenges. The Festival of Code is an annual celebration of programming held over a week in multiple centres, culminating in a huge showcase weekend where everyone presents their solutions (this year in Plymouth, hosted by Plymouth University).

A Python script interprets the RSS feed for all bills before parliament. It then visits the <link> for each bill, scraping the sponsor information (as this is not included in the RSS feed). HTML pages are then generated, which are served by nginx. In future, I’m planning to automatically email members  of parliament (using contact details from TheyWorkForYou) when they submit a new bill asking them to upload a video.

After my initial presentation, I was nominated for “Best In Show” – although sadly I was knocked out by “YouDraw” (who later won Best In Show). Having gained valuable feedback from presenting to the Speaker’s Commission on Digital Democracy and Tracy Green (head of online services at parliament), I’m now intending on building a second beta with more functionality.

If you’d like to work for Mythic Beasts in your gap year, you need to demonstrate how you can use Google to find our jobs page and work the rest out from there.