Raspberry Pi on Raspberry Pi

June 22nd, 2019 by

Question: Is the Raspberry Pi 4 any good?
Answer: It’s good enough to run its own launch website with tens of millions of visitors.

Raspberry Pi 4 with PoE mounting points already attached.

The Raspberry Pi 4 is out. It’s a quad core ARM A72 running at 1.5Ghz with 4GB of RAM and native 1Gbps ethernet. This means that according to our benchmarks (PHP 7.3 and WordPress) it’s about 2.5x the speed of the 3B+, thanks to the much faster core design and slight clock speed boost. The downside is that it uses more power. Idle power consumption is up slightly to about 3W, peak is now around 7W, up from 5W. It has some improved video features too and USB3.

We obtained an early sample and benchmarked it running the Raspberry Pi website. We used the main blog, which hosts the www.raspberrypi.org blog, and has historically been the most CPU-intensive site to provide. We now see complete page generation in about 0.8s, compared to 2.1s for the 3B+. Obviously in normal operation, most pages are served from a cache, and so the typical end user experience is much faster.

We were really excited by the Pi 4 and wanted to have them available in our cloud for launch day. Unfortunately, Eben had some bad news for us: netboot on the Pi 4 is only going to be added in a future firmware update. Netboot is critical to the operation of our cloud, as it prevents customers from bricking the servers. Our dreams were shattered.

Our standard Pi Cloud unit consists of 6x9x2 blocks of Pi 3B servers connected to PoE switches with just one wire per server. They all net boot and are controlled through our control panel and API for customer use. Since the lack of netboot means we couldn’t just drop the Pi 4 in as a faster version at this time, we went back to the lab and we built an alpha Pi 4 Cloud on a smaller scale: 18 Pi 4s that Raspberry Pi have very generously given to us, all connected with gigabit ethernet so we can try out the 2.5x faster CPUs, 3x faster Network and 4x RAM capacity. We deployed this to our Sovereign House data centre where it connects to our core network.

In full production, we’ll have six racks of Pi 4 stacked back to back.

What we needed then was a test application. We suggested running the main Raspberry Pi website, as we once did with the Pi 3. But with over twice the horsepower per machine we thought we’d dream bigger. How about hosting the Raspberry Pi website on the Raspberry Pi 4, on the Raspberry Pi 4 launch day?

We’ve set up 14 Pi 4s for PHP processing for the main website (56 cores, 56GB RAM), two for static file serving (8 cores, 8GB RAM) and two for memcached (8 cores / 8GB RAM). Late on Friday night we started moving production traffic from the existing virtual machines to the Pi 4 cluster, completing the move shortly after midnight. Every page from the blog after Sat 22nd June has been generated on a Raspberry Pi 4.

Unfortunately, this configuration isn’t yet ready to become the standard, production environment for the Raspberry Pi website. As noted above, the Pi 4s don’t yet support netboot, and so these ones have local SD card storage rather than netboot and network file storage. This means they can’t be remotely re-imaged and have comparatively unreliable storage. The configuration is also only deployed in a single data centre with all servers on a single switch, whereas in normal usage the Raspberry Pi website is simultaneously hosted in two different data centres for redundancy.

To make things more nerve wracking, Pi 4 requires Debian Buster which is a pre-release version of the operating system (full release July 6th). So it’s a cluster of brand new hardware, with a pre-release operating system and a single point of failure. We very strongly advise our customers not to use this for a mission critical super high profile website under-going the most significant production launch in their history. That really isn’t a very good idea.

We once advised Eben that Raspberry Pi probably wouldn’t sell very many computers. He didn’t listen to us then either.

We haven’t moved the entire stack to the Pi 4. The front-end load balancers, download and apt servers are still on non-Pi hardware, split across three data centres (two in London, one in Amsterdam). The Pi 4 hardware looks well-suited to taking over these roles too, although we’ve kept the current arrangement for now, as it’s well tested and allows us to switch back to non-Pi 4 back-ends quickly if needed.

We haven’t moved the databases to the Pi 4 yet either. We’re not going to do that until we can have nice reliable mirrored storage on enterprise SSDs with high write reliability and long write lifetimes attached to the Pis.

Where do we go from here?

Once netboot on Pi 4 is available, we’ll be adding 4 core A72 / 4GB servers to our Pi Cloud, at a slightly higher price than the existing Pi 3 servers, reflecting the higher hardware and power costs. We are also planning to investigate virtualisation as 1 core / 1GB Raspberry Pi VMs may be of interest to existing Pi3 users. 64 bit kernel support and potentially a 64 bit userland would also now be worth investigating.

If you like the idea of Pi 4 in the cloud, a Pi 4 VM in the cloud or 64 bit ARM in the cloud, tell us your plans at sales@mythic-beasts.com.

Raspberry Pi 3B+

March 14th, 2018 by

Today is Pi Day where we celebrate all things mathematical. Today is a super special Pi day, because a new Raspberry Pi has been released.

It takes the previously excellent Raspberry Pi 3 (or 3B, to give it its full name) and upgrades it with an extra 200Mhz of CPU speed and gigabit ethernet over USB 2. It fixes many of the netboot issues which Pete highlighted at the last big Pi Birthday Party and will soon have a new smaller and cheaper Power over Ethernet HAT. These new features are of particular interest for our Raspberry Pi Cloud service, as we use netbooted Pis, with network file storage and Power over Ethernet to enable remote powercycling.

Raspberry Pi 3B+.

We’ve had one to play with, and we’ve run our favourite benchmark – Raspberry Pi’s own website. We installed the full stack (MySQL, WordPress & PHP7) under Debian Stretch onto a Pi 3B and a Pi 3B+, and tried it out with 32 concurrent connections. We’re running near identical setups on the two servers: both have their files stored over the network on an NFS file server and it’s the same operating system and applications; only the kernel differs.

Model Pages/second
Raspberry Pi 3B 3.15
Raspberry Pi 3B+ 3.65

The new model is about 15% faster than the old one which is almost exactly as expected from the boost in clock speed; WordPress is CPU limited.

Checksumming the 681MB database file shows up the gigabit ethernet rather effectively. All our storage is over the network so reading files is a benchmark of the network speed.

Model Elapsed time Data rate
Raspberry Pi 3B 54.4s 11.25MB/s
Raspberry Pi 3B+ 28.1s 22.1MB/s

This is very nearly twice as fast as the previous model.

When is it coming to the Raspberry Pi Cloud?

The Raspberry Pi 3B+ is an obvious upgrade for our Raspberry Pi Cloud. We need to wait for the PoE HAT to become available. That will allow us better density and lower capital costs. However, the 3B+ consumes more power than the 3B so we need to do some thermal and airflow work before we can make it generally available.

Raspbian Stretch now available in the Raspberry Pi Cloud

August 31st, 2017 by

A very short service announcement.

Raspbian Stretch is now available for Raspberry Pis hosted in our Raspberry Pi Cloud. This joins Raspbian Jessie and Ubuntu Xenial as available images. With all of these you can upload an SSH key through our control panel and log in directly. Re-imaging and rebooting can both also be done directly from our control panel.

Re-imaging your Raspberry Pi will reset the image and delete all data on your Cloud Pi.

On the server side the most significant upgrade is PHP 7, which should double the performance of PHP-based applications running on the Raspberry Pi.

FRμIT: Federated RaspberryPi MicroInfrastructure Testbed

July 3rd, 2017 by

The participants of the FRμIT project, distributed Raspberry Pi cloud.

FRμIT is an academic project that looks at building and connecting micro-data-centres together, and what can be achieved with this kind of architecture. Currently they have hundreds of Raspberry Pis and they’re aiming for 10,000 by the project end. They invited us to join them, we’ve already solved the problem of building a centralised Raspberry Pi data centre and wanted to know if we could advise and assist their project.  We recently joined them in the Cambridge University Computer Lab for their first project meeting.

Currently we centralise computing in data centres as it’s cheaper to pick up the computers and move them to the heart of the internet than it is to bring extremely fast (10Gbps+) internet everywhere. This model works brilliantly for many applications because a central computing resource can support large numbers of users each connecting with their own smaller connections. It works less well when the source data is large and in somewhere with poor connectivity, for example a video stream from a nature reserve. There are also other types of application such as Seti@Home which have huge computational requirements on small datasets where distributing work over slow links works effectively.

Gbps per GHz

At a recent UK Network Operator Forum meeting, Google gave a presentation about their data centre networking where they built precisely the opposite architecture to the one proposed here. They have a flat LAN with the same bandwidth between any two points so that all CPUs are equivalent. This involves around 1Gbps of bandwidth per 1GHz of CPU. This simplifies your software stack as applications don’t have to try and place CPU close to the data but it involves an extremely expensive data centre build.

This isn’t an architecture you can build with the Raspberry Pi. Our Raspberry Pi cloud is as about as close as you can manage with 100Mbps per 4×1.2GHz cores. This is about 1/40th of the network capacity required to run Google architecture applications. But that’s okay, other applications are available. As FRμIT scales geographically, the bandwidth will become much more constrained – it’s easy to imagine a cluster of 100 Raspberry Pis sharing a single low bandwidth uplink back to the core.

This immediately leads to all sort of interesting and hard questions about how to write a scheduler as you need to know in advance the likely CPU/bandwidth mix of your distributed application in order to work out where it can run. Local data distribution becomes important – 100+ Pis downloading updates and applications may saturate the small backbone links. They also have a variety of hardware types, the original Pi model B to the newer and faster Pi 3, possibly even some Pi Zero W.

Our contribution

We took the members of the project through our Raspberry Pi Cloud is built, including how a Pi is provisioned, how the network and operating system are provisioned and the back-end for the entire process from clicking “order” to a booted Pi awaiting customer login.

In discussions of how to manage a large number of Federated Raspberry Pis we were pleased to find considerable agreement with our method of managing lots of servers: use OpenVPN to build a private network and route a /48 of IPv6 space to it.   This enables standard server management tools work, even where the Raspberry Pis are geographically distributed behind NAT firewalls and other creative network configurations.

Donate your old Pi

If you have an old Raspberry Pi, perhaps because you’ve upgraded to a new Pi 3, you can donate it directly to the project through PiCycle. They’ll then recycle your old Raspberry Pi into the distributed compute cluster.

We’re looking forward to their discoveries and enjoyed working with the researchers. When we build solutions for customers we’re aiming to minimise the number of unknowns to de-risk the solution. By contrast tackling difficult unsolved problems is the whole point of research. If they knew how to build the system already they wouldn’t bother trying.

Cambridge Beer Festival, Raspberry Pi powered Apps for Beer

May 22nd, 2017 by

We drew the architecture diagram for the beer festival on a beer mat.

Today marks the first day of the Cambridge Beer Festival, the longest running CAMRA beer festival, one of the largest beer festivals in the UK and in our obviously correct opinion, by far the best. Not only have we run the web back-end for many Cambridge CAMRA websites for many years, this year we’ve been involved with Cambridge App Solutions who run the iPhone Beer Festival App. They’d been having some trouble with their existing hosting provider for the back-end. In frustration they moved it to their Cloud Raspberry Pi which worked rather better. They then suggested that we keep the production service on the Raspberry Pi, despite it being a beta service.

Preparing for production

We’ve set up all our management services for the hosted Pi in question, including 24/7 monitoring and performance graphing. We then met up with Craig, their director in the pub to discuss the app prior to launch. The Pi 3 is fronted by CloudFlare who provide SSL. However, the connection to the Pi3 from Cloudflare was initially unencrypted. We took Craig through our SSL on a Raspberry Pi hosting guide and about a minute later we had a free Let’s Encrypt certificate to enable full end-to-end data security.

 

 

The iPhone app that runs the Cambridge Beer Festival (also found at Belfast and Leestock)

 

The iPhone Beer Festival App tracks which beers are available and the ratings for how good they are. Availability is officially provided, ratings are crowd sourced.  The app is continuously talking to the back end to keep the in app data up to date. All this data is stored and served from the Raspberry Pi 3 in the cloud.

Proximity

The festival also has some Estimote beacons for proximity sensing which use Bluetooth Low Energy to provide precise location data to the phone. On entry to the beer festival the app wakes up and sends a hello message.

Raspberry Pi Cloud upgrades

May 12th, 2017 by

We’ve made some improvements to our Raspberry Pi Cloud.

  • Upgraded kernel to 4.9.24, which should offer improved performance and a fix for a rare crash in the network card.
  • Minor update to temperature logging to ease load on our monitoring server and allow faster CPU speeds.
  • Upgrade to the NFS fileserver to allow significantly improved IO performance.
  • Recent updates applied to both Debian and Ubuntu images.

Thanks to Gordon Hollingworth, Raspberry Pi Director of Engineering for his assistance.


PHP7 on Pi 3 in the cloud (take 2)

March 24th, 2017 by

On Wednesday, we showed you how to get PHP7 up and running on one of our Pi 3 servers. Since then, we’ve implemented something that’s been on our to do list for a little while: OS selection. You can now have Ubuntu 16.04 and the click of a button, so getting up and running with PHP7 just got easier:

1. Get yourself a Pi 3 in our cloud.

2. Hit the “Reinstall” button:

3. Select Ubuntu 16.04:

4. Upload your SSH key (more details), turn the server on, SSH in and run:

apt-get install apache2 php7.0 php7.0-curl php7.0-gd php7.0-json \
    php7.0-mcrypt php7.0-mysql php7.0-opcache libapache2-mod-php7.0
echo "<?=phpinfo()? >" >/var/www/html/info.php

Browse to http://www.yourservername.hostedpi.com/info.php and you’re running PHP7:

PHP7 on a Raspberry Pi 3 in the cloud

March 22nd, 2017 by
Rasberry Pi 3

Two Raspberrys PI using PHP7 during the Pi 3 launch.

Last April we moved the main blog for Raspberry Pi to a small cluster of Raspberry Pi 3s. This went so well we made it commercially available and you can now buy your Raspberry Pi 3 in the cloud.

If you’d like to have PHP 7 running on your Raspberry Pi 3 in the cloud, this guide if for you. Click the link, buy a Pi 3 and install your ssh-key and log in. This should take no more than about a minute.

PHP 7 isn’t yet part of the standard Raspbian OS, so we need to get it from somewhere else.

A brief aside about CPU architectures, Raspbian and Debian

Debian provides three versions for ARM processors:

  • armel – 32 bit and ARMv5
  • armhf – 32 bit, ARMv7 and a floating point unit
  • arm64 – 64 bit ARMv8 and a floating point unit

The Raspberry Pi uses three different architectures:

  • Raspberry Pi A, B, Zero & Zero W – 32 bit ARMv6 with floating point
  • Raspberry Pi 2 – 32 bit ARMv7 with floating point
  • Raspberry Pi 3 – 32/64 bit ARMv8 with floating point unit

Raspbian is an unofficial port for 32bit ARMv6 and a floating point unit, which matches the hardware for an original Raspberry Pi model B. Because we’re working here with the Pi 3 – ARM8 and floating point, we can take official debian armhf packages and run them directly on our Pi 3.

Ondřej Surý is the Debian PHP maintainer who also has a private repository with newer versions of PHP built for Debian and Ubuntu. So we can use 32 bit Debian packages for ARM 7 (armhf) and install directly on top of Raspbian.

PHP 7 packages aren’t available for armel, so this won’t work on an original Raspberry Pi, or a Pi Zero/Zero W.

Add the PHP 7 repository

deb.sury.org includes newer PHP packages built for armhf, which we can use directly. Following the instructions here here we can set up the repository:

apt-get install apt-transport-https lsb-release ca-certificates
wget -O /etc/apt/trusted.gpg.d/php.gpg https://packages.sury.org/php/apt.gpg
echo "deb https://packages.sury.org/php/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/php.list
apt-get update

Now we can install everything we need for php7 and apache2.4:

apt-get install apache2 php7.0 php7.0-curl php7.0-gd php7.0-json \
    php7.0-mcrypt php7.0-mysql php7.0-opcache libapache2-mod-php7.0
echo "<?=phpinfo()?>" >/var/www/html/info.php 

Wait a few moments and we have a webserver running PHP7 on our Pi3 in the cloud.

You’ll note we’ve included php7-opcache. This should accelerate our PHP performance by a factor of two or so.

Now for an application…

Try WordPress

WordPress needs a MySQL server & PHP library for accessing MySQL. We need to restart Apache to make PHP 7 pick up the additional library.

apt-get install php7.0-mysql  mysql-server
apache2ctl restart
mysql -u root -p

mysql> create database wordpress;
mysql> grant all privileges on wordpress.* to wordpress identified by 'password';

We strongly recommend you invent a better password.

cd /var/www/html
wget https://wordpress.org/latest.tar.gz
tar -zxvf latest.tar.gz
chown -R www-data:www-data wordpress

Then navigate to http://www.yourpiname.hostedpi.com/wordpress and finish the install through your browser.

Next steps

For information on how to host on your own domain name, and how to enable HTTPS see our previous blog post on hosting a website on a Raspberry Pi.

Hosting a website on an IPv6 Pi part 2: PROXY protocol

March 10th, 2017 by

In our previous post, we configured an SSL website on an IPv6-only Raspberry Pi server, using our IPv4 to IPv6 reverse proxy service.

The one problem with this is that our Pi would see HTTP and HTTPS requests coming from the proxy servers, rather than the actual clients requesting them.

Historically, the solution to this problem is to have the proxy add X-Forwarded-For headers to the HTTP request, but this only works if the request is unencrypted HTTP, or an HTTPS connection that is decrypted by the proxy. One of the nice features of our proxy is that it passes encrypted HTTPS straight to your server: we don’t need your private keys on the proxy server, and we can’t see or interfere with your traffic.

Of course, this means that we can’t add X-Forwarded-For headers to pass on the client IP address. Enter PROXY protocol. With this enabled, our proxies add an extra header before the HTTP or HTTPS request, with details of the real client. This is easy to enable in our control panel:

You also need to configure Apache to understand and make use of the PROXY protocol header. This is a little more involved, as the necessary module isn’t currently packaged as part of the standard Apache distribution (although this is changing), so we need to download and build it ourselves. First some extra packages are needed:

apt-get install apache2-dev git

This will install a good number of packages, and take a few minutes to complete. Once done, you can download, install and build mod_proxy_protocol

git clone https://github.com/roadrunner2/mod-proxy-protocol.git
cd mod-proxy-protocol
make

At this point you should just be able to type make install but at time of writing, there seems to be some problem with the packaging. So instead do this:

cp .libs/mod_proxy_protocol.so /usr/lib/apache2/modules/

Now you can load the module:

echo "LoadModule proxy_protocol_module /usr/lib/apache2/modules/mod_proxy_protocol.so" > /etc/apache2/mods-available/proxy_protocol.load
a2enmod proxy_protocol

You also need to configure Apache to use it. To do this, edit /etc/apache2/sites-enabled/000-default.conf and replace each line that contains CustomLog with the following two lines:

	ProxyProtocol On
	CustomLog ${APACHE_LOG_DIR}/access.log "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\""

This tells Apache to use Proxy Protocol, and to use the supplied IP address in its log files. Now restart Apache:

systemctl reload apache2

Visit you website, and if all is working well, you should start seeing actual client IP addresses in the log file, /var/log/apache2/access_log:

93.93.130.44 - - [24/Feb/2017:20:13:25 +0000] "GET / HTTP/1.1" 200 10701 "-" "curl/7.26.0"

Trusting your log files

With the above configuration, we’ve told Apache to use the client IP address supplied by our proxy servers. What we haven’t done is told it that it can’t trust any random server that pitches up talking PROXY protocol. This means that it’s trivial to falsify IP addresses in our log files. To prevent this, let’s set up a firewall, so that only our proxy servers are allowed to connect on the HTTP and HTTPS ports. We use the iptables-persistent package to ensure that our firewall is configured when the server is rebooted.

apt-get install iptables-persistent

ip6tables -A INPUT -s proxy.mythic-beasts.com -p tcp -m tcp --dport 80 -j ACCEPT
ip6tables -A INPUT -s proxy.mythic-beasts.com -p tcp -m tcp --dport 443 -j ACCEPT
ip6tables -A INPUT -p tcp --dport 80 -j REJECT
ip6tables -A INPUT -p tcp --dport 443 -j REJECT

ip6tables-save

And we’re done! Our IPv6-only Raspberry Pi3 is now hosting an HTTPS website, and despite being behind a proxy server, we’re tracking real client IP addresses in our logs.

Attending a 5 year old’s birthday party

March 7th, 2017 by

Over the weekend Pete went to the Raspberry Pi Big Birthday Weekend. Rather larger than previous years, they booked out the whole of the Cambridge Junction allowing them to have far more guests.

A very small Eben reviewing the extraordinary growth of Raspberry Pi over the last 12 months.

Eben Upton started the day with a keynote about how Raspberry Pi is doing. In short, they’ve made a lot of computers. Raspberry Pi Trading (the company that makes and sells computers) has generated several million pounds of profit that’s been returned to the Raspberry Pi Foundation (the charity that educates people). This makes them a very unusual startup compared to the current Silicon Valley competition of trying to lose money as fast as possible. Pi Zero W, the newest member of the family has sold over 100,000 units in the four days since launch.

Picture of the Pi Logo done on a sense hat by a young girl in the Astro-Pi workshop.

Pete joined David Honess for his Astro-Pi workshop, programming the Sense HAT and writing code that could eventually end up running on the International Space Station. One girl ran away with one of the examples, turning the black and white X into a full colour Raspberry Pi logo using the Sense HAT.



We were particularly interested in the presentation by Gwiddle as it’s a project that we sponsor, and the Pi Party was first opportunity we had to meet the founders in person. Gwiddle provide free Web hosting for people in full time education and are having significant success supporting nearly 1,000 students. Yasmin and Joshua fielded their questions well. We’re massively proud of what Gwiddle have achieved so far and their plans for future expansion into a full multi-national with world domination look exciting.

Pete giving a talk on Pi in the Cloud, photo by Joshua Bayfield. View the slides

Pete gave a talk on Raspberry Pi in the data centre and how we’ve built a Raspberry Pi Cloud. We’ve put the slides up at the request of the Japanese Raspberry Pi Users Group who’d flown over to be at the Party. We’d love it if you translate the slides Japanese and put them on your website!

Sam Aaron and Sonic Pi

Sam Aaron demonstrated new features in Sonic Pi in a live performance. Pete was particularly impressed by the distortion guitar amp effect from the Pi, and also the details of the new Erlang back-end to reduce timing jitter around a millisecond. Somebody needs to take the library and implement an easy real-time mode for the Pi Zero W to ease building Internet of Things applications.

A fantastic, but incredibly tiring weekend. We’re already looking forward to the 6th birthday party!