Benchmarking Single Board Computers
We run several SBCs through their paces and compare their performances.
Benchmarking is a favorite pastime among many computer enthusiasts. Many tests exist that evaluate various performance aspects of computers, and with the rise in popularity of single board computers (SBCs), we would like to have a method of comparing them to each other.
We want to test the performance of several SBCs
Testing SBCs is nothing new. However, we wanted to create a set of (somewhat) standardized tests that we can apply to new SparkFun SBCs. We just want to get a comparison of relative performance; our tests do not have to be super accurate, but should show how SBCs perform against each other in the same aspects.
The Phoronix Test Suite provides a useful set of tests, and we chose only a few to construct our array of tests. Additionally, since power is a common concern among SBC users, we measured average current draw for idle and loaded CPU states. If you want to see the full report, complete with test installation instructions and results, click the link:
Results
If you are still here, then we can happily give you a summary of that report. We tested the following single board computers:
Each of the tests are briefly described in their respective sections below.
CPU (Single Core)
Times how long it takes to convert a .WAV audio file to a FLAC file. The raw test results can be found here.
CPU (Multi-Core)
Computes floating point operations per second (in MFLOPS) by timing how long it takes to solve a pressure Poisson equation. The raw test results can be found here.
RAM
Tests aggregate RAM read and write speed. The raw test results can be found here.
Flash Memory
Transfers files to and from flash memory to compute the read and write rates. The raw test results can be found here and here.
Note that the BeagleBone Black Rev C test was slightly different, as we had to use a 512 MB test file (instead of the 2 GB file used for the other tests). There was simply not enough space on the BeagleBone Black's onboard eMMC.
Graphics (2D)
Times how long it takes to draw a set number of filled circles (many thousands) in a window. The raw test results can be found here.
Note that we left out the Edison as it does not have a video out port and therefore no default GUI.
Power
Compute average power consumption based on measured current draw (averaged over 10 seconds) and measured DC voltage. We created a current logging device to measure the instantaneous current draw. The raw test results can be found here.
Conclusion
So, you might ask, which single board computer is the best? Well, that really depends on how you plan to use it. If you want a lightweight server that consumes little power, low power might be your biggest concern (an Edison could be a good choice). Alternatively, if you want a tiny home theater PC, graphics is important to note (in which case, a pcDuino3 or Raspberry Pi 2 would be good options).
Personally, I am still a big fan of the Edison, especially when you consider performance per Watt (in CPU, RAM, and flash memory read/write). If I need some kind of video output, well, that's another story...
We plan to update the SBC Benchmarks tutorial once in a while with a new set of computers (hence the idea of "rounds"). If you are curious about how SBCs perform, definitely keep an eye on that page.
What SBC is your favorite to use right now (if you use them at all)? Are there other tests that you would like to see performed?
Something that I feel is being forgotten is that all of these are running different flavors of Linux with different overheads. As such, the amount of time the processor is dedicating to each benchmark is going to be vastly different.
For example, the Edison is running Yocto Linux which is a very minimalist version of Linux. In fact, it is so minimalist that it is very difficult to install any packages or other softwares on it - they assume you will build a new version of Yocto with any packages you need when you need them. It has very few background processes and as a result has a much greater amount of CPU time it can dedicate to running your benchmark code.
Conversely, if we look at the Raspi, it is running a comparatively heavy version of Linux (Raspian in this case). Instead of a super bare-bones experience like Yocto, Raspian is designed to be almost kid-friendly. There is A LOT happening in the background to give you endless tools, super easy USB interfaces (keyboards, mice, wifi/bluetooth dongles, ect), a GUI with two video outputs, external RAM, and endless other things. The end result is that the Raspi has a much greater CPU overhead and much less CPU time it can dedicate to a given test.
When you compare the two without taking this into consideration, it looks like the Edison is the best thing since sliced bread. But, I’d bet you money if you built a version of Yocto for the Raspi (and the other two boards) and ran your tests again, you’d see something completely different. I very highly doubt 400MHz dual core chip can run around 3.5-4 times faster than a 1.5MHz quad core chip (especially in both single and multicore tests). Intel does some great work, don’t get me wrong, but so do the folks that design the ARM architectures.
You also have to look at the fact that PHP is an interpreted language and not a compiled one. Different instruction sets and chip architectures are going to define just how well a bit of scripted code will run depending on its ability to do things like pipelining, prefetching instructions, and the like. I would personally love to see some benchmarks done in compiled languages like C/C++ and see how they compare to the PHP benchmarks.
Yocto indeed enables building minimalistic Linux distros. However Yocto build for Edison has all heavier things that are normally not included in minimalist linux: eglibc (rather than uclibc), real linux utils (rather than busibox), g++/binutils, node.js, systemd, dbus. (Though no heavy GUI). Besides, when running CPU and/or RAM intensive benchmarks, several daemons running in background in Raspbian or Yocto/Edison barely consume CPU cycles.
You are right that PHP interpreter could be not the best benchmark. It might be better optimized for x86/Atom than for Arm, so running C benchmark could tell us more about bare compute performance difference. So what we got here is h/w + middleware performance, and middleware performance (php interpreter, node.js) differs on different platform and could also be decision point.
Comparing Mhz for different architectures means nothing. If we assume same s/w, the performance would be proportional to (Mhz * IPC) * (numCores * scalability).
p.s. I do full time performance measurement and optimization since 2005. p.p.s. disclaimer: I work for Intel.
Wow, the Edison really stands out in these tests.
Inforce Computing’s SBC should definitely be part of the above Benchmarking report. Their boards have excellent IOs, are rightly priced and come with full free support for Android and Ubuntu Linux based BSPs.
Add in the fact that Edison has built in Wifi and bluetooth (which may have been running during the power benchmark?) and its a no brainer for your embedded needs. I am running two of them, both battery powered.
Nope! I made sure to turn them off during the power tests. :) I was definitely able to see the current drop by several 10s of mA when I did (at least for WiFi. Bluetooth seemed to be pretty quiet if it wasn't connected to anything).
If I am not mistaken, BBB C is the oldest one of the bunch, so it is at disadvantage when compared with newer models. I think older models should also be included in this test, like RPi v1.
It is, but it works as a useful comparison to see how the newer boards perform. I would like to do a "Round 0" with things like the RPi v1 B+ and BBB (something like the Rev B if I can get my hands on one).
Picking a favorite SBC is pretty difficult because a lot of it boils down to how you are planning on using it. Since they all have little features and quirks that make one just slightly better than another for each individual project.
Of the three SBCs I have used so far I would have to say my favorite is the Odroid C1. The Banana Pi Pro is a close second and the Raspberry Pi gets an honorable mention for being a solid system to take the plunge into SBCs.
I've heard good things about the Banana Pi, but haven't heard of the Odroid before. I just looked it up, and it seems like a solid competitor for the Raspberry Pi.
With the 2D graphics test, why did the pcDuino3 outperform the Acadia? The Acadia is just a quad core version of the pcDuino3. It should have at least been the same speed.
They're not actually the same. The Acadia is a quad core ARM Cortex-A9, whereas the pcDuino3 is a dual core ARM Cortex-A7, which means a slightly different architecture. However, a lot of that could be software (e.g. Linux or video drivers). I experienced some video problems with the Acadia while running the tests, which makes me believe that it has not been optimized yet.