Benchmark sparks
Benchmark sparks
“Figures don’t lie, but liars figure,” they used to say back in my engineering days. You can be scrupulously careful about quoting numbers accurately and still be blowing smoke at people if you are choosy about which numbers you choose to report. I've been thinking about this problem recently because it always comes to mind in the lead-up to a US election.
Dear Linux Pro Reader,
"Figures don't lie, but liars figure," they used to say back in my engineering days. You can be scrupulously careful about quoting numbers accurately and still be blowing smoke at people if you are choosy about which numbers you choose to report. I've been thinking about this problem recently because it always comes to mind in the lead-up to a US election. At this writing, two candidates are both claiming they are ahead in the race and quoting different polls and surveys. To each, the other's polls and surveys don't even exist – their world does not have room for information that conflicts with their story.
This same phenomenon was also in the High Tech news this month, when AMD posted a video on YouTube criticizing Intel's use of the SYSmark benchmarking tool for measuring PC performance. SYSmark shows the latest Intel processors outperforming AMD equivalents by a large margin. In the video, AMD spokesmen John Hampton and Tony Salinas argued that the alternative PCMark 8 benchmark shows a much more competitive race between the Intel and AMD chips, and they demonstrate some practical use cases that show the PCMark 8 is more accurate than anything Intel is using.
They even quote a 2010 Federal Trade Commission (FTC) ruling that forced Intel to include some fine print with their SYSmark citations stating, "Software and workloads in performance tests may have been optimized only on Intel microprocessors."
The problem, according to Hampton and Salinas, is that SYSmark places the emphasis on the CPU only, whereas PCMark 8 looks at the CPU, GPU, and video subcomponents together, which is more like how the computer actually behaves.
I try not to take sides in vendor disputes, but in this case, since the FTC has taken sides, it does seem very likely that AMD has a point. If SYSmark is not an accurate benchmark for comparing real-world PC system performance, why does Intel keep using it? Because marketing departments of major corporations don't exactly put the emphasis accuracy.
Hampton invokes another recent news story when he mentions "the recent debacle over the emissions standards provided by a major auto maker …" This apparent reference to the Volkswagen emissions scandal highlights another case where customers were misled with numbers. Of course, as Intel would readily point out, Volkswagen is actually providing inaccurate information, whereas Intel is reporting the results of the SYSmark benchmarks very accurately. Is using the wrong facts as bad as giving the wrong answer? If it leads people to incorrect conclusions, the effect is the same.
We get so many numbers thrown at us every day that we really can't chase down all of them. Still, if you're going to go to the trouble of buying a computer that you will have to work with every day for at least a couple of years, it really might be worth taking a closer look at the numbers. The next time I read a triumphant claim of success that references a computational benchmark, I'm going to Google the benchmark and find out what it is really testing.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
Endless OS 6 has Arrived
After more than a year since the last update, the latest release of Endless OS is now available for general usage.
-
Fedora Asahi 40 Remix Available for Macs with Apple Silicon
If you've been anticipating KDE's Plasma 6 for your Apple Silicon-powered Mac, then you're in luck.
-
Red Hat Adds New Deployment Option for Enterprise Linux Platforms
Red Hat has re-imagined enterprise Linux for an AI future with Image Mode.
-
OSJH and LPI Release 2024 Open Source Pros Job Survey Results
See what open source professionals look for in a new role.
-
Proton 9.0-1 Released to Improve Gaming with Steam
The latest release of Proton 9 adds several improvements and fixes an issue that has been problematic for Linux users.
-
So Long Neofetch and Thanks for the Info
Today is a day that every Linux user who enjoys bragging about their system(s) will mourn, as Neofetch has come to an end.
-
Ubuntu 24.04 Comes with a “Flaw"
If you're thinking you might want to upgrade from your current Ubuntu release to the latest, there's something you might want to consider before doing so.
-
Canonical Releases Ubuntu 24.04
After a brief pause because of the XZ vulnerability, Ubuntu 24.04 is now available for install.
-
Linux Servers Targeted by Akira Ransomware
A group of bad actors who have already extorted $42 million have their sights set on the Linux platform.
-
TUXEDO Computers Unveils Linux Laptop Featuring AMD Ryzen CPU
This latest release is the first laptop to include the new CPU from Ryzen and Linux preinstalled.