Written by Mike James
Thursday, 13 April 2017
Benchmarking is both essential and a distortion. The trouble is that once there is a benchmark that anyone pays any attention to, the tendency is to tweak for a high score - even if this results in a poorer real world performance.
Octane contained cut down real world code and you might have assumed that this would make it more relevant, but no. As the Chromium blog explains:
At the beginning of last year, the V8 team started to measure performance with higher fidelity by instrumenting snapshots of popular web pages such as Reddit, Twitter, Facebook, and Wikipedia. This analysis revealed that while peak performance benefits certain types of large web applications, browsing typical websites relies more on “startup” performance, or the speed it takes to start running script. Using insights gleaned from this real-world performance data, the V8 team implemented optimizations which improved mean page load between Chrome 49 and Chrome 56 by 10-20%, depending on CPU architecture.
From the data gathered the team put together a new benchmark - Speedometer - with added automotive reference. It includes chunks of code from common web frameworks such as React, Angular, Ember and jQuery. The important point is that while Octane didn't respond to the startup optimizations, Speedometer did - suggesting that optimizing with respect to Speedometer might yield real world speedups. As a consequence:
What do we conclude?
The first is that benchmarking isn't easy and never has been easy. An established benchmark that becomes detatched from the real world workload distorts the optimization made. The Google V8 team seem to have the right idea:
Python is currently very popular as a beginner's programming language and has become the preferred language for introductory computer science courses. It is also a very versatile language, cropping up [ ... ]