Mersenne Twister Considered Harmful 
Written by Mike James  
Wednesday, 16 October 2019  
The Mersenne Twister is a very common random number generator. It is used in C, Python, Mathematica, Excel, PHP, Ruby ... A new paper suggests it shouldn't be. Random Number More cartoon fun at xkcd a webcomic of romance,sarcasm, math, and languag The quality of the pseudo random numbers we use is important and the adoption of the Mersenne Twister generator, which is a special case of an F2 linear generator, was partly in response to the discovery that the current generators were flawed. As a new research paper by Sebastiano Vigna, inventor of xorshift+ and xoroshiro128+ used in the JavaScript engines of Chrome, FIrefox and Safari, says: "Once language implementors decided to use the Mersenne Twister, the choice was doomed to be cemented for a very long time. Also, the problems of the Mersenne Twister are not immediately detectable in everyday applications, and most users really interested in the quality of their PRNG will make an informed choice, rather than relying on the stock PRNG of whichever programming environment they are using. So there was no strong motivation to move towards a better PRNG." Part of the argument for not moving from the Mersenne Twister is that the tests it did fail were not important in the real world use of the generator. In particular, if you take the bits generated as uniformly distributed in the interval 0 to 1 and then apply a nonlinear transformation to a different distribution then the defects go away. This is indeed the way that many simulations use pseudo random numbers but the new paper gives examples where this is not the case. The bottom line of all of these examples is: "F2linear generators fail statistical tests which can have an impact on actual applications. They should be used with care, and only in context where there it is certain that strongly nonlinear operations will be applied to their output, so their defects are diluted or completely hidden. In particular, linear generators should never be used as generalpurpose generators, unless their output is suitably scrambled by combining it with other, nonlinear generators, or by applying nonlinear maps. The current, dangerous ubiquity of the Mersenne Twister as basic PRNG in many environments is a historical artifact that we, as a community, should take care of. Moreover, several touted advantages of the Mersenne Twister, such as reparameterization, do not actually work properly when examined closely." There also seems to be an easy fix to the problem  the use of a scrambler as in the author's own xoshiro256++ means that we have a generator which does not fail any known statistical test. The point is there are better ways of doing the job and we are not using them.
More InformationIt is high time we let go of the Mersenne Twister Related ArticlesNeural Networks Learn to Produce Random Numbers ERNIE  A Random Number Generator How not to shuffle  the Knuth FisherYates algorithm Canada's RAND Immigration Lottery Not Random! Random Means Random  The Green Card Fiasco Randomness Restored In Chrome 49 To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.
Comments
or email your comment to: comments@iprogrammer.info


Last Updated ( Wednesday, 16 October 2019 ) 