Arvids Blog

On C++ Random Number Generator Quality

tl;dr: Do not use <random>’s generators. There are plenty good alternatives, like PCG, SplitMix, truncated Xorshift32*, or Xorshift1024*.

A recent tweet reminded me of the <random> facilities in the C++ standard library. Having just recently studied and implemented a couple random number generators myself, I was curious how they hold up in a modern test.
From the get-go, I knew this is going to end badly, but I gave it a shot anyway.
I build a little test program, to feed into PractRand:

#include <random>
#include <stdio.h>
#include <stdint.h>
#include <stddef.h>

template<class Rand, class T>
static void
run()
{
    std::random_device rd;
    Rand gen(rd());
    std::uniform_int_distribution<T> dis(0);

    T val;
    while (1) {
        val = dis(gen);
        fwrite((void *)&val, 1, sizeof(T), stdout);
    }
}

int main()
{
    freopen(nullptr, "wb", stdout); /* only required for windows */
    run<std::ranlux24_base, uint32_t>();
    return 0;
}

The test program is printing unsigned 32bit numbers to stdout, which will be piped into the RNG_Test executable from the PractRand suite.
I compiled a single executable for each generator, and started the tests.

Results

Most of the tests were over in just a couple seconds.

The table shows after how many iterations the generator fails the statistical tests.
The earlier it fails, the worse it is. Modern PRNGs, like PCG, are known to not fail the statistical tests at all.
That’s only have the story, though.
With the table above showing the statistical quality of each generator, the following is listing it’s properties:

While both the 32-bit and 64-bit mersenne twister fail (mt19937 & mt19937_64) at some seemingly high amount of data (256G and 512G, respectively), they are fundamentally flawed. They’re easily predictable, large (2.5KiB of state) and multiple times slower than any other alternative.

If you must use any generator from <random>, use either ranlux24 or ranlux48, the only two which showed a reasonable result in the test. At the expense of performance.

While <random> has the building blocks to build a decent generator, with good statistical quality, it requires knowledge of the domain, and is not trivial. The predefined generators didn’t stand the test of time, and are weak throughout. To make matters worse, the default_random_engine often defaults to mt19937, which, while not as bad as the other choices, is neither fast, lean nor unpredictable.
There is a certain irony, that it’s known that rand() is terrible generator, while everybody is fine that C++ includes almost the exact same generator (rand() is usually implemented as an LCG).

Raw Test Results

Following are the raw tests results linked, for your viewing pleasure.

• ranlux24_base
• ranlux48_base
• minstd_rand
• minstd_rand0
• knuth_b
• mt19937
• ranlux24
• ranlux48

ranlux24_base

$ ./ranlux24_base | ./PractRand/RNG_test stdin32 -multithreaded
RNG_test using PractRand version 0.93
RNG = RNG_stdin32, seed = 0xc6851f49
test set = normal, folding = standard (32 bit)

rng=RNG_stdin32, seed=0xc6851f49
length= 256 megabytes (2^28 bytes), time= 3.5 seconds
 Test Name Raw Processed Evaluation
 BCFN(2+1,13-2,T) R= +9.7 p = 1.5e-4 mildly suspicious
 BCFN(2+2,13-3,T) R= +9.0 p = 4.6e-4 unusual
 [Low8/32]BCFN(2+0,13-3,T) R= +91.1 p = 6.3e-43 FAIL !!!
 [Low8/32]BCFN(2+1,13-3,T) R= +24.4 p = 2.4e-11 FAIL
 [Low8/32]BCFN(2+2,13-4,T) R= +12.3 p = 2.4e-5 mildly suspicious
 [Low1/32]BCFN(2+0,13-5,T) R= +45.5 p = 5.4e-18 FAIL !
 [Low1/32]DC6-9x1Bytes-1 R=+485.8 p = 1.8e-256 FAIL !!!!!!
 ...and 117 test result(s) without anomalies

ranlux48_base

$ ./ranlux48_base | ./PractRand/RNG_test stdin32 -multithreaded
RNG_test using PractRand version 0.93
RNG = RNG_stdin32, seed = 0x13294a68
test set = normal, folding = standard (32 bit)

rng=RNG_stdin32, seed=0x13294a68
length= 256 megabytes (2^28 bytes), time= 2.8 seconds
 Test Name Raw Processed Evaluation
 BCFN(2+1,13-2,T) R= +28.1 p = 6.0e-14 FAIL
 [Low8/32]BCFN(2+0,13-3,T) R=+251.2 p = 1.0e-118 FAIL !!!!!
 [Low8/32]BCFN(2+1,13-3,T) R= +20.5 p = 1.6e-9 VERY SUSPICIOUS
 [Low1/32]BCFN(2+0,13-5,T) R= +16.0 p = 1.9e-6 very suspicious
 [Low1/32]DC6-9x1Bytes-1 R= +49.8 p = 3.4e-26 FAIL !!
 ...and 119 test result(s) without anomalies

minstd_rand

$ ./minstd_rand | ./PractRand/RNG_Test stdin32 -multithreaded
RNG_test using PractRand version 0.93
RNG = RNG_stdin32, seed = 0x665a9f8e
test set = normal, folding = standard (32 bit)

rng=RNG_stdin32, seed=0x665a9f8e
length= 256 megabytes (2^28 bytes), time= 3.9 seconds
 Test Name Raw Processed Evaluation
 FPF-14+6/16:(0,14-0) R= -7.9 p =1-6.2e-7 mildly suspicious
 FPF-14+6/16:(1,14-0) R= -7.1 p =1-2.8e-6 mildly suspicious
 FPF-14+6/16:(2,14-0) R= -6.3 p =1-1.8e-5 unusual
 FPF-14+6/16:all R= -13.4 p =1-1.0e-12 FAIL
 FPF-14+6/16:all2 R= +24.9 p = 2.8e-11 VERY SUSPICIOUS
 [Low8/32]FPF-14+6/16:all R= -6.7 p =1-4.2e-6 suspicious
 ...and 118 test result(s) without anomalies

minstd_rand0

$ ./minstd_rand0 | ./PractRand/RNG_Test stdin32 -multithreaded
RNG_test using PractRand version 0.93
RNG = RNG_stdin32, seed = 0x3b74f60f
test set = normal, folding = standard (32 bit)

rng=RNG_stdin32, seed=0x3b74f60f
length= 128 megabytes (2^27 bytes), time= 2.2 seconds
 Test Name Raw Processed Evaluation
 BCFN(2+2,13-3,T) R= +7.8 p = 1.8e-3 unusual
 FPF-14+6/16:(0,14-0) R= -6.5 p =1-1.0e-5 unusual
 FPF-14+6/16:all R= -8.2 p =1-1.5e-7 very suspicious
 FPF-14+6/16:all2 R= +10.6 p = 1.3e-5 mildly suspicious
 ...and 113 test result(s) without anomalies

rng=RNG_stdin32, seed=0x3b74f60f
length= 256 megabytes (2^28 bytes), time= 4.6 seconds
 Test Name Raw Processed Evaluation
 FPF-14+6/16:(0,14-0) R= -9.4 p =1-2.6e-8 very suspicious
 FPF-14+6/16:(2,14-0) R= -8.7 p =1-9.5e-8 suspicious
 FPF-14+6/16:all R= -13.5 p =1-8.8e-13 FAIL
 FPF-14+6/16:all2 R= +31.8 p = 5.2e-14 FAIL
 [Low8/32]FPF-14+6/16:all R= -4.2 p =1-1.2e-3 unusual
 [Low1/32]BCFN(2+1,13-5,T) R= -6.5 p =1-5.1e-4 unusual
 ...and 118 test result(s) without anomalies

knuth_b

$ ./knuth_b | ./PractRand/RNG_Test stdin32 -multithreaded
RNG_test using PractRand version 0.93
RNG = RNG_stdin32, seed = 0x2ba764e1
test set = normal, folding = standard (32 bit)

rng=RNG_stdin32, seed=0x2ba764e1
length= 128 megabytes (2^27 bytes), time= 2.9 seconds
 no anomalies in 117 test result(s)

rng=RNG_stdin32, seed=0x2ba764e1
length= 256 megabytes (2^28 bytes), time= 5.8 seconds
 Test Name Raw Processed Evaluation
 FPF-14+6/16:all R= -5.5 p =1-6.4e-5 mildly suspicious
 ...and 123 test result(s) without anomalies

rng=RNG_stdin32, seed=0x2ba764e1
length= 512 megabytes (2^29 bytes), time= 10.9 seconds
 Test Name Raw Processed Evaluation
 FPF-14+6/16:(0,14-0) R= -6.8 p =1-5.5e-6 unusual
 FPF-14+6/16:all R= -8.7 p =1-4.7e-8 very suspicious
 FPF-14+6/16:all2 R= +8.9 p = 4.2e-5 mildly suspicious
 ...and 129 test result(s) without anomalies

rng=RNG_stdin32, seed=0x2ba764e1
length= 1 gigabyte (2^30 bytes), time= 20.8 seconds
 Test Name Raw Processed Evaluation
 FPF-14+6/16:(0,14-0) R= -12.8 p =1-1.9e-11 VERY SUSPICIOUS
 FPF-14+6/16:(2,14-0) R= -6.7 p =1-7.3e-6 unusual
 FPF-14+6/16:all R= -15.4 p =1-1.1e-14 FAIL !
 FPF-14+6/16:all2 R= +44.4 p = 2.4e-19 FAIL !
 ...and 137 test result(s) without anomalies

mt19937

$ ./mt19937 | ./PractRand/RNG_test stdin32 -multithreaded
RNG_test using PractRand version 0.93
RNG = RNG_stdin32, seed = 0xadda9d07
test set = normal, folding = standard (32 bit)

rng=RNG_stdin32, seed=0xadda9d07
length= 512 megabytes (2^29 bytes), time= 3.5 seconds
 no anomalies in 132 test result(s)

rng=RNG_stdin32, seed=0xadda9d07
length= 1 gigabyte (2^30 bytes), time= 7.0 seconds
 no anomalies in 141 test result(s)

rng=RNG_stdin32, seed=0xadda9d07
length= 2 gigabytes (2^31 bytes), time= 13.6 seconds
 no anomalies in 148 test result(s)

rng=RNG_stdin32, seed=0xadda9d07
length= 4 gigabytes (2^32 bytes), time= 26.5 seconds
 no anomalies in 156 test result(s)

rng=RNG_stdin32, seed=0xadda9d07
length= 8 gigabytes (2^33 bytes), time= 52.4 seconds
 no anomalies in 165 test result(s)

rng=RNG_stdin32, seed=0xadda9d07
length= 16 gigabytes (2^34 bytes), time= 104 seconds
 no anomalies in 172 test result(s)

rng=RNG_stdin32, seed=0xadda9d07
length= 32 gigabytes (2^35 bytes), time= 203 seconds
 no anomalies in 180 test result(s)

rng=RNG_stdin32, seed=0xadda9d07
length= 64 gigabytes (2^36 bytes), time= 404 seconds
 no anomalies in 189 test result(s)

rng=RNG_stdin32, seed=0xadda9d07
length= 128 gigabytes (2^37 bytes), time= 794 seconds
 no anomalies in 196 test result(s)

rng=RNG_stdin32, seed=0xadda9d07
length= 256 gigabytes (2^38 bytes), time= 1620 seconds
 Test Name Raw Processed Evaluation
 DC6-9x1Bytes-1 R= -5.3 p =1-3.6e-3 unusual
 [Low8/32]BRank(12):12K(1) R= +6116 p~= 4e-1842 FAIL !!!!!!!!
 ...and 202 test result(s) without anomalies

ranlux24

$ ./ranlux24 | ./PractRand/RNG_test stdin32 -multithreaded
RNG_test using PractRand version 0.93
RNG = RNG_stdin32, seed = 0xd8322a9c
test set = normal, folding = standard (32 bit)

rng=RNG_stdin32, seed=0xd8322a9c
length= 64 megabytes (2^26 bytes), time= 2.1 seconds
 no anomalies in 108 test result(s)

rng=RNG_stdin32, seed=0xd8322a9c
length= 128 megabytes (2^27 bytes), time= 4.6 seconds
 no anomalies in 117 test result(s)

rng=RNG_stdin32, seed=0xd8322a9c
length= 256 megabytes (2^28 bytes), time= 9.2 seconds
 no anomalies in 124 test result(s)

rng=RNG_stdin32, seed=0xd8322a9c
length= 512 megabytes (2^29 bytes), time= 17.6 seconds
 no anomalies in 132 test result(s)

rng=RNG_stdin32, seed=0xd8322a9c
length= 1 gigabyte (2^30 bytes), time= 33.7 seconds
 no anomalies in 141 test result(s)

rng=RNG_stdin32, seed=0xd8322a9c
length= 2 gigabytes (2^31 bytes), time= 65.8 seconds
 no anomalies in 148 test result(s)

rng=RNG_stdin32, seed=0xd8322a9c
length= 4 gigabytes (2^32 bytes), time= 129 seconds
 no anomalies in 156 test result(s)

rng=RNG_stdin32, seed=0xd8322a9c
length= 8 gigabytes (2^33 bytes), time= 256 seconds
 no anomalies in 165 test result(s)

rng=RNG_stdin32, seed=0xd8322a9c
length= 16 gigabytes (2^34 bytes), time= 510 seconds
 no anomalies in 172 test result(s)

rng=RNG_stdin32, seed=0xd8322a9c
length= 32 gigabytes (2^35 bytes), time= 1008 seconds
 no anomalies in 180 test result(s)

rng=RNG_stdin32, seed=0xd8322a9c
length= 64 gigabytes (2^36 bytes), time= 1920 seconds
 no anomalies in 189 test result(s)

rng=RNG_stdin32, seed=0xd8322a9c
length= 128 gigabytes (2^37 bytes), time= 3743 seconds
 no anomalies in 196 test result(s)

rng=RNG_stdin32, seed=0xd8322a9c
length= 256 gigabytes (2^38 bytes), time= 7372 seconds
 no anomalies in 204 test result(s)

rng=RNG_stdin32, seed=0xd8322a9c
length= 512 gigabytes (2^39 bytes), time= 14681 seconds
 Test Name Raw Processed Evaluation
 [Low1/32]DC6-9x1Bytes-1 R= +4.5 p = 0.010 unusual
 ...and 212 test result(s) without anomalies

rng=RNG_stdin32, seed=0xd8322a9c
length= 1 terabyte (2^40 bytes), time= 29803 seconds
 no anomalies in 220 test result(s)

rng=RNG_stdin32, seed=0xd8322a9c
length= 2 terabytes (2^41 bytes), time= 60168 seconds
 Test Name Raw Processed Evaluation
 [Low1/32]DC6-9x1Bytes-1 R= +5.6 p = 8.4e-3 unusual
 ...and 227 test result(s) without anomalies

ranlux48

$ ./ranlux48 | ./PractRand/RNG_test stdin32 -multithreaded
RNG_test using PractRand version 0.93
RNG = RNG_stdin32, seed = 0xd97451a6
test set = normal, folding = standard (32 bit)

rng=RNG_stdin32, seed=0xd97451a6
length= 64 megabytes (2^26 bytes), time= 2.1 seconds
 no anomalies in 108 test result(s)

rng=RNG_stdin32, seed=0xd97451a6
length= 128 megabytes (2^27 bytes), time= 4.5 seconds
 no anomalies in 117 test result(s)

rng=RNG_stdin32, seed=0xd97451a6
length= 256 megabytes (2^28 bytes), time= 8.9 seconds
 no anomalies in 124 test result(s)

rng=RNG_stdin32, seed=0xd97451a6
length= 512 megabytes (2^29 bytes), time= 17.2 seconds
 Test Name Raw Processed Evaluation
 DC6-9x1Bytes-1 R= +5.5 p = 5.4e-3 unusual
 ...and 131 test result(s) without anomalies

rng=RNG_stdin32, seed=0xd97451a6
length= 1 gigabyte (2^30 bytes), time= 33.1 seconds
 Test Name Raw Processed Evaluation
 [Low8/32]DC6-9x1Bytes-1 R= +6.1 p = 3.3e-3 unusual
 ...and 140 test result(s) without anomalies

rng=RNG_stdin32, seed=0xd97451a6
length= 2 gigabytes (2^31 bytes), time= 64.6 seconds
 no anomalies in 148 test result(s)

rng=RNG_stdin32, seed=0xd97451a6
length= 4 gigabytes (2^32 bytes), time= 127 seconds
 no anomalies in 156 test result(s)

rng=RNG_stdin32, seed=0xd97451a6
length= 8 gigabytes (2^33 bytes), time= 253 seconds
 no anomalies in 165 test result(s)

rng=RNG_stdin32, seed=0xd97451a6
length= 16 gigabytes (2^34 bytes), time= 501 seconds
 no anomalies in 172 test result(s)

rng=RNG_stdin32, seed=0xd97451a6
length= 32 gigabytes (2^35 bytes), time= 1001 seconds
 no anomalies in 180 test result(s)

rng=RNG_stdin32, seed=0xd97451a6
length= 64 gigabytes (2^36 bytes), time= 2014 seconds
 no anomalies in 189 test result(s)

rng=RNG_stdin32, seed=0xd97451a6
length= 128 gigabytes (2^37 bytes), time= 4182 seconds
 Test Name Raw Processed Evaluation
 DC6-9x1Bytes-1 R= -4.3 p = 0.989 unusual
 ...and 195 test result(s) without anomalies

rng=RNG_stdin32, seed=0xd97451a6
length= 256 gigabytes (2^38 bytes), time= 8169 seconds
 no anomalies in 204 test result(s)

rng=RNG_stdin32, seed=0xd97451a6
length= 512 gigabytes (2^39 bytes), time= 16166 seconds
 no anomalies in 213 test result(s)

rng=RNG_stdin32, seed=0xd97451a6
length= 1 terabyte (2^40 bytes), time= 32123 seconds
 no anomalies in 220 test result(s)