{"id":157,"date":"2015-12-17T17:15:25","date_gmt":"2015-12-18T01:15:25","guid":{"rendered":"http:\/\/www.howardklein.net\/?p=157"},"modified":"2015-12-17T17:15:25","modified_gmt":"2015-12-18T01:15:25","slug":"random-adventures","status":"publish","type":"post","link":"https:\/\/www.howardklein.net\/?p=157","title":{"rendered":"Random() Adventures"},"content":{"rendered":"<p style=\"text-align: center;\"><em>Better to remain silent and be thought a fool than to speak and to remove all doubt.<\/em><\/p>\n<p style=\"text-align: center;\">Often, but <a href=\"http:\/\/quoteinvestigator.com\/2010\/05\/17\/remain-silent\/\" target=\"_blank\">probably incorrectly<\/a>, attributed to Mark Twain<\/p>\n<p>&nbsp;<\/p>\n<p>The idea of writing about random number generation (I mean, <em>pseudo<\/em>-random number generation, but that term involves way too much typing) makes me extremely nervous.\u00a0 When it comes to that topic, I think of two groups of people:<\/p>\n<ol>\n<li>The High Priesthood. The individuals (one or two dozen, in my imagination) who develop, test and understand these generators<\/li>\n<li>The thousands who make fools of themselves writing about random numbers on the Internet.<\/li>\n<\/ol>\n<p>I am definitely not in the first category; I\u2019ll keep my fingers crossed that this post doesn\u2019t put me solidly into the second.<\/p>\n<p>Back in the day, when dinosaurs roamed the earth and computers were Big Machines in Large Rooms with Raised Floors, there was a popular expression in the industry \u2013 <em><a href=\"http:\/\/corporatevisions.com\/blog\/2007\/06\/11\/no-one-ever-got-fired-for-buying-ibm\/\" target=\"_blank\">No one ever got fired for buying IBM<\/a>.<\/em> The<a href=\"http:\/\/www.math.sci.hiroshima-u.ac.jp\/~m-mat\/MT\/emt.html\" target=\"_blank\"> Mersenne Twister<\/a> is the IBM of pseudo-random number generators.\u00a0 It\u2019s not the fastest or most efficient generator.\u00a0 It may or may not deliver the \u201cbest\u201d results (a debate I\u2019ll leave to the aforementioned priesthood). But it is fast enough, good enough and best-known of the modern vintage of random number generators \u2013 so it has become a de facto standard.\u00a0 If you were to recommend a different generator that I haven\u2019t heard of, I\u2019d probably ask how it compares to the Mersenne Twister.\u00a0 Python\u2019s <code>random<\/code> module uses MT19937, the most commonly used (and heavily tested) version of Mersenne Twister.\u00a0 The standard library designers definitely could have chosen worse.\u00a0 But is it good enough for our discrete event simulator?<\/p>\n<p><!--more--><\/p>\n<p><a href=\"http:\/\/www.amazon.com\/Simulation-Mcgraw-Hill-Industrial-Engineering-Management\/dp\/0073401323\" target=\"_blank\">Law<\/a> (Ch. 7) lists five properties of a \u201cgood\u201d random number generator for simulation purposes:<\/p>\n<ol>\n<li>The output should be uniformly distributed on the interval [0, 1] and uncorrelated.<\/li>\n<li>It should be efficient \u201cenough\u201d, in terms of both speed and memory requirements.<\/li>\n<li>The output should be reproducible \u2013 i.e., given an initial state (such as seed), it should always generate the same output.<\/li>\n<li>To facilitate simulation output analysis, it should be able to generate multiple independent (uncorrelated) streams of sufficient length.<\/li>\n<li>It should be portable, ensuring that given an initial state, the output is the same on every target computing platform.<\/li>\n<\/ol>\n<p>The generator provided by the random module fulfills properties 1, 2, 3 and 5.\u00a0 The fourth is, however, a bit more problematic.<\/p>\n<p>The canonical way to generate multiple independent streams is to divide a single stream into multiple non-overlapping substreams, each of which should meet the same statistical randomness tests as the main stream and be independent of each other.\u00a0 Many random number generators include functions that can be used to initialize substreams from a single stream, either through a jump-ahead or other means.\u00a0 The Python 3.x random module does not.<\/p>\n<p>&nbsp;<\/p>\n<p>First, a bit of history is in order\u2026<\/p>\n<ul>\n<li>The random module initially (or at least as of Python 2.0) included a <code>jumpahead()<\/code> method, which was supported by what was then the default random number generator &#8211; <a href=\"http:\/\/wwwf.imperial.ac.uk\/~ayoung\/m3s9\/wichmannhill.pdf\" target=\"_blank\">Wichmann-Hill<\/a>. The Mersenne Twister replaced Wichmann-Hill in <a href=\"https:\/\/docs.python.org\/2\/library\/random.html#module-random\" target=\"_blank\">version 2.3<\/a>, released in 2003.<\/li>\n<li>The <a href=\"http:\/\/www.math.sci.hiroshima-u.ac.jp\/~m-mat\/MT\/ARTICLES\/mt.pdf\" target=\"_blank\">original Mersenne Twister publications<\/a> (1998) did not include a true jump ahead algorithm. As a result, the behavior of Python jumpahead() was changed in version 2.3 as well. Quoting the documentation: <em>Instead of jumping to a specific state,\u00a0n\u00a0steps ahead,\u00a0jumpahead(n)\u00a0jumps to another state likely to be separated by many steps.<\/em><\/li>\n<li>The <code>jumpahead()<\/code> function was eventually<a href=\"http:\/\/bugs.python.org\/msg115990\" target=\"_blank\"> removed from Python as an API defect<\/a>, as the implementation no longer really matched the function&#8217;s implied semantics.<\/li>\n<li>In 2013, Mersenne Twister jumpahead algorithms were <a href=\"http:\/\/www.iro.umontreal.ca\/~lecuyer\/myftp\/papers\/jumpf2.pdf\" target=\"_blank\">published<\/a> (code <a href=\"http:\/\/www.math.sci.hiroshima-u.ac.jp\/~m-mat\/MT\/JUMP\/index.html\" target=\"_blank\">here<\/a>). Unfortunately, this functionality has not yet been incorporated into the Python <code>random<\/code> module.<\/li>\n<\/ul>\n<p>So where does that leave us? Before I dive in, a couple of questions to consider:<\/p>\n<ul>\n<li>How many streams do I need?<\/li>\n<li>How long is a \u201csufficiently long\u201d stream?<\/li>\n<\/ul>\n<p><em>How many streams do I need?<\/em> In a perfect world, the answer is obvious \u2013 as many as I want! Failing that, we need to think about the two ways that multiple random number streams are used:<\/p>\n<ol>\n<li>Dedicating streams to particular components of the simulation model, which is usually done to facilitate variance reduction during output analysis.<\/li>\n<li>Running replications of the simulation in order to (for example) generate confidence intervals on the output measures. This requires an independent stream (or set of streams) for each replication; the more replications, the lower the variance (and the tighter the confidence intervals).<\/li>\n<\/ol>\n<p>The numbers required is, of course, model and context-dependent.\u00a0 In my prototype simulator, I\u2019ve assumed up to 100 streams per model, and up to 100 replications \u2013 which implies 10,000 independent random number streams.\u00a0 That\u2019s probably the right order of magnitude for the models I\u2019m thinking of.<\/p>\n<p><em>How long is \u201csufficiently long\u201d?<\/em>\u00a0 For a given model, this depends on the number of random variates used (model size, essentially) and the desired length of the simulation run.\u00a0 What might this be for some mythical \u201cmaximal\u201d model?\u00a0 It\u2019s worth a quick back-of-the-envelope estimation\u2026<\/p>\n<p>Let\u2019s consider the M\/M\/1 queuing model described <a href=\"https:\/\/www.howardklein.net\/?p=52\">here<\/a>.\u00a0 It takes roughly two seconds to execute a run of one million simulated time units, with an interarrival mean of ten time units.\u00a0 That translates into roughly 100,000 processes serviced, requiring 200,000 random values (an interarrival time and a service time for each process serviced).\u00a0 So roughly speaking, this model requires 100,000 random numbers per second, 360 million per hour, or roughly 8.6 billion per day (all on my laptop, of course). \u00a0I\u2019m going to go out on a limb and say that if a simulation takes days to execute, than my simulator is probably not the right tool for the job. \u00a0So I&#8217;ll guess that streams of 100 billion are almost certainly more than sufficient for virtually all cases\u00a0\u2013 though being conservative on these sorts of things, I\u2019d really like to add some zeros to that number.<\/p>\n<p>Say we\u2019d like 10,000 independent random number streams, each with a period of at least 100 billion (say roughly 2<sup>37<\/sup>). Given the current limitations of Python\u2019s <code>random<\/code> module, how should we proceed?<\/p>\n<p>For starters, there\u2019s always the brute force method; call the built-in <code>random()<\/code> function one quadrillion (10<sup>15<\/sup>) times while saving the state (obtained via <code>Random.getstate()<\/code>) \u00a0every 100 billion calls.\u00a0 When we\u2019re done, we\u2019ll have the states for 10,000 substreams of length 100 billion. \u00a0That should only take about four years of runtime on my laptop \ud83d\ude42 .<\/p>\n<p>A few other options worth considering, if we don\u2019t have the patience to wait that long:<\/p>\n<ol>\n<li><u>Create a separate generator for each required stream using different seeds<\/u>. This is certainly the easiest solution; unfortunately, there are no guarantees that these streams are independent. While there are differences of opinion, this is not an approach I would use for anything other than demoware\u00a0(until or unless someone publishes a table of seeds that have been shown to have good results in this regard)<\/li>\n<li><u>Use a different random number generator<\/u>. We could use a Python implementation (or wrapper for) a generator that does support multiple independent streams.\u00a0 There is a <a href=\"https:\/\/pypi.python.org\/pypi\/dcmt\" target=\"_blank\">package<\/a>\u00a0wrapping <a href=\"http:\/\/www.math.sci.hiroshima-u.ac.jp\/~m-mat\/MT\/DC\/dc.html\" target=\"_blank\">DCMT<\/a>; unfortunately, the most recent version is pretty old (2011), and there appears to be no support for Python 3.x.\u00a0 There is another package implementing the Well1024a generator, but it does not include jump-ahead functionality.\u00a0 In other words, if I want a Python package that provides multiple independent streams by replacing or augmenting the existing random module, it looks like I\u2019ll have to build at least some of it myself.<\/li>\n<li><u>Generate (and save) the MT19937 states for 10,000 streams offline using the jump-ahead algorithm<\/u>. If I don\u2019t want to go to the trouble of creating, maintaining and delivering a Python wrapper, I could write, cobble together or (better yet) find a non-Python MT19937 implementation that incorporates the jump-ahead algorithm noted above. \u00a0We could then run that code offline to generate the state vectors for all 10,000 substreams, and save them to a file of some sort. \u00a0At simulation run time, we should then be able to simply create generators from the standard random module and initialize their states using values from this file.\u00a0 Using jump-ahead allows us to make the substream length considerably longer, perhaps 2<sup>50<\/sup>.\u00a0 (The MT19937 generator has a period of 2<sup>19937<\/sup> \u2013 1.)<\/li>\n<\/ol>\n<p>Being the intellectually lazy person that I am, my current vote is for option (3). The C++ 2011 standard library includes a <code>&lt;random&gt;<\/code> module that includes an MT19937 generator.\u00a0 The API includes a <code>discard()<\/code> function.\u00a0 While the default implementation is typically dumb\/brute-force (<code>discard(n)<\/code> invokes the generator <code>n<\/code> times), the <a href=\"http:\/\/www.boost.org\/\" target=\"_blank\">Boost<\/a> version of the library implements <code>discard()<\/code> via the aforementioned jump-ahead algorithm (for large values of n).\u00a0 So it\u2019s time to dust off my rusty C++ skills:<\/p>\n<pre class=\"brush: cpp; title: ; notranslate\" title=\"\">\r\n#include &lt;iostream&gt;\r\n#include &lt;sstream&gt;\r\n#include &lt;fstream&gt;\r\n#include &lt;boost\/random\/mersenne_twister.hpp&gt;\r\n#include &lt;math.h&gt;\r\n\r\n\r\nusing namespace std;\r\n\r\nint main(int argc, char** argv)\r\n{\r\n    unsigned long long substream_sz = pow(2,50);\r\n    int nstreams = 10000;\r\n    const char* outfilename = \u201cmt19937_states.dat\u201d;\r\n\r\n    boost::random::mt19937 rng;\r\n    ofstream fout(outfilename, ios::out|ios::binary);\r\n\r\n    for (int i = 0; i &lt; nstreams; ++i) {\r\n        rng.discard(substream_sz);\r\n        std::stringstream input;\r\n        input &lt;&lt; rng;\r\n \r\n        std::vector&lt;unsigned&gt; state;\r\n        unsigned p;\r\n        while (input &gt;&gt; p) {\r\n            state.push_back(p);\r\n        }\r\n\r\n        fout.write(reinterpret_cast&lt;const char*&gt;(&amp;state&#x5B;0]), \r\n                   state.size() * sizeof(unsigned));\r\n    }\r\n}\r\n<\/pre>\n<p>&nbsp;<\/p>\n<p>This small program generates the 10,000 substream state vectors and writes them to a binary file.\u00a0 It runs in about 18 minutes on my laptop.<\/p>\n<p>At this point, I\u2019d like to run that binary file through a short Python script in order to turn it into a portable (machine-independent) <a href=\"http:\/\/docs.scipy.org\/doc\/numpy\/neps\/npy-format.html\" target=\"_blank\">NPY<\/a>\u00a0format:<\/p>\n<pre class=\"brush: python; title: ; notranslate\" title=\"\">\r\nstate_array = numpy.fromfile(\u201cmt19937_states.dat\u201d)\r\nstate_array.reshape(10000, 624)\r\nstate_array.save(\u201cmt19937_states.npy\u201d)\r\n<\/pre>\n<p>&nbsp;<\/p>\n<p>And finally, just to close the loop, a snippet of Python code demonstrating how a simulator might initialize random number generator instances for each of these streams:<\/p>\n<pre class=\"brush: python; title: ; notranslate\" title=\"\">\r\n# Load the .npy file, which contains an nstreams x 624 array \r\n# of unsigned ints (type numpy.uint32)\r\nnparray = numpy.load(\u201cmt19937_states.npy\u201d)\r\nnstreams = nparray.shape&#x5B;0]\r\n\r\n# Create and initialize a generator for each defined stream\r\nrng = &#x5B;]\r\nfor i in range(nstreams):\r\n    # Convert the array of numpy.uint32s to a list of Python ints\r\n    state_list = &#x5B;int(x) for x in nparray&#x5B;i]]\r\n    \r\n    # Append the required value of 624 to the list\r\n    state_list.append(624)\r\n    \r\n    # Define the required internalstate tuple and use it to \r\n    # set the state for a new RNG instance.\r\n    rng_state = (3, tuple(state_list), None)\r\n    rng.append(random.Random())\r\n    rng&#x5B;-1].setstate(rng_state)\r\n<\/pre>\n<p>&nbsp;<\/p>\n<p>A few notes on the code above:<\/p>\n<ul>\n<li>The argument to <code>Random.setstate()<\/code> is the tuple <code>(version, internalstate, gauss_next)<\/code>. <code>version<\/code> is 3 for the Python 3.x distributions that I use, <code>gauss_next<\/code> is always <code>None<\/code> when initializing a Random instance, as near as I can tell.<\/li>\n<li><code>internalstate<\/code> is a tuple containing 635 Python ints \u2013 the 624 elements of the MT19937 state vector plus a value of 624 at the end. <code>Random<\/code> is very particular about types here; <code>internalstate<\/code> <em>must<\/em> be a tuple (<u>not<\/u> a list!), and the elements <strong>must<\/strong> be ints (<span style=\"text-decoration: underline;\">not<\/span> NumPy types such as uint32). Hence the extra data massaging in the code above.<\/li>\n<\/ul>\n<p>All in all, I\u2019m pretty comfortable with this approach, at least until the Python standard library (or, more likely, NumPy) incorporates jump-ahead functionality and\/or an additional, pluggable random number generator such as DCMT. \u00a0And hopefully I haven\u2019t made an utter fool of myself writing about it!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Better to remain silent and be thought a fool than to speak and to remove all doubt. Often, but probably incorrectly, attributed to Mark Twain &nbsp; The idea of writing about random number generation (I mean, pseudo-random number generation, but that term involves way too much typing) makes me extremely nervous.\u00a0 When it comes to&#8230;&nbsp;(<a href=\"https:\/\/www.howardklein.net\/?p=157\">read more<\/a>)<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"nf_dc_page":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[4,5,2],"tags":[],"class_list":["post-157","post","type-post","status-publish","format-standard","hentry","category-goals-requirements","category-python","category-simulation"],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/www.howardklein.net\/index.php?rest_route=\/wp\/v2\/posts\/157","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.howardklein.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.howardklein.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.howardklein.net\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.howardklein.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=157"}],"version-history":[{"count":16,"href":"https:\/\/www.howardklein.net\/index.php?rest_route=\/wp\/v2\/posts\/157\/revisions"}],"predecessor-version":[{"id":173,"href":"https:\/\/www.howardklein.net\/index.php?rest_route=\/wp\/v2\/posts\/157\/revisions\/173"}],"wp:attachment":[{"href":"https:\/\/www.howardklein.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=157"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.howardklein.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=157"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.howardklein.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=157"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}