Seedy Random Number Generation

One of the advantages of the multiple small runs strategy (described last time) was that it makes it easy to replicate any individual run. That gives us the capability to examine the details of a specific run, such as one in which the iterations did not converge, or that aborted.

There are two ways to do this; keep all of the individual data sets, or be able to re-generate each and every data set whenever we want.

Whether we are using a strategy of generating all of the data, or of sampling from a large dataset, or re-sampling (for a jackknife- or bootstrap-based study), we will need to generate a series of (pseudo-) random numbers. All of the pseudo-random number generators make use of a seed. We input some value, and then a series of numbers is generated. Once the seed is provided, the series is completely determined. Most implementations provide two possibilities for the seed:

1. explicitly provide a seed value
2. implicitly have the seed generated for us

Most generators make it simpler to one or the other of these, but pretty much every generator I have used allows both options. A common strategy for implicit seeding is to use the system's clock.

The key to being able to replicate runs is to capture the seed for every run. We can do this by using an implicit seed, and recording that seed (say by appending it to a file). I have found this approach to be extremely error prone, and don't advise it. Instead, my approach is to generate a seed file by using a random number generator (often with an implicit seed) to generate a file containing a large number of seed values., one for each replication. Then I copy the seed file to a backup (a separate disk or flash drive). In general, I will have one seed file per condition of the study. That file should contain at least the number of iterations I want. In general, I have some overage (at least 10% if I'm optimistic, 100% if I'm feeling pessimistic). That way if a specific run hiccups, my simulations can continue without running out of seeds.

Say that I call the file something clever, like "seed.dat". Each replication:

1. creates a backup copy of the file--"seed.bak",
2. reads the N values in the seed file,
3. removes the top value, and
4. writes N-1 values (without the top value) back to same file (seed.dat) for the next replication's use.

The purpose of the backup is to be able to kill the current run. I often want to pause the simulation to do some interim data analyses. Using this strategy, I can copy seed.bak to seed.dat, and resume the simulation.

This approach gives me two advantages:

1. I can don't have to keep the individual data sets around, reducing the possibility that I'l run into storage problems
2. I can re-generate any specific data set by simply using the corresponding seed value
3. Finally, when I go to archive the files associated with the study, I only have to save the seed files, secure in the knowledge that I can recreate anything (from a single run to the entire study) at a moment's notice.

Doing the Replications: (Again, and again and again...)

We'll consider how to get the data for the replications in another installment. But first, I want to talk about the general approach to doing a large simulation study.

There are two schools of though there:

1. One BIG RUN. It goes for 4 weeks, performs all 50,000 replications, and then prints out a 7. This kind of thing is routinely done in the physical sciences; I've read stories about work on subatomic particles that do this one-long-run approach.
2. 50,000 small runs. Then we pull the results from all of the files, and create a big data set.

I fall firmly into the second camp. There a several reasons.

1. I have had the experience of the BIG RUN go for 3 weeks, 6 days, and facilities turns off the power for some needed maintenance. This always results in me saying a long spew of bad words, that my mother would not approve of.
2. Somewhere during the process, a few runs hiccup. It may be that iterations fail to converge; that the dataset is defective; something. Now I have 49,936 replications, and no idea what happened to the other 64 runs. And actually, it may be that the 64 runs are trying to tell me something (maybe more interesting that what I thought I was looking for). However, I don't know what dataset produced the problem, why, or anything else. In short, I'm SOL [1].
3. There may be something else about the data, like the skewness of the statistic, or the standard error of the estimated standard errors; (it's always weird) that a journal reviewer wants to know about before approving the manuscript. If you've done the BIG RUN, you have little choice but to redo it--more bad words. If I have the dataset of individual runs, I can re-analyze it. Sometimes, the reviewer is on to something, or they may be full of hot air. Either way, I can go back to the data. With the BIG RUN, it's not an option.
4. It is easier to monitor the progress. If the conference paper is due in 3 days and I'm only on replication 17, it is time to re-group.
5. (and this is my favorite), if I structure the replications so that each of the conditions is executed in round-robin fashion, I can do some preliminary analysis after 10 or 50 replications. Based on these, I can do things like power analyses for the effects I'm looking for and realize that I need more (or fewer!) replications. Or, things may look different than I expected, and I can shut it down, call it a pilot, and go off in another direction.

In short, I've been burned a bunch of different ways by the BIG RUN. In the physical sciences like subatomic physics, they may understand the domain to a depth that this approach makes sense. In statistical simulations, there are some domains where that is true, but in general we are working with newer, lesser known quantities, and circumspection is well-advised.

There are ways to reduce the risks/downsides of the BIG RUN. But, overall, I find that doing a bunch of little runs gives me many more degrees of freedom to adapt to conditions that arise. I've tried it both ways, and lots of small runs is better.

There are downsides to doing small runs:

1. Disk space can become a problem,
2. You often can't use simulators that are baked into the software,
3. Figuring out how to run a commercial program (especially one that is based on a GUI) can pose a major problem,
4. With so many tiny pieces, it is easy to get lost in the detail.
5. Pulling the results together from lots of individual files can be a major undertaking.

In the next post, I'll talk about my general approach. Then we'll consider, how to solve some of the trickier problems.

[1] American slang meaning "Sh*t Out of Luck."