Why Simluate?

So you want to do a simulation. Go lie down and wait for the feeling to go away. Really, you'll thank me.

The most important thing to ask yourself is: "why?" There are a couple of very bad reasons to do a simulation, and a couple of really bad ones.

The bad ones are:

• It seems easy. That may be true, but it isn’t easy to do right. There are many mistakes to make (trust me, I've made most of them). Doing it right takes a lot of thought and an immense effort.
• Well I’ve got this computer sitting here doing nothing…. There is an old saying, “A fool with a tool is still a fool." or is that "... still a tool." No matter. As a scientist you have only one thing of value, your reputation. If you get the reputation for doing simple, careless work, then it doesn't matter how brilliant hyou are. No one will listen or care.

The only good reason to to a simulation is:

• To understand something that we can’t develop a good analytic result for.

That’s right. That is the only reason to do statistical simulation is as a poor proxy for an analytic result.

Remember:

One good derivation wipes away several decades of simulations.

So why is simulation worth doing at all?

1) We can’t do the derivation. This may be the fault of our personal training, or the difficulty of the task, whatever. But realize that if we are turning to simulation, we are announcing that we

2) The derivation is asymptotic. Most of us don’t live in the blessed land of Asymptopia, the land of infinite sample sizes where everything is linear and/or distributed normally. Many of the statistical tools that we have (Taylor series, first-order asymptotics, etc.) rely on large sample theory in order for their results to hold.

3) The procedure makes an assumption (normality, independence of observations, etc.) that probably isn’t true. As practitioners, we would like know how sensitive to the assumptions the procedure is.

4) We want to look at the properties of a particular implementation of a procedure. In the early 1980’s, confirmatory factor analysis and structural equations became practical. There was a great deal of interest in comparing software for estimating the parameters: particularly LISREL v. EQS. In IRT we have LOGIST v. BILOG v. PARSCALE v. MULTILOG. More recently, methods of handling missing data (based on Little and Rubin’s (1976) work) became available in general-use software such as SPSS and SAS, and there was a flurry of work comparing the implementations.

So, make sure you have a good reason for entering into the simulation arena. A good simulation brings useful information to a generation of practitioners. But, beware; here be dragons. There are an awful lot of ways to go wrong.