Fake data, part 3: Bypassing the central limit theorem

May 2017: This is from my other blog that's no longer online. The original comments are no longer available, but you are welcome to add more.

Now that I have my black swan distribution, and I verified that it fits market returns remarkably well while being mathematically tractable, I want to use it to generate artificial market data.

Generating a series of closing prices is easy.

  1. Select a starting price, take the logarithm
  2. Select a mean μ and standard deviation σ of log returns ln(P0/P1) where P0 is the most recent price and P1 is the previous price.
  3. Generate a uniformly-distributed random probability p between 0 and 1. Plug it into the inverse black swan distribution (using a=1.6):
    $$B^{-1}(p;\mu,s) = \mu - 2 s \, \sinh \left[\frac {\tanh^{-1}(1-2p)} {a} \right]$$
  4. Add the result to a running total.
  5. Go to step 3. Repeat as often as desired.
Then simply calculate the antilog or exponential of the values in the running total to get prices.

This works fine. But what if we want to generate more than just single prices? How do I get a series of daily price bars, with high, low, and closing values for my fake market? The way to mimic real market behavior is to generate a series of n values for each "day" and record the highest, lowest, and final price for high, low, and close of the day. That is, define Sd as a daily step and Si as an intraday step:

$$S_d = \sum_{i=1}^n S_i$$ where $$S_i = B^{-1}(p,\mu_i,s_i) = \mu_i - 2 s_i \, \sinh \left[\frac {\tanh^{-1}(1-2p)} {a} \right]$$ $$\mu_i = \frac{\mu_d}{n}, \quad s_i = \frac {s_d}{\sqrt n}$$ Black swan random walk sampling every n steps

The scaling of the intraday mean μi and standard deviation σi (corresponding to the shape parameter si) is necessary to preserve the desired mean μd and standard deviation σd for daily values. As described in the previous part, the approximation for the distribution shape parameter s is:

$$s \approx \frac{\sqrt{6}\,\sigma} {\pi \tanh^{-1} \left(\frac{1}{a}\right) \sqrt{\tanh^{-1} \left(\frac{1}{a}\right)^2+2}}$$ ...using a=1.6. But look at what happened! A disaster. The figure shows the resulting distributions for 1, 10, and 100 intraday steps. Look at the big rounded top made up of green dots representing n=100. As the number of intraday steps n grows, the resulting distribution of daily values approaches a normal distribution.

I'm back where I started! The whole point was to find something better than a normal distribution to model the markets. I don't want to end up with a normal distribution, I want to end up with my black swan distribution.

So what happened? The central limit theorem, evidently the "sovereign law" of probability theory, came in and took over. It says that the sum of many random variables with a finite mean and variance will be normally distributed, regardless of the underlying distribution we start with.

Although the black swan distribution has infinite kurtosis, it does have a finite mean and variance. This means, if we generate a bunch of tiny black swan steps to build large "daily" steps in a random walk, the large steps will approximate a normal distribution.

How can I break this law?

After much experimentation, I found a way to get past the central limit theorem. If we perturb μd to vary black-swanly with each daily step (not each intraday step), we get values that still appear distributed according to black swan, and so are the sums! $$\large \varepsilon = \begin {cases} \frac{ B^{-1}(p,0,s_d)}{\sqrt{n}\sqrt{n+1}} = \frac{\mu_d - 2 s_d \, \sinh \left[\frac {\tanh^{-1}(1-2p)}{a} \right]}{\sqrt{n}\sqrt{n+1}} & n>1 \\ 0 & n=1 \end {cases}$$ Re-generate ε once per day, and use that value for all intraday steps (if you generate a new ε for each intraday step, you end up with a normal distribution again). For each intraday step Si, use these values of μi and si: $$\mu_i = \frac{\mu_d}{n}+\varepsilon, \quad s_i = \frac {s_d}{n}$$ For large n, the adjustment to μd basically perturbs the mean each day by a small black swan distribution having a standard deviation of σ/n. For n>50 or so, one can simply use n by itself in the denominator of the expression for ε. Again, p is a random probability between 0 and 1.

Here's how it worked out:

That seems to work. I can now generate an artificial series of high-low-close prices that have similar statistical properties to actual markets. Now I must investigate whether any dependencies exist between successive values in the series.

I want to mention that the plots in this article use the Rnd() function in Excel's Visual Basic. The Visual Basic random number generator is fairly crude by modern standards, which may explain the noisy tails of the distributions shown (using 16,000 samples) versus the relatively cleaner tails of the distributions measured from actual market data (using less samples). I can't know for sure unless I try a better random number generator, but I'm not inclined to do so now — I'm satisfied with these results.

Comments

Post a Comment

Popular posts from this blog

Syncing Office 365 Outlook to Google calendar using Power Automate

Image
When my employer switched over from the Google office suite to Microsoft's Office 365, it caused unexpected disruption in my family. I had been sharing my work calendar with my wife (sharing no details, just busy times). It became an integral part of her calendar, to help her coordinate (between her job and my job) who picks up and drops off my son from school, and to let her know when I was free to take a phone call during my work day. Since the switchover to O365, my ability to share a calendar externally was eliminated. Based on documentation I could find, it should be possible, but the option doesn't appear in my employer's implementation of Outlook. I can view my personal Google calendar in my work Outlook calendar, but I can't share it the other way. Microsoft's Power Automate provides a way to solve this problem. It lets you create flows of data, triggered manually or automatically by events, to move data around, not only among the Microsoft products but
Read more

New approach to screw threads in OpenSCAD

Image
This is an intellectual exercise with a practical end-result. There are better approaches to this problem, but the approach I describe here is interesting. A screw thread from a stack of discs Imagine a stack of 50 coins, stacked so that each coin is slightly offset from the one below it, with the offset moving around in a circle with each new coin. Here is what it looks like for fifty coins 20 mm in diameter and 0.5 mm thick, offset by 2 mm from a center of rotation, and stacked with a rotation offset of 30 degrees per coin ( click to enlarge any image in this article ): Obviously, this is a spiral stack of coins. But it also approximates screw threads. The thread profile looks like a sinewave rather than the trapezoidal profile of machine screw threads. Indeed, others have observed this, and used the freeware parametric CAD program OpenSCAD to make screw threads by extruding an offset circle while twisting it, as in this model on Thingiverse , for example. From a distan
Read more

The water rocket: Thrust from water

Image
Launch tube thrust Thrust from air Once the rocket has left the launch tube, the water thrust phase of the flight begins. But before we begin those calculations, we need to have equations for ballistic flight, which is the last step described in the introduction . Why? Because during ballistic flight (coasting through the air), the only forces acting on the rocket are gravity and wind drag — but these are acting on the rocket during the thrust phase too. Ballistic flight is exactly the same, just without the thrust. Ballistic flight We need to know how gravity and drag affect acceleration, velocity, and ultimately altitude during all thrust phases as well as the ballistic trajectory afterward, so we'll start with the ballistic flight equations. Air resistance First we need to get the ambient air density. Denser air results in higher drag, and dry air is more dense than moist air. To get the density, we first need to know the partial pressure of water vapor in the air. T
Read more