I should probably explain exactly what I'm doing in my forecast so people understand. As some have commented, there are many ways to do it, and I just chose one. I will probably continue to tweak this methodology until we can get some good forecasts, not ones that look so random.
First things first, why am I even doing this kind of forecast? Far too many trading systems are backward looking. I am trying to find something that looks ahead and can tell me more about the future. I have other trading signals I also use, but it would be good if I had a reference that could help me know when to hedge my core positions, whether they are long or short. Supposed I'm short the market, and I get a consensus forecast that points to a bounce -- it would make sense to buy some protection against my short positions (maybe index call options or similar).
Here is a step by step of my approach:
1) I have gathered all Dow history going back to 1927 or so. I got this data directly from Dow Jones. DO NOT trust Yahoo for financial data. Their data is not correct. I want to look at the Dow because it has a far richer history of data, going back through many recessions and cycles.
2) I calculate all daily returns, plus the high and low returns for each day.
3) I enter a date that I want to forecast, so for today 10/20/2010
4) The program goes through all the data and finds the closest match to the daily returns of a 2-5 period look back. Here's an example of a two period study:
10/19/2010 posted a return of -1.48%
10/20/2010 posted a return of +1.18%
Find the closest match to a two day cluster in history:
You can see that these two periods are nearly identical. So what happened next? That's what I'm plotting on the charts.
Now, there are many ways to do this. I could take the top 30 closest matches and average them. I could take more period samples, say, 2-20, and average the best historical matches. As for what is correct or incorrect, I have yet to really figure that out, so I can try out other methods until I find something that's reliable. I will say that a 6 period look back in my studies was correct about next day direction 60% of the time, and 76% of the time the market moved over 0.25% beyond what direction it predicted (so if it was positive, the market advanced at least 0.25% the next day at some point). Ironically, I'm not showing the 6 period look back. I will take a look at alternatives and do more testing.