What does it mean to be a good forecaster? In short, it means that you are right more than you are wrong. That’s the most relaxed and most basic qualification for being able to call yourself a forecaster.
A good forecaster is someone who’s right significantly more than they’re wrong. Being right 80% of the time is a good place to start. So let’s say I’m forecasting the weather for you each night. Each night I tell you whether the following day will be sunny, or cloudy. If I’m right on Monday, Tuesday, Wednesday, and Thursday but get it wrong on Friday I’d be performing with an 80% success rate. Would you employ me as a weather forecaster if I could keep up that kind of performance? I would hope so because 80% accuracy is pretty darn good when you’re essentially telling the future.
Now when I’m wrong, how do we measure how wrong I am? It’s pretty simple when you’re just predicting sunny or cloudy, heads or tails. Most forecasting situations are a little more complicated. Let’s add a little more detail to the previous example and say that instead of just telling you that tomorrow’s weather will be sunny or cloudy I also tell you the expected high temperature. So it’s Sunday night I and I tell you that Monday will be Sunny and the high temperature will be 66 degrees. Monday rolls around and it is Sunny but the high temperature is actually 70 degrees! Whoops! I got the sun forecast but missed the temperature by 4 degrees. I was 100% correct on my sunlight call but only 94% correct on my temperature call. That gives me a solid average of 97% for Monday. Not Bad!
Let’s consider another example. It’s Monday night and I tell you that Tuesday is going to be sunny again, and the high temperature will be 72 degrees. When all is said and done, Tuesday turned out to be a cloudy day and the high was only 51 degrees. Now I’ve really done it. My success in the sunlight call is 0% and I was off by 21 degrees, 30% error in my temperature call. When we average the success percentages of 0% for the sunlight call and 70% for the temperature call we get an overall success rate of 35% for Tuesday. That’s not too great, but every forecaster will have those days. Every forecaster will be wrong. It’s part of the job.
Now I’d like to consider financial forecasting in light of the examples above.
I’ve been trying to understand the stock market for about a year now. I’ve read a bunch of books, looked at a lot of charts, and dabbled in some methods of technical analysis. I’ve learned about reading charts for short term trading, and long term trading as well. On August 14th I signed up for a forecast service which will remain unnamed for this post. Since I started reading their forecasts, they were by and large wrong.
I expect forecasting services to be wrong sometimes, but when I am a customer of the service, I would like to understand what went into the decision making process that lead to an incorrect forecast. I want to know what happened, and why the forecaster thought the way that he did. This is not because I want to see him hung for making a mistake, but I see myself as his student, and I would like to learn how he does what he does by understanding his reasoning. It’s useful to know what reasoning has lead to success and what reasoning has lead to failure. You have to know both sides.
So after reading these forecasts for two months and watching the predictions fail time and time again I wrote a detailed email to the people that run the service. I cited sentences, phrases, and numbers from their forecasts and calculated the percentage of error they had shown on several occassions. This was all meant to show that I was earnestly trying to understand their work. I asked what sort of percentage of accuracy they were aiming for and what success rate I should expect as a subscriber. I also asked what went wrong in their reasoning.
I was very disappointed to see that the only email I recieved was a vague and defensive response cited that they “aim for 100%” accuracy and “all forecasters are wrong sometimes.” They dropped the ball on that one. I was expecting better customer service than that. As a forecaster, when you’re wrong whoops isn’t good enough.
