When Can I Stop Backtesting?

Posted by jmot in Thursday, October 15th 2009 under: General No Tags

Allow me to share a dilemma that I am facing recently. It has to do with backtesting.

I am backtesting a trading system with daily charts - manually. I have to do it for over 5 instruments. I confirm that it is tough to do manual backtesting Tough not in the menial way but mentally. As I am working my way through the charts, I keep thinking of the following:

I can only carry out limited testing

274808215_51854b0261_m I am only backtesting the trading system itself - the entry signal, the entry, stop loss and profit target. What I did not include is money management and my psychology.

Well, it is still possible to simulate money management. Just that it will take even longer for the backtesting. As I am trading more than 1 instrument, my money management has to take into account for which instrument the entry signal appear first. You can imagine I switching between 5 charts and determining if the entry signal has appeared. Hence the longer time for backtesting.

It is not possible to stimulate the psychology part though. For example, I may choose to skip a trade because I have quarreled with my wife… for instance only

What can I tell from the backtest results then?

Since I am not able to simulate money management and psychology, what can I get out of my backtesting? We all know that most trading systems have success rate of around 50%. Am I to just prove that my trading system is near average?

How much history data to use for backtesting?

Ray Barros suggests that he would backtest 5 years worth of downtrend data, uptrend data and range data. I found that amazing. Firstly, I have to determine uptrend, downtrend and range data accurately. Then it is the painful process of working through the data! Tough job!

Which instrument to backtest?

Do I want to trade with instruments that are uncorrelated? How can I be sure that the instruments that I have chosen are uncorrelated.

Will backtesting using different charts reap the same results?

For instance, the daily candle on InterbankFX chart closes at 8 am SGT while the daily candle on Oanda closes at 10 am SGT (if I remember correctly, anyway, they definitely close at different times). Will I get the same backtesting results with the 2 different charts?

Then again, would I want to do my homework now or do I prefer to pay the market to learn? Meanwhile, the backtesting continues…

If you find this post or this blog is helpful to you in any way...do consider buying jmot a kopi (aka coffee)!

29 Comments Already

coconut Said,

October 16th, 2009 @9:54 am

mmm, backtesting, this is a tough one.
tough one for you that is.

to make the answer short, you should STOP backtesting. for the reason that you lack of time. not that i against backtesting.

the easier way is to create your own chart, yes your own chart to simlulate all kinds of market condition. that you need to have a little bit of creativity mind which is a essential part of successful trading. that goes to your personalize trading system as well.

we used to say that history will always repeat itself. but my understanding is that (market) history will never repeat itself. (although they look the same especially on chart pattern.)

lastly, if backtesting can booooost your confident in trading, be my guest.

coconut Said,

October 16th, 2009 @10:11 am

oh ya, one more thing,

don’t forget to include all the hidden cost when you do your P/L,

commission
spread
skid
roll over
interest etc etc etc….

coconut Said,

October 16th, 2009 @11:01 am

look jmot,

i know that you are a nice guy, a man full of responsibilities, that i respect you from “inside out”.

although i trade for a living (that’s your term), except managing my own money, i don’t feel as successful as you are.

i spend most of my time doing nothing but thinking (which is good for trading) except that moment of executions of trade. i don’t seem to be interested in any other things except trading (and golf, which i pick up the same time as i did for trading).

i can’t turn you to become a successful trader, but i can certainly help you to shorten the journey that you seek to discover yourself (to be a trader or not).

right now, you are still not at the starting gate yet.

Forex Educator Said,

October 18th, 2009 @1:59 pm

Back testing is a very important part of my trading process. When I build a system, I will back test it. If it does not do well in my back test, how will I be convinced that it can do well in the future?

However by doing well in the past does not mean that it will do well in the future. Assumption is made that the past will repeat itself. Weakness about this assumption comes in during earning season (January, April, July, October). If I have not traded during earning season, I will have profitable months for the whole year.

As I use my own system to trade, that is call forward testing. I still manage to achieve more wins than losses, so I will continue to use my current system. At the same time teaches my students about this system.

There are no perfect system but I am contented with 2,929 pips profit since start of 2009.

coconut Said,

October 20th, 2009 @12:41 pm

don’t blame the earning seasons.

you can (learn to) trade the reverse during that period.

everything else i agreed.

coconut Said,

October 20th, 2009 @12:51 pm

the farmer plant rice during the wet season

they go hunting during they dry season.

coconut Said,

October 20th, 2009 @1:35 pm

provided that their rice paddies are well taking care of.

coconut Said,

October 20th, 2009 @1:56 pm

“Then again, would I want to do my homework now or do I prefer to pay the market to learn? Meanwhile, the backtesting continues…”

backtesting is not homework, you don’t learn anything from it in a trading prospective.

“expert” used super computer to back test more than 150 years of data and still cannot find the magic bullet.

the only way to learn to trade is to trade.

and please don’t be so negative (minded) as the market also pay you while you learn.

coconut Said,

October 23rd, 2009 @1:05 pm

well, i just came to know that some joker (”expert”) had even backtested financial data for more than 800 years.

800 years!!! (i wander where they get the data from?)

5 years of backtesting is like kindergarden.

coconut Said,

October 23rd, 2009 @1:08 pm

man, it went right through the dark ages.

coconut Said,

October 23rd, 2009 @1:51 pm

quantum physics, if anyone knew, had over thrown Einstein famous quote “God does not play dice”.

based on the uncertainty principle of quantum mechanic (which theory had been tested), it literally says that even God does not know what is going to happen in the future.

so what good does backetesting do to us?

although the theory make me sick, but just look at the bomb! i certainly would not want to stand in its way.

coconut Said,

October 23rd, 2009 @4:19 pm

” the boundary condition of the universe is that it has no boundary” - stephen hawking.

coconut Said,

October 26th, 2009 @3:00 pm

” the pattern recognition of the market is that it has no pattern” .

jmot Said,

October 28th, 2009 @5:24 pm

coconut, how can we tell if a trading system has an edge?

coconut Said,

October 29th, 2009 @11:26 am

yes n no

no trading system has an edge over a long period of time. just like market, it performence changes accordingly.

personally, i think i can tell if a trading system is good or not, based on my personal “trading system” as a guide.

i (think i) can trade with any trading system out there by making some vital adjustment to it. provided it is not too complicated and too mechanical.

i’m both mechanical and descretional trader.( half man half machine)

to conclude, no single trading system is ever lasting, whether it is trend following or contrarian system.

the best solution is to have 2 sets of contraditing system combine into one system, or have 2 sets of system to use in difference market condition, but thats require experience, thats why i said, trade trade trade.

coconut Said,

October 29th, 2009 @11:28 am

the samurai carrying 2 sword

one is to kill his enemy

the other is to kill himself.

coconut Said,

October 29th, 2009 @2:36 pm

ya, one word sum up a good trading system - flexibility.

coconut Said,

October 30th, 2009 @12:29 pm

a simple example to elustrate the above

the structure of water (trading system), its is both liquid or solid depend on the temperature (market condition). hence flexible in its form (structure).

the key words here are structure and flexibility, not water, ice or any other thing.

damn my english.

nomad Said,

October 30th, 2009 @7:25 pm

I see now that you have settle for a trading system and is backtesting it for its potential. If this is your own developed system…let me congratulate you for making it come to fruition. However, if this is something which is not your own..then maybe you can still learn something out of it….see how its developed and try to improve on its shortcomings perhaps. In either case…don’t be too hung up on backtesting…spending lots of time on historical data( a reasonable period should be enough) becos historical data are just historical data…they’re history. A system that works well during backtesting may not necessarily be so in live market. Also, most people like to tweak their system so closely to be in line with the historical data..thereby running the risk of curve fitting..giving them a false sense of security..thinking that their system is profitable and when it’s tested on the live market..doesnt live up to expectations.

Since you’re trading on a part-time basis as you have a day job…my advice to you is to try to concentrate only on a couple of major currency pairs and trade them well rather than to spread yourself thin trying to grapple with too many markets unless you really have the time to concentrate unlike full time traders.
Remember…you dont have to be profitable to trade a wide range of markets.

I feel if you really want to evaluate the effectiveness of a system…forward testing is the way to go. Never mind if initially its only demo trading..although live trading with real money would be the ultimate choice. At the end of the day..when the “homework’s” done..and your system’s developed and backtesting completed…you’ve got to pay the market to learn. Keep an open mind and soak in whatever the market teaches you for this is where you will learn the most…never mind the hundreds of forex books you’ve read..the courses you’ve taken…the superior system you believed you’ve developed…as all these will come to nought if your skills and knowledge cannot stand up to the test of the market…and so be prepared to go back to the drawing broad and make whatever adjustments that’s necessary. Through this you will learn to maximise your strength and minimise your weaknesses.
As the saying goes….” When the student is ready…the master will appear”

michael Said,

October 30th, 2009 @7:55 pm

As the saying goes….” When the student is ready…the master will appear”

Yes..Here comes the coconut…

coconut Said,

October 31st, 2009 @11:56 am

wrong,

the “master” will disappear.

coconut Said,

October 31st, 2009 @11:57 am

just like when everyone is ready to go long, the market crash!

coconut Said,

October 31st, 2009 @12:24 pm

its always though for me to read loooong paragraph.

“Through this you will learn to maximise your strength and minimise your weaknesses”

that is precisely what you should not do, if i read it correctly.

trading is the reverse of other thing u have learned.

you should improve on your weakness and tame your strength.

damn, i’m going to have lots of enemy soon.

coconut Said,

October 31st, 2009 @12:45 pm

i know its difficult to swallow, i’ll explain,

if you have a pair of wings, one side big (strong) and the other small (weak), you certainly will crash very time trying to fly. or at lease always have bumpy ride.

the solution is obvious. strenghten the side that is weak or reduce the side that is too strong.

coconut Said,

October 31st, 2009 @12:54 pm

or stop trading!

coconut Said,

October 31st, 2009 @4:49 pm

to nomad
if you’re angry, i apologise.

just to clearify, it is only this sentence that i disagree.

on the other 99% context, i agree with what you said. but since i agreed, i’ve nothing to add on.

coconut Said,

October 31st, 2009 @5:22 pm

i’m also sensing that my time on this blog is coming to an end soon. i’ll give my last burst of fire.

nomad Said,

November 1st, 2009 @2:04 am

hey coconut, there is no need to apologise. I’m not offended in anyway. In fact…I always welcome alternative views..thats how we learn. You are right. Perhaps, I should have worded the sentence as “maximise your strength while eliminating your weaknesses(ie. correcting one’s bad trading habits)

I enjoyed reading some of your comments on the blog here. They’re interesting and actually liven things up a bit. Must say I’ve also learned something from you… looking at things from a different perspective.

As for the saying “when the student is ready..the master will appear”, this statement is open to many interpretations and different people interpret it in different ways.
For me, my interpretation would be that if a trader has a closed mind and one who has no strong will to change his bad trading habits for the better(even though he is aware of his weaknesses), have no real passion to learn, receive, and practise the true teachings/the proper rules of of trading, then he has not really learn anything at all.
Only when he has an open mind…and is prepared to commit himself and to learn how to be a good trader…and readily soak in the knowledge and teachings will he be ready to become a student and when he becomes a student…and in his continued quest for knowlege to better himself…the master in him will appear. In this case, the master is not some third-party but actually the trader himself.

coconut Said,

November 2nd, 2009 @9:55 am

“Perhaps, I should have worded the sentence….”

no, you don’t, its alright to make “mistake” (i did it all the time). question is, how fast can you correct them? i think you have done it well.

its funny just by “talking” one can tell who are the one that can really trade and who can’t.

since me and the market will be getting busy for a while, i wish you and every trader best of no luck.

yes, luck is a dirty word in trading.