But now, I am seeing more and more areas where AI is being applied and they are all striking closer and closer to home!
Reading up on it I found that major thrust of AI is in the development of computer functions associated with human intelligence, such as reasoning, learning and problem solving. While this is true for probably everything in life, I realise it can be particularly true for the markets.
Trading and investing in the market is nothing but a series of reasoning, based on the data and solving the problem of forecasting the future direction of stock prices. The age old way was fundamental and technical analysis. But they were created from a century ago when data was limited and markets were like country clubs.
Now in the 2000s, there is a virtual explosion of data and the field of data science has gone into Hyperloops! I have been around in the markets when everything was done manually- from trades recorded in a ‘Soda book’, to records submitted to the exchange to reconciliation of mismatched trades, to maintenance of records and accounts etc. etc.
All this gave way to computerisation, first at the exchange and then progressively to brokers and sub-brokers. What really got done was that whatever could be automated was automated. And efficiency improved all round.
That is the objective of AI after all. It is augmenting human beings to think and act better. One has to be free from mundane activities in order to think deeply about something.
From there, better actions can emerge, that will lead to better outcomes. AI, therefore, leads to intelligence augmentation of humans (hence the cliché AI leads to IA!). People are able to get better information that is more organised in a way that it becomes immediately usable in a better way. This is what leads to all round improvements in any process and creates efficiency.
When the trading ring was shut down in 1994, many of who were part of it thought that the open outcry system cannot be reduced to automation. But we found out very quickly that it could be done far, far better by the machines. The adoption then was very swift, and today, very few people can really imagine that business was actually done that way!
Every 20 years or so, life changes because of new inventions in the world. Now, more and more this is happening in the areas related to technology. We can perhaps call this a technological industrial revolution! In the last few years, and I believe in the coming decade or two, AI and its associated aspects are going to be play game changer in our lives. Here is a brilliant illustration that I found on the net that highlights the prospects. It does not show the markets of course, but I feel markets are a bit of all those elements!
ET CONTRIBUTORSArtificial intelligence and its prospects
The natural progress that the world makes is to get more knowledge about certain areas and then perhaps gather skills there and work in that area. The problem with knowledge is that it is vast (to the point of being limitless), much of it is not well-organised or even well formatted and it is subject to constant changes. Here is where AI can step in to organise and use the knowledge in an efficient manner.
With AI, we can perceive knowledge better, modify it as new inputs indicate errors of former ways and make it usable so that the speed of execution can be improved.
More than ever, data is being generated at a humongous level and pace. This is happening in the financial markets too. Some 50 years ago, we barely got closing prices of traded instruments. Now we get data up to great levels of granularity and at lightning speed too.
If you can afford to pay, you can get the data. The time is not far when the high cost of granular data (as of now) will become cheap. The main game is therefore to get that data organised.
In the 1970s and 1980s, I used to draw charts by hand on a chart paper. It was not until about 1993 or so when the exchange started providing us with data on a file. We had to stand in a queue every day at about 6 pm to collect this data file on a floppy drive and then upload the data into our programs (I had designed software to chart by that time).
It was just end of day data. Intraday data began to be provided much, much later. But now free or minimally costing charting software are available easily with live data streamed into it.
To a small extent, charting software was like an AI application in technical analysis, in that, it automated the charting routine. Stringing together a few elements to form some queries or scans was an extension of the same. Technical analysis software has pretty much remained there since the 1980s. There was some attempt to introduce machine learning into market trading (programs like Vantage and The Brain etc.), but these Neural Net-based software were perhaps a bit ahead of time and didn’t really succeed.
But slowly, some of these are seen to be making a comeback of sorts. In the field of fundamental analysis, we saw the creation of programs like Capital Market and a few others that collected the vast fundamental data and got it organised for easier use by the analysts. Bloomberg, Reuters etc. rule the institutional houses.
With the advent of cheap broking platforms like Zerodha and AliceBlue etc., automated strategies are now springing up all over. Execution platforms like TradeTron and several others have also come up swiftly to cater to such service providers. This again is one step further to the look-at –the-chart and find the signal to trade situation. A hybrid version of this was happening using Amibroker as software to generate trade signals, but that has now given way to these strategy platforms. Their success rate and longevity is unknown as of now. We, too, have designed a new AI-based product, called NeoTrader, which organises data and presents in a way that makes it easy to use by the trader/investor and does many of the things that AI is supposed to do in a field.
AI is used for the purpose of curating of the signals so that the user can control the kind of trades that he receives from the software. Data is the base on which analysis is done.
Not all data can be logicised. But that it cannot be logicised doesn’t mean that there is no solution. Data often aids reach a solution. This is particularly so in probability-based, sentiment-led activities like trading. The construction of the logic then becomes an art. Like detection of cancer from the reading of an MRI is based on the reading of the pattern by the radiotherapist. We know that every MRI report is accompanied by a request to correlate it to clinical symptoms and examination. Trading and investing requires that the user traverses the ‘last mile’.
So these market products are often a bridge between the science of data crunching and analysis and scenario analysis by individuals. It is a probabilistic model, because beyond data analysis comes the imagination and purpose of the user. The idea is to enhance those aspects.
For example, various people have designed RSI and BB and candlesticks etc. But not everyone knows them, and not everyone understands them well. These are used extensively in a very random fashion, based on some very loosely defined or understood way. Most tools crunch the past and present data and give you outputs that actually facilitate the formation of some foresight to how the future can play out. With this, the user can predict many probabilistic outcomes.
How well you understand and how well you use such tools is the key. Some of these tool can help overcome the innate stupidity of traders; which is the lack of focus on what they are doing or what they need to do. They take off the mundane and allow you to concentrate on planning and executing. Thus, they become a core necessity to engage with the market.
