Progress Apama : How Deep Does HFT’s Rabbit Hole Go ?

[Versiunea romaneasca] [ in romana] [English edition]

This month, while waiting for the new Strategy Tester, I felt myself pretty void of ideas. I realised that the content that we produce is effectively drying out. The backtest is the creed of the trader, what gives him confidence in the trading strategies that he attempts.

By the mid of march I stumble upon some Progress Apama recorded webinars entitled “Build Quickly, Run Fast” and “From Concept to Profit in No Time Flat”. Sure, our study objective is not the Progress Apama platform. That is an institutional, high frequency platform, that we’ll never put our hands on, however it’s interesting to see more of their approach on HFT, as well as some of their concepts in both building and running. Not even big market players use it, but a very tiny elite of them, la crème de la crème, since they have about 120 clients globally.

Problem, is HFT is becoming mainstream in institutional trading. And when the institutionals are doing something, better open the ears wider, cause they’re always getting all the money and they’re probably right!According to a Tabb Group research, HFT firms in US are just 2% of 20K trading firms, and make up 73% of the total equity volume traded. It’s looking like the entire equity market is under a “hostile takeover” from HFT, and the process will not stop. More and more trading firms are getting new trading technologies, or, if not possible, at least new HFT-like algos to run on their own platforms, so this trend will not stop. It is important that the retail environment remains aware of what is out there, so it would not have the market perception completely distorted by paradigms too old to fit the current developments.

It seems that , from the beginning of markets, traders always struggled for latency, to be the first in getting a certain trade. The first example that comes in my mind is Nathan Rothschild, that found out the first that Napoleon was defeated at Waterloo and lead the entire market into believing that Napoleon was victorious to crash it enough for him to buy it a pound for a penny. Another example more close to our days is the character from “Reminiscences of a stock operator” who loses his grip on the market and cuts his latency by using the newly technology of the time, the telex. With every major innovation in telecom and computing, a new layer was added, and what it was in years before the known level II markets, where human traders were manually placing and executing trades, is now the realm of high frequency trading machines. At first, it was all about seconds – as it is the case for fellow MetaTrader folks – then they started going deeper and deeper to milliseconds, microseconds, and even nanoseconds, that they think about right now. This happened because HFT technologies adapted, and modified market behaviours on every time layer : when milliseconds started to become available, the algos running on slower stations to become faulty, now, when it is microseconds time, the current millisecond-based stations are in the process of upgrading, and tomorrow, nanosecond-based stations will replace the ones working on millisecond level.

Let me be sarcastic here a bit: a microsecond is 10^-6 of a second, and a nanosecond is 10^-9. There is a very long technology road towards the Planck time, which is supposed to be the minimum amount of time: 5.4 × 10^-44 s. At the current rate, we should expect latency being cut by 2 powers of 10 by year, unless something really happens to change the curvature of latency cut.

What happens to algorithms due to improvements in execution and expansion of HFT desks? As Louis Lovas, Chief Architect for Progress Apama, explains “it becomes either obsolete , reverse engineered, or the spreads become so tight that is no longer profitable”

Dan Hubscher, Principal Product Marketing Manager – Capital Markets for Progress Apama citing an AITE Group research, says that “even in a relatively stable environment, the lifespan of an algorithm can be as short as three months”. This was the reason for developing the Progress Apama Studio, a rapid strategy development environment. Because if the lifetime of an algo is limited, it’s logical that if you cut it’s development time you get extra profits by running that system for a longer time, with the extra time gained when others are struggling to replicate it on other platforms with worser development environments.

The following is an example from their demo. The strategy is a Statistical Arbitrage between two stocks. A mean is calculated , along with Bollinger Bands at 1.8 standard deviations. The following screen depicts the strategy running in one selected instance (Bank of America vs Citigroup) , and the Scenario Instances window below shows what is the algo doing on each instance (first line is Caterpillar vs Chevron, second is Bank of America vs Citigroup). It is a predefined strategy, it comes as a template with the station, there is nothing secretive and important about it.

A few notes about the strategy.
1. As you can see, only one spread was set up to be displayed – and you’ll see below, strangely, they picked the spread between mid prices. Normally, if the strategy would be based on market executions, you have at least two spreads to monitor, and the exit criteria would use the spread which is opposite to the one taken ; if the strategy would be based on quoting, there are four spreads to monitor.
2. The quantities are predetermined, as parameters, first is 10, second is 20. The algo doesn’t adjust them by hedge ratios.
3. I can see no tick count or time span of ticks used for average and Bollinger Band calculus.
4. The max quantity fields are used for position control. Yes, that hated position that MT5 has too now. If you look at Scenario Instances, in the Current Position fields, you can see that they are not 10 and 20, it’s not having only these quantities at a time, rather positions are bulked up with new and new orders of 10 and 20, until closed or touch the max quantity parameter. Such order bulking may appear if one of the Bollinger Bars is crossed repeatedly without the touch of the other bar.
5. The order timeout seems to have the role of a position decay, positions being flattened after 20 sec. It can’t be an execution timeout because a second is a century now in HFT, and waiting 20 sec for an order confirmation doesn’t sound to me HFT at all. Or was I wrong about HFT ? See the article about MetaTrader , Tick Data & Charting : Why Not a Level II MetaTrader ? !
6. Note the time scale of the chart. There are 12 seconds between time labels, and there seem to be from 3 to 7 executions per interval. Know any EA trading like that?

What is interesting in Progress Apama is how strategies are built. Strategies can be built even without coding , but unlike environments in other fields, that attempt building without coding, Progress Apama comes with a complex, yet simple to use concept. As Dan Hubscher says, “according to AITE Group, the amount of time it takes to develop a new algorithm, from data acquisition and analysis, to strategy deployment, and thru to production is typically 10 to 28 weeks”. So it was quite natural to come with rapid application environment, because this is the problem that hits other direct competitors on the institutional platform market, and put Progress Apama into a more favorable position. In their development system, all variables that have to be either directly watched, or calculated, originate from smart blocks, in a block diagram. Each block takes data , either from the market or from another block, and outputs it as an input for another block, generating at the same time internal variables (which you can see in the green panel on the right).

Then the strategy is designed on a workflow basis, kinda similar to the diagrams used when learning algorithmics. Each block contains conditions and commands written in plain english, all conditions , commands and variables being selectable from a menu appearing upon right clicking.

As you can see, it has a pretty fancy development environment, that has to answer quick to the ever expanding ideas and complexity degree of trading algorithms, but the real power of the station unleashes when running the strategies. This is what makes it a top notch platform.

The Statistical Arbitrage example is quite simple, as it follows the spread as its driver. But real HFT strategies are more complicated, each strategy may be following asynchronous events that happen on multiple assets:

When did we ever have such an approach ? When did it matter the time restraint in which the signals were supposed to come? Oh, I forgot, we didn’t even have the backtester…
An interesting thing to note about their view, is the WHEN approach. Repeated times, Dan Hubscher and Louis Lovas explain that WHEN is not the same as IF. No comment.

To make this kind of conditions usable for the strategy designer, Progress Apama comes with a smart thing called Complex Event Processing:

Dan Hubscher defines Complex Event Processing as “the ability to relate two or more events, even if these events occur in a span of time, and then determining meaning from that relationship immediately, such as detecting a trading signal, or an alpha-generating profit opportunity”.

But we don’t even have OnNews(), ain’t that right?

It’s clear, that “immediately” word there means that you must have the lowest possible latency. I believe that the CEP can be translated into MetaTrader, by starting from OnChart() and OnTrade() implementations, like the ones from my past articles, and transforming these methods into Complex Event Processors, that translate tick and trade events into intelligible events, such as “MSFT above 10-second Bollinger Band”, followed by a time stamp, and put into a queue. The queue might be then taken over by another procedure, that applies the time constraints, generates signals, and passes them to decision modules. Check our article Anatomy of a simple CEP engine.

However, the thing that makes Progress Apama the million buck machine is none of these things, because these are simple tools, that can be used by retail stations too, if their developers would include such facilities. Sure, the rapid development tools matter only when compared with competitors. However, it is the ability to run these strategies in hundredths of instancies, taking advantage of multicore infrastructures, without losing the latency advantage. And this is done by Event Correlator(s).

Progress Apama has also a programming feature. They come with the proprietary Monitor Script language, a Java-like language, although simpler, side by side with a full Java implementation:

As I said above, I include below parts of the Statistical Arbitrage code , the ones that explain the core of the strategy, found in the webinar:

This is the beginTrading() action, that quite shows the candies of MonitorScript. As you can see, the route in this case, are getting the data from the market. Bust most important here, is the on statement. This statement defines listeners, having a syntax like:

on [all] <event expression> <action part>

The first on that you see checks for the destruction of the instance by the user (in a similar manner to WM_CLOSE being sent to WindowProc() ), then it removes the variables and closes the instance. The secondary on is more interesting. Here, the second and the third one are written in a quite redundant manner, because the first condition which is an and is replicated inside the code, with an or. As Louis Lovas notes in the comment , “Make sure we have at least one tick from each before moving on” it might appear that the Depth() event is triggered by a new tick, but it actually seems that the Depth() event is triggered when the structure of the orderbook changes, meaningly when at least one volume changes on either the Bid or the Ask side, not only when a tick comes (change in core Bid/Ask). Depth() seems to trigger similarly to the OnBookEvent() that was not yet completed by MetaQuotes, which I explained in other articles that should trigger way more frequent than OnTick(). Leaving this comparison aside and continuing with the code, see the second condition, which is written by Louis just to show how flexible is the syntax, and then, what it triggers, the processNextSpread() action.
An interesting thing is to be remarked about listeners. By looking at the listeners we see a very advanced syntax, way beyond the rigidity of the event-driven system that MetaQuotes is struggling to produce, and which exists in other stations. First, the listeners apply to every kind of event that is linked to subscribed data. Beyond this, the event description appears in the listener condition, and it can be anything. Even if you would like to get a tick on another instrument in MT5 you need to run a separate EA (daca nu cumva va deveni posibil prin OnBookEvent() ) because OnTick() answers only to current instrument. At the same time, each listener is event condition – action implementation, deployable quite everywhere inside the code, giving a high degree of freedom to the programmer, whereas in the event driven model, the treating for every event type is confined within a single code body, where it analyses all the possible events of that type, which, if translated in MonitorScript, would mean that all listeners working with a certain event type would have to reside in a single code body.

As you can see, the spread is really calculated with the mid prices. What’s more important, is that the method incapsulates all the strategy itself, and you can see how it sends orders, without checking previous orders, thus demonstrating once again the positional system.

Seems the entire SubmitOrder() action has been written with the purpose of updating strategy variables. Probably the entire code could have rested inside processNextSpread(). Strangely, the position variables are calculated , by adding or subtracting volume, instead of simply using a function to retrieve the position per each instrument, in the parts of the code where this is necessary. What is not clear in the code is where the maximum position constraints apply, as well as the order timeout.

What is essential about Progress Apama is that it can run millions rules concurrently and make use of multiple cores environments.

Here you can see a strategy with simulated data running in 500 instances on a quad core machine:

So , if Louis Lovas says that it works on a quad core machine, and we don’t have any reason not to believe it, means that Irene Aldridge was right, and gamers ever increasing requirements pushed the hardware to the point that machines able to run HFT are in every home ; but , which broker will give you low latency for your poor retail account and where will you get from a platform that mustn’t waste processor resources?

Of course, we will never be able to do that. MetaTrader folks don’t even have the cash to meet the margin requirements of running portfolios that rebuild indexes with required weights, so low latency trading is out of discussion. MetaTrader will never ever have low latency. However, some high frequency trading elements, such as algos and complex event processing for low scaled strategies, that span on a very limited number of assets, can be ported there. The purpose of this article was to introduce readers in the HFT world and its specific issues.