MQL5 : Implementation of a simple CEP engine

[Versiunea romaneasca] [MQLmagazine.com in romana] [English edition]

This edition was harder to make than any of the previous editions. The late release of Strategy Tester and the large number of bugs that surged while making the CEP engine took a lot of time. From this moment, magazine issues will not be monthly anymore, because they will contain a lot of work and updates of past works. This is why, this time we are adding the Forum to the Magazine.

Now back to the CEP engine.

The CEP engine was some of the hardest stuff possible to write in MQL5. It was hard not only for me as programmer, but it had implications over MT5 as well, because the spaghetti data structure that it uses put MetaTrader to the test ; it needed a lot of bug filtering, especially on objects and memory allocations. The code managed to run with minor issues barely on build 270 (yes, already in May), and completely clean on build 271. It’s a long hard code, and I’ll chop it to pieces for commenting. I also left inline comments and Print lines intact to be better understood by readers. The algorithm is quite modified, not the same as the one presented in Anatomy of a Simple CEP Engine .

The following file it’s gonna be called CEPEngine.mqh :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
//+------------------------------------------------------------------+
//|                                                    CEPEngine.mqh |
//|                                       Copyright Bogdan Caramalac |
//|                                           http://mqlmagazine.com |
//+------------------------------------------------------------------+
#property copyright "Bogdan Caramalac"
#property link      "http://mqlmagazine.com"
 
#property version "1.0";
 
//*******************************************
 
string Replicate(string s,int n)
   {
   string res="";
   int k=1;
   if (n<1)
      return(res);                    
   for (k=1;k<=n;k++) 
       {
        StringConcatenate(res,res,s);
       }
    return(res);
   }          
 
string BoolToString(bool b)
  {
   if (b==true)
     return(".T.");
   else
     return(".F.");   
  }
 
//*******************************************

These are very simple string functions, that are needed for the larger functions that translate structures into string messages, descriptions of several CEP structures. The first function will replicate a string, and the second one will return a string for a boolean , in a FoxPro-ish style.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
struct SimpleEvent
  {
   public:
   uint EventID;
   uint EventTime;
   uint Reserved;
   ushort BindCount;
   uint Binds[100];
  };
 
   bool BindEvent(uint eventid,SimpleEvent& s)
     {
      bool fd;
      if (s.BindCount==0)
        {
         s.Binds[0]=eventid;
         s.BindCount++;
         return(true);
        }
      else
        {
         fd=false;
         //checking previous bind
         for (int i=0;i<s.BindCount;i++)
            {
             if (s.Binds[i]==eventid)
               {
                fd=true;
                break;
               }
            }//for (int i=0;i<s.BindCount;i++)
         if (fd==false)
           {
            if (s.BindCount<ArrayRange(s.Binds,0)-1)
              {
               s.Binds[s.BindCount]=eventid;
               s.BindCount++;
               return(true);
              }
            else
              return(false);
           }
         else
           return(false);
        }//else if (s.BindCount!=0)
     }
 
   int FindBind(uint eventid,SimpleEvent& s)
     {
      bool fd;
      if (s.BindCount==0)
        {
         return(-1);
        }
      else
        {
         fd=false;
         //checking previous bind
         for (int i=0;i<s.BindCount;i++)
            {
             if (s.Binds[i]==eventid)
               {
                return(i);
                break;
               }
            }//for (int i=0;i<s.BindCount;i++)
         return(-1);
        }
     }     
 
   bool UnbindEvent(uint eventid,SimpleEvent& s)
     {
      if (s.BindCount==0)
        return(false);
      int fd;
      fd=-1;
      for (int i=0;i<s.BindCount;i++)
         {
          if (s.Binds[i]==eventid)
            {
             fd=i;
             break;
            }
         }//for (int i=0;i<s.BindCount;i++)
      if (fd==0)
        return(false);
      else
        {
         for (int j=fd+1;j<s.BindCount;j++)
            {
             s.Binds[j-1]=s.Binds[j];
            }
         s.Binds[s.BindCount-1]=0;
         s.BindCount--;
         return(true);
        }//if (fd==0)
     }

The SimpleEvent structure is at the core of the engine. The events queue that I described in my previous article it is made by tiny SimpleEvent structures. EventID plays the ID, EventTime is the time (in ticks – milliseconds since Windows start – not datetime format), BindCount tells to how many complex events the simple event is binded, and the Binds[] contains these events. The BindEvent(), FindBind() and UnbindEvent() functions are to work with this structure. Normally, I could have opted for a class instead of a structure, but what you gain in class functionality you lose in data maneuvrability : you cannot copy objects, but field by field, otherwise you get Structure have objects and cannot be copied.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
struct SimpleEventArrangement
  {
   int EventCount;
   uint EventArray[100];
   int Prebinds[100];
   int NegationsCount[100]; //how many negation events before each event from the array
   uint NegationsArray[10][100]; //store negation events before each event from the array
   bool NegationsChecked[100];//checks of the first negations that match
  };
 
struct ComplexEvent
  {
   uint ComplexID;
   uint TimeConstraint;
   ushort ArrangementsCount;
   SimpleEventArrangement ArrangementsArray[100];
  };

The ComplexEvent structure is the base unit of the complex events table. Remember the pic from the previous article? That is quite what can be found in the ComplexEvent structure. Thus, we have the ID , time constraint, arrangements count (number of rows) , or versions of event queues that make up the same complex event, and finally an arrangements array, made up by SimpleEventArrangement(s).

The SimpleEventArrangement structure describes what can be called to be a row of that matrix. Thus we have a count, and the row (meaningly, the EventArray[]) , which contains the required simple events. The Prebinds[] will note down of course, the prebinds of these events, meaningly the position in the event queue for each simple event that is found. The NegationsCount[] will contain how many negation events are required not to appear before every simple event that is required. Thus, one count per each required simple event. The NegationsArray[] will contain these events required not to appear, on columns, rather than on lines. The NegationsChecked[] array is a sort of shortened prebind array. Instead of storing where the negations appear, a simple negation flag is raised in this array, thus the array is unidimensional.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
void MakeSimpleEventArrangement(string s,SimpleEventArrangement& arrg)
  {
   int len=StringLen(s+" ");
   string rdnow="";
   string c;
   uint number,swap;
   int normcount;
   arrg.EventCount=0;
   string first;
   int intg,d,crt;
   double dbl;
   s=s+" ";//to make sure it registers last word
   ArrayInitialize(arrg.EventArray,0);
   ArrayInitialize(arrg.NegationsCount,0);
   ArrayInitialize(arrg.NegationsArray,0);
   ArrayInitialize(arrg.Prebinds,-1);
   ArrayInitialize(arrg.NegationsChecked,false);
   normcount=-1;
   if (len!=0)
     {
      for (int i=0;i<len;i++)
         {
          c=StringSubstr(s,i,1);
          if (c==" ")
            {
             if (rdnow!="")
               {
                first=StringSubstr(rdnow,0,1);
                if (first=="-"||first=="!"||first=="n"||first=="N")
                  {
                   number=StringToInteger(StringSubstr(rdnow,1,StringLen(rdnow)-1));
                   if (normcount!=-1)   
                     {
                      arrg.NegationsArray[normcount][arrg.NegationsCount[normcount]]=number;
                      arrg.NegationsCount[normcount]++;
                     }                  
                  }
                else
                  {
                   normcount++;
                   number=StringToInteger(StringSubstr(rdnow,0,StringLen(rdnow) ));
                   arrg.EventArray[normcount]=number;
                   arrg.EventCount=normcount+1;
                  }
                rdnow="";
               }//if (rdnow!="")                
            }//if (c==" ")
          else
            {
             StringConcatenate(rdnow,rdnow,c);
            }
         }//for (int i=0;i<len;i++)
       if (rdnow!="")
         {
          first=StringSubstr(rdnow,0,1);
          if (first=="-"||first=="!"||first=="n"||first=="N")
            {
             number=StringToInteger(StringSubstr(rdnow,1,StringLen(rdnow)-1));
             if (normcount!=-1)   
               {
                arrg.NegationsArray[normcount][arrg.NegationsCount[normcount]]=number;
                arrg.NegationsCount[normcount]++;
               }                  
            }
          else
            {
             normcount++;
             number=StringToInteger(StringSubstr(rdnow,0,StringLen(rdnow) ));
             arrg.EventArray[normcount]=number;
             arrg.EventCount=normcount+1;
            }
          rdnow="";
         }//if (rdnow!="")
 
      //reversing event order
      if (arrg.EventCount>0)
        {
         dbl=NormalizeDouble(arrg.EventCount/2,0);
         intg=NormalizeDouble(MathRound(NormalizeDouble(arrg.EventCount,0)/2),0);
         if (dbl==intg)
           d=0;
         else
           d=1;           
         crt=0;         
         for (int i=0;i<=intg-1-d;i++)
            {             
             swap=arrg.EventArray[i];
             arrg.EventArray[i]=arrg.EventArray[arrg.EventCount-1-crt];
             arrg.EventArray[arrg.EventCount-1-crt]=swap;             
             for (int l=0;l<ArrayRange(arrg.NegationsArray,0);l++)
                {                 
                 swap=arrg.NegationsArray[l][i];
                 arrg.NegationsArray[l][i]=arrg.NegationsArray[l][arrg.EventCount-1-crt];
                 arrg.NegationsArray[l][arrg.EventCount-1-crt]=swap;
                }
             crt++;
            }
        }
 
     }//if (len!=0)
   return;
  }

The MakeSimpleArrangement() function will build a simple arrangement, from a string, as instructed by the user. For instance, “10 !6 11″ means “10, followed by 11, with a 6 that has not to appear before 11″. Spacebar is separator, and negation flag can be “-”,”!”,”n” or “N”. Note that at the end of the function, data is reversed, from last to first, as I said in the previous article, that complex event arrangements will start from the nearby events going to the older events. This function has not to be called by the user, rather it is called by MakeComplexEvent().

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
void MakeComplexEvent(string s,uint eventid,uint time,ComplexEvent &cplx)
  {
   int len=StringLen(s+" ");
   string rdnow="";
   string c;
   string arrg;
   ushort cnt;
   s=s+" ";
   cplx.ComplexID=eventid;
   cplx.TimeConstraint=time;
   cplx.ArrangementsCount=0;
   if (len!=0)
     {
      for (int i=0;i<len;i++)
         {
          c=StringSubstr(s,i,1);
          if (c==";")
            {
             if (rdnow!="")
               {
                MakeSimpleEventArrangement(rdnow,cplx.ArrangementsArray[cplx.ArrangementsCount]);
                cplx.ArrangementsCount++;
                rdnow="";
               }//if (rdnow!="")
            }//if (c==" ")
          else
            StringConcatenate(rdnow,rdnow,c);
         }//for (int i=0;i<len;i++)
      //for last word
      if (rdnow!="")
        {
         MakeSimpleEventArrangement(rdnow,cplx.ArrangementsArray[cplx.ArrangementsCount]);
         cplx.ArrangementsCount++;
         rdnow="";
        }//if (rdnow!="")
     }//if (len!=0)
   return;
  }

The MakeComplexEvent() function is to be called by user when telling the engine how the complex events look like. The definition is a big string with more simple event arrangements, separated by semicolon. The other parameters are the complex event ID, the time constraint and a complex event variable to store the result in.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
string ComplexEventToString(ComplexEvent& c)
  {
   string res="";
   res=res+"ComplexID "+DoubleToString(c.ComplexID,0)+"n";
   res=res+"Arrangements "+DoubleToString(c.ArrangementsCount,0)+"n";
   for (int i=0;i<c.ArrangementsCount;i++)
      {
       res=res+"Events: "+DoubleToString(c.ArrangementsArray[i].EventCount,0)+"n";
       for (int j=0;j<c.ArrangementsArray[i].EventCount;j++)
          {
           res=res+" "+DoubleToString(c.ArrangementsArray[i].EventArray[j],0);
          }
       res=res+"n";       
       res=res+"Negations:n";
       for (int j=0;j<10;j++)
          {
           for (int k=0;k<c.ArrangementsArray[i].EventCount;k++)
              {
               res=res+" "+DoubleToString(c.ArrangementsArray[i].NegationsArray[j][k],0);
              }
           res=res+"n";
          }
       res=res+"n";
      }
   return(res);
  }

This function was made for commenting reasons only. The function describes a complex event in a string form. You can output the resulting string with Comment().

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
class CEPEngine
  {
   //setup elements
   private:
   uint QueueTimeConstraint;
   ComplexEvent ComplexEventsTable[30];
   ushort ComplexEventsCount;
   //data
   SimpleEvent EventsQueue[1000];    
   int OldestIndex;
   uint OldestTime;
 
   public:
   void CEPEngine();//constructor
   void ClearEvents();//cleans event table
   void AddComplexEvent(ComplexEvent &c);   
   virtual void EventsCallback(uint EventID);//here you treat raised events from the CEP Engine
   void SetupEngineMilliseconds(uint ms);
   void RaiseEvent(uint EventID);
   void EngineRun();
   string StatusString();
   };

Finally, we got to the CEP engine class.
QueueTimeConstraint is an important variable, which tells which is the length in time for which the engine will look in the queue. Incoming simple events, that are found in the EventsQueue[] , and go past this “event horizon” are overwritten. The OldestIndex is the end of the array ; it is a dynamic end, calculated at every new RaiseEvent() call from the outside, abiding the QueueTimeConstraint, and it has the time , in ticks, in OldestTime.

Now the small stuff:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
   void CEPEngine::ClearEvents()  //cleans event table
     {
      for (int i=0;i<ArrayRange(ComplexEventsTable,0);i++)
         {
          ComplexEventsTable[i].ArrangementsCount=0;
         }
      for (int i=0;i<ArrayRange(EventsQueue,0);i++)
          {
           EventsQueue[i].EventID=0;
           EventsQueue[i].EventTime=0;
           EventsQueue[i].BindCount=0;
           ArrayInitialize(EventsQueue[i].Binds,0);
          }
      OldestIndex=-1;
      OldestTime=0;
      ComplexEventsCount=0;
     }  
 
   void CEPEngine::CEPEngine()
     {
      ClearEvents();
     }
 
   void CEPEngine::AddComplexEvent(ComplexEvent &c)
     {
      ComplexEventsTable[ComplexEventsCount]=c;
      ComplexEventsCount=ComplexEventsCount+1;
     }
 
 
   void CEPEngine::SetupEngineMilliseconds(uint ms)
     {
      QueueTimeConstraint=ms;
      return;
     }
 
 
   void CEPEngine::EventsCallback(uint EventID)//here you treat raised events from the CEP Engine
     {
     }   
 
    string CEPEngine::StatusString()
       {
        string res="";
        string eventtext;
        int etl;
        if (ComplexEventsCount!=0)
          {
           for (int c=0;c<ComplexEventsCount;c++)
              {
               eventtext="["+DoubleToString(ComplexEventsTable[c].ComplexID,0)+"]::";
               etl=StringLen(eventtext);
               res=res+eventtext;
               if (ComplexEventsTable[c].ArrangementsCount!=0)
                 {
                  for (int a=0;a<ComplexEventsTable[c].ArrangementsCount;a++)
                     {                     
                      if (ComplexEventsTable[c].ArrangementsArray[a].EventCount!=0)
                        {                         
                         for (int e=0;e<ComplexEventsTable[c].ArrangementsArray[a].EventCount;e++)
                            {
                             res=res+DoubleToString(ComplexEventsTable[c].ArrangementsArray[a].Prebinds[e],0)+"  ";
                            }
                         res=res+"n"+Replicate(" ",etl);
                         for (int e=0;e<ComplexEventsTable[c].ArrangementsArray[a].EventCount;e++)
                            {
                             res=res+BoolToString(ComplexEventsTable[c].ArrangementsArray[a].NegationsChecked[e])+"  ";
                            }
                         res=res+"n";
                         if (a!=ComplexEventsTable[c].ArrangementsCount-1)
                           {
                            res=res+Replicate(" ",etl);
                           }                      
                        }//if (ComplexEventsTable[c].ArrangementsArray[a].EventCount!=0)                   
                     }//for (int a=0;a<ComplexEventsTable[c].ArrangementsCount;a++)
                 }//if (ComplexEventsTable[c].ArrangementsCount!=0)
              }//for (int c=0;c<ComplexEventsCount;c++)           
          }//if (ComplexEventsCount!=0) 
        return(res);
       }

So, the ClearEvents() clears the simple events queue. This is the reason for which it is called from the CEPEngine() constructor. The AddComplexEvent() adds a complex event to the table. It has to be called by user, after making complex events with MakeComplexEvent(). The engine is ready to work after its time constraint is set up by the user with SetupEngineMilliseconds(). And finally, the EventsCallback() is the virtual method that has to be overridden by the user, being called when the CEP engine raises a complex event.
The StatusString() function produces a comment version of the engine status.

Now some average complicated stuff: RaiseEvent() : when the user raises a simple event

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
    void CEPEngine::RaiseEvent(uint EventID)
      {
       long time=GetTickCount();
       bool isforbidden=true;
       uint opposite=0;
       uint difft;
       if (OldestIndex>0)
         {
          for (int i=0;i<ArrayRange(EventsQueue,0);i++)
             {
              if (EventsQueue[i].EventTime==0)
                {
                 OldestIndex=i-1;
                 OldestTime=EventsQueue[i-1].EventTime;
                 break;
                }
              if (time-EventsQueue[i].EventTime>QueueTimeConstraint)
                {
                 OldestIndex=i-1;
                 OldestTime=EventsQueue[i-1].EventTime;
                 break;
                }//if (time-EventsQueue[i].EventTime>QueueTimeConstraint)  
             }//for (int i=0;i<OldestIndex+1;i++)
         }//if (OldestIndex!=0)
       else
         {
          if (OldestIndex<=0)
            {
             OldestIndex=0;
             OldestTime=time;
            }
         }
       for (int j=OldestIndex+1;j>0;j--)
          {
           EventsQueue[j].EventID=EventsQueue[j-1].EventID;
           EventsQueue[j].EventTime=EventsQueue[j-1].EventTime;
           EventsQueue[j].BindCount=EventsQueue[j-1].BindCount;
          }
       ArrayInitialize(EventsQueue[0].Binds,0);
       EventsQueue[0].EventID=EventID;
       EventsQueue[0].EventTime=time;
       EventsQueue[0].BindCount=0;
       OldestIndex=OldestIndex+1;
 
       if (ComplexEventsCount!=0)
         {
          EngineRun();
         }         
       return;
      }//raise event ends

The function has a moderate degree of complexity.
Lines 7 – 32 establish the event horizon within the queue. That is, which is the oldest event within the queue, abiding the time constraint of the whole queue.
Lines 33 – 38 move the queue one step lower , making room for a new simple event.
Lines 39 – 43 insert the new simple event coming from the user program.
Lines 45 – 48 call the EngineRun() method is to process the new data, if at least one complex event is registered in the table.

At the heart of the CEP engine lies the EngineRun() method. It is the most complicated piece of code of the entire class. But, there would be no car without the engine, right?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
   void CEPEngine::EngineRun()
     {
      long time=GetTickCount();
      bool any_binded;
      int CrtColumn[100][100];//address CrtColumn as CrtColumn(event,arrangement)
      bool bypass=false;
      int q,p;
      int j,lastj;
      int test;
      if (ComplexEventsCount==0)
        return;
      ArrayInitialize(CrtColumn,0);
      for (int c=0;c<ComplexEventsCount;c++)
         {
          Print("Arrangements in complex event ",c," are ",ComplexEventsTable[c].ArrangementsCount);
          if (ComplexEventsTable[c].ArrangementsCount!=0)
            {                          
             for (int a=0;a<ComplexEventsTable[c].ArrangementsCount;a++)
                {      
                 //Check if the current simple event within arrangement within current complex event is to be binded
                 //by checking with simple events from the queue
                 p=0;                                       
                 while (time-EventsQueue[p].EventTime<=ComplexEventsTable[c].TimeConstraint)
                   {
                    Print("Looking on entry ",p," where is simple event ",EventsQueue[p].EventID," comparing with ",ComplexEventsTable[c].ArrangementsArray[a].EventArray[CrtColumn[c][a]]," on complex event ",c,", arrg. ",a," (column ",CrtColumn[c][a],") vs count ",ComplexEventsTable[c].ArrangementsArray[a].EventCount);                       
                    if (EventsQueue[p].EventID==ComplexEventsTable[c].ArrangementsArray[a].EventArray[CrtColumn[c][a]])
                      {
                       Print("Found simple event ",EventsQueue[p].EventID," for column ",CrtColumn[c][a]," of arrangement ",a," of event ",c," meaningly id=",ComplexEventsTable[c].ComplexID); 
                       ComplexEventsTable[c].ArrangementsArray[a].Prebinds[CrtColumn[c][a]]=p;
                       CrtColumn[c][a]++;                                                                                        
                       bypass=false;
                       if (CrtColumn[c][a]>1)
                         {
                          q=ComplexEventsTable[c].ArrangementsArray[a].Prebinds[CrtColumn[c][a]-2];
                          //-2 instead of -1 because we already incremented the CrtColumn by ++
                          Print("Checking negations from previous bind, ",q," to current element ",p);
                          q=q+1;
                          //Checking negations; if test passes, simple event is prebinded;                                                          
                          //q is an older event than the current one, p, so q>p
                          if (ComplexEventsTable[c].ArrangementsArray[a].NegationsCount[a]!=0)
                            {                                                
                             while (q<p)
                               {
                                for (int j=0;j<ComplexEventsTable[c].ArrangementsArray[a].NegationsCount[a];j++)
                                   {
                                    lastj=j;
                                    Print("Comparing event on ",q," meaningly ",EventsQueue[q].EventID," with negation ",ComplexEventsTable[c].ArrangementsArray[a].NegationsArray[j][CrtColumn[c][a]-2+1]);
                                    if (EventsQueue[q].EventID==ComplexEventsTable[c].ArrangementsArray[a].NegationsArray[j][CrtColumn[c][a]-2+1])
                                      {
                                       ComplexEventsTable[c].ArrangementsArray[a].NegationsChecked[j]=true;
                                       break;
                                      }
                                   }
                                 if (ComplexEventsTable[c].ArrangementsArray[a].NegationsChecked[lastj]==true)
                                   break;
                                 q++;
                                }
                            }//if (ComplexEventsTable[c].ArrangementsArray[a].NegationsCount!=0)
                          if (ComplexEventsTable[c].ArrangementsArray[a].NegationsChecked[lastj]==true)
                            {
                             Print("Erasing last bind on ",CrtColumn[c][a]-1,". Decreasing current column for complex ",c,", arrangement ",a);
                             ComplexEventsTable[c].ArrangementsArray[a].Prebinds[CrtColumn[c][a]-1]=-1;//erase prebind;                                  
                             CrtColumn[c][a]--;
                             bypass=true;
                             break;
                            } 
                         }//if (CrtColumn[c,a]>1)                            
                       if (bypass==false)//meaning,if no negations are checked                          
                         {
                          if (ComplexEventsTable[c].ArrangementsArray[a].EventCount==CrtColumn[c][a])//all prebinds are done
                            {
                             //checking time constraint again
                             if (GetTickCount()-EventsQueue[ComplexEventsTable[c].ArrangementsArray[a].Prebinds[ComplexEventsTable[c].ArrangementsArray[a].EventCount-1]].EventTime<=ComplexEventsTable[c].TimeConstraint)
                               {
                                //before binding, will check if any of these events is previously binded to the same event
                                any_binded=false;
                                for (int e=0;e<ComplexEventsTable[c].ArrangementsArray[a].EventCount;e++)
                                   {
                                    Print("Prebind",ComplexEventsTable[c].ArrangementsArray[a].Prebinds[e]," bind location of complex event id ",ComplexEventsTable[c].ComplexID," : ",FindBind(ComplexEventsTable[c].ComplexID,EventsQueue[ComplexEventsTable[c].ArrangementsArray[a].Prebinds[e]]));
                                    if (FindBind(ComplexEventsTable[c].ComplexID,EventsQueue[ComplexEventsTable[c].ArrangementsArray[a].Prebinds[e]])!=-1)
                                      {
                                       any_binded=true;
                                       Print("One event is already binded to complex event ",ComplexEventsTable[c].ComplexID);
                                       break;
                                      }
                                   }
                                if (any_binded==false)
                                  {
                                   //binding event
                                   for (int e=0;e<ComplexEventsTable[c].ArrangementsArray[a].EventCount;e++)
                                      {
                                       Print("Binding simple event on ",ComplexEventsTable[c].ArrangementsArray[a].Prebinds[e]," to ",ComplexEventsTable[c].ComplexID);
                                       BindEvent(ComplexEventsTable[c].ComplexID,EventsQueue[ComplexEventsTable[c].ArrangementsArray[a].Prebinds[e]]);
                                      }                       
                                   EventsCallback(ComplexEventsTable[c].ComplexID);//raise complex event for user to handle                                   
                                  }//if (any_binded==false)                                
                                break;//search exhausted for current arrangement of current complex event
                                      //it's either complete, so pointless to look forward
                                      //or one of the simple events is binded to the same complex event, again pointless
                               }//if (GetTickCount()-EventsQueue[ComplexEventsTable[c].ArrangementArray[a].Prebinds[...]].EventTime<=ComplexEventsTable[c].TimeConstraint)
                             else//it's a flop: completed too late; deleting binds
                               {                                      
                                for (int f=0;f<ComplexEventsTable[c].ArrangementsArray[a].EventCount;f++)
                                   {
                                    Print("Unbinding simple event from ",ComplexEventsTable[c].ArrangementsArray[a].Prebinds[f]," to ",ComplexEventsTable[c].ComplexID);
                                    UnbindEvent(ComplexEventsTable[c].ComplexID,EventsQueue[ComplexEventsTable[c].ArrangementsArray[a].Prebinds[f]]);
                                    ComplexEventsTable[c].ArrangementsArray[a].Prebinds[f]=-1;
                                    ComplexEventsTable[c].ArrangementsArray[a].NegationsChecked[f]=false;
                                   }//for (int f=0;f<ComplexEventsTable[c].ArrangementsArray[a].EventCount;f++)
                                 break;//no point to look forward , time`s up already
                               }//else if (GetTickCount()-EventsQueue[ComplexEventsTable[c].ArrangementArray[a].Prebinds[...]].EventTime<=ComplexEventsTable[c].TimeConstraint)
                            }//if (ComplexEventsTable[c].ArrangementsArray[a].EventCount==CrtColumn[c][a])
                         }//else if (bypass==true)                                                                                                 
                      }//if (EventsQueue[p].EventID==ComplexEventsTable[c].ArrangementsArray[a].EventArray[CrtColumn[c,a]])
                    p++;
                   }//while (time-EventsQueue[p].EventTime<=ComplexEventsTable[c].TimeConstraint)
                }//for (int a=0;a<ComplexEventsTable[c].ArrangementsCount;a++)
            }//if (ComplexEventsTable[c].ArrangementsCount!=0)
         }//for (int c=0;c<ComplexEventsCount;c++)       
 
      for (int c=0;c<ComplexEventsCount;c++)
         {          
          if (ComplexEventsTable[c].ArrangementsCount!=0)
            {
             for (int a=0;a<ComplexEventsTable[c].ArrangementsCount;a++)
                {                     
                 if (ComplexEventsTable[c].ArrangementsArray[a].EventCount!=0)
                   { 
                    if (CrtColumn[c][a]<ComplexEventsTable[c].ArrangementsArray[a].EventCount)
                      {
                       //removing prebinds
                       for (int f=0;f<ComplexEventsTable[c].ArrangementsArray[a].EventCount;f++)
                          {
                           ComplexEventsTable[c].ArrangementsArray[a].Prebinds[f]=-1;
                           ComplexEventsTable[c].ArrangementsArray[a].NegationsChecked[f]=false;
                          }//for (int f=0;f<ComplexEventsTable[c].ArrangementsArray[a].EventCount;f++)                         
                      }//if (CrtColumn[c][a]<ComplexEventsTable[c].ArrangementsArray[a].EventCount)
                   }//if (ComplexEventsTable[c].ArrangementsArray[a].EventCount!=0)                       
                }//for (int a=0;a<ComplexEventsTable[c].ArrangementsCount;a++)
            }//if (ComplexEventsTable[c].ArrangementsCount!=0)
         }//for (int c=0;c<ComplexEventsCount;c++)          
      return;
     }

The function contains two loops. One is the main loop, lines 13 – 117 , and the second loop is 120 – 140. The second loop has a cleanup role to be explained below.

The main loop will go thru the ComplexEventsTable, to check which appear. For each complex event within this table, the arrangements will be checked, and for each arrangement, a series of procedures will be performed.
Inside this loop, another while loop will pass thru the EventsQueue[], (lines 23 – 115) from the first to the last, looking for events that match the time constraint of the current complex event. One of the most important variables inside this method is the CrtColumn[] array. This one stores the current pointer (current column in the unidimensional EventArray[]). Thus, the procedure checks the current required simple event from the table with the one found in the queue (line 26) , prebinds it and increases the CrtColumn[] value for the current complex event arrangement (lines 29 – 31). Since CrtColumn[] is initialized with zeros, if it finds first event, it is increased to 1. Now, if the CrtColumn[] is higher than one (meaning that the second event was found, and preparing to find the third) it is necessary to check if between the previous two events (e.g. first and the second, in this case) there are negation events (the ones that are forbidden from appearing), only if the NegationsCount[] for that specific simple event is nonzero (lines 40 – 59). If one negation event appears, it will break the search – pointless for the current arrangement. It also sets the bypass variable to true, for the following code, however that is just a programming technique to make the listing look better – the following code will be executed only if no negations appear in the meanwhile. Since events array is zero based, the maximum event that can be binded is EventCount-1, and, if CrtColumn[] for the current arrangement is equal to EventCount, then all events were prebinded. The complex event is like already done. What follows happens only if the oldest event still matches the time constraint. It will check if at least one of the pre-binded events of this pass was binded for real to the analysed complex event, already. In this case, will raise the any_binded flag to be true, and break. If the flag is not raised, then the complex event is considered completed in this arrangement, simple events are binded to the complex event , EventsCallback() is called to notify the user and current search is breaked (lines 90-96) . Otherwise, pre-binding is failed – happens too late. Events are unbinded, negations and prebinds are erased, search it’s breaked – no point to look forward if it’s too late already.

Then what comes is a secondary loop, lines 121 – 141. Since events are mapped from the nearest to the oldest, they are reported to the user creating incomplete complex events whose “shadowing” that manifests into future prohibiting same complex event from being reported, should it appear again, but made by other simple events. All the prebinds are removed, only definitive binds remain valid. Note, however, that the same complex event, conceptually, may be triggered more times than once, by different arrangements, if these contain completely different simple event streaks (e.g. 15 and 16 or 11 and 10 , if all four appear , meaning the same event, both arrangements will be reported as complex events with the same ID, but distinct) because the engine will not be able know if the event appeared again or these mean the same one.

You may question about negations, as you see negations are event specific, each event having its own negation events before it and the previous event. The current engine does not support global negations. However, if you want some events to be global negations, they have to be set up between every two consecutive simple events. The first event in the queue may have negations assigned, however these will not be checked, because there would be no known timeframe where to check for them. Negations are valid only between consecutive simple events.

This is a script example using the CEP engine in a test. The test consist in passing abstract events to the CEP engine. Let’s call it ceptest.mq5 .

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
//+------------------------------------------------------------------+
//|                                                      ceptest.mq5 |
//|                                        Copyright Bogdan Caramalac|
//|                                           http://mqlmagazine.com |
//+------------------------------------------------------------------+
#property copyright "Copyright 2010, MetaQuotes Software Corp."
#property link      "http://www.mql5.com"
#property version   "1.00"
 
#include <CEPEngine.mqh>
 
 
//our class: we needed it to override EventsCallback;
class LocalCEPEngine : public CEPEngine
  { 
    void EventsCallback(uint EventID)
     {
      Print("COMPLEX EVENT RECOGNIZED: ",EventID);
     }   
  };
 
//the CEP engine object
LocalCEPEngine myengine;
 
//+------------------------------------------------------------------+
//| Script program start function                                    |
//+------------------------------------------------------------------+
void OnStart()
  {
   uint tickstart,tick;
   ulong AbstractEvent;
   ComplexEvent c;
   //setting up engine time span
   myengine.SetupEngineMilliseconds(10000);
   //creating complex events
   MakeComplexEvent("10 !6 11 ; 10 12 13 ; 10 13 12",1111,2000,c); //the event for MSFT;  
   myengine.AddComplexEvent(c);   
   //note the events with ! are negations
   MakeComplexEvent("5 !7 15 ; 5 !7 16 17 ; 5 !7 17 !7 !9 16",2222,3000,c); //the event for ^GSPC;
   myengine.AddComplexEvent(c); 
 
   tickstart=GetTickCount();
   tick=tickstart;
 
//alternate tests:
   myengine.RaiseEvent(10);   
   Print("*****************************************************************************************************************");
   myengine.RaiseEvent(6);   
   Print("*****************************************************************************************************************");   
   myengine.RaiseEvent(11);   
   Print("*****************************************************************************************************************");
   myengine.RaiseEvent(10);
   Print("*****************************************************************************************************************");
   myengine.RaiseEvent(11);   
   Print("*****************************************************************************************************************");
   return;
}

This one generates for instance the following output:

I invite you to check the logs, look how the system is working. Try other tests:
Like this:

1
2
3
4
5
6
//alternate tests:
   myengine.RaiseEvent(10);   
   Print("*****************************************************************************************************************");
   myengine.RaiseEvent(11);
   Print("*****************************************************************************************************************");
   return;

Or like this:

1
2
3
4
5
6
7
//alternate tests:
   myengine.RaiseEvent(10);   
   Print("*****************************************************************************************************************");
   myengine.RaiseEvent(6);   
   Print("*****************************************************************************************************************");   
   myengine.RaiseEvent(11);   
   Print("*****************************************************************************************************************");

Or like this:

1
2
3
4
5
6
7
8
//alternate tests:
   myengine.RaiseEvent(10);   
   Print("*****************************************************************************************************************");
   myengine.RaiseEvent(11);   
   Print("*****************************************************************************************************************");
   myengine.RaiseEvent(10);
   Print("*****************************************************************************************************************");
   return;

Or like this:

1
2
3
4
5
6
7
8
9
10
//alternate tests:
   myengine.RaiseEvent(10);   
   Print("*****************************************************************************************************************");
   myengine.RaiseEvent(11);   
   Print("*****************************************************************************************************************");
   myengine.RaiseEvent(10);
   Print("*****************************************************************************************************************");
   myengine.RaiseEvent(11);   
   Print("*****************************************************************************************************************");
   return;

Now you gonna ask me how to find out which combinations of simple events have to be wired up into complex events. Which complex events stay at the basis of the mid frequency algos? How to scout for them?

Well, I didn’t graduate Paul Wilmott’s CQF. I cannot tell you that. I am a simple romanian finance bachelor, possibly going to be fried very soon in the coming mess to the public sector where I sadly work. I gave a minor suggestion about the mechanics of this search, in the article about self-tuning systems. If you got the brains and the power to go forward, and if you believe in the face value of the things, fairness of education and other blah blah stuff – try getting yourself a Computational Finance masters – or a Quantitative Finance masters – should be opening enough. If you can’t finance one, try Finland. I see Hanken is pretty fair – if you have the nerve to pass thru papers and questions. Otherwise, study alone, implement at home – with all the limits coming from this!

File links:
CEPEngine.mqh
ceptest.mq5