The devil is in the details

Easy:
getting real-time data from a financial exchange

Harder:
ensuring that data is correct

Harder Still:
interacting with the exchange and sending markets/orders cancels etc.

Harder Than That:
maintaining a book with offsetting positions

Really Really Really Hard:
dealing with partial fills for new positions and/or exits

Approaching Impossible:
crossing those positions internally to save a trip and the spread, including partial filling, waiting for confirmation that outstanding markets

Worthy of Arthur:
doing all that, without loops, and a subsytem to minimize paying the spread via fancy bid-> agress conversion logic.

E.g. imagine you are long 5@100, looking for the market to goto 110$ in the next hour. Then you get short 10 @101, expecting the market to goto 99 in the next 30 minutes, in 30 minutes the market is 100- what do you do?
Lets say you cross internally. You take the 5 longs off the book, so you cancel the offers for those 5 longs- but wait- while thats happening the market goes to 107/109.

Now instead of 2 positions imagine 100, and instead of 1 price per position- imagine N distinct prices- and a market that jumps all over the place.

Some useful financial functions

thought these might be handy to a few people. I can't take credit for most:

Drawdown
drawdown:{[x]v:u?max u:(maxs x)-x;(u v;x?x[v]+u v;v)};

This returns a vector, the drawdown value, the index of the start and end of that period.


EMA
ema:{[n;x]b:1-a:2%n+1;c:(sum n#x)%n;((n-1)#0n),c,c{[a;b;x;y](a*y)+b*x}[a;b]\n _x};

Smooth Moving Average
smavg:{[n;x]((n-1)#0n),i,{(z+(x-1)*y)%x}[n]\[i:avg n#x;n _ x:0^x]};

Max Consecutive Losses
MaxConsectiveLosers:(max count each "0"vs raze string pl<0)

Convert Tics to Float and visa versa

f2tic:{[x]a:x - floor x;b:a%(1%32);c:floor(a -(floor b)%32)%(1%256);if[c=4;c:"+"];raze raze string(floor x),"-",string floor b, string floor c};

tic2f:{("I"$n#x)+((8*"I"$x n+1 2)+"0123+567"?x 3+n:x?"-")%256.}


Back testing in Q

I get this all the time- how do you backtest a strategy.
Well, here's a simple way. Every trade has 4 components- it's entry time, it's profit objective (ge: good exit)) it's stop limit (se- stop exit) and a time exit (te).
The code below will, given a table with a column "entry" which is boolean and a price will backtest the strategy (this is a version for going long). I like to back test against quotes, but this is for prices (trades).




btL:{[t;gep;sep;tep]
i_eb:where t`entry; //where are the entry indicies
i_te:(count t)^((t`time) bin/: ((t@i_eb)`time)+tep*1000); //find the indicies for the time exits
rng:{x+key floor (y-x)}'[i_eb;i_te]; //define the ranges
f_u:{[xe;limit;p;x]limit&x+xe>p x};f_l:{[xe;limit;p;x]limit&x+xe<p x}; //functions for upper and lower limits
e_pr:(t`entryprice)i_eb; //define the entry prices
i_ge:f_u[(e_pr+gep);i_te;t`bid1]/[i_eb]; //indices for the good exits
i_se:f_l[(e_pr-sep);i_te;t`bid1]/[i_eb]; //indices for the stop exits
i_xe:min each v:(count t)^flip (i_te;i_ge;i_se); //define the exit action
x_ty:(`te`ge`se)@/:i_x:first each iasc each v; //define the exit types
x_pr:(t`bid1) i_xe; //the exit price is the bid at the exit index
x_pl:x_pr-(e_pr); //the exit pnl is the exit price- entry price
`entrytime`exittime`entryprice`exittype`exitprice`exitindx`pnl!((t`time)i_eb;(t`time)i_xe;e_pr;x_ty;x_pr;i_xe;x_pl)}





An example

n:1000;
t:`time xasc flip `time`price`entry`entryprice!(n?`time$.z.Z;n?10;n?01b;n?10)
btL[t;2;1;30] //go for 2$,risk 1$, hold for 30 seconds


1 billion queries

Prior to a life in finance, I had a brief stint in root cause analysis for diseases for the Department of the Navy (NMIC).
They problem was always the same- a ton of variables and some measure of sickness. For example you might have weight, age, blood pressure etc and on the right hand side you have a weighted value of sickness (visits, stage of disease etc). If I had q and arthur back then, I think we could have cured cancer. The code below does 1 billion queries in a couple of seconds, finding all 3 variable combinations. For example if the variables are `a`b`c, and the weighted death value is D then an example output is

if a>2,b<3,d>5 then the sum of D is 2345


n:500000
m:5
t:([]a:n?1.0;b:n?1.0;c:n?1.0;d:n?1.0;e:n?1.0)
il:`b`c`d`e //variable list il (independent list)

k)f3:{s2'+\(m;0N)#@[(m*m*m)#0.0;m/:(x;(m-1)-y;z);+;t.a]}
k)u:m .q.xrank't il
k)r3:{u f3[x]\:/:u}':u

CEP

Complex Event Processing- is overly complex.
I don't go to trade shows anymore- or talk to vendors. This reminds me of the first large installations of networks in the 90's (WAN). The idea's were simple- and so was the base technology- but in order to make a buck vendors made the whole process so complicated and full of jargon that no-one knew what the hell was going on.

Seriously

Event processing is this

kdb+tic-> subscriber
alert on subscriber-> push out
chain another subscriber.

How hard is that?

kdb+ as the infrastructure

We use kdb+ as our sole technology infrastructure. Some people find that odd. What? No java? No C#? No Tibco? And then the inevitable.. "How do you..." So I thoughts I'd anwser some common questions.

How do you?

Q... Build GUI's
A: In general we don't. We do use flash (adobe flex) to connect to kdb+ via q's built in XML parser- but that's only for show. We use the kdb+Excel link for real-time information. But in general we have few GUI's.

Q... Handle Risk Management
A: kdb+tic is all you need for an event driven system. Risk management is just an event based decision process. Requested positions come in, they are checked against a bank of risk management rules, and either they are rejected or passed to the execution engine.

Q... Execute trades
A: The feedhandler works both ways. It can both capture data from the exchange and send it. This ensures that you are not making decisions you can't trade on.

Q...Why kdb+
A: Speed and simplicity. Our entire trading system- which is data capture, back testing, real time screening, risk management, execution and position management is about 8 pages and very very fast.

Send in some questions and Ill post them here