j/k

A New Computer

2009-01-30T05:16:00.000-08:00

It's probably been 10 years since I've bought a new computer. I feel like it's a good habit to develop on older, less advanced hardware- it makes you write tighter code (256MB of memory anyone). But the house computer died, and it was time to build a new media center. Since this is a once a decade purchase- I got the signoff from the boss lady to do it "my way".

So I went with a true media center.
I got a 902 Euni Case off ebay, with the board, CPU and Fan off eBay for 700$.

This is the setup (to start)

MSI Platinum 975x Power Up Edition MotherBoard
Intel Dual Core E6600 @ 2.4Ghz
HD Tuner Card
On Board 5.1 Sound

Now to tune this up I put in dual OCZ 64GB SSD Drives. These drives are rates 155/90 MBs Read/Write. I put them in Raid 0. Then some more memory, OCZ 4GB DDR800 (29$ ?!? I remember when 4GB of memory was 1k!).
Then Off to the OS. I went with Vista... my first experience. I chose the 32bit version for ompatibility (nobody writes 64bit drivers yet, and it is a family computer). After the intense install, and the absurd driver dectective work... Vista was up. After 5 hours of registry hacks, service disabling, process removal, etc... Vista was usable.

Let me be the first to say, SSD drives are amazing. The applications load instantly, the OS boots from cold in 20 seconds (be sure to enable the boot manager to use both cores... why this isn't enabled by default is beyond me). I threw in a 7900GT graphics card, because honestly it's good enough and I'm not a gamer. HD picture is perfect, video's are clear and crisp. Just a wonderful little machine. Ans those SSD drives? See below

Read Rates

File ReaD and Write Rates (SSD write is not up to par just yet, but still absurdly quick).

Access time..... nuff said

A little h2benchw (from the German mag C't) sums it up nicely
**********************************************************************
Interface transfer rate w/ block size 128 sectors at 0.0% of capacity:
Sequential read rate medium (w/out delay): 273626 KByte/s
Sequential transfer rate w/ read-ahead (delay: 0.26 ms): 1201217 KByte/s
Repetitive sequential read ("core test"): 1757337 KByte/s

Sustained transfer rate (block size: 128 sectors):
Reading: average 270946.7, min 147388.7, max 279534.2 [KByte/s]

Random access read: average 0.22, min 0.17, max 0.38 [ms]
Random access read (<504 MByte): average 0.23, min 0.03, max 0.55 [ms]

Application profile `swapping': 64632.6 KByte/s
Application profile `installing': 32218.5 KByte/s
Application profile `Word': 99457.5 KByte/s
Application profile `Photoshop': 91775.7 KByte/s
Application profile `copying': 220664.1 KByte/s

riddles

2007-09-07T11:23:00.000-07:00

I was under the impression that riddles were out of style for interviews. I was wrong. Don't get me wrong, I love a good brain teaser- but as a criteria for hiring?
My favorite are the "personal probing riddles"
I actually got this today

q:"if you were stuck on a desert island what 3 items would you bring with you"
me:"seriously?"
q:"yeah"
me:"ok. a boat, a sat phone. and a tank of gas"
q:"oh"
me:"what would you take?"
q:"well, generally people say their favorite book or something"
me:"that wont get you off the island."
q:"no, it wouldn't.
me:"next question"

the times, they are a changing

2007-09-04T18:43:00.000-07:00

So, the dust has settled... mostly.
The world has calmed... mostly.
And the market has stabalized- er, well at least it's not tanking- er, at least not as bad as a couple of weeks ago.

And now is the time for change.

At my firm, it began with mortgages (pronounced "morgue-ages"). Then onto CDO. I get that. Then onto repo. OK. And now, it has began to spread everywhere. My firm was not a trading firm- purely brokerage. Riskless. That was until my division came around. As a whole, I think we did ok, but the scare has changed things. Books are flattening, and you can taste change in the air.

And so, after 2 years, and my best month of trading ever (seriously- it was ridiculous. 100% win rate one day. 122mm traded. 100%! Come on) I find my self looking for another job, which is a real shame because I loved my job. I loved my boss. And I loved that the solutions to the problems we were solving were always just out of our reach- which kept us reaching. I loved my team, small and fast, the way all things should be. Most of all I loved my freedom- it was a lab first- a trading desk second. And although at the time I sweated the actual trading- looking back it was good for me personally.

I'm not afraid of change, I was always ready to leave if required. I've had the number for a truck driving school as my desktop for almost 2 months. Change is good. It means a new challenge, new problems, and new people/friends. And so on that note, if you're looking for either

a. A professional surfer for your southern California team. Or...
b. A new statistical arbitrage group. Or...
c. A kdb+ programmer (I've decided not to work in another language except if absolutely necessary)

Then drop me a line.

Fridays (again)

2007-07-20T08:13:00.000-07:00

Yup. Worst day ever. Just further reinforcement that coming in on Friday's is bad for me, bad for business. All over by 10AM.
Damn.
j

redundancy

2007-07-19T04:31:00.000-07:00

I've always said (and I picked it up from somewhere) that for every 9 you want to add to your uptime, multiply your initial cost by 9.
So if you have a 10mm USD data center, and you want to go from 99 to 99.9% uptime, you need a 90mm data center infrastructure. Why? Well lets take a look

You need back up generators, back up network lines, which means changing the data center architecture to support all that. Redundant gas lines for the generators. Failover for power sources etc (would have been cheaper to build it like that in the first place)-8mm
You need double the hardware, and multiple components in all the hardware you have (dual nic etc)- 7mm
You need a DR data center in another location, with the same setup- 15mm.
You need a new high speed line for the center links, and a new line for the DR site (different clec)- 2mm
You need to upgrade the SAN with real time LUN level replication, and buy one for the DR site - 5mm
You need clustering, on everything (oracle,sybase, windows, unix, linux ...), custom coded apps need to be re-written for active failover- 25mm (includes services)
You need the buy staff to deal with active/active failover, and 24x7 operation- 4mm
You need load balancing and/or fail over network operations for inbound and outbound connections (data feeds etc)- 10mm (at least!)
Since you have 2 different physical locations now (hopefully not the same state) you need new services contracts- 1mm

So thats only 78mm more, or 8x the original cost. But I'm sure I could spend the extra 12mm on something I've forgotten.

The best DRS I've ever seen was at the Depart of the Navy. Everything was a virtual machine. Live snapshots of the VM's was taken every hour or so. The snapshots were saved to an EMC SAN- which had real time replication to 5 other locations. All locals replicated to all remotes. Every remote site had a small cluster of "failover machines". The network was designed by Cisco and everything could automagically routed wherever. So, the entire data center in VA gets blown up (or whatever). The alarm fires, the VM's are started at the primary failover site (NC), they come online, routers do their thing (the DoD has the benefit of their own network) and wallah. Magic data center moved. Worst case loss, 1 hour. Failover time for the entire data center- 5-10 minutes (AND no application restart- they are hot snapshot loads). beautiful.

So, having built a couple of data centers in my day, and coded many an application for active failover, and having deployed clustering on every version of windows since NT4 and Red hat- when a data center is down for say 7 hours, I think people should not only be fired, but any contractors should be sued. I'm not saying who's data center went down, but let's just say it was bad.
Oh, kdb+ failover is trivial. Everything I do is in a pub/sub model. So aside from an extra machine in my data center- I push everything to my desktop. So when the lights went out, I still knew my positions.
J

BSC- sorry about that

2007-07-18T05:22:00.000-07:00

Sorry, we lost all the money. Well- 90% of it, but the other 10% is ours.
In other news, we have some free flights to Florida for you- and all you need to do is attend a 2 hour presentation on the wonderful oppertunity for real estate in the greater Miami area.

maths

2007-07-11T06:08:00.001-07:00

I've tossed this around to a bunch of people, so I thought I'd toss it out here and maybe someone would know an answer:

Given 3 series, A B and C
Solve for the next value of each series subject to the constraints:


correlation of A&B>=.9
correlation of B&C>=.9
correlation of A&C>=.85

deviation of A is <=.051
deviation of B is <=.051
deviation of C is <=.051

There are multiple correct answers, I'm looking for as many as possible as quickly as possible.

Here's some sample data

A B C
2.47 3.453 4.263
2.476 3.405 4.211
2.484 3.429 4.228
2.46 3.377 4.178
2.395 3.309 4.119
2.387 3.298 4.115
2.46 3.394 4.215
2.582 3.494 4.296
2.591 3.508 4.293
2.55 3.456 4.24
2.469 3.363 4.161
2.485 3.411 4.197
2.485 3.404 4.188
2.469 3.36 4.137
2.436 3.343 4.126
2.478 3.377 4.166

Amending to a Matrix

2007-07-11T04:40:00.000-07:00

Jamie did an excellent job on explaining this and I thought I share it (cleaning out old emails)

A way of amending multiple points in a square matrix in one go.
Take the following matrix:


q)4 4#0
(0 0 0 0;0 0 0 0;0 0 0 0;0 0 0 0)

Let's say we want to add 1 to the diagonal (make it the identity matrix). We could get ourselves a list of coordinates (0 0;1 1;2 2;3 3) and do it one at a time, using over to pass the previous result forward each time:


q).[;;+;1]/[4 4#0;(0 0;1 1;2 2;3 3)]
(1 0 0 0;0 1 0 0;0 0 1 0;0 0 0 1)

This would get pretty slow if the list of coordinates is large, since they are essentially scalar operations. So we can do as Arthur has done, flatten the matrix and use a bit of code to map the 2 dimensional coordinate to it's one dimensional equivalent:


q)(4*4)#0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
q)4 sv flip (0 0;1 1;2 2;3 3)
0 5 10 15
q)@[(4*4)#0;4 sv flip (0 0;1 1;2 2;3 3);+;1]
1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1

The cut command gives us back the matrix from the flat vector:


q)4 cut @[(4*4)#0;4 sv flip (0 0;1 1;2 2;3 3);+;1]
(1 0 0 0;0 1 0 0;0 0 1 0;0 0 0 1)

Data Mining in kdb+

2007-07-10T12:36:00.000-07:00

I'm revisting the "billion query code" because many people have sent questions in.
Mining in Practice
In general, if you can brute force all the solutions to a given search space (or phase space as my physics friends and wife say) - thats what you want to do. If you can't, time for a heuristic.
What do I mean by space? Well lets consider an example:

K- Maximum Sum Subarray Problem

We have a table (arrayable) T


n:10
t:ungroup flip (`$/:.Q.a)!enlist each (26 0N#(26*n)?5)*(26 0N#(26*n)?-1 1)

Which looks like
q)show flip t
a| 0  0  4  -2 -3 3  0  -4 1  -1
b| 0  3  1  -4 -3 2  4  3  0  -3
c| 4  -3 0  -1 -2 0  -3 -4 1  0
...
z| -1 2  0  3  -2 2  1  -3 0  2

Now I ask you to find the maximum sum of the column z, using any combination of the other variables. E.g. let's sort by column a

We can see that if we use the range a>=0&a<=5 then we have


q)show flip `a xasc t
a| -4 -3 -2 -1 0  0  0  1  3  4
b| 3  -3 -4 -3 0  3  4  0  2  1
c| -4 -2 -1 0  4  -3 -3 1  0  0
...
z| -3 -2 3  2  -1 2  1  0  2  0

What if we add the condition b>=3? Then the intersection is...


q)show flip `a xasc t
a| -4 -3 -2 -1 0  0  0  1  3  4
b| 3  -3 -4 -3 0  3  4  0  2  1
c| -4 -2 -1 0  4  -3 -3 1  0  0
...
z| -3 -2 3  2  -1 2  1  0  2  0 //the -1 falls out

Ok now what's the best we can do? How hard is this problem? If we consider all 3 dimensional solutions (e.g. using 1 a, 1 b and 1 d) the problem has */26 26 26 5 5 5 solutions (about 2 million). But in real life we have lots more values (more than 5) and generally more variables.

Breaking down the problem:
A good place to start is to reduce the dimensionality by bucketing values. Consider placing value in m uniform buckets. The code below does this- and quickly.
Then you can do the search, this is ludicrously fast in q (arthur code of course).



n:500000
m:10
t:([]a:n?1.0;b:n?1.0;c:n?1.0;d:n?1.0)
f:`b`c`d

/running totals in 2 dims
s2:+\+\'(m;0N)#
/ 3d aggrs
f3:{s2'+\(m;0N)#@[(m*m*m)#0.0;m/:(x;(m-1)-y;z);+;t.z]}

\t u:m .q.xrank't f
\t r3:{u f3[x]\:/:u}'u

Stay home Friday's

2007-07-06T11:17:00.001-07:00

So-
yesterday and the day before the holiday were great days for me. I trade a pattern recognition system (which is just jargon for "matched patterns built from data mining to streaming data"). Most of the time the system trades as normal- never that big of a position- and life is fine. I win, I lose, but overall I'm up with a very small draw down. But the market has been wicked recently- like a skittish, rabid, cocaine addicted rabbit.

So today I walk in, knowing it's going to be a bad day. Why? Well first of all it seems every time I trade on Friday (my day off) I lose- as if to reinforce the fact that I should have stayed home. But also after 2 big winning days- I was in for a loss.
So the day goes like this

5 AM- wake up/news
5:30 Out the door
5:45 At the office
6:00 Turn on trading system (start q process), check last night's data, get coffee
wait
8:00 (or about) start trading
8:30 lose $7500 on one trade- ok
8:45 lose another 3k- still ok
9:00 down another 10k mark to market- that sucks
9:30 risk management and other stuff begins triggering (q is awesome)
10:00 system fights back from down 20k to down only 11
11:00 flat (down only 1k)
12:00 market grinds down, no trades. At this point I contemplate going home.
12:00:00.001 get long 10 mm
12:10 get long another 10 mm (why not)
12:30 %*^*&*@!!!
13:30 more %*^*&*@!!!
14:00 give up, risk management rolls out of positions, end the day down 30k.
Fuck.

The devil is in the details

2007-06-13T10:39:00.000-07:00

Easy:
getting real-time data from a financial exchange

Harder:
ensuring that data is correct

Harder Still:
interacting with the exchange and sending markets/orders cancels etc.

Harder Than That:
maintaining a book with offsetting positions

Really Really Really Hard:
dealing with partial fills for new positions and/or exits

Approaching Impossible:
crossing those positions internally to save a trip and the spread, including partial filling, waiting for confirmation that outstanding markets

Worthy of Arthur:
doing all that, without loops, and a subsytem to minimize paying the spread via fancy bid-> agress conversion logic.

E.g. imagine you are long 5@100, looking for the market to goto 110$ in the next hour. Then you get short 10 @101, expecting the market to goto 99 in the next 30 minutes, in 30 minutes the market is 100- what do you do?
Lets say you cross internally. You take the 5 longs off the book, so you cancel the offers for those 5 longs- but wait- while thats happening the market goes to 107/109.

Now instead of 2 positions imagine 100, and instead of 1 price per position- imagine N distinct prices- and a market that jumps all over the place.

Some useful financial functions

2007-06-11T12:37:00.001-07:00

thought these might be handy to a few people. I can't take credit for most:

Drawdown

 drawdown:{[x]v:u?max u:(maxs x)-x;(u v;x?x[v]+u v;v)};

This returns a vector, the drawdown value, the index of the start and end of that period.

EMA
ema:{[n;x]b:1-a:2%n+1;c:(sum n#x)%n;((n-1)#0n),c,c{[a;b;x;y](a*y)+b*x}[a;b]\n _x};

Smooth Moving Average
smavg:{[n;x]((n-1)#0n),i,{(z+(x-1)*y)%x}[n]\[i:avg n#x;n _ x:0^x]};

Max Consecutive Losses
MaxConsectiveLosers:(max count each "0"vs raze string pl<0)

Convert Tics to Float and visa versa


f2tic:{[x]a:x - floor x;b:a%(1%32);c:floor(a -(floor b)%32)%(1%256);if[c=4;c:"+"];raze raze string(floor x),"-",string floor b, string floor c};

tic2f:{("I"$n#x)+((8*"I"$x n+1 2)+"0123+567"?x 3+n:x?"-")%256.}

Back testing in Q

2007-05-25T05:24:00.000-07:00

I get this all the time- how do you backtest a strategy.
Well, here's a simple way. Every trade has 4 components- it's entry time, it's profit objective (ge: good exit)) it's stop limit (se- stop exit) and a time exit (te).
The code below will, given a table with a column "entry" which is boolean and a price will backtest the strategy (this is a version for going long). I like to back test against quotes, but this is for prices (trades).



btL:{[t;gep;sep;tep]
 i_eb:where t`entry;  //where are the entry indicies
 i_te:(count t)^((t`time) bin/: ((t@i_eb)`time)+tep*1000); //find the indicies for the time exits
 rng:{x+key floor (y-x)}'[i_eb;i_te]; //define the ranges 
 f_u:{[xe;limit;p;x]limit&x+xe>p x};f_l:{[xe;limit;p;x]limit&x+xe<p x}; //functions for upper and lower limits 
 e_pr:(t`entryprice)i_eb;  //define the entry prices 
 i_ge:f_u[(e_pr+gep);i_te;t`bid1]/[i_eb];  //indices for the good exits 
 i_se:f_l[(e_pr-sep);i_te;t`bid1]/[i_eb]; //indices for the stop exits 
 i_xe:min each v:(count t)^flip (i_te;i_ge;i_se); //define the exit action 
 x_ty:(`te`ge`se)@/:i_x:first each iasc each v; //define the exit types
 x_pr:(t`bid1) i_xe; //the exit price is the bid at the exit index
 x_pl:x_pr-(e_pr); //the exit pnl is the exit price- entry price 
 `entrytime`exittime`entryprice`exittype`exitprice`exitindx`pnl!((t`time)i_eb;(t`time)i_xe;e_pr;x_ty;x_pr;i_xe;x_pl)}

An example


n:1000;
t:`time xasc flip `time`price`entry`entryprice!(n?`time$.z.Z;n?10;n?01b;n?10)
btL[t;2;1;30] //go for 2$,risk 1$, hold for 30 seconds

1 billion queries

2007-05-25T05:17:00.000-07:00

Prior to a life in finance, I had a brief stint in root cause analysis for diseases for the Department of the Navy (NMIC).
They problem was always the same- a ton of variables and some measure of sickness. For example you might have weight, age, blood pressure etc and on the right hand side you have a weighted value of sickness (visits, stage of disease etc). If I had q and arthur back then, I think we could have cured cancer. The code below does 1 billion queries in a couple of seconds, finding all 3 variable combinations. For example if the variables are `a`b`c, and the weighted death value is D then an example output is

if a>2,b<3,d>5 then the sum of D is 2345

n:500000
m:5
t:([]a:n?1.0;b:n?1.0;c:n?1.0;d:n?1.0;e:n?1.0)
il:`b`c`d`e //variable list il (independent list)

k)f3:{s2'+\(m;0N)#@[(m*m*m)#0.0;m/:(x;(m-1)-y;z);+;t.a]}
k)u:m .q.xrank't il
k)r3:{u f3[x]\:/:u}':u

CEP

2007-05-25T05:13:00.000-07:00

Complex Event Processing- is overly complex.
I don't go to trade shows anymore- or talk to vendors. This reminds me of the first large installations of networks in the 90's (WAN). The idea's were simple- and so was the base technology- but in order to make a buck vendors made the whole process so complicated and full of jargon that no-one knew what the hell was going on.

Seriously

Event processing is this

kdb+tic-> subscriber
alert on subscriber-> push out
chain another subscriber.

How hard is that?

kdb+ as the infrastructure

2007-05-25T05:05:00.000-07:00

We use kdb+ as our sole technology infrastructure. Some people find that odd. What? No java? No C#? No Tibco? And then the inevitable.. "How do you..." So I thoughts I'd anwser some common questions.

How do you?

Q... Build GUI's
A: In general we don't. We do use flash (adobe flex) to connect to kdb+ via q's built in XML parser- but that's only for show. We use the kdb+Excel link for real-time information. But in general we have few GUI's.

Q... Handle Risk Management
A: kdb+tic is all you need for an event driven system. Risk management is just an event based decision process. Requested positions come in, they are checked against a bank of risk management rules, and either they are rejected or passed to the execution engine.

Q... Execute trades
A: The feedhandler works both ways. It can both capture data from the exchange and send it. This ensures that you are not making decisions you can't trade on.

Q...Why kdb+
A: Speed and simplicity. Our entire trading system- which is data capture, back testing, real time screening, risk management, execution and position management is about 8 pages and very very fast.

Send in some questions and Ill post them here