|
Homepage > Latest
News
Activity Log
1 March 2010
The 3D visualisation project has progressed to the
stage where we can draw an interesting range of
primitives to represent different types of data.
Getting the interactivity right seems to be a
difficult problem - you really need datagloves (or
other pointing devices) with a speech interface to
make it all "natural", but that will come some day
(hopefully the current fad in "touch" interfaces
will spill over into the third dimension) - but
right now we have to be content with mouse and
keyboard - it seems OK, so far.
There are other problems to be aware of, like
rendering very large or high-dimensional datasets,
but these will be tackled as they arise - the
techniques to deal with them are known; for example
you usually wish to get an overview of a dataset,
then search or filter on certain aspects, then
perhaps drilling down to microscopic detail - you
may have visualisations for companies, various
aspects of the business model, then down to
individual people and so on even into their
personal data and characteristics (a name, a photo,
their home address, their masters thesis, GPS data
from their car ... potentially, at least), plus a
forest of relational
connectives between all of these things; this is
potentially a huge dataset - you can't simply load
it all at startup and render it as you go in a dumb
and simplistic manner. (We will add level of detail
hierarchies and k-d trees when the time comes ...
don't worry, we know what to do!)
There is thus a 3D browser application that has
been developed separately from StockWave - the hope
is then to "drop" it into whatever datasets
StockWave generates. As a first step it could be
used to generate concept models from the news
archives, or interactive timelines or show the
webagent as it spiders between websites. It is very
general and is designed to allow practically any
dataset you want to be visualised interactively.
Getting this to work further opens up new areas for
analysis, e.g. a news concept model will help to
build (or test the feasibility of) a "predictive
news model" which we can then incorporate into the
monte carlo simulations as a new kind of "noise" source.
And still there's more - the 3D stuff now gets put
aside for a while and the StockWave project brought
up to speed; there is much in this that has not
really been tested in earnest - the datacapture and
monte carlo algorithms were re-written to spawn
more threads to use multicore systems (and to scale
up as large or as fast as your hardware will
allow) and the scripting should be a very exciting
addition; it opens up the system to the full
"algorithmic trading systems" people as well as
allowing the less tech-savvy user the ability to
"one-click" his way through many things; in
retrospect, StockWave was a tricky application to
use and could be off-putting.
1 September 2009 Processors and Benchmarks
One of the worrying things we found when creating
StockWave was the length of time it took for the
algorithms to run - at the time there were no
multi-processors on the desktop and so it was a
tough sell, i.e. the prospect of having the user
wait some hours before getting a result; havng said
that, the program had bags of other stuff which
made it useful anyway, but the centrepiece of the
analysis toolkit ... simply took too long.
Since then technology has caught up with us and our
methods mostly take well to multiple threads and
processors; once 8 or 16 cores are standard on the
desktop I think this could lead to a revolution in
online trading, vast new territories of trading
possibilities could open up to the retail investor.
(To any of you that are curious - the advanced
analyzers do NOT map onto forms which can run on
the GPU, not even with CUDA, however the GPU will
do basic random walks very quickly.)
The point I'm really getting at is - for us,
getting what you want out of StockWave depends on
what number crunching you've got available, for us,
there is never too much, and corrrespondingly you
need to be able to get the best bang for your buck
computer-wise - the very top processors often
provide only slighter better performance than their
second-bests while being ridiculously more
expensive; Intel's "Extreme" series processors come
to mind here - you really have to be a
more-money-than-sense stupid prick to buy one of
them.
Actual performance results often produce surprises
here - the number one rule about performance and
optimisation, is to actually measure it in the
first place; some people forget this (- the true
performance determinators / bottlenecks are often
quite unexpected). I have been using a benchmarking
program - free 30 day trial - called Performance
Test - it uses a large collection of real world
benchmarks and lets you access their database of
results for comparison; it gives breakdown of
benchmarks for CPU / memory / graphics / disk / CD
plus an overall figure. Note that the one most
interesting to us is CPU, especially floating point
performance.
The results are quite surprising in many ways - I
present these as a list of miscellaneous
observations without wishing to draw any premature
conclusions -
- The most basic observation is that the usual
expected variational dependencies do exist, but are
often much weaker than you might think they should
be; but also, apparently similar systems with the
same specifications - same processor, ram, graphics
card - can vary quite a bit in performance among
themselves! The practical repercussions are obvious
here, both potentially useful and worrying at the
same time - you could pay modestly for an apparent
middling system which is in reality a real flying
machine, or, pay well over the odds for a relative
dud. With a few hours tuition and watching it done
a couple of times, we could all, every one of us,
build our own PC to our precise requirements,
there's no real excuse for it except laziness - you
want it quiet, or you want it fast, or you want it
for games, or you want it just for web and email?
Build it yourself!
- Intel's chips fall into two main classes -
"Nehalem", branded as i7, and "Penryn", branded as
Core 2 Quad; there is a range of models,
differentiated by clockspeed and cache size; i920
940 965 975 and Q8300 8400 9300 9400 9450 9550
9650. I will ignore older chips and anything with
less than 4 cores.
- Dear old AMD's best new chips are only getting
to grips with Intel's older line - the Penryns, so
I wont't bother with them; like lots of people I
have a soft spot for AMD, they always gave good
value and often beat Intel in performance, but now
a gap has opened up - however, and this is very
important to remember, AMD's chips are the best
bang for the buck for almost all PC users; weirdos
like me who want to do monte carlo simulations and
manipulate large datasets are very much a minority.
- The Nehalem class beats the Penryn class by
significant margin, but there is a weak dependence
within each class - "inferior" processors can often
exceed "better" ones, or be easily overclocked to
do so.
- There is a weak dependence on clockspeed within
classes - if you think the 3.2 GHz model will just
"melt" the 2.4 GHz chip, it probably will, but not
always, and the difference may be trivial anyway;
it's a pity the faster chip cost 3 times the
cheaper one!
- The actual clockspeeds of each chip can vary
quite a bit from what is on the box - e.g. my 2.66
is only 2.48.
- Overclocking can make a huge difference - or
almost non at all; this is one aspect that you
would expect to find a nice linear relationship,
but there isn't. Note that Intel seems to put in a
lot of safety room for what it clocks it's chips at
- a 30% overclock without much tomfoolery is
relatively straighforward; Nehalems can go to 4GHz
or better, and the Penryns to about 3.2GHz.
- Higher-end Penryns can be overclocked into
low-end Nehalem country, and
- Low-end Nehalems can be overclocked to equal
their top-end rivals.
- Every processor is subtly different, a unique
individual - some just come out superstars, while
others are a bit "retarded", sorry "special", but
not in a good way.
- There is a weak dependence in going from 4G to
8G RAM; whether large amounts,16,24,48 give you
anything is unknown; what is sure is that less than
2G is going to hurt you - this is the minimum
amount for a modern system.
- There is a weak variation between Windows XP,
Vista and Windows 7 operating systems when using
the same number of bits, but
- The biggest surprise I found is that the
largest single performance increment seems to be in
moving from a 32bit operating system to a 64bit
one; the XP, Vista, 7 dependence is tiny but from
32bit to 64bit is big. Naively you can see this is
correct - you are shifting bigger chunks of data,
it is more efficient hence faster, with less
swapping to the hard drive etc.
- ASUS seem to make good motherboards - they are
well represented among the fastest systems in the
database.
- Dell systems are rather average performers -
it's easy to buy from them and their support is
great, but their PCs aren't flying machines.
- The current value-for-money chip has got to be
the i920 or Q9550, although the new i5 class may
well muddy this. So, in conclusion the "sweetspot"
would be a good Penryn (since the motherboard and
memory will be cheaper), any 64bit OS and 8G RAM,
with an ASUS motherboard; this is probably the
cheapest, really fast machine you can build. The
very fastest OS is probably 64bit XP Professional
SP3, though I don't think you can get hold of it
very easily.
BTW, if you like this sort of thing, then the
AnandTech and TechReport websites are where you
should be hanging out - for them it's a full time
job.
Taking a wider view for a moment, I must say that
for ordinary folks, performance considerations are
totally pointless, and why do some of them
overclock? - it's like boy racers modifying their
cheap, crappy cars into ugly, lower, louder, cheap
crappy cars. If all you need to do is to websurf
and email then any PC will do the job well - it's
the speed of your broadband that will be the
limiting factor. And if you are a big gamer wanting
to enjoy the latest eyecandy, then you should be
spending your money on the baddest graphics card
you can afford, with a nice large, wide monitor.
Looking forward to my next trade-up, I really like
the look of the Dell T7500 workstation with the
dual sockets, the tech geek in me drools slightly
over those xeons - 8 cores with hyperthreading ...
mmmm (sounds like homer simpson contemplating a hot
dog) ... on the other hand the cynic and the
skeptic in me, who are disgustingly, almost
unerringly correct in their every judgement and
prognostication kicks in here; aren't "Xeons" just
a marketing brand-name, a ploy to get corporate
types to pay a premium for "server/workstation
class" chips? Consider the long running outrageous
scam that operates in the graphics card market for
"certified" cards like Quadro and FireGL - isn't it
just the same? Let's check the figures on the
database - dual xeon system at 3.2GHz is around
16000 on the CPU mark, my humble Q9450 is 3500
(though some systems have been scored about 5000) -
let's be generous and say the top of the line
system will do 5 times the work of my current one,
but costs, alas about 10 times the other one! In
terms of doing StockWave number crunching, it may
reduce an analysis runtime down from half an hour
down to 5 minutes - but that could be worth quite a
bit of money as while you were waiting the extra
time for your full analysis, the prices could have
moved a lot against you. Which is all very
interesting - these "value for money" questions
have to be settled from a global view point; if
it's the difference between profit and loss, the
very fastest chip may pay for itself, easily!
8 June 2009
Numbers, even after much crunching, don't mean much
- that's why we need graphics or rather
visualisation to help us "understand"; the heatmaps
in StockWave show time dependent probability
distributions for where the share price might go, a
lot of numbers condensed to a little picture that
tells you what you need; the news-chart and event
stream shows stories alongside share price moves
and also gives an indicator of the news sentiment
at the time; these straightforward visualisations
seem to do the trick, but what happens when you've
got some arbitrary, crazy, data-set that you might
hope has some useful information buried in it -
what happens when there is no clear way to look at
this data, no obvious approach?
The new feature we are working on is a generalised
solution to the data visualisation problem - for
any old weird set of datatypes, in any number of
dimensions; to create a 3D picture which generates
itself automatically from anything you stick into
it and then lets you explore it, tune it, morph it
into something which lets you extract useful
information. Then once you have something which you
like, graft onto it the standard types of
algorithms that will work on particular aspects of
it - stick a kohonen map here, do a k-means over
there, run an SQL query on this, and tie it
together into a hidden markov model; whatever you
want, totally open.
So, obviously we have to do 3D graphics - this is
just a matter of being clear what you are doing,
refreshing your vector and matrix maths, and at all
times remembering what direction is up. Still, your
debugging sessions can be very invigorating.
Graphics cards these days are incredible beasts,
capable of incredible fill rates and polygon
counts, but all they are really doing is drawing
texture-mapped triangles - that's all you really
need to know how to do.
One has to make a choice about what to use as a
rendering library; for some time now games
programmers have been using DirectX / Direct3D
almost exclusively, but their concerns - to get the
latest hardware shader special effect working - are
not ours, which is a good thing, as I think DirectX
is based on COM (- the underlying design I mean),
which is not a good thing. The other way to go is
OpenGL, which we have some previous experience of,
so that's that then!
On the face of it, OpenGL seems like the best
choice for us - it is open, and well-supported, and
all those touchy-feely good-time things and it
seems quite straightforward if you've done a
computer graphics class. But it is very low-level,
and frankly, I hate it. What you have to remember
that it is designed as a State Machine; this
project and most large projects these days use
object-oriented design principles - it just doesn't
go together naturally, so not only do we have to
use OpenGL but we have to create an "object-ified"
wrapper for it.
Of course there are plenty of 3D engines out there
- too many, with every variety of sophistication -
but none of them seem quite like what we need, and
when you use these you often have to include
practically "the kitchen sink", and still need to
do a bit of hacking to make it fit.
State machines are annoying to use when in the
"object" mind-set as you must never forget that the
order of operations matters, and when you set a
switch it stays on until you reset it - it does not
reset once you leave the current scope; if you have
a black screen, or funny colours which change
unpredictably, or objects which translate oddly or
rotate about odd vectors, textures which disappear,
or the wrong textures on the wrong objects, then
it's all because you've probably done something and
forgotten to undo it, e.g. that classic of Opengl
newbies, forgetting to pop the matrix stack.
The documentation is not as good as you might think
- the "Red Book" (now availabe for free online)
contains lots of examples, but they contain very
-simple- examples only. There is no advice on
design, on how to use OpenGL within a larger
project - no "how to put it all together",
certainly nothing on making OpenGL into an object
design.
Obscure problems, things you would never imagine,
and answers to which can only be found on the
newsgroups - are quite common, e.g. some commands
don't work inside others, or need to be done in a
certain order (!), otherwise they don't work; this
can lead to mammoth debugging sessions with oodles
of trial and error - this kind of grind makes
programmers very depressed.
I am proposing a new marketing slogan to the Open
Standards Group people -
"OpenGL - not quite as shit as DirectX"
On a completely different matter, I seem to have
lost access to the company email accounts - Doh! -
I will set-up a gmail or yahoo alternative as soon
as I can, not that there's much to really talk
about in the immediate short term; scripting and
Python is in, which I think will really open up the
system for people, but there is loads of testing to
be done, as soon as this 3D stuff is ready.
Update on the hardware problems - if you have had
problems with the firmware bug on your Seagate
Barracudas, Seagate will fix them free of charge;
they won't exactly track you down and do it for
you, but if you logon to their i365 site and check
the serial nos, you can open up a recovery case and
they will fix the drives and ship them back to you
in about a week. For free!
22 March 2009
I have just noticed something interesting about the
news event analysis tools - they seem to work
better than expected! In the documentation there is
some description of them, plus the usual caveats
about it being an experimental feature, well, some
way down the road, and with quite a lot of news
stories in the database, I have been creating news
event streams, i.e. "market sentiment" indicators
for various stocks. For the FTSE 100 it seems to
give a good leading - yes, a leading - indicator
for trend direction; it runs about a month ahead,
but note that it is changeable and is prone to
whipsawing around, neither is the news model in any
way predictive (- more on that later). Still,
that's not bad for a "side dish".
I'm going to try and push on with the "super web
agent" next - this is a web agent which can, while
it is spidering, suck out all the goodness - names,
people, companies, organisations, events, states,
and show it graphically. This will have a lot of
practical use I think - being a great user of the
web it is very frustrating just how much crap there
is to dig through and how low the signal to noise
ratio is.
The fundamentals database can go on the back-burner
for now; fundamentals are fictions anyway as the
latest round of swindling has shown - all these
companies going down the stank had their accounts
signed-off true and proper, and their debt all
rated AAA; data as corrupted as this cannot be
data-mined for anything worthwhile - it's like
building a house on quicksand. Companies go to
great lengths to create useful fictions about
themselves - so it must be the side channels we
need to examine to get our "information leakage"
and any signs of anomalous behaviour - which is
better suited to the web agent.
17 March 2009
At the risk of this becoming a blog, a literary
form I detest (- if you can actually write, then go
write a bloody book) I need to vent and blow-off a
little. Firstly, some product reviews -
Fancy new PC has had two Seagate Barracuda 7200.11
drives fail on me - one after 1000 hours the other
after 700; given a claimed MTBF of 750000 hours
this is "impossible". Dell support is not up to
much, "next business day" ends up as "one week
later"; talking to them is like conversing with
polite but rather slow 12 year olds (- who can be
scary smart at times!); you need to tell them 3
times to do something, then they phone you up to
confirm it; not too quick at answering emails, and
not too good at giving direct answers. While
waiting, I just ordered a new 1T Hitachi drive from
Misco, got it next day, and was back up and running
in 4 hours. How hard is that?
The hard drive failures made me look into disk
cloning solutions; Acronis True Image, which is
incredibly highly rated over the web-o-sphere (- by
the company's shills?) is the most useless backup
and cloning software there is. Don't take it even
if offered it for free. (I returned mine for a
refund.)
Opentick has gone and done what I guessed they
might do. Shame I've now got a ton of their API
calls in my codebase, but who knows, maybe one day
they'll come back. Taking a wider view, looking
back, it simply didn't make sense - how could they
make money on this, if it was not for some other
reason? The reason seems to be that the whole
enterprise was a large-scale beta test to iron out
the bugs in their real product, Xasax. Question:
will all that dormant Opentick code be able to
interface with Xasax?
How StockWave will support datafeeds in the future
is thus - not at all; really I just can't be
bothered. OK, actually we will, but here's how -
you get the standalone feed reader application from
your supplier, then you phone their helpdesk and
ask how to export data in CSV format, if they get
suspicious, tell them you want to put it in an
Excel spreadsheet; they send you a script which can
do this for you. Now you email us, and we show you
how to create a "FilePriceSource" - this will read
in the exported CSV text files on a periodic basis.
Voila! And no messing around with their APIS, DLLs
and signing NDAs. Our hands are kept clean, and a
hundred hours or so of developer time is saved for
something worthwhile.
My "can't be bothered" remarks point to something
of a deeper malaise; fact is folks, programming is
... shit. Yes, honestly, it is. Not a good career,
a refuge for second rate intellects who think they
are the cream of the crop; a lot of social inepts
and aspberger freaks. It is only bearable, if you
are working on your own project, which has some
potentially deep or important innovation. Software
such as this cannot be effectively project managed
- it is similar to blue-sky research, and has
payoffs, if any, which are highly nonlinear and
unpredictable. It's a tough gig just to keep going.
There is a highly rated programming blog which is
obsessed with the idea of "rock star programmers" -
this is a risible concept; if you are working in a
corporate environment, for someone else, then, by
definition, you cannot be a "rock star programmer".
You are a maintenance programmer with a
nice-sounding job title. No genuine "rock star
programmer" would ever call himself such, wish to
be, or seek to be known as such; they are also
pretty-much un-hireable as they simply will not do
other people's tedious shit.
(I am not a rock-star programmer, I am an
independent scientist who does a bit of "karoake"
programming on the side; programming is not
creation, it is implementation, something the hired
help does. And when the hired help is not
available, you have to roll your own sleeves up.)
Getting the latest StockWave out has been hard as
there is a lot of "tedious shit" to do - no big
issues, no design innovations, no novel algorithms,
just fiddly crap. And herein lies the real problem
- we have another project on the go, it is very
open-ended and wide-ranging (- it will feed back
into StockWave eventually) in the general area of
information processing, and equally importantly,
visualisation; a lot of effort has been spent on
this, for no visible benefit as yet. But the other
project is a lot more interesting at the moment as
StockWave is quite mature.
Must stop this, starting to moan like my old Aunt
Nelly.
5 March 2009
Normally I don't bother much with what other people
are doing; let them - so what if they say this, or
claim that, but recently I've been looking at a
number of websites for trading software companies
for comparison as our site needs a bit of a revamp;
it's too dark and forbidding. Obviously these guys
are all furiously hype-ing their wares, as is their
right, but some of the claims they make are really
annoying, as the gap between their claims and the
reality is often a mile wide. Naming no names, to
protect the guilty.
Practically everyone claims to offer
"advanced analytics" - what does this actually
mean? For example -
- Do you have monte carlo simulation?
- Do you have machine learning algorithms?
- Do you use signal processing?
- Do you have news event analysis?
- Do you have specialised web agents to seach the
web for you?
No, none of the above - what you really get is a
few dozen technical indicators drawn on your chart;
this is not "advanced analytics" - this is
technicolour voodoo for the self-brainwashed
hopeless optimist.
"Datamining" is another popular buzzword they like
to use -
- Do you get advanced visualisations of high
dimensional datasets?
- Do you have pattern extraction techniques to
identify signals of good and bad companies?
No, they have a very basic scanner which can pick
out certain stock criterion like market
capitalisation, PE ratios and volatility; this is
not datamining it's just an arbitrary
pre-selection.
Everybody is into "advanced trading sysems" as
well, but - is this a customised probability model
which can search for specialised and novel trading
opportuntities?
Again, no - it's a formula editor with a backtest
facility; this is not an "advanced trading system"
- it's simplistic guesswork and a "wet finger in
the air".
An "easy-to-use programming interface" - well, you
just have to have one of those now, don't you!
So, does this API use one of the most popular and
powerful, loosely-typed scripting languages, which
most people can get up to speed with in under a
week, and is freely available with many supporting
libraries?
Er, yet again no, not really - it is instead our
own custom language which we have invented all by
ourselves just the way we like it - with a mass of
idiosycrasies and unfamilair syntax, all inspired
by the company founder's notions of just what good
coding style really is (- because he was a
real cool cobol coder-dude or was a visual basic
"guru" in his pomp); also comes with a boatload of
bugs and library interdependencies to test you
with, and all guaranteeing in the end, that no one
but us will ever use it - that way we've got you
forever!
Somebody should get these guys under the Trade
Descriptions Act.
not-really-sure-what-day-it-is, January 2009
Happy New Year to everyone! And Merry Christmas
before that, as well; our turkey was moist, and our
kapitan, our glorious leader, well-pleased with his
gift, a pair of Oakley sunglasses to hide his eyes
while he tortures us to higher productivity at
Stalag StockWave. (- that's enough of that!
Mocking sarcasm, 5 seconds into the New Year.
Hmm.)
What's been going on, in no particular order, or
even logical structure -
Teletext is I think by now mostly kaput everywhere,
goodbye little friend! The code to use it is still
in there, so it will still work if the service is
available and you have WinTV/VTPlus installed, but
please don't send us emails asking why things have
stopped working - it's the governments doing, not
ours. Write to your MP - try sending him anonymous
threats - it's always worked for us!
Have added a treeview component to the main form as
well as the usual scrollable grid; this took a
little bit of time to get working, and it does look
OK, but it does make the user interface busier, and
on reflection, it adds absolutely nothing
that the user really needs; make a mental note here
- do not add pointless eye-candy which just "looks
kewl" without a good functional reason; remember
this the next time someone wants to be able to dock
all the application sub-forms into the main form (-
what a bloody mess that would be! If you're a
"docking junkie" then try Eclipse.)
Being megalomaniacs at heart we'd like
everyone to use our program - and for
everyone to use "StockWave" as a proxy phrase for
"trading software" just like the way, back in the
day, "Hoover" was synonymous with "vacuum cleaner";
to have a chance at this you have to cater for the
needs of folks in "other countries" of course, now
while China and Japan would be too much of a
stretch, we did try to get India and Australia
covered; not sure it actually worked though,
probably down to glitches in timezone handling code
and rules for daylight saving time. Now, you might
think we're idiots for having trouble here - surely
you just read it from the system clock, and take a
bit away or add a bit on, well, to use a
wonderfully crazy phrase I heard only recently :
"in theory, theory and practice are the same, but
in practice, they are different". Anyone actually
living out there, genuinely trying to use
StockWave, give us some feedback please. Obviously
there a lot of potential users out there for us -
intelligent, well-educated, English-speaking folks,
... and Australians too, and we would like to reach
them.
Finally, moving to proper source control, which
should allow us to be a bit faster and more
aggressive in our work. Note to young developers
out there - winzip with a timestamp is not source
control.
An improved form for displaying option chains has
been added - it can also download these from Yahoo,
furthermore the option prices can be shown on the
stockchart as well which is nice. We have also
created an on-chart visualisation of level 2 data -
this lets you see individual orders and fills
"dance around" in realtime, alas we don't have a
level 2 feed to test it with! (God, we're so
stingy ...)
WTF is happening with opentick, we ask ourselves?
When opentick started to offer free non-delayed
streaming data we thought "Wow, let's support that
- just what the world is waiting for"; downloading
their libraries and API we were further thrilled to
see that the programming interface is a thing of
clarity and simplicity, which is not a small issue;
other providers' APIs typically induce mental
torment, they also charge you an arm-and-a-leg, and
also make you sign ridiculous NDAs to stop you
"stealing their trade secrets" (- like what?! -
reading data through a socket in a thread ... no
one's ever seen that before - better patent it
quick!); generally they act like they're doing you
a huge favour. Supporting lots of feed providers
means writing the "same" code over and over again
each time dealng with the often bizarre and
peculiar idiosyncrasies of their APIs, then keeping
up-to-date with their frequent and often arbitrary
changes. This leads to major code bloat, is a
maintenance nightmare and leads to significant
customer problems as the public can get
-justifiably- very irate if their feed goes
haywire, which is always seen as the application
developers fault and not the feed provider - since
"it worked OK on my other software"! Then they log
on to some traders website and trash you
mercilessly. But then Opentick came along - seeming
like a breath of fresh air, a clean design and free
apart from exchange fees (- a few bucks!) - it
seemed like a new dawn.
Anway, we quickly knocked up a standalone grid-view
based downloader, and it worked really well, we
even put flashing red and green backgrounds for
price direction when updating (- woo hoo!); this
was quite groovy in itself, but it never got put in
to StockWave down to what I would call "niggles" -
little bugs we couldn't get to the bottom of;
rather than get sidetracked by a side-issue like
this we got onto other more important things (-
sometimes in these situations, it resolves itself,
or the solution suddenly becomes obvious). This was
well over a year ago; we've only recently
integrated the opentick downloader into StockWave,
only to find it doesn't work at all now - we logon,
get a few messages, but don't receive any ticks,
then the connection breaks after a while (- yes,
the test account we are using is still active ...
thanks for the suggestion!) Have contacted them
about this, but there is no apparent solution.
Opentick are owned by a company called Xasax, their
prime interest is in really high quality, high
frequency fault resistant datafeeds and order
routing (- impressive technology using custom
hardware and specialised chips, i.e. FPGAs) - it's
designed for the next frontier, the remaining
undiscovered country of trading - high-frequency
algorithmic trading; you are reading data at the
sub-millisecond resolution, doing some analytics,
then maybe making dozens or hundreds of trading
decisions on this per second; this is way beyond
any human decision making. There are quite a few
companies investing in very expensive,
high-bandwidth "plumbing" hoping to give themselves
an edge; at very high frequencies you are hoping
that there are numerous inefficiencies which you
can exploit with some relatively straightforward
algorithms, but if you can also gain an edge by
getting your price quotes a fraction of a second
sooner than the next guy, or can find a price
improvement at execution time ... that also helps.
A lot.
Xasax product does not seem to be available yet,
but don't hold your breath little "retail investor"
because it will be stratospherically beyond your
price range; opentick stopped taking on customers a
while back, and so it might all have been just a
hobby for them - a kind of "big beta" experiment to
act as a testbed for the "real deal" which people
are going to be paying for; presumably once the
big-paying real customers arrive, opentick might be
allowed to wither on the vine - which would be a
real shame. Let's hope this is not the case.
A few words about this high frequency business, and
why the retail investor shouldn't worry about it (-
too much ...) - high frequency is not and will
probably never be available to the ordinary
investor - this is big boys' toys for big boys'
games; it's just an arms race in trading technology
infrastructure, which will probably end up in a
stalemate. Market regulators ultimately must ensure
that the playing field is (somewhat) level. Note
also that at such high frequencies, you can't do
much in the way of analysis, certainly not the
deeply multi-faceted monte-carlo based processing
that StockWave does, it's simple stuff running
really really fast fast FAST. I would rather make
fewer trading decisions based on a deeper analysis
than lots of fast ones based on some simplistic
scalping technique which to work needs to have a
speed advantage over everyone else. And for the
trading to become fully automated - I just don't
like, the human being should never be taken out of
the decision making loop; human beings have "common
sense" and intuition, they also are implicitly
aware of context, and are good "bullshit
detectors". Proponents of high frequency would
argue that at short enough timescales things will
become simpler and that the "fusionist" philosophy
of StockWave is not required.
Multiple core processors are now fully supported in
StockWave - we always had extensive use of threads,
but now we can spawn even more "worker threads"
depending on how many cores there are available -
this allows us to take full advantage of the
hardware. We've got a new machine to test this, a
Dell XPS430 - a very sweet machine, at a good
price; it's fast, very fast, even though it's not
even the latest Core i7 processor, but "only" a
Core2 Quad Q9450; a full StockWave build takes only
a few minutes. Other observations of potential
interest; Windows Vista (Home Premium) seems, ugh
... "quite good", I hate to say (- I'd have had XP
installed by default had I the option), still it
would be madness to install it as an upgrade on an
old PC; if you are a trader than 23" or better
widescreen single monitor set-up should be good
enough for you - multiple monitor configurations
will go the way of the dodo. Anyway, the main thing
is that our monte carlo goes like the clappers -
it's great to see the algorithm start, ramp up the
cores to maximum, then crash back to idle.
Python and scripting - getting there, but there's
something I need to do with a library called SWIG.
There loads of other stuff still to be tested and
integrated.
Have also been thinking about adding more content
to the website and help files - the glossary in
particular has grown into a collection of
mini-essays, originally it was conceived as an
antidote to the usual "explain it but tell you
nothing" guides you often come across, hopefully
delivering its insights in the guise of an acerbic
one-liner, but some things need a bit more
explaining, and some things shouldn't really be
joked about, even if your humour is blacker than
the inside of Satan's coal scuttle (- which ours
is). And since everything affects everything else,
we need to look at wider issues - the danger here
is descending into "punditry", i.e. lazy,
opinionated, self-serving wind-baggery (- think of
all these experts currently being wheeled out to
explain the complex details and reasons behind the
ongoing market turmoils - a year ago most of them
were probably saying the "fundamentals of the world
economy are sound", and we should all just buy,
buy, buy. Arseholes.)
Despite being locked in a lonely garret we are in
conversation with users from all over the world; we
get some truly interesting emails from some very
intelligent folks and also some hilarious bollocks
from real crackpots - both types of conversation
are worthy of a wider audience, so they might get
put on the website.
1 June 2008 Oh dear me, is that the time ... ??
What have we been doing!
Apologies for the current tardiness with regards to
our email correspondence - work comes first but
we're trying to respond to the backlog. Please be
patient.
This is just an update for all our avid fans out
there especially the Singaporean who looked at
almost every single page on our site the other day
- "big-up" to you (- just a wee hint though : the
freebie download has all the documentation already
with it!)
Interesting times re: the (various) market turmoils
- we hope that no one got badly burned out there,
still has a house, and money for the meter; of
course we would, rather arrogantly, hope that
having read some of our screeds you may have been a
little forewarned, and therefore cautious in your
dealings - the markets are intrinsically unstable
and will fall apart, irregularly, every ten years
or so, but only the big guys get the bailouts, so
"Little Man" be-ware; and note also that this
situation will continue indefinitely, or until some
form of socialism is adopted in the major western
economies, staring with the USA ...
(fits of giggles, rolling around laughing,
biting the carpet, tears rolling down cheeks)
- pardon us, but we do like our little jokes; the
facts are we're all stuck with capitalism until we
die ... that's just the way it is (- of course, all
of these corporate and banking bailouts are a kind
of "welfare-state socialism for the rich", but
that's not what we're really talking about.)
So where are we, exactly?
Technically, we've been laying the foundations of
"future plans", basically it's coming together, and
crucially, everything planned is do-able - and will
be done, eventually. Specifically -
- Opentick support will be in - but there no
plans to support any other feeds; we became
physically ill looking at the API documentation for
some of the most popular feed providers, and our
teams of lawyers fell asleep while reading their
various NDAs.
- Scripting with Python. Yup.
- Moving the underlying data management to a
database, probably PostgreSQL, but that will be
left open, architecturally speaking.
- 3D visualisation stuff ... "getting there"
- in particular, the confusing mess that was the
TradeCreator form has been cleaned up, becoming
"almost usable"; in a way, this was the crux of the
application - to find custom, specifically-crafted,
probabilistically-optimised trading opportunities -
the fact it was so hard to use was a little
embarrassing.
Commercially, we are going for a change of tack;
getting rid of "basic edition" and "advanced
edition" - never liked all of that nonsense, same
goes for version numbers, 1.2,1.42,1.7833537.839393
build 28782738237 - oh dear lord almighty ... give
it a rest!
There is only one product, it is called StockWave -
- it is not free anymore, but
- it has a generous trial period, and
- it is very inexpensive if you want to buy it
There will be an "uber-StockWave", not generally
available to the public, which will be our own
research and trading/testing platform; it will
contain things which are unproven, untested,
hard-to-use, or computationally so demanding as to
be outside the utility of the general public; we
might sell you this version if you really want it,
but it will be expensive, and we won't be
supporting it; leading edge, or bleeding edge, it's
experts-only with fat wallets.
Note that StockWave will be almost exactly-the-same
as "uber-StockWave" except for the things which you
probably can't use effectively anway, i.e. - it
will not be "hobbled" to the extent that Basic
Edition was compared with Advanced Edition.
There is still a planned "big test" to do - the
good news is we don't need investors anymore as
now, e.g. an 8-core machine can do an advanced
analyzer in about 10 minutes! We thought we needed
£50K+ and 4 months, but we can probably get
by with £10K and 6 months; it has also been
brought to our attention that "cloud computing" is
now available with the (back-of-an-envelope
calculation) capability to do an advanced study in
2 minutes at a cost of 17p! Which is something to
think about ...
Website will be updated - the lack of proper online
shop is, really, a bit of an embarrassment, and the
use of mail order makes us seem like a bunch of
scammers (- BTW your legal consumer protections are
probably at least as strong with mail order as with
online credit card payments, at least in the UK.)
(hi ho hi ho ...)
20 January 2008 Garbage In, Garbage Out; one of the
good features of StockWave is the free high
frequency historical price data - to get this we
have to aggregate it ourselves from multiple
sources, alas, it has been pointed out of late that
the data is getting a bit ropey due to mismatches
among the end of day and intra-day stuff - this is
due to stock splits, reverse stock splits, various
adjustments, bad reads, etc and so on. Look at ANTO
and BARC for some chart-silliness.
There are tools within the program to maintain data
integrity, and we could tell you how to go about
fixing things when you come across a crazy chart
but rather than bore or alienate our users ...
we're going to just fix it ourselves.
To get the best available data when you open a
chart go to the first drop down menu and 'download
intra-day prices from web'; this will be the best
available but if it still looks crazy then email
us.
Since we are going through this lot by hand it
might take while - 1.5 was to have the nifty and
opentick feed support plus updated stocks lists -
but we shall have to fix this first. When we're
done we'll release a 1.42 version in the interim.
10 January 2008 Analogue signal turn-off; the UK is
turning off the 'normal' TV signal fairly soon,
which means that it will no longer be possible to
get share prices (cheaply!) from teletext, thus
this feature will be removed. We could investigate
the possibility of looking at the data services
provided on digital TV, but we aren't going to
bother, as it seems almost all users are relying on
Yahoo for streaming prices and our own backfill
service. Ho hum - it was a nice, cheap, facility
while it lasted. If anyone thinks getting data from
DigiText is a good idea we might reconsider, but in
the meantime, we'll be working on other stuff.
1 January 2008 Investment Prospectus
StockWave Software Ltd invites approaches from
investors and potential business partners.
StockWave Software is a small software company
based in the west of Scotland; it is employee-owned
with no debt; it has developed a highly innovative
product for stock market data analysis using a
synthesis of advanced scientific techniques. (The
designer of StockWave holds advanced degrees from
respected universities and has extensive work
experience in high-tech engineering and software
development.)
Our original intention was to provide scientific
tools for the mass market of private investors but
up against a mature market dominated by data-feed
providers' packages and brokerage platforms, our
success has been limited; a further problem for us
is that our target customer market, the retail
investor, is wedded to "Classical Technical
Analysis" which he seems loathe to give up.
Commercial realities are that marketing to the
masses is prohibitively expensive requiring a
dedicated full-time sales team plus support staff,
trainers, materials and advertising in print and
web; although our - relatively feature-poor,
freeware Basic Edition has found approximately
10000 users, this has not proved to be a bridge to
Advanced Edition (- our full featured product) -
the transfer rate is too small to be viable.
A different approach is therefore needed as alas it
seems we cannot reach the "man in the street" (-
please note that this "different approach" will not
involve altering our product to meet the whims of
dyed-in-the-wool "technicians") - we intend now to
market StockWave as a niche software product to
various financial experts and professionals such as
quants, analysts, fund managers and professional
traders. To be convincing to such skeptical and
highly critical persons, we need to do a
large-scale, statistically significant test; 4-6
months to map the sweet-spot of our advanced
algorithms, and another 4-6 months to trade a
typical portfolio.
Investment is thus required to purchase the
necessary hardware to conduct our tests.
What we offer - appropriately, for what is, after
all, a derivatives analysis package - is a Call
Option, giving the holder the right to buy a large
equity stake in StockWave Software Ltd for a very
low price at a future date. This call option will
be 'European-style' with an expiry date one month
after the test results have been presented; the
option holder will get a first, and exclusive, look
at these results.
The premium paid for the call option will be used
to purchase hardware; since we have a rough idea of
what we need, and the target timescale for doing
this study, we do have a ballpark figure in mind
for the sum required - prospective investors should
be able to work this out themselves, but we shall
not mention any figure explicitly. We should point
out that the figure required will be a modest
amount to anyone we would see as a viable investor.
Any early bid which significantly exceeds our
reserve will be accepted as we want to get started
as soon as possible - otherwise there will be an
auction.
Further Notes
StockWave may contain patentable technologies,
although we have made no study of this, we are
confident that much of what has been done is
innovative and unique.
For the above reason we cannot go into too much
detail in explaining our ideas to prospective
investors - we mention (some of) the ingredients,
but not the recipe itself; having said that, our
documentation and help files already contain as
much information as we are going to provide to
anyone - we would therefore encourage any potential
investor to seek independent technical experts who
can make some reasonable critical assessment of our
approach.
The true value of our software may not be to
private investors with their relatively limited
computational resources but to, e.g. hedge funds or
other large organisations interested in high
frequency trading, statistical or event based
arbitrage; StockWave is already multi-threaded and
is scalable.
Advanced Edition may be removed from sale for the
duration of the testing period, if so requested.
31 December 2007 This fruits of this year's
technical development and new features will be
drip-fed into StockWave over the coming months on a
bi-monthly cycle going up through versions 1.2,
1.3, and 1.4. A 2.0 release will be made 6 months
later. Advanced users will be afforded free uprades
through 1.2 to 1.4; 2.0 will be a new product
requiring a separate license, although existing
users will have 50% discount.
19 December 2007 Merry Christmas and Happy New
Year! We're shutting down over the holiday period,
so see you in 2008.
1 December 2007 We found some old demo videos lying
around - the quality is not good, and they are
rather large, but they could be of help to anyone
who is having trouble and is too shy to send us an
email. To download these to your hard drive, do a
right-click and then "save link as".
Advanced Filtering
Random Walk
Chart Analysis
News on Chart
Old style technical analysis. Bleh.
Main Window
Market Summary
Option Prices
Portfolio
Stock Chart
System Manager
Using the built-in Help.
29 October 2007 Basic Edition 1.41 released - this
is a minor bugfix version.
28 September 2007 6000 users of Basic Edition.
27 August 2007 We've fallen a bit behind on our
intended release schedule for Basic Edition 1.5 and
Advanced Edition 1.2 - this is because everyone is
just back from/on/just about to go on, holiday and
those that are here want to work on the groovy
stuff outlined in the Future Plans section of the
site. We shall be cracking-the-whip soon though.
Monday 16 April Get the updated datasources from
here;
unzip the file to your install directory, your old
datasources will be over-written. Any problems,
send us an email.
Friday 13 April 2007 Black Friday Fun! Yahoo seem
to have changed their time formats, outputting
everything in Eastern US time; fix applied locally
- updated datasource files will be available for
download on Monday; backfill will be available as
usual. This brings us neatly onto discussion of
Basic Edition 1.5 - we have decided to support the
OpenTick
datafeed; if you don't know what this is, it is a
FREE (- apart from exchange fees which are nominal
for non-professionals) realtime, non-delayed tick
datasource, including such excellent things as
option chains and Level 2 data. We also intend to
add the "Nifty" stocks from the Indian stock
market. Advanced Edition is also undergoing a mild
upgrade to version 1.2. Release should be around
the end of May for both versions.
12 March 2007 Changes to daylight saving time in
the US have caught us out!
10 March 2007 3000 users of Basic Edition; keep the
feedback coming.
26 February 2007 Basic Edition 1.4 released;
includes stocks for Indian and Australian stock
exchanges; major new feature is the Option Strategy
Browser.
1 January 2007 Spammers have started sending out
emails using our domain; please ignore any
unsolicited emails - they are not from us; we do
not market ourselves in this way, and in particular
we do not tout stock market tips.
15 November 2006 Preparing Basic Edition 1.4; we
have had requests for support of Indian Stocks, so
we shall we be offering Bombay Stock Exchange BSE
and Nifty stocks in our next release, as well as
updated stocks lists for all other groups, which we
admit are a little out-of-date by now. 1.4 should
be ready mid-December.
1 November 2006 Basic Edition 1.3 Minor bugfix
release. We now have over 1300 users of our Basic
Edition software - thank-you all, and please do not
hesitate to offer feedback!
21 October 2006 Basic Edition 1.3 in preparation.
Would the people who have been trying to download
the advanced edition by pointing their browsers at
http://www.stockwavesoftware.co.uk/download/install_stockwave_advanced_edition.exe
and variations thereof, please note that the
advanced edition is available by MAIL ORDER ONLY;
it was hoped that online purchase via credit card
would be available by now, but there appears to be
a real problem with the "website guy" - it could be
an extended holiday, a nervous breakdown, or a
terminal case of asshole-itis ... no one knows -
we seek him here, we seek him there ...
25 September 2006 Upgraded Basic Edition released -
now at version 1.2; this is a collection of minor
bugfixes and enhancements; some controls have been
moved for better accessibility.
19 August 2006 New Contact page added, as the old
one was a bit too complicated and er, ... rather
crap.
18 August 2006 Installer glitches found in Basic
Edition 1.1 - many apologies folks; now fixed, but
if you are having any trouble at all, please email
me direct at help@stockwavesoftware.co.uk
5 August 2006 Upgraded Basic Edition released -
this is now version 1.1; new features are free
streaming prices from the internet for UK, US and
EU (- NB this is 15 minute delayed), and a
'Classical Technical Analysis' package for those of
you who like that sort of thing. Windows 95/98/Me
are no longer supported - the program should still
work, but we are no longer specifically supporting
these OSes.
21 July 2006 Minor visual alignment problems found
on Basic Edition when using high screen resolutions
(1280 by 1024 and above); fixed.
19 June 2006 Installer program can now handle
non-standard Windows partitions - for those of you
who have windows installed on your "O:" drive.
6 May 2006 Basic Edition installer re-instated;
please contact if you have any problems with the
installation process - please note that under
Windows XP you may need to have
Administrator-level privileges enabled.
30 April 2006 Added web-update for users of Basic
Edition (- i.e. 'backfill' capability); this means
you can now download FREE historical intra-day
price data (15 minute resolution) for use in your
analyses. This data will be updated weekly.
29 April 2006 Bug reported in Basic Edition
installer; removed temporarily.
14 April 2006 Online, at last!
|
