The small in Big Data

Waltzing along the Thames from the Tower Bridge to Westminster, in amongst the plethora of skyscrapers struggling to reach the gods and tourists snapping photos, there lies the White Tower, its stone weathered by a thousand years of rain and wind, battering the British landscape in all seasons. The history of a nation lies there carved into the stone of Caen.

Generations have passed. Ages have passed since its construction. This island nation fell at Hastings, a nation forced to kneel at the sword of William the Conqueror from across the Channel, the man who built the White Tower. In the royal court, the foreign sound of French replaced that of English, the language of the Anglo-Saxons. As this green and pleasant land beckoned before him, the question for William was simple. What was this land which I have conquered?

William sent out his men throughout the land, to seek answers (and taxes), by conducting a survey of the nation’s wealth, recording the holdings of landowners, in nearly 13,500 places across the kingdom. The result was the Domesday Book which was completed in 1086, a truly epic work for its time. Indeed, for medieval times, the amount of data collected was truly astonished. Perhaps this was an example of medieval Big Data?

Leaping across a millennium to today, the term Big Data is as ubiquitous as it appears to be misunderstood. The term has seemingly captured an almost ethereal quality. Despite, the regularity with which the term appears, it seems to be rarely defined in the popular press. Essentially, Big Data refers to massive data sets.

The sheer quantity of data makes it computationally very difficult to analyse. Yet, beneath this veneer of complexity, the supposed promise of Big Data is that we can find simple and wonderfully intuitive results and relationship between the data, which can be visualised in novel ways. Big Data is only useful if we can make it “small” data that we can interpret. The web has given rise to masses of Big Data. Simply think of Google and Facebook and the reams of data which their servers trawl through every second. As oil was the way to profit from the twentieth century, is data the basis of alchemy in the twenty-first century?

Clearly, for the aforementioned institutions, data has proved to be valuable (we, the consumers, have freely given it to them in our droves). However, does simply throwing more data at a problem help create solutions? The difficulty is that more data can often mean more noise.

A model with more variables does not mean a better model, in the same way that having more lights on a motorcycle might not improve it (see photograph above). Financial markets are plagued by noise. Every minute newswires buzz with more stories, some crucial for markets, whilst others can be discarded as noise. Human traders have (always) used news to trade markets and have continually needed to make these decisions. We can apply a similar approach to trade markets by examining large amounts of news data (maybe we should call this Big News).

Whilst Big Data is not a panacea, through diligence it can improve our understanding of financial markets. In a sense it is like baking a cookie. We can see Big Data as a set of complicated ingredients, which only taste good once baked. Indeed, I have written about this topic in an earlier blog article, where I demonstrated how Big Data can be used directly for trading. I showed how RavenPack news data can be used to create trading filters for reducing the drawdowns associated with carry trades in the currency markets. The method I employed relied upon relatively straightforward concepts, notably understanding how news volume is related to market volatility and also the impact of the labour market on risk sentiment. Whatever approach we choose to Big Data, a modicum of old fashioned trading intuition needs to be there to start us on our way.

The key is to filter the signal from the noise, as Nate Silver might say, so we can find the small in Big Data.

My book Trading Thalesians - What the ancient world can teach us about trading today is out in late October on Palgrave Macmillan, also has some colour on lateral thinking to a trading idea and much more (mixed in with a bit of ancient history). You can pre-order the book on Amazon.