To many, the term “Big Data” is hard to define. Is it just more data in a standard relational database or is it something else?
The next question usually is… How can it help me?
Let me start by comparing it to the standard relational database approach. In this architecture source files are imported through an ETL (extract/transform/load) process into a data warehouse where the information is then accessible through various applications.
For an investment management firm, a simple example would be client account data being combined with holdings and transaction data and then exported to a monthly statement.
Big Data is something else and does not replace the standard relational database model.
Big Data streamlines the process and removes the need for the ETL process and speeds the loading of data to end applications by splitting and then re-combining data when requested by the end applications. This simplistic explanation may not be clear, but what you need to know is that it is fast and can quickly combine both structure and unstructured data.
Let’s get to the more interesting topic… How it can help
Investment Managers that use trading models know that their value comes from three main areas: data, how it is being analyzed, and the speed at which it is processed. The flexibility of these models is also critical and it is always better to reduce development time (and cost).
All of these make trading models a perfect fit for Big Data solutions.
Solutions (meaning, the front and back-end infrastructure) that can analyze both structured and unstructured data and be integrated to automatically execute and report (ex. pre- and post-trade risk engines and management and client reporting tools, etc…) are immensely valuable to the investment management industry.
A system that can quickly analyze large volumes of both historical and real-time data enables analysts and traders to find market inefficiencies and ultimately alpha for their clients.
Complex trading strategies can be tested and revised at significantly less cost than previously possible giving investment managers an edge over retail and institutional investors that have not invested in the tools and back-end infrastructure required to be competitive in the current market.
Investment management firms all value security expertise and many are understanding the value of developers and quants (data scientists), but just as important is the technology infrastructure.
Big Data solutions are relatively new and exciting technology. I urge firms to assess their technology to gain (or maintain) a competitive advantage.