Technological changes within the financial markets often come in cycles, with trends rapidly grabbing headlines before attention shifts to the next big thing.
In reality, markets are always evolving, and it is not necessarily the large paradigm shifts that generate big wins for both large and small entities. Instead, it is often the incremental changes that deliver for organisations.
A clear trend is the move away from delayed, or even end-of-day, data towards real-time streaming data, as the UK and European regulators look to introduce consolidated tapes across multiple Fixed Income asset classes over the next few years.
This development presents firms with multiple decisions, not least of which is whether they need real-time data; that is, do the necessary changes and hurdles justify the speed increase?
Once a decision to transition to a lower latency solution has been made, considerations must turn to the practical aspects. In a traditional polling-based model, applications typically run on a fixed schedule, querying trading venues and APAs to collect data and populate downstream databases or data warehouses. While this approach inevitably introduces latency between market events and data availability, it allows simplistic operation through the use of workflow management and scheduling tools, such as Apache Airflow.
To take advantage of streaming delivery models, such as those expected from the UK and EU, along with FINRA’s TRACE real-time feed, firms will need to adopt different architectural approaches and technologies. In particular, this points towards greater use of event-driven logic, leveraging messaging queue technologies.
Event-driven architecture enables systems to respond to market events as they occur, rather than periodically checking for updates, which inevitably introduces artificial latency. While this can lead to efficiency gains, many underestimate the organisational and cultural shift required to operate in this mode.
Events must be processed sequentially, rather than opportunistically when an orchestrator (a workflow management tool) executes, which is a universal challenge when switching to this model. This can expose operational weaknesses quickly, including unclear data ownership, poorly defined Service Level Agreements (SLAs), and brittle downstream consumer infrastructure.
Monitoring the size of a database table at a given point in time is relatively straightforward; however, it is significantly harder to observe and manage rates of change with event-based processing. Components must therefore either be natively scalable, with appropriate buffering strategies in place to ensure no data is lost or processed out of sequence when resource constraints are reached.
Historically, there has been a reluctance within the industry to outsource data cleansing and quality checking. However, with increasing complexity, many firms are turning to external specialists to address these challenges. With a polling-based (pull) approach, data quality issues can often be handled during batch processing. However, streaming (pull) architecture forces real-time processing and decisions, including whether to block, divert, or publish potentially erroneous data.
Without clear policies and automated controls, firms risk either propagating poor-quality data faster or introducing instability through overly aggressive validation. While the validation of a single trade can be applied easily, the nature of event-driven systems makes complex validations more difficult to implement.
The downstream destinations for these data streams also require careful consideration as the traditional, on-premises relational database (which has always been well suited to this workload, handling high volumes of incremental row inserts efficiently) may no longer be the most appropriate choice.
Adoption of cloud data warehouses, which excel at large-scale analytical queries, is increasing. This can, however, require architecture changes, as they often depend on separate ingestion pipelines given the typical batch-oriented focus. An emerging alternative for those seeking scalability and portability is a lakehouse architecture, using technologies such as Apache Iceberg. This is one area where specialist aggregation service providers can offer efficient and cost-effective solutions that single firms simply cannot achieve alone.
Overall, the changes driven by the switch towards streaming Fixed Income post-trade data markets mirror, in many respects, the earlier electronification seen in equities and FX. As with those transitions, the technology itself is not the limiting factor.
In conclusion, in order to differentiate and achieve a competitive edge, industry players will need to rapidly adapt their operating models, governance, and engineering practices to support always-on data flows. There is a significant opportunity for data teams to build scalable, event-driven systems on modern data stacks, unlocking greater value from market data. Alongside these benefits, however, come new difficulties with regard to scaling, monitoring, and maintaining data quality.
For many financial institutions, there are additional obstacles aside from just technology costs, time, and resource constraints. The move towards streaming data, including real-time, reflects a broader structural evolution in how institutions organise, operate and compete.
Download The Financial Technologist magazine here.
.png)
.png)
.png)