Designing a Data Layer for Real-World Assets: Five Decisions That Compound

Designing a data layer for real-world assets is not primarily a technology problem. The technology choices matter, but they are downstream of a set of data modeling decisions that most teams make without fully understanding the long-term consequences. By the time those consequences are apparent, the system has grown around the early decisions in ways that make them expensive to revisit.

We have spent the past year building the OpenAssets data model from the ground up, working with a set of specific regulatory and operational requirements as the design constraints. These are the five decisions we think matter most, and what gets expensive when they are made incorrectly.

Decision One: Event Sourcing vs. State Storage

The most consequential early decision is whether to store the current state of ownership records or the events that produced them. Most teams default to state storage because it is simpler to build and easier to query. You have a table that says Account A owns X units of Asset Y as of today. Simple, fast, familiar.

The problem emerges during regulatory review or audits. When an examiner asks you to show the ownership history of a specific position, including every transfer, every adjustment, and every reconciliation action, state storage cannot give you that answer. You have the current state. You may have some log of changes. But reconstructing a complete, ordered, timestamped history of every event affecting a specific position is not something a state-storage system was designed to do.

An event-sourced model stores each ownership event as an immutable record. The current state is derived from the event history by replaying the events in order. This is more complex to implement and requires more storage, but it gives you the audit trail that regulators expect and the ability to reconstruct the state of any position at any point in time. For compliance infrastructure, this is not a nice-to-have. It is foundational.

Decision Two: Canonical Identifiers from Day One

Most data layers start with the identifiers that are most convenient: whatever the first custodian feed uses. That might be ISINs, or CUSIPs, or internal codes the portfolio management system assigned. Then a second custodian feed arrives, using different identifiers for the same instruments. And a third, using different ones again.

If the canonical identifier was not defined before the first feed was integrated, each subsequent integration creates a translation problem. The system accumulates a collection of mappings, aliases, and exception handling logic that grows with every new instrument and every new data source. What starts as a technical inconvenience becomes a genuine compliance risk when the system cannot reliably identify whether position A in feed one and position B in feed two refer to the same underlying asset.

The right approach is to define a canonical instrument identification scheme before integrating any feeds, and to enforce it at the ingest boundary. Every record that enters the system must carry a canonical identifier. If the source record uses a different identifier type, the translation happens at ingest and the source identifier is preserved as metadata, not as the primary key.

Decision Three: Data Lineage Is Not an Afterthought

Data lineage means being able to trace any derived value back to the source records that produced it. For a reconciled ownership record, lineage means: which custodian feed did this data come from, what was the raw record, what transformations were applied, and what was the basis for any adjustments.

Teams that defer lineage to a later phase of development typically find that adding it retroactively requires rearchitecting core parts of the system. Lineage requires that every transformation carry forward a reference to its inputs. If the transformation pipeline was built to optimize for query performance, every optimization that drops intermediate results also drops lineage information.

In practice, lineage is one of the two or three things that audit examiners check most carefully for compliance infrastructure. The standard question is: show me where this number came from. If the answer requires a developer to manually trace through pipeline logic, the system does not have real lineage. It has a system that happens to produce numbers that look right, which is a different thing entirely.

Decision Four: Separation of Operations and Compliance Records

Operational systems are optimized for the operations team. They show current positions, highlight exceptions, and support the workflow of reconciling and resolving breaks. Compliance records need to be something different: immutable logs of what happened, when, and who acted on it.

When these two functions live in the same data layer without architectural separation, compliance records become mutable. Operators resolving breaks modify the same records that should be preserved as evidence. A position that was wrong and then corrected should show both states: the original incorrect record, the correction event, and the resulting corrected record. A system that simply overwrites the original gives you the right answer at end of day and an incomplete audit trail for the next examination.

The separation does not require two completely separate systems. It requires that the data model distinguish between operational state, which can be updated, and compliance records, which cannot. Every update to an operational record creates a new compliance record rather than modifying an existing one. This is a design constraint that needs to be established early because the difference between a mutable and an immutable record design runs through the entire write path of the system.

Decision Five: Retention Policies in the Schema, Not the Policy Document

Regulatory retention requirements for ownership records vary by record type and by regulatory regime. SEC Rule 17a-4 requires that certain electronic records be retained for six years and that the first two years of retention be in an accessible location. CFTC Regulation 1.31 has parallel requirements for derivatives records. GLBA requires retention of customer financial records for defined periods.

The common failure is to document these retention requirements in a compliance policy but not to encode them in the data system. When a retention period ends, the deletion requires someone to identify the records in scope and take action. That works at small scale with a small number of record types. It becomes unreliable at scale across multiple asset classes, each with its own regulatory retention clock.

The more maintainable approach is to encode retention policy metadata at the record level, at creation time. Each record carries a retention class that maps to a minimum retention period. The system enforces deletion restrictions until the period expires and generates a deletion record when retention ends. This transforms retention compliance from a periodic manual process into an automated operational function.

Why These Five Are the Ones That Compound

These five decisions share a common characteristic: they interact. Event sourcing gives you the raw material for lineage, but only if your canonical identifier scheme makes records unambiguously traceable. Your separation of operational and compliance records is only meaningful if the compliance records are truly immutable, which requires the write architecture to enforce immutability rather than relying on operational discipline. Your retention policy encoding is only reliable if the record types are well-defined, which comes back to the canonical identifier scheme.

Teams that make these decisions thoughtfully early in the build cycle tend to find that the system scales in a reasonably predictable way. Teams that make them ad hoc find that what looked like a simple tactical choice in month two becomes a structural constraint in month eighteen that requires a migration to fix. We made some of these mistakes ourselves in the first iteration of the data model. The ones we got right early cost nothing. The ones we got wrong cost considerably more to correct than they would have cost to design correctly from the start.

This is not a theoretical concern about architectural purity. These are the specific decisions that determine whether your ownership data layer will hold up to a regulatory examination two years from now, when the data volume has grown and the examination team is asking questions you cannot anticipate today.