Building a Compliant Asset Data Layer from the Ground Up

When we started building the OpenAssets data model in late 2024, we made a decision that shaped everything else: we would design for audit and compliance first, and figure out operational ergonomics second. That is the opposite of how most data systems in financial services get built. Most start with operational requirements, get something working that the team can use, and then retrofit compliance properties onto the operational layer when an examination or an audit surfaces the need.

A year in, we can say which decisions aged well and which ones we revised at cost. This post is a candid account of what the first year of building an RWA compliance data layer actually looked like, written for teams who are considering building something similar or evaluating whether a commercial layer might save them from repeating the expensive mistakes we made.

What We Got Right from the Start

The event sourcing decision was the most consequential right call we made early. We modeled every ownership change as an event, not as a state update. Every position in the system is the result of replaying its event history in order. This means we can reconstruct the state of any position at any point in time by replaying events up to that moment. It also means that the audit trail is structural rather than additive: there is no way for a user to change the current state of a position without the change appearing as an event in the immutable log.

Regulators and auditors ask questions that assume event sourcing: "show me who held this position on November 15, 2025." If your system stores current state rather than event history, answering that question requires either having maintained a separate historical state table (usually incomplete) or admitting you cannot reconstruct past state accurately. We avoided that problem by deciding upfront that the event log was the source of truth.

The second early decision that held up was separating instrument reference data from position data at the schema level. Instrument reference data (what an asset is, its identifiers, its jurisdictional classification, its corporate action history) is maintained in a separate store with its own versioning and update workflow. Position records reference instrument IDs rather than embedding instrument properties. This means that when an instrument's properties change due to a corporate action or a reclassification, we update the reference data once and the change is reflected in all position records without a migration.

The Identifier Mapping Problem That Cost Us Six Weeks

We underestimated the identifier mapping problem. We designed the system assuming that ISIN would be the canonical identifier for the majority of instruments we would ingest, with fallback handling for other identifier types. What we found in practice was that the mix of identifier types across custodian feeds was more variable than expected, and that our fallback handling was not robust enough.

The specific failure mode: a custodian feed would deliver a position record using an internal reference code that the feed documentation stated was equivalent to a CUSIP. The equivalence mapping was sometimes stale. A new instrument would arrive in the feed using an identifier that had not yet been added to our reference database. The record would fail to map, trigger an exception, and sit in the exception queue until someone manually researched the identifier and added the mapping.

We rebuilt the identifier mapping layer twice before arriving at an architecture we were satisfied with. The current version maintains a priority hierarchy for identifier resolution: ISIN preferred, CUSIP as first fallback, LEI for issuer-level disambiguation, internal codes last. For records that cannot be resolved through the hierarchy, a separate enrichment pipeline tries to match against the instrument reference database using available metadata fields (issuer name, asset class, maturity date for fixed income) before creating an exception. The exception rate dropped from roughly 4 percent of ingested records to under 0.3 percent after the second rebuild.

The Compliance Record Immutability Problem We Solved Wrong the First Time

We knew from the start that compliance records needed to be immutable. What we got wrong was the implementation boundary. The first version made the event log immutable but allowed mutable operational views derived from the event log. An analyst could resolve a break by marking it closed in the operational interface, and that resolution was an event in the log. But the exception record itself, in the operational interface, was mutable. An analyst could re-open it, change the resolution code, re-close it, and the final state in the operational interface did not reflect the intermediate states.

The regulatory standard is not that the final state of an exception is correct. It is that the complete history of actions taken on an exception is preserved. We caught this gap during an internal audit simulation before we went to external clients. Fixing it required adding a separate compliance event layer for exception lifecycle events, separate from the position event log but using the same immutability guarantees. The compliance event log captures every state transition on every exception, by which user, with what timestamp and resolution code.

The practical lesson: immutability at the source record level is necessary but not sufficient. The complete process trail, including exception handling, manual overrides, and correction workflows, needs to be in the immutable log. Anything that is only in the operational state layer will eventually have its history lost when someone updates the state.

Retention Management Was an Afterthought Until It Was Not

We built the initial version with retention management as a future feature. The data retention requirements under SEC Rule 17a-4 and CFTC Regulation 1.31 were documented in a policy document but not encoded in the data system. In practice, this meant that retention periods were tracked manually, deletion of records at retention expiry was a manual process, and the distinction between records in the two-year accessible period and records in the four-year subsequent period was maintained in a spreadsheet.

When we onboarded the first early-access clients and began holding their ownership records, we recognized that the manual retention process was not scalable and was creating audit exposure. We spent about three weeks building retention metadata into the record schema and implementing automated retention enforcement. Every record now carries a retention class (determined by the regulatory regime applicable to that record type), a creation timestamp, and a retention expiry timestamp. The system enforces deletion restrictions and generates deletion records when retention expires.

The lesson is not that retention management requires extensive engineering. It does not. The lesson is that building it in after the fact requires a schema migration and a backfill process for existing records, both of which take time and introduce risk. Building it in from the start would have cost a few days of additional design work and saved three weeks of remediation. If you are building a compliance data layer, retention policy encoding should be in the initial schema design, not a future sprint.

What We Would Do Differently

The identifier mapping and retention management mistakes were expensive because we deferred them. Both required rework at a point where the system was already handling real data, which meant the fixes had to be coordinated with data migrations and client communication.

The one design decision we debated and have not needed to reverse is the event sourcing architecture. It adds query complexity (deriving current state requires replaying history or maintaining a materialized current state view) and storage cost (events accumulate). Both costs are real. But the compliance properties it enables, complete auditability, point-in-time reconstruction, structural immutability, are foundational to the regulatory environment we are operating in. We would not redesign it.

What we would tell teams starting this work: the compliance requirements are not details to be addressed later. They are constraints that shape the architecture from day one. Build the immutable log first. Design the identifier scheme before integrating the first feed. Encode retention policy in the schema. The investment in those decisions at the start is small relative to the cost of remediation later, and the compliance risk in the interim is real.