Logical Datasets, Not Raw Tables: The Catalog That Decouples Strategies From Storage

Crypto market data is a special kind of chaotic. One table calls the timestamp timestamp, another calls it ts. OHLCV stores symbols as the unified BTC/USDT:USDT; the funding table right next to it stores BTCUSDT. Venues fund on different schedules. The "symbol" column means a different thing depending on which table you're in. None of this is anyone's fault — it's what you get when data accretes from a dozen exchange APIs over years.

The wrong response is to make every strategy deal with it. The moment a strategy hardcodes SELECT timestamp, symbol, close FROM t1_ccxt_prices_ohlcv WHERE broker_name = ..., you've welded your research to a physical schema. Rename a column, migrate a table, add a venue, and you're editing strategies. That doesn't scale past the first inconvenience.

The dataset catalog is the layer that absorbs the mess. A strategy asks for a logical dataset — "ohlcv", "funding" — and the catalog knows how to read it from a physical table. Strategies never see a raw table name. Ever.

A logical name maps to a physical reality

The core abstraction is a DatasetSpec: how to read one logical dataset from one physical table.

@dataclass(frozen=True)
class DatasetSpec:
    """How to read one logical dataset from a physical table."""
    name: str                    # logical name a strategy references ("ohlcv")
    kind: Kind                   # 'bars' | 'funding' | 'series' | 'spec' | 'events'
    physical_table: str          # the actual t1 table
    column_map: Dict[str, str]   # logical key -> physical column
    interval_seconds: Optional[int] = None
    symbol_namespace: str = "native"
    freshness_ttl_seconds: Optional[int] = None  # point-in-time as-of tolerance
    knowledge_time_col: Optional[str] = None      # transaction-time column for PIT travel
    align: str = "asof_ffill"
    filters: Dict[str, str] = field(default_factory=dict)

The column_map is the heart of it. A strategy's engine asks for the close price by the logical key "c"; the catalog translates that to whatever the physical column is actually called:

DatasetSpec(
    name="ohlcv", kind="bars", physical_table="t1_ccxt_prices_ohlcv",
    column_map={
        "ts": "timestamp", "symbol": "symbol", "venue": "broker_name",
        "o": "open", "h": "high", "l": "low", "c": "close", "v": "volume",
        "interval_col": "interval", ...
    },
)

Change where the data lives and you change one catalog row. Every strategy keeps working, because every strategy only ever spoke the logical language.

It encodes the gotchas so strategies don't have to

The catalog isn't just a rename table — it's where the venue-specific landmines are defused, once, with the knowledge written down. The seed catalog's own comments capture exactly the kind of trap that costs an unwary analyst a day:

# Key gotcha: `symbol` means DIFFERENT things per table —
#   * ohlcv `symbol` = UNIFIED  (BTC/USDT:USDT)  -> namespace ccxt_unified
#   * funding/open_interest `symbol` = NATIVE (BTCUSDT) -> namespace native
# Both are the indexed + currently-updated columns; `input_symbol` is the other form.

That symbol_namespace field means the engine can join a unified-symbol bars table against a native-symbol funding table correctly, automatically, because the catalog records which is which. A strategy author asking for BTC funding never learns that the symbol formats differ — the catalog reconciled it.

The filters field handles another classic: one physical table that mixes multiple venues. Pin a venue with an equality predicate and you get a narrower logical dataset out of the same table:

# extra equality predicates so one mixed table backs narrower datasets, e.g.
# {"broker_name": "KUCOINFUTURES"} to pin a single venue's funding cadence.

So "Kucoin funding" and "Binance funding" can be two clean logical datasets backed by one messy shared table — each with its own correct funding cadence — without a single strategy knowing the table is shared.

Point-in-time semantics live here, too

Crucially, the catalog is also where freshness and knowledge-time controls are declared per dataset — freshness_ttl_seconds, knowledge_time_col, align. This is what lets the engine's as-of joins (the no-look-ahead machinery) bind the right TTL to each dataset automatically. The data's temporal behavior is part of its definition, not something each strategy re-derives. A dataset isn't just "where the numbers are" — it's "where the numbers are, how to align them in time, and how stale is too stale."

Seed catalog plus a database table

The catalog merges two sources, with a sane precedence: a built-in seed catalog describing known tables, plus a fund.datasets table loaded at runtime, with the database winning:

"""Two sources, merged (DB wins):
  1. A built-in seed catalog (SEED_DATASETS) describing known t1 tables, so the
     data layer + engine run against t1 immediately, before fund.datasets exists.
  2. The fund.datasets table (when present), loaded via load_db_datasets."""

That ordering is the operationally smart choice. The platform works out of the box against known tables with zero configuration — the seed catalog ships the knowledge. But anything you register in the database overrides it, so onboarding a new dataset is a row insert, never a deploy. New CSV feed, new SQL table, new API-sourced series? One catalog row and every strategy can ask for it by name.

Why decoupling is the architecture that lets you grow

A research platform's long-term velocity is decided by one question: what does it cost to add a data source? If the answer is "edit the engine and every strategy that uses it," you've capped your growth at how fast your engineers can refactor. If the answer is "add a catalog row," you've decoupled data growth from code change entirely.

That's the whole point of the catalog. Strategies depend on stable logical names. Storage evolves underneath them. New data flows in through configuration, not code. The mess of crypto data stays contained in one well-defined layer whose job is precisely to know the mess so nothing above it has to.

Clean strategies, messy reality, one translation layer between them. That's how a research platform scales past its first dataset without scaling its headaches with it.

---

Register any table, CSV, or feed as a logical dataset and reference it from any strategy by name. Explore the data catalog →