generate_cleaned_vault_datasets

Documentation for eth_defi.research.wrangle_vault_prices.generate_cleaned_vault_datasets function.

generate_cleaned_vault_datasets(vault_db_path=PosixPath('/home/runner/.tradingstrategy/vaults/vault-metadata-db.pickle'), price_df_path=PosixPath('/home/runner/.tradingstrategy/vaults/vault-prices-1h.parquet'), cleaned_price_df_path=PosixPath('/home/runner/.tradingstrategy/vaults/cleaned-vault-prices-1h.parquet'), settlement_db_path=None, logger=<built-in function print>, warning_logger=None, display=<function display>, diagnose_vault_id=None, vault_db=None, *, denomination_families=None, daily_materialisation=False, daily_price_df_path=None, crypto_min_tvl_usd=None, raw_vault_specs=None)

Clean raw scanned vault prices and atomically write an analysis dataset.

  • Reads vault-prices-1h.parquet and generates a cleaned Parquet dataset

  • Calculates returns and related fields included with price data

  • Cleans abnormal returns

  • Stamp the cleaned Parquet with the current Docker metadata.version provenance, matching vault scanner JSON exports

Note

Defaults to stablecoin-only selection for compatibility. The isolated crypto bundle passes explicitly reviewed stablecoin, ETH and BTC families.

Parameters

  • vault_db_path (pathlib.Path) – Vault metadata pickle path.

  • price_df_path (pathlib.Path) – Raw scanner Parquet source.

  • cleaned_price_df_path (pathlib.Path) – Atomic cleaned Parquet destination.

  • settlement_db_path (Optional[pathlib.Path]) – Optional settlement database used to annotate cleaned observations.

  • logger (Callable[[str], None]) – Progress callback accepting one message.

  • warning_logger (Optional[Callable[[str], None]]) – Optional warning callback for recoverable sidecar failures. Defaults to logger for notebook callers without structured log levels.

  • display (Callable[[pandas.DataFrame], None]) – Optional DataFrame display callback for diagnostics.

  • diagnose_vault_id (Optional[str]) – Optional canonical vault identifier to show between cleaning stages.

  • vault_db (Optional[eth_defi.vault.vaultdb.VaultDatabase]) – Optional preloaded vault metadata database. Supplying it avoids a second pickle read when callers derive raw_vault_specs from the same metadata.

  • denomination_families (Optional[collections.abc.Iterable[eth_defi.vault.denomination.DenominationFamily]]) – Denomination families to retain; defaults to stablecoins.

  • daily_materialisation (bool) – Whether the primary output itself should contain one row per UTC day.

  • daily_price_df_path (Optional[pathlib.Path]) – Optional atomic daily sidecar destination. When supplied, the sidecar is materialised from the settlement-annotated hourly frame before the frame is released, allowing the crypto bundle to avoid rereading the full stablecoin Parquet.

  • crypto_min_tvl_usd (Optional[decimal.Decimal]) – Optional USD TVL threshold for crypto-denominated vaults.

  • raw_vault_specs (Optional[collections.abc.Collection[eth_defi.vault.base.VaultSpec]]) – Optional source vault identities to push down to the Parquet reader. Raw Parquet stores these as chain and address rather than the derived id column. The crypto-only cleaner uses this to avoid materialising unrelated raw histories before denomination selection. A targeted clean may produce a typed zero-row output. When omitted, a zero-row result is rejected to protect the last valid public output.

Returns

None. The verified output is atomically installed at cleaned_price_df_path.

Return type

None

Baseline (2026-09-22 production log): the default stablecoin cleaner processed 22,552,978 raw rows to 10,116,623 cleaned rows in about 11m25s. The production-shaped rerun processed the same row counts in 146.02s (4.7x), with 20.4 GiB peak process RSS. The measured input fingerprint was the 333,430,642-byte, 38-column raw Parquet copy from 2026-09-22; output had 49 columns and 10,116,623 rows. This timer covers metadata filtering, all Python transforms, settlement annotation, Arrow conversion, write, verification and the optional sidecar when requested.