process_raw_vault_scan_data
Documentation for eth_defi.research.wrangle_vault_prices.process_raw_vault_scan_data function.
- process_raw_vault_scan_data(rows, prices_df, logger=<built-in function print>, display=<function <lambda>>, diagnose_vault_id=None, *, denomination_families=None, crypto_min_tvl_usd=None)
Preprocess vault data for further analysis.
Assign unique names to vaults
Add denormalised vault data to prices DataFrame
Filter out non-stablecoin vaults
Calculate returns, rolling metrics
- Parameters
rows (Union[dict[eth_defi.vault.base.VaultSpec, eth_defi.vault.vaultdb.VaultRow], eth_defi.vault.vaultdb.VaultDatabase]) – Metadata rows from the vault database, or a loaded database instance.
prices_df (pandas.DataFrame) – Raw price observations containing at least chain, address, timestamp, share-price, total-assets and total-supply columns.
logger (Callable[[str], None]) – Notebook, console or structured-log callback accepting one message.
display (Callable[[pandas.DataFrame], None]) – Optional DataFrame display callback used for one-vault diagnostics.
diagnose_vault_id (Optional[str]) – Optional canonical vault identifier to show between cleaning stages.
denomination_families (Optional[collections.abc.Iterable[eth_defi.vault.denomination.DenominationFamily]]) – Denomination families to retain; defaults to stablecoins.
crypto_min_tvl_usd (Optional[decimal.Decimal]) – Optional USD TVL threshold converted into ETH/BTC denomination units.
- Returns
Cleaned, denormalised price observations ordered by vault and time.
- Return type
Baseline (2026-09-22 production run): stablecoin cleaning processed 22,552,978 raw rows and took about 11 minutes 25 seconds. The first optimisation pass filters before wide enrichment, reuses integer group codes and replaces the inactive-lead and outlier callbacks with positional array masks. The production-shaped rerun processed 22,552,978 raw rows, selected 10,338,606 stablecoin rows and produced 10,116,623 cleaned rows in 146.02 seconds, versus 11m25s in the source log (4.7x). Peak process RSS was 20.4 GiB versus no RSS measurement in the original log. The stage breakdown is emitted by this function’s timer logs; the exact input was the 333,430,642-byte, 38-column Parquet copy from 2026-09-22.