research.wrangle_vault_prices

Documentation for eth_defi.research.wrangle_vault_prices Python module.

Clean vault price data.

  • Denormalise data to a single DataFrame

  • Remove abnormalities in the price data

  • Reduce data by removing hourly changes that are below our epsilon threshold

  • Generate returns data

The input is the raw scanner parquet conforming to RawVaultPriceRow. The output is a cleaned DataFrame conforming to CleanedVaultPriceRow, consumed by calculate_lifetime_metrics().

Module Attributes

HYPERCORE_ZERO_NAV_EPSILON

NAV at or below this value counts as a complete Hypercore wipe-out.

HYPERCORE_PNL_NAV_LAG_ABSOLUTE_TOLERANCE

Maximum absolute mismatch when recognising a delayed PnL/NAV API update.

HYPERCORE_PNL_NAV_LAG_RELATIVE_TOLERANCE

Maximum relative mismatch when recognising a delayed PnL/NAV API update.

MIN_HYPERCORE_PNL_NAV_LAG_CHECKPOINTS

PnL-first, NAV-confirming API lag needs a baseline and two later checkpoints.

MIN_HYPERCORE_RECAPITALISATION_ASSETS

New capital must reach this NAV before a recapitalised vault is tracked again.

PRIORITY_SORT_IDS

For manual debugging, we process these vaults first

VAULT_STATE_COLUMNS

Vault state and pass-through columns added by the historical scanner.

Functions

add_denormalised_vault_data(rows, prices_df)

Add denormalised data to the prices DataFrame.

approximate_hypercore_share_prices_from_pnl_nav(...)

Build an approximate Hypercore economic-performance index from PnL and NAV.

assign_unique_names(rows, prices_df[, ...])

Ensure all vaults have unique human-readable name.

calculate_vault_returns(prices_df[, logger])

Calculate returns for each vault.

check_missing_metadata(rows, price_ids, ...)

Check that we have metadata for all vaults in the prices DataFrame.

clean_by_tvl(rows, prices_df[, logger, ...])

TVL-based threshold filtering of returns.

clean_returns(rows, prices_df[, logger, ...])

Clean returns data by removing rows with NaN or infinite values.

derive_deposit_closed_reason(prices_df)

Derive unified deposit_closed_reason from protocol-specific columns.

discard_hypercore_pre_recapitalisation_history(...)

Start a recapitalised Hypercore vault at its new meaningful capital base.

ensure_vault_state_columns(prices_df)

Ensure vault state columns are present in the DataFrame.

filter_unneeded_row(prices_df[, logger, epsilon])

Dedpulicate data rows with epsilon.

filter_vaults_by_stablecoin(rows, prices_df)

Reduce vaults to stablecoin vaults only.

fix_outlier_share_prices(prices_df[, ...])

Fix out rows with share price that is too high.

forward_fill_vault(vault_df)

Forward fill missing vault prices up to max_gap_hours.

generate_cleaned_vault_datasets([...])

A command line script entry point to take raw scanned vault price data and clean it up to a format that can be analysed.

get_vaults_by_id(rows)

Build a dictionary of vaults by their chain-address id.

process_raw_vault_scan_data(rows, prices_df)

Preprocess vault data for further analysis.

remove_inactive_lead_time(prices_df[, logger])

Remove initial inactive period from each vault's price history.

replace_cleaned_vault_histories(vault_ids, *)

Rebuild and atomically replace cleaned histories for selected vaults.

sort_and_index_vault_prices(prices_df, ...)

Set up the order of vaults for processing.

Classes

CleanedVaultPriceRow

Schema for a single row in the cleaned vault price DataFrame.
HYPERCORE_ZERO_NAV_EPSILON = 1e-06

NAV at or below this value counts as a complete Hypercore wipe-out.

HYPERCORE_ECONOMIC_CHECKPOINT_INTERVAL = Timedelta('0 days 04:00:00')

Fixed UTC interval used to select publishable Hypercore economic checkpoints.

MAX_HYPERCORE_PNL_NAV_CONFIRMATION_DELAY = Timedelta('1 days 02:00:00')

Maximum elapsed time accepted around a delayed Hypercore NAV confirmation.

HYPERCORE_PNL_NAV_LAG_ABSOLUTE_TOLERANCE = 1.0

Maximum absolute mismatch when recognising a delayed PnL/NAV API update.

HYPERCORE_PNL_NAV_LAG_RELATIVE_TOLERANCE = 0.001

Maximum relative mismatch when recognising a delayed PnL/NAV API update.

MIN_HYPERCORE_PNL_NAV_LAG_CHECKPOINTS = 3

PnL-first, NAV-confirming API lag needs a baseline and two later checkpoints.

MIN_HYPERCORE_RECAPITALISATION_ASSETS = 1000.0

New capital must reach this NAV before a recapitalised vault is tracked again.

MIN_HYPERCORE_RECAPITALISATION_RECOVERY_DELAY = Timedelta('7 days 00:00:00')

Ignore isolated zero-NAV observations that recover before this delay.

class CleanedVaultPriceRow

Bases: TypedDict

Schema for a single row in the cleaned vault price DataFrame.

This is the enriched format produced by the cleaning pipeline in this module and consumed by calculate_lifetime_metrics().

It extends RawVaultPriceRow with denormalised metadata columns (id, name, event_count, protocol) and computed columns (returns_1h). The DataFrame uses a DatetimeIndex built from the timestamp column.

Columns are grouped by availability:

  • General columns are present for all vault protocols.

  • ERC-4626 only columns come from on-chain ERC-4626 calls and are NaN / empty for native protocols.

  • Lending only columns are populated for lending protocol vaults (IPOR, Euler, Morpho, Gearbox, etc.) and NaN for others.

  • Hypercore only columns come from the Hyperliquid native vault API and are NaN for all other protocols.

  • Native protocol flow columns are populated for native protocols that provide daily deposit/withdrawal data (Hypercore, GRVT, Lighter, Hibachi) and NaN for ERC-4626 vaults.

chain: int

EVM chain id (e.g. 1 for Ethereum, 8453 for Base).

Native (non-EVM) protocols use synthetic in-house chain ids:

The full mapping lives in CHAIN_NAMES.

General — present for all protocols.

address: str

Vault contract address, lowercase.

Address formats vary by protocol:

  • EVM vaults: 0x-prefixed hex (e.g. "0xabcd...")

  • Hypercore: 0x-prefixed hex (Hyperliquid vault addresses)

  • GRVT: platform-specific id (e.g. "vlt:xxx")

  • Lighter: synthetic id (e.g. "lighter-pool-281474976710654")

  • Hibachi: synthetic id (e.g. "hibachi-vault-2")

See is_good_multichain_address() for the validation function that accepts all these formats.

General — present for all protocols.

block_number: int

Block number of the on-chain read. For native protocols without blocks this is a synthetic sequence number.

General — present for all protocols.

timestamp: pandas.Timestamp

Naive UTC timestamp (also used as the DatetimeIndex).

General — present for all protocols.

share_price: float

Share price in denomination token units.

For ERC-4626 vaults this is read directly from the contract (convertToAssets(1e decimals)). GRVT, Lighter, and Hibachi provide native share prices from their respective APIs. Hypercore (native Hyperliquid vaults) does not expose a historical share price or supply. Its cleaned value is a PnL/NAV economic-performance index starting at 1.0 for each retained capital epoch. The scanner’s synthetic input remains available in raw_share_price.

General — present for all protocols.

total_assets: float

Total assets under management (TVL) in denomination token units.

General — present for all protocols.

total_supply: float

Total supply of vault share tokens. Hypercore has no exposed historical token supply; its value is synthetic index units calculated as total_assets / share_price and must not be interpreted as an on-chain share count.

General — present for all protocols.

performance_fee: float

Performance fee at time of read (e.g. 0.20 = 20%). NaN if unknown.

General — present for all protocols.

management_fee: float

Management fee at time of read (e.g. 0.02 = 2%). NaN if unknown.

General — present for all protocols.

errors: str

Comma-separated RPC error messages, or empty string if no errors.

Example values: "total_supply call failed", "total_assets zero: 0", "total_supply call missing". Always empty for native protocols.

General — present for all protocols (always empty for native protocols).

vault_poll_frequency: str

Dynamic poll frequency used when taking this sample. Empty string if not set.

Example values: "1h", "4h", "24h". The scanner adjusts frequency based on vault TVL and activity; low-TVL vaults may be polled less frequently.

General — present for all protocols (may be empty for native protocols).

id: str

Vault identifier string: "<chain_id>-<address>".

General — present for all protocols.

name: str

Human-readable vault name (unique within the dataset).

General — present for all protocols.

event_count: int

Total deposit + redeem events observed for this vault.

Zero if the protocol does not support on-chain deposit/redeem event tracking (e.g. native vaults like GRVT, Lighter, Hibachi).

General — present for all protocols.

protocol: str

Protocol name (e.g. "Morpho", "Yearn", "Hyperliquid").

General — present for all protocols.

returns_1h: float

Hourly return as pct_change() of share_price within each vault group. Despite the name, for native protocols (Hypercore, GRVT, Lighter) the interval may be daily or irregular — the column name is kept for backward compatibility.

General — present for all protocols.

max_deposit: float

Maximum deposit amount allowed (ERC-4626 maxDeposit). NaN if unknown.

ERC-4626 only — NaN for native protocols.

max_redeem: float

Maximum redeem amount allowed (ERC-4626 maxRedeem). NaN if unknown.

ERC-4626 only — NaN for native protocols.

deposits_open: str

Whether deposits were open: "true", "false", or "".

ERC-4626 only — empty for native protocols.

redemption_open: str

Whether redemptions were open: "true", "false", or "".

ERC-4626 only — empty for native protocols.

trading: str

Whether the vault was actively trading: "true", "false", or "". Currently only supported for D2 Finance vaults.

Protocol-specific — empty for most protocols.

available_liquidity: float

Available liquidity for immediate withdrawal in denomination token units. NaN if not applicable.

Lending only — IPOR, Euler, Morpho, Gearbox, etc. NaN for other protocols.

utilisation: float

Utilisation ratio (0.0–1.0) for lending vaults. NaN if not applicable.

Warning

This metric measures capital deployment efficiency (how much of the vault’s AUM is lent out), not redeemable liquidity. For single-market vaults (Euler EVK, Gearbox, Silo) high utilisation does mean low available liquidity. For multi-market aggregators (Morpho, Euler Earn, IPOR) a vault can show 95% utilisation yet have substantial instantly redeemable liquidity in low-utilisation underlying markets. See README-vault-redeemable.md and README-utilisation.md in eth_defi.erc_4626.vault_protocol for details.

Lending only — IPOR, Euler, Morpho, Gearbox, etc. NaN for other protocols.

deposit_closed_reason: str

Unified reason why deposits are closed (e.g. "Vault deposits disabled"). Empty string if deposits are open. Derived from deposits_open for ERC-4626 vaults or set directly by native protocol exporters.

General — present for all protocols (empty when deposits are open).

written_at: pandas.Timestamp

When this price row was actually written/fetched (naive UTC). NaT for old data.

General — present for all protocols.

vault_settlement_at: pandas.Timestamp

Latest asynchronous vault settlement timestamp in the interval ending at this price row.

General — populated by merging vault-settlements.duckdb after cleaning. NaT means no known settlement occurred since the previous cleaned price row.

leader_fraction: float

Fraction of vault assets controlled by the leader (0.0–1.0).

Hypercore only — NaN for all other protocols.

leader_commission: float

Commission rate charged by the vault leader (0.0–1.0).

Hypercore only — NaN for all other protocols.

follower_count: float

Number of followers in the vault.

Hypercore only — NaN for all other protocols.

account_pnl: float

Cumulative PnL of the vault leader account in USD.

Hypercore only — NaN for all other protocols.

cumulative_volume: float

Cumulative trading volume of the vault in USD.

Hypercore only — NaN for all other protocols.

hypercore_source: str

Hypercore scanner source: "daily" or "hf".

Used during wrangling to reconcile overlapping synthetic share prices. Hypercore only — NaN for all other protocols.

epoch_reset: bool

The row starts a new performance epoch after a complete wipe-out.

Hypercore only — false for ordinary observations.

hypercore_repair_status: str

Provenance of the cleaned Hypercore PnL/NAV approximation.

approximated_pnl_nav marks a four-hour economic checkpoint, approximated_pnl_nav_clipped a positive checkpoint capped at 100%, approximated_pnl_nav_lag_repaired a gain whose NAV confirmation arrived within the bounded confirmation window, approximated_pnl_nav_wipe_out a terminal NAV-corroborated loss, and approximated_pnl_nav_carried a row that adds no performance, either because it is not the selected checkpoint for its four-hour UTC bucket, because its PnL awaits NAV confirmation, or because the epoch is already at zero after a terminal loss. A later NAV confirmation can revise the recent provisional PnL-only checkpoint and subsequently compounded prices. deferred_pnl_nav means inputs were missing and deferred_pnl_nav_outlier means an uncorroborated negative PnL step was not allowed to zero a funded vault. Hypercore only — empty for ordinary observations and other protocols.

daily_deposit_count: float

Number of deposit events in the latest day.

Native protocol flow — Hypercore, GRVT, Lighter, Hibachi. NaN for ERC-4626 vaults.

daily_withdrawal_count: float

Number of withdrawal events in the latest day.

Native protocol flow — Hypercore, GRVT, Lighter, Hibachi. NaN for ERC-4626 vaults.

daily_deposit_usd: float

Total USD deposited in the latest day.

Native protocol flow — Hypercore, GRVT, Lighter, Hibachi. NaN for ERC-4626 vaults.

daily_withdrawal_usd: float

Total USD withdrawn in the latest day.

Native protocol flow — Hypercore, GRVT, Lighter, Hibachi. NaN for ERC-4626 vaults.

__init__(*args, **kwargs)
__new__(**kwargs)
clear()

Remove all items from the dict.

copy()

Return a shallow copy of the dict.

fromkeys(value=None, /)

Create a new dictionary with keys from iterable and values set to value.

get(key, default=None, /)

Return the value for key if key is in the dictionary, else default.

items()

Return a set-like object providing a view on the dict’s items.

keys()

Return a set-like object providing a view on the dict’s keys.

pop(k[, d]) → v, remove specified key and return the corresponding value.

If the key is not found, return the default if given; otherwise, raise a KeyError.

popitem()

Remove and return a (key, value) pair as a 2-tuple.

Pairs are returned in LIFO (last-in, first-out) order. Raises KeyError if the dict is empty.

setdefault(key, default=None, /)

Insert key with a value of default if key is not in the dictionary.

Return the value for key if key is in the dictionary, else default.

update([E, ]**F) → None.  Update D from mapping/iterable E and F.

If E is present and has a .keys() method, then does: for k in E.keys(): D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

values()

Return an object providing a view on the dict’s values.

PRIORITY_SORT_IDS = ['8453-0x0d877dc7c8fa3ad980dfdb18b48ec9f8768359c4']

For manual debugging, we process these vaults first

get_vaults_by_id(rows)

Build a dictionary of vaults by their chain-address id.

Parameters

rows (dict[eth_defi.vault.base.VaultSpec, eth_defi.vault.vaultdb.VaultRow]) – Metadata rows from vault database

Returns

Dictionary of vaults by their chain-address id

Return type

dict[str, eth_defi.vault.vaultdb.VaultRow]

VAULT_STATE_COLUMNS = {'account_pnl': nan, 'available_liquidity': nan, 'cumulative_volume': nan, 'daily_deposit_count': nan, 'daily_deposit_usd': nan, 'daily_withdrawal_count': nan, 'daily_withdrawal_usd': nan, 'deposit_closed_reason': '', 'deposits_open': '', 'epoch_reset': False, 'follower_count': nan, 'hypercore_repair_status': '', 'leader_commission': nan, 'leader_fraction': nan, 'max_deposit': nan, 'max_redeem': nan, 'redemption_open': '', 'trading': '', 'utilisation': nan, 'vault_settlement_at': NaT, 'written_at': NaT}

Vault state and pass-through columns added by the historical scanner. Ensure these are always present in cleaned data, even when processing old scan data that lacks them. See CleanedVaultPriceRow for column semantics.

ensure_vault_state_columns(prices_df)

Ensure vault state columns are present in the DataFrame.

  • Adds missing columns with default values for backward compatibility with raw scan data generated before these fields were added.

Parameters

prices_df (pandas.DataFrame) –

Return type

pandas.DataFrame

derive_deposit_closed_reason(prices_df)

Derive unified deposit_closed_reason from protocol-specific columns.

For Hyperliquid vaults, deposit_closed_reason is already set by build_raw_prices_dataframe() with specific reason strings.

For ERC-4626 vaults, the deposits_open string column (“true”/”false”/””) is converted to a generic reason.

Parameters

prices_df (pandas.DataFrame) – DataFrame with deposit_closed_reason and deposits_open columns.

Returns

DataFrame with deposit_closed_reason filled in for both vault types.

Return type

pandas.DataFrame

assign_unique_names(rows, prices_df, logger=<built-in function print>, duplicate_nav_threshold=1000)

Ensure all vaults have unique human-readable name.

  • Rerwrite metadata rows

  • Find duplicate vault names

  • Add a running counter to the name to make it unique

Parameters
Return type

pandas.DataFrame

add_denormalised_vault_data(rows, prices_df, logger=<built-in function print>)

Add denormalised data to the prices DataFrame.

  • Take data from vault database and duplicate it across every row

  • Add protocol name and event count columns

Parameters
Return type

pandas.DataFrame

filter_vaults_by_stablecoin(rows, prices_df, logger=<built-in function print>)

Reduce vaults to stablecoin vaults only.

Parameters
Return type

pandas.DataFrame

calculate_vault_returns(prices_df, logger=<built-in function print>)

Calculate returns for each vault.

Filter out reads without a share price and add the compatibility returns_1h column. Consecutive zero prices after a complete loss carry a zero return: the loss was already recorded by the first transition to zero, and leaving later 0 / 0 changes as NaN makes the terminal curve unnecessarily difficult for downstream consumers to use.

Example of input data:

     chain                                     address  block_number           timestamp  share_price  ...  errors                                                id  name  event_count            protocol
207  42161  0x487cdc7d21ac8765eff6c0e681aea36ae1594471      13294721 2022-05-30 19:59:22          1.0  ...          42161-0x487cdc7d21ac8765eff6c0e681aea36ae1594471  LDAI           17  <unknown ERC-4626>
Parameters

  • prices_df (pandas.DataFrame) – Price rows containing string id and numeric share_price columns, ordered by vault and timestamp.

  • logger (Callable[[str], None]) – Notebook, console, or structured-log adapter accepting one message.

Returns

Price rows with returns_1h calculated between consecutive rows.

Return type

pandas.DataFrame

clean_returns(rows, prices_df, logger=<built-in function print>, outlier_threshold=0.5, display=<function <lambda>>, returns_col='returns_1h')

Clean returns data by removing rows with NaN or infinite values.

  • In returns data we have outliers that are likely not real returns, or one-time events that cannot repeat.

    • Floating point errors: [Share price may jumps wildly when a vault TVL is near zero](https://x.com/0xSEM/status/1914748782102630455)

    • Bugs: Vault share price method to estimate returns does not work for a particular airdrop

    • Airdrops: Vault gets an irregular rewards that will not repeat, and thus are not good to estimate the future performance

  • We clean returns by doing an assumptions - Daily returns higher than static outlier - Daily TVL max does not make sense - Daily TVL min does not make sense - Daily TVL % below lifetime average TVL

Parameters
Return type

pandas.DataFrame

clean_by_tvl(rows, prices_df, logger=<built-in function print>, tvl_threshold_min=1000.0, tvl_threshold_max=99000000000, tvl_threshold_min_dynamic=0.02, returns_col='returns_1h')

TVL-based threshold filtering of returns.

Hypercore keeps its PnL/NAV price-derived return because rewriting the return alone would make profit disagree with the cleaned share price. Its low-TVL observations still receive tvl_filtering_mask=True so investment-suitability consumers can exclude them. Other protocols retain the existing zero-return behaviour.

Parameters

  • rows (dict[eth_typing.evm.HexAddress, eth_defi.vault.vaultdb.VaultRow]) – Vault metadata keyed by address. Retained for compatibility with the existing cleaner interface.

  • prices_df (pandas.DataFrame) – Timestamp-indexed price rows containing id, chain, total_assets, and the selected return column.

  • logger – Notebook, console, or structured-log adapter accepting one message.

  • tvl_threshold_min – Absolute minimum NAV in USD.

  • tvl_threshold_max – Absolute maximum NAV in USD.

  • tvl_threshold_min_dynamic – Minimum NAV as a fraction of the vault’s all-time average NAV.

  • returns_col – Name of the return column to clean.

Returns

The input frame with TVL audit columns and protocol-appropriate return cleaning applied.

Return type

pandas.DataFrame

filter_unneeded_row(prices_df, logger=<built-in function print>, epsilon=0.0025)

Dedpulicate data rows with epsilon.

  • Reduce data size by elimating rows where the value changes is too little

  • Remove rows where the total asset/share price/total supply change has been too small

Note

This filter conly yields 2% savings in row count, so it turned out not to be worth of the problems.

Parameters

  • prices_df (pandas.DataFrame) – Assume sorted by timestsamp

  • epsilon – Tolerance for floating point comparison

Return type

pandas.DataFrame

remove_inactive_lead_time(prices_df, logger=<built-in function print>)

Remove initial inactive period from each vault’s price history.

  • At the beginning of a vault’s lifecycle, total supply may remain constant while the vault is inactive (e.g., 1, 1000, etc.)

  • When the vault activates, the share price may jump, causing abnormal returns

  • This function removes the initial rows where total_supply hasn’t changed

  • Uses exact equality for comparison

  • Skips initial rows with zero or NaN total_supply to find first valid value

Parameters

prices_df (pandas.DataFrame) – Price data with ‘id’ and ‘total_supply’ columns. Assumes data is sorted by timestamp within each vault.

Returns

DataFrame with inactive lead time removed for each vault

Return type

pandas.DataFrame

approximate_hypercore_share_prices_from_pnl_nav(prices_df, logger=<built-in function print>, max_positive_return=1.0)

Build an approximate Hypercore economic-performance index from PnL and NAV.

Hyperliquid does not expose historical vault share supply or an investable unit price through its vaultDetails API. Its rolling NAV and PnL windows can refresh at different timestamps, so the scanner-derived ERC-4626-like supply can change units between daily and HF reads. The July 2026 production investigation found that this left raw multi-hundred-percent moves in cleaned data and that a partial daily/HF repair itself created Order Block Hunter’s +275.4% clean return. Ledger flows prove NAV accounting for some intervals, but their intra-period ordering is unavailable and therefore cannot recover exact time-weighted investor returns.

The cleaned Hypercore price is consequently a conservative performance index. One freshest usable checkpoint is selected per fixed four-hour UTC bucket while retaining its original API timestamp. Missing buckets are not manufactured, and older history remains daily or weekly where Hyperliquid has already downsampled it. Between checkpoints, cumulative account-PnL change is divided by the larger of opening NAV, closing NAV, and one dollar; the resulting returns are compounded from 1.0 within each recapitalisation epoch. This denominator prevents unknown capital-flow timing and small NAV from manufacturing performance. Positive returns are capped at max_positive_return per selected checkpoint. A return at or below -100% is accepted only when NAV is zero and does not recover in the same epoch; otherwise the price is carried because applying one questionable negative PnL baseline would permanently zero the index. Non-checkpoint rows are also carried, avoiding duplicate daily/HF returns without interpolating future information backwards.

A July 2026 follow-up found a narrower timestamp problem in Fish Market. Cumulative PnL increased by $2,169.43 on 17 March while NAV remained unchanged; the same $2,169.43 appeared in NAV on 18 March with no additional PnL or capital flow. Treating the first half of this staggered API update as a complete checkpoint produced a clipped +100% return. When a checkpoint within 26 hours exhibits this exact PnL-then-NAV pattern, within a small numerical tolerance, this function carries the premature checkpoint and applies the return at the confirming checkpoint using the larger opening/confirmed NAV. Flat intermediate checkpoints may occur; any other economic change disqualifies the repair. This gives approximately +54.5% for Fish Market. The bounded, positive-only and value-matching conditions avoid suppressing large but reconciled gains in Magixbox, IKAGI and Satori Quantum HF Vault. Because confirmation arrives later, the most recent provisional checkpoint and prices compounded after it may be revised on the next cleaning run.

Input uses a timestamp index and requires id (string), chain (integer), share_price (float), total_assets (float), and cumulative PnL as either exported account_pnl (float) or scanner-shaped cumulative_pnl (float). Optional written_at timestamps select the freshest scanner batch, hypercore_source strings break otherwise equal ties in favour of HF, and boolean epoch_reset values separate performance epochs. The output preserves scanner values in raw_share_price, writes audit values to hypercore_repair_status, and recalculates Hypercore total_supply as synthetic index units so that total_assets == share_price * total_supply remains true. Hypercore total_supply is not an actual token supply.

Parameters

  • prices_df (pandas.DataFrame) – Timestamp-indexed cleaned-price input containing the columns described above. Rows must already be ordered by vault and timestamp.

  • logger (Callable[[str], None]) – Notebook, console, or structured-log adapter accepting one message.

  • max_positive_return (float) – Maximum approximate return applied at one four-hour checkpoint. The default is 1.0 (100%).

Returns

A copy with every Hypercore row expressed on its PnL/NAV performance index; all non-Hypercore rows are unchanged.

Return type

pandas.DataFrame

discard_hypercore_pre_recapitalisation_history(prices_df, logger=<built-in function print>, min_recapitalisation_assets=1000.0, min_recovery_delay=Timedelta('7 days 00:00:00'))

Start a recapitalised Hypercore vault at its new meaningful capital base.

A complete wipe-out followed by new deposits cannot be represented by one continuous share-price series. The old investors have a -100% return, while the new investors must not inherit the destroyed share supply. When a vault has meaningful NAV, reaches zero, and does not regain any positive NAV until after min_recovery_delay, discard its earlier observations from the cleaned output. The raw parquet remains unchanged.

Recovery duration and the new tracking threshold are intentionally separate. The delay is measured to the first value above HYPERCORE_ZERO_NAV_EPSILON, even when that value is below min_recapitalisation_assets. This prevents a sequence such as $2,000 -> $0 -> $900 next day -> $1,000 after seven days from erasing valid history merely because the recovery crossed the display threshold later. Once a durable recovery is established, the first retained observation must have at least min_recapitalisation_assets in NAV and is marked epoch_reset.

Raw scanner epoch_reset values are cleared before applying this rule. They mark arbitrary resets of the reconstructed synthetic supply, including funded vaults, and are not evidence of an economic wipe-out. Only the duration/NAV-qualified marker produced here may split the cleaned performance index.

The July 2026 production snapshot contained four qualifying episodes across 569 Hypercore vaults. HODL My Perps, HLP Liquidator, Rehobot LR, and Sifu all still qualify when measuring the delay to the first positive NAV, removing 369 rows from cleaned output. The stricter definition was chosen because the same snapshot contained hundreds of transient zero observations which must not reset lifetime performance.

Parameters

  • prices_df (pandas.DataFrame) – Vault price data indexed by timestamp, with id, chain, and total_assets columns. It must be sorted by vault and timestamp.

  • logger – Notebook or console logging function.

  • min_recapitalisation_assets (float) – Minimum NAV in USD needed before tracking the new investment epoch.

  • min_recovery_delay (pandas.Timedelta) – Minimum elapsed time between zero NAV and the first later positive NAV.

Returns

Price data without the superseded pre-recapitalisation epochs.

Return type

pandas.DataFrame

fix_outlier_share_prices(prices_df, logger=<built-in function print>, max_diff=0.33, look_back_hours=24, look_ahead_hours=24)

Fix out rows with share price that is too high.

  • Sometimes share price jump to an outlier value and back

  • This caused abnormal returns in returns calculations, messing all volatility numbers, sharpe, charts, etc.

  • The root cause is bad oracles, fat fingers, MEV trades, etc.

  • The lookback window is time-based (hours), not row-based, so it works correctly for vaults with non-hourly polling intervals

  • See check-share-price script for inspecting individual prices

Case Fluegel DAO:

timestamp

chain

address

block_number

share_price

total_assets

total_supply

2024-07-16 15:02:57

8453

0x277a3c57f3236a7d458576074d7c3d7046eb26c

17176415

1.60

373,740.21

232,929.92

2024-07-16 16:02:57

8453

0x277a3c57f3236a7d458576074d7c3d7046eb26c

17178215

1.63

379,832.59

232,929.92

2024-07-16 17:02:57

8453

0x277a3c57f3236a7d458576074d7c3d7046eb26c

17180015

0.33

75,744.97

232,929.92

2024-07-16 18:02:57

8453

0x277a3c57f3236a7d458576074d7c3d7046eb26c

17181815

1.64

382,282.78

232,929.92

Case Untangle Finance:

Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-12 23:14:19 (3206): fixing: 1.038721 -> 1.038721, prev: 1.038827, next: 0.444865
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-13 00:14:13 (3207): fixing: 1.038931 -> 1.038931, prev: 1.038801, next: 0.444865
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-13 01:14:09 (3208): fixing: 1.038931 -> 1.038931, prev: 1.038801, next: 0.444865
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-13 02:14:03 (3209): fixing: 1.038931 -> 1.038931, prev: 1.038801, next: 0.444865
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-13 03:14:01 (3210): fixing: 1.038931 -> 1.038931, prev: 1.038801, next: 0.444865
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-13 04:13:56 (3211): fixing: 1.038931 -> 1.038931, prev: 1.038801, next: 0.444865
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-13 05:13:53 (3212): fixing: 1.038931 -> 1.038931, prev: 1.038801, next: 0.468629
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-13 06:13:51 (3213): fixing: 1.039134 -> 1.039134, prev: 1.038439, next: 0.468629
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-13 07:13:45 (3214): fixing: 1.039134 -> 1.039134, prev: 1.038439, next: 0.468629
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-13 08:13:40 (3215): fixing: 1.039134 -> 1.039134, prev: 1.038439, next: 0.468629
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-13 09:13:37 (3216): fixing: 1.039134 -> 1.039134, prev: 1.038439, next: 0.482511
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-13 10:13:27 (3217): fixing: 1.039134 -> 1.039134, prev: 1.038439, next: 0.482511
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-13 11:13:21 (3218): fixing: 1.039134 -> 1.039134, prev: 1.038439, next: 0.482511
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-14 00:11:51 (3230): fixing: 0.444865 -> 1.0405275, prev: 1.038721, next: 1.042334
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-14 01:11:42 (3231): fixing: 0.444865 -> 1.0406325, prev: 1.038931, next: 1.042334
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-14 02:11:33 (3232): fixing: 0.444865 -> 1.0407335, prev: 1.038931, next: 1.042536
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-14 03:11:36 (3233): fixing: 0.444865 -> 1.0407335, prev: 1.038931, next: 1.042536
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-14 04:11:26 (3234): fixing: 0.444865 -> 1.0407335, prev: 1.038931, next: 1.042536
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-14 05:11:17 (3235): fixing: 0.444865 -> 1.0407335, prev: 1.038931, next: 1.042536
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-14 06:11:10 (3236): fixing: 0.468629 -> 1.0407335, prev: 1.038931, next: 1.042536
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-14 07:11:01 (3237): fixing: 0.468629 -> 1.040835, prev: 1.039134, next: 1.042536
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-14 08:10:52 (3238): fixing: 0.468629 -> 1.0406445, prev: 1.039134, next: 1.042155
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-14 11:10:26 (3239): fixing: 0.468629 -> 1.0406445, prev: 1.039134, next: 1.042155
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-14 12:10:18 (3240): fixing: 0.482511 -> 1.0406445, prev: 1.039134, next: 1.042155
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-14 13:10:09 (3241): fixing: 0.482511 -> 1.0406445, prev: 1.039134, next: 1.042155
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-14 14:10:01 (3242): fixing: 0.482511 -> 1.0406445, prev: 1.039134, next: 1.042155
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-15 04:08:39 (3254): fixing: 1.042334 -> 1.042334, prev: 0.444865, next: 1.04251
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-15 05:08:33 (3255): fixing: 1.042334 -> 1.042334, prev: 0.444865, next: 1.04251
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-15 06:08:29 (3256): fixing: 1.042536 -> 1.042536, prev: 0.444865, next: 1.04251
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-15 07:08:24 (3257): fixing: 1.042536 -> 1.042536, prev: 0.444865, next: 1.04251
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-15 08:08:20 (3258): fixing: 1.042536 -> 1.042536, prev: 0.444865, next: 1.04251
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-15 09:08:12 (3259): fixing: 1.042536 -> 1.042536, prev: 0.444865, next: 1.04251
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-15 10:08:07 (3260): fixing: 1.042536 -> 1.042536, prev: 0.468629, next: 1.042519
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-15 11:08:03 (3261): fixing: 1.042536 -> 1.042536, prev: 0.468629, next: 1.042519
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-15 12:07:55 (3262): fixing: 1.042155 -> 1.042155, prev: 0.468629, next: 1.042519
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-15 13:07:51 (3263): fixing: 1.042155 -> 1.042155, prev: 0.468629, next: 1.042519
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-15 14:07:49 (3264): fixing: 1.042155 -> 1.042155, prev: 0.482511, next: 1.042519
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-15 15:07:42 (3265): fixing: 1.042155 -> 1.042155, prev: 0.482511, next: 1.042519
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-15 16:07:36 (3266): fixing: 1.042155 -> 1.042155, prev: 0.482511, next: 1.042354
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-23 02:04:05 (3433): fixing: 1.036482 -> 1.036482, prev: 1.126302, next: 0.487429
Abnormal share price detected for 42161-0x4a3f7dd63077cde8d7eff3c958eb69a3dd7d31a9 at index 2025-10-24 06:01:21 (3457): fixing: 0.487429 -> 1.041995, prev: 1.036482, next: 1.047508
Parameters

prices_df (pandas.DataFrame) –

Return type

pandas.DataFrame

sort_and_index_vault_prices(prices_df, priority_ids)

Set up the order of vaults for processing.

  • If we do debugging we want vaults we debug go first, as the pipeline takes several minutes to run

Parameters
process_raw_vault_scan_data(rows, prices_df, logger=<built-in function print>, display=<function <lambda>>, diagnose_vault_id=None)

Preprocess vault data for further analysis.

  • Assign unique names to vaults

  • Add denormalised vault data to prices DataFrame

  • Filter out non-stablecoin vaults

  • Calculate returns, rolling metrics

Parameters
Return type

pandas.DataFrame

check_missing_metadata(rows, price_ids, prices_df, logger=<built-in function print>)

Check that we have metadata for all vaults in the prices DataFrame.

Vault id is in format: 56-0x10c90bfcfb3d2a7ae814da1548ae3a7fc31c35a0

If there are vaults with price data but no metadata, they are logged at error level and their IDs returned so the caller can drop them.

Parameters

  • rows (dict) – Metadata rows from vault database

  • prices_df (pandas.DataFrame) – The full prices DataFrame, used to extract context for missing vaults.

  • price_ids (pandas.Series) –

Returns

Set of vault IDs that are missing from the metadata. These should be dropped from the price data before further processing.

Return type

set[str]

generate_cleaned_vault_datasets(vault_db_path=PosixPath('/home/runner/.tradingstrategy/vaults/vault-metadata-db.pickle'), price_df_path=PosixPath('/home/runner/.tradingstrategy/vaults/vault-prices-1h.parquet'), cleaned_price_df_path=PosixPath('/home/runner/.tradingstrategy/vaults/cleaned-vault-prices-1h.parquet'), settlement_db_path=None, logger=<built-in function print>, display=<function display>, diagnose_vault_id=None)

A command line script entry point to take raw scanned vault price data and clean it up to a format that can be analysed.

  • Reads vault-prices-1h.parquet and generates cleaned-vault-prices-1h.parquet

  • Calculate returns and various performance metrics to be included with prices data

  • Clean returns from abnormalities

  • Stamp the cleaned Parquet with the current Docker metadata.version provenance, matching vault scanner JSON exports

Note

Drops non-stablecoin vaults. The cleaning is currently applicable for stable vaults only.

Parameters
replace_cleaned_vault_histories(vault_ids, *, vault_db_path=PosixPath('/home/runner/.tradingstrategy/vaults/vault-metadata-db.pickle'), raw_price_df_path=PosixPath('/home/runner/.tradingstrategy/vaults/vault-prices-1h.parquet'), cleaned_price_df_path=PosixPath('/home/runner/.tradingstrategy/vaults/cleaned-vault-prices-1h.parquet'), settlement_db_path=None, logger=<built-in function print>)

Rebuild and atomically replace cleaned histories for selected vaults.

The normal cleaner is deliberately whole-dataset: it reads every raw row, applies its transformations, and emits a new Parquet file. A historical repair only changes a small number of vaults, however, and each cleaner transformation is independent between vault ids. Recompute the complete raw history for the selected ids, then stream-copy all other cleaned row groups into a replacement Parquet. This avoids expensive pandas cleaning for unrelated vaults while preserving their existing cleaned rows and physical id, timestamp order.

The destination remains a single Parquet file, so its bytes must still be rewritten before the atomic replace. The function does not silently drop columns: a selected vault whose cleaned columns do not match the existing output raises an error before replacing the original file.

Parameters

  • vault_ids (set[str]) – Canonical lower-case chain_id-address ids to replace.

  • vault_db_path (pathlib.Path) – Metadata database used for denormalisation and stablecoin filtering.

  • raw_price_df_path (pathlib.Path) – Raw scanner Parquet containing the replacement histories.

  • cleaned_price_df_path (pathlib.Path) – Existing cleaned Parquet to update atomically.

  • settlement_db_path (Optional[pathlib.Path]) – Optional settlement database applied to the selected cleaned rows.

  • logger (Callable[[str], None]) – Progress callback.

Returns

Number of cleaned rows written for the selected vaults.

Return type

int

forward_fill_vault(vault_df)

Forward fill missing vault prices up to max_gap_hours.

  • For displaying, calculating metrics, etc. we want continuous time series

  • Align random sample interval to 1h

Parameters

vault_df (pandas.DataFrame) –

Price data for a single vault.

Assume 1h price data.

Return type

pandas.DataFrame