research.metrics_freshness

Documentation for eth_defi.research.metrics_freshness Python module.

Low-TVL vault metrics freshness gate.

Vault metrics calculation is expensive per vault, and most vaults are tiny: roughly three quarters of stablecoin-denominated vaults hold less than USD 5,000. This module implements the freshness gate that recalculates low-TVL vaults only every LOW_TVL_METRICS_MAX_AGE while keeping every export-relevant vault fresh on every run.

A vault is low TVL when its current TVL is below the hardcoded family threshold (LOW_TVL_THRESHOLD_USD / LOW_TVL_THRESHOLD_BTC / LOW_TVL_THRESHOLD_ETH). Thresholds and the max age are deliberately hardcoded module constants, not env vars.

The gate state lives in a small per-bundle JSON file (vault-metrics-state.json, crypto-vaults/crypto-vault-metrics-state.json) because the export JSONs only contain vaults that passed the export filter: the never-exported low-TVL vaults are the bulk of the savings and would otherwise be due on every run.

The due rules are evaluated from one cheap groupby("id") pass over the raw price frame, before any metrics work:

1. no state entry (new vault), or

2. current TVL is null/non-finite (safe default), or

3. current TVL >= the family freshness threshold (promotion is immediate), or

4. peak TVL >= the per-vault export threshold (the vault can enter or re-enter the export, so its row must stay fresh), or

5. metrics_updated_at plus a deterministic per-vault stagger offset is older than LOW_TVL_METRICS_MAX_AGE.

Everything else is skipped. Without the stagger, the first full run would give every low-TVL vault the same timestamp and three days later the entire cohort would become due in one giant spike run; the offset spreads expiry evenly across the window with no extra state.

The freshness thresholds are intentionally independent of the export thresholds (MIN_TVL with per-protocol overrides, crypto CRYPTO_VAULTS_MIN_TVL_USD converted at guideline rates): the two mechanisms answer different questions, and due rule 4 uses the real per-vault export threshold, not the freshness constant.

Module Attributes

LOW_TVL_THRESHOLD_USD	Current TVL below this USD value marks a stablecoin/USD-family vault low TVL.
LOW_TVL_THRESHOLD_BTC	Current TVL below this BTC value marks a BTC-family vault low TVL.
LOW_TVL_THRESHOLD_ETH	Current TVL below this ETH value marks an ETH-family vault low TVL.
LOW_TVL_METRICS_MAX_AGE	Low-TVL vaults are recalculated when their last metrics update is older than this.
VAULT_METRICS_STATE_SCHEMA_VERSION	Schema version of the metrics freshness state document.
VAULT_METRICS_CALCULATION_VERSION	Recompute cached rows when endpoint or annualisation definitions change.
VAULT_METRICS_STATE_FILENAME	Default freshness state filename for the stablecoin bundle.
CRYPTO_METRICS_STATE_FILENAME	Freshness state filename for the crypto bundle, stored in its bundle directory.
LOW_TVL_THRESHOLDS_BY_FAMILY	Freshness thresholds in family units, keyed by DenominationFamily value.

Functions

clear_period_rankings(record)	Clear ranking fields on a replayed or patched export record.
compute_vault_tvl_observations(prices_df)	Cheap per-vault current and peak TVL from raw price rows.
free_memory()	Release freed buffers back to the operating system.
load_metrics_state(path, now)	Load freshness state, self-healing from a corrupt file.
load_valid_previous_crypto_records(...)	Load previous crypto-bundle records that are safe to patch.
low_tvl_threshold_for_family(family)	Resolve the low-TVL freshness threshold for one denomination family.
make_empty_metrics_state(now)	Create an empty freshness state document.
partition_due_vault_ids(seen_ids, state, ...)	Split vaults into due (recalculate) and skipped (keep previous record).
refresh_metrics_state(state, seen_ids, ...)	Refresh the freshness state after one metrics run.
resolve_metrics_state_path(data_dir)	Resolve the metrics freshness state path for the stablecoin bundle.
save_metrics_state(state, path)	Atomically write freshness state.

LOW_TVL_THRESHOLD_USD = 5000.0

Current TVL below this USD value marks a stablecoin/USD-family vault low TVL.

LOW_TVL_THRESHOLD_BTC = 0.1

Current TVL below this BTC value marks a BTC-family vault low TVL.

LOW_TVL_THRESHOLD_ETH = 2.5

Current TVL below this ETH value marks an ETH-family vault low TVL.

LOW_TVL_METRICS_MAX_AGE = datetime.timedelta(days=3)

Low-TVL vaults are recalculated when their last metrics update is older than this. High-TVL and export-relevant vaults are always recalculated.

VAULT_METRICS_STATE_SCHEMA_VERSION = 1

Schema version of the metrics freshness state document.

VAULT_METRICS_CALCULATION_VERSION = 2

Recompute cached rows when endpoint or annualisation definitions change.

VAULT_METRICS_STATE_FILENAME = 'vault-metrics-state.json'

Default freshness state filename for the stablecoin bundle.

CRYPTO_METRICS_STATE_FILENAME = 'crypto-vault-metrics-state.json'

Freshness state filename for the crypto bundle, stored in its bundle directory.

LOW_TVL_THRESHOLDS_BY_FAMILY: dict[str, float] = {'btc': 0.1, 'eth': 2.5, 'stablecoin': 5000.0}

Freshness thresholds in family units, keyed by DenominationFamily value. Unknown or unsupported families fall back to the USD threshold, which only affects how eagerly they are skipped; due rules 1, 2, 4 and a missing metrics_updated_at still force recomputation.

free_memory()

Release freed buffers back to the operating system.

gc.collect() alone is not enough under pandas 3, where string column buffers belong to Arrow’s allocator: memory is freed logically but can stay in the allocator and keep RSS high. pa.default_memory_pool().release_unused() returns those buffers to the OS. Call after deleting a large frame.

See pyarrow.MemoryPool.release_unused.

Return type

None

low_tvl_threshold_for_family(family)

Resolve the low-TVL freshness threshold for one denomination family.

Unknown or unsupported families deliberately fall back to the USD threshold: the fallback only affects how eagerly such a vault is skipped, and every unsafe case (no state entry, null TVL, export relevance, missing timestamp) still forces recomputation through the other due rules.

Parameters

family (Optional[str]) – DenominationFamily value (stablecoin, eth, btc), or None/unknown for a safe USD fallback.

Returns

Freshness threshold in the family unit.

Return type

float

resolve_metrics_state_path(data_dir)

Resolve the metrics freshness state path for the stablecoin bundle.

VAULT_METRICS_STATE_PATH is an explicit override, following the VAULT_EXPORT_STATE_PATH precedent. Otherwise the state file lives under the pipeline data directory. The crypto bundle does not use this helper: its state file is always derived from the bundle’s sticky-state path, so one env override can never point both bundles at the same file.

Parameters

data_dir (pathlib.Path) – Pipeline data directory.

Returns

State file path.

Return type

pathlib.Path

make_empty_metrics_state(now)

Create an empty freshness state document.

Used on first deploy, after state loss, and when quarantining a corrupt state file; all three cases intentionally degrade to one full recomputation run.

Parameters

now (datetime.datetime) – Current naive UTC datetime.

Returns

Empty state mapping.

Return type

dict

load_metrics_state(path, now)

Load freshness state, self-healing from a corrupt file.

A corrupt state file (malformed JSON, wrong schema version, wrong structure) is moved aside with a .corrupt-<timestamp> suffix and an empty state is returned: one full recompute follows and the gate applies from the next run.

A valid state from an older calculation version also triggers a full recomputation, without quarantining the state or changing price history.

Parameters

• path (pathlib.Path) – State file path.

• now (datetime.datetime) – Current naive UTC datetime.

Returns

Loaded state mapping.

Return type

dict

save_metrics_state(state, path)

Atomically write freshness state.

The write goes through the same atomicwrites pattern as the sticky export state, so a crash mid-write never leaves a truncated state file behind.

Parameters

• state (dict) – State mapping.

• path (pathlib.Path) – Destination path.

Return type

None

compute_vault_tvl_observations(prices_df)

Cheap per-vault current and peak TVL from raw price rows.

One groupby("id") pass over the price frame: the last non-null total_assets observation is the current TVL and the maximum is the peak TVL. No metrics calculation is needed for the due/skip decision. The frame may be unfiltered (every denomination): the lookups simply cover every vault in it, and callers restrict the due/skip decision to their own vault set afterwards. When the total_assets column is missing, empty lookups are returned so every vault falls back to the safe “due” default.

Parameters

prices_df (pandas.DataFrame) – Vault price rows with id and total_assets columns.

Returns

(current_tvl_by_id, peak_tvl_by_id) with finite floats or None per vault.

Return type

tuple[dict[str, float | None], dict[str, float | None]]

partition_due_vault_ids(seen_ids, state, current_tvl_by_id, peak_tvl_by_id, family_by_id, export_threshold_by_id, now, patchable_ids=None)

Split vaults into due (recalculate) and skipped (keep previous record).

See the module docstring for the due rules. The optional patchable_ids precondition is used by the crypto bundle, which has no sticky record replay: a vault may only be skipped when a validated previous export record exists to patch into the output. The stablecoin bundle passes None because its sticky export state replays skipped vaults automatically.

Parameters

• seen_ids (set[str]) – Vault ids present in the current price data.

• state (dict) – Loaded freshness state mapping.

• current_tvl_by_id (dict[str, float | None]) – Current TVL per vault from compute_vault_tvl_observations().

• peak_tvl_by_id (dict[str, float | None]) – Peak TVL per vault from compute_vault_tvl_observations().

• family_by_id (dict[str, str]) – DenominationFamily value per vault.

• export_threshold_by_id (dict[str, float]) – Per-vault export threshold (with protocol overrides) in the vault’s denomination unit.

• now (datetime.datetime) – Current naive UTC datetime.

• patchable_ids (Optional[set[str]]) – Optional set of vault ids with a validated previous record to patch.

Returns

(due_ids, skipped_ids) partitioning seen_ids.

Return type

tuple[set[str], set[str]]

refresh_metrics_state(state, seen_ids, computed_ids, current_tvl_by_id, peak_tvl_by_id, family_by_id, now)

Refresh the freshness state after one metrics run.

metrics_updated_at advances only for vault ids actually returned in the metrics frame: a vault that failed metric calculation stays due next run instead of being stamped fresh for three days. The cheap TVL observations are refreshed for every vault seen in the price data, and vaults no longer present in the price data are dropped.

Parameters

• state (dict) – Mutable loaded state mapping.

• seen_ids (set[str]) – Vault ids present in the current price data.

• computed_ids (set[str]) – Vault ids actually returned by the metrics calculation.

• current_tvl_by_id (dict[str, float | None]) – Current TVL per vault.

• peak_tvl_by_id (dict[str, float | None]) – Peak TVL per vault.

• family_by_id (dict[str, str]) – DenominationFamily value per vault.

• now (datetime.datetime) – Current naive UTC datetime.

Returns

The mutated state mapping.

Return type

dict

clear_period_rankings(record)

Clear ranking fields on a replayed or patched export record.

Rankings are computed over the vaults that were recalculated in the same run, so a record replayed from an earlier run carries ranks from a different vault universe and would produce duplicate, non-comparable curator ranks. Cleared in place; skipped vaults show no current ranks.

Parameters

record (dict) – Exported vault JSON row with a period_results list.

Returns

The same record, mutated.

Return type

dict

load_valid_previous_crypto_records(metadata_path, *, schema_version, whitelist_sha256)

Load previous crypto-bundle records that are safe to patch.

A previous record may only be re-attached when the whole document is compatible with the current process: matching schema_version and denomination_whitelist_sha256. A missing, unreadable or mismatched document yields an empty mapping so every affected vault is recomputed. Per-record family and threshold validation stays at the call site, which owns the vault metadata and threshold inputs.

Parameters

• metadata_path (pathlib.Path) – Previous crypto-vault-metadata.json path, read before it is overwritten.

• schema_version (int) – Current CRYPTO_VAULTS_SCHEMA_VERSION.

• whitelist_sha256 (str) – Current get_denomination_whitelist_digest() value.

Returns

Mapping of vault id to previous record; empty when the document cannot be trusted.

Return type

dict[str, dict]