vault.top_vaults_json

Documentation for eth_defi.vault.top_vaults_json Python module.

Multi-chain vault analysis + safe JSON export.

Features: - Performs lifetime metric analysis for all available chains. - Filters and formats results for the top-performing vaults. - Safely exports to JSON with NaN/Inf -> null sanitization. - Normalizes column keys into snake_case. - Uses column-wise .map(parse_value) to comply with modern pandas. - Uses allow_nan=False to guarantee strict JSON validity.

To test out:

OUTPUT_JSON=/tmp/top-vaults.json VAULT_EXPORT_STATE_PATH=/tmp/vault-export-state.json python -m eth_defi.vault.top_vaults_json

The legacy wrapper also works:

OUTPUT_JSON=/tmp/top-vaults.json VAULT_EXPORT_STATE_PATH=/tmp/vault-export-state.json python scripts/erc-4626/vault-analysis-json.py

To test out Pandas warning issues in calculate_lifetime_metrics(), enable strict warnings:

Functions

add_exported_vault(vaults_by_key, key, ...)	Add a vault export row with deterministic deduplication.
annotate_current_record(record, state_entry, ...)	Annotate an exported current row.
annotate_fallback_record(record, state_entry)	Annotate a stale fallback export.
annotate_sticky_record(record, state_entry, ...)	Annotate a sticky current-row export.
apply_sticky_export_state(lifetime_data_df, ...)	Apply append-biased sticky export state to lifetime metrics.
copy_export_record(record)	Create a detached copy of an exported row.
find_non_serializable_paths(obj[, path, results])	Recursively traverses a Python object (dict or list) and collects paths to non-serializable values or invalid keys.
format_state_timestamp(ts)	Format a naive UTC timestamp for sticky export state.
is_blacklisted_record(record)	Check if an exported vault row is blacklisted.
is_blacklisted_risk_value(value)	Check if a row or exported record risk means hard blacklist.
is_current_record_export_safe(record)	Check if a current row can replace the stored export record.
is_export_record_key_safe(record)	Check if a stored exported row has the key fields needed for replay.
is_row_fresh(record, now, stale_warning_age_days)	Check if an exported current row is inside the freshness warning window.
load_sticky_export_state(path, now)	Load sticky export state or create a new empty state.
main([data_dir, vault_db_path, ...])	Main execution function for vault analysis and JSON export.
make_current_vault_row(row, now, ...)	Normalise one metrics dataframe row for sticky processing.
make_empty_sticky_export_state(now)	Create an empty sticky export state document.
make_state_entry_from_current_row(...[, ...])	Create or update sticky state for an exported current row.
make_vault_export_state_key(chain_id, address)	Create a canonical sticky export state key.
make_vault_export_state_key_from_record(record)	Create a sticky export state key from an exported vault record.
mark_state_entry_suppressed(state, key, ...)	Persist structural suppression for a vault.
normalise_datetime_to_naive_utc(value)	Normalise timestamp-like values to naive UTC datetimes.
resolve_sticky_export_state_path(data_dir)	Resolve the sticky export state path.
save_sticky_export_state(state, path)	Atomically write sticky export state.
validate_sticky_export_state(state)	Validate sticky export state structure.
validate_strict_json_serialisable(obj)	Validate JSON serialisability with strict NaN handling.

Classes

CurrentVaultRow	Normalised current dataframe row for sticky export processing.
StickyExportResult	Result of applying sticky vault export state.
StickyExportStats	Counters emitted by sticky vault export state processing.

class CurrentVaultRow

Bases: object

Normalised current dataframe row for sticky export processing.

__init__(key, record, export_safe, fresh)

Parameters

• key (str) –

• record (dict) –

• export_safe (bool) –

• fresh (bool) –

Return type

None

class StickyExportResult

Bases: object

Result of applying sticky vault export state.

__init__(vaults, state, stats)

Parameters

• vaults (list[dict]) –

• state (dict) –

• stats (eth_defi.vault.top_vaults_json.StickyExportStats) –

Return type

None

class StickyExportStats

Bases: object

Counters emitted by sticky vault export state processing.

The counters are intentionally small and line-oriented, because this script is often run from cron and pipeline logs are the first place operators diagnose data gaps.

__init__(loaded_state_entries=0, current_filter_passed=0, sticky_additions=0, sticky_fallback_exports=0, current_row_structural_fallbacks=0, structurally_suppressed_vaults=0, stale_warning_vaults=0, missing_protocol_slugs=0, missing_curator_slugs=0, previous_current_filter_count=None)

Parameters

• loaded_state_entries (int) –

• current_filter_passed (int) –

• sticky_additions (int) –

• sticky_fallback_exports (int) –

• current_row_structural_fallbacks (int) –

• structurally_suppressed_vaults (int) –

• stale_warning_vaults (int) –

• missing_protocol_slugs (int) –

• missing_curator_slugs (int) –

• previous_current_filter_count (Optional[int]) –

Return type

None

add_exported_vault(vaults_by_key, key, priority, record)

Add a vault export row with deterministic deduplication.

Higher priority wins. The priority rule is a safety net for malformed duplicate inputs, because normal current/sticky paths are intended to be mutually exclusive.

Parameters

• vaults_by_key (dict[str, tuple[int, dict]]) – Accumulator keyed by canonical vault key.

• key (str) – Canonical vault key.

• priority (int) – Priority number, higher wins.

• record (dict) – Export row.

Return type

None

annotate_current_record(record, state_entry, fresh)

Annotate an exported current row.

Parameters

• record (dict) – Exported row.

• state_entry (Optional[dict]) – Previous or current state entry.

• fresh (bool) – Whether the row is fresh.

Returns

Annotated copy.

Return type

dict

annotate_fallback_record(record, state_entry, fallback_reason=None)

Annotate a stale fallback export.

Parameters

• record (dict) – Stored last_exported_record.

• state_entry (dict) – Sticky state entry.

• fallback_reason (Optional[str]) – Optional reason for fallback.

Returns

Annotated copy.

Return type

dict

annotate_sticky_record(record, state_entry, fresh)

Annotate a sticky current-row export.

Parameters

• record (dict) – Exported current row.

• state_entry (dict) – Sticky state entry.

• fresh (bool) – Whether the row is fresh.

Returns

Annotated copy.

Return type

dict

apply_sticky_export_state(lifetime_data_df, state, *, now, threshold_tvl, stale_warning_age_days)

Apply append-biased sticky export state to lifetime metrics.

Current rows passing the peak TVL filter qualify a vault forever. Later missing, stale, below-threshold, or structurally incomplete rows do not make a previously qualified vault disappear unless exact blacklist/invalid-fallback suppression applies.

Parameters

• lifetime_data_df (pandas.DataFrame) – Calculated lifetime metrics.

• state (dict) – Mutable sticky export state.

• now (datetime.datetime) – Current naive UTC datetime.

• threshold_tvl (float) – Active peak TVL export threshold.

• stale_warning_age_days (int) – Warning age in days.

Returns

Final vault rows, mutated state, and counters.

Return type

eth_defi.vault.top_vaults_json.StickyExportResult

copy_export_record(record)

Create a detached copy of an exported row.

Parameters

record (dict) – Exported row.

Returns

JSON-round-tripped copy.

Return type

dict

find_non_serializable_paths(obj, path=None, results=None)

Recursively traverses a Python object (dict or list) and collects paths to non-serializable values or invalid keys.

Args:

obj: The object to check (dict, list, or nested combination). path: Current path (list of keys/indices; internal use). results: List to collect issues (internal use).

Returns:

List of tuples: (path_list, issue_description) for each problem found. Empty list if everything is serializable.

format_state_timestamp(ts)

Format a naive UTC timestamp for sticky export state.

State files use Z suffixes for readability, while comparisons use naive UTC datetimes internally.

Parameters

ts (datetime.datetime) – Naive UTC datetime.

Returns

ISO 8601 string with Z suffix.

Return type

str

is_blacklisted_record(record)

Check if an exported vault row is blacklisted.

Parameters

record (dict) – Exported vault row.

Returns

True when the row carries the hard blacklist risk.

Return type

bool

is_blacklisted_risk_value(value)

Check if a row or exported record risk means hard blacklist.

Exported records serialise VaultTechnicalRisk.blacklisted as "Blacklisted" through get_risk_level_name(). Current dataframe rows may still carry the enum before export.

Parameters

value – Risk enum or serialised risk value.

Returns

True if the value is the exact blacklist marker.

Return type

bool

is_current_record_export_safe(record)

Check if a current row can replace the stored export record.

Missing non-key fields are treated as a transient metadata gap for sticky vaults. The caller can still replay a previous last_exported_record.

Parameters

record (dict) – Exported vault row.

Returns

(safe, reason).

Return type

tuple[bool, str | None]

is_export_record_key_safe(record)

Check if a stored exported row has the key fields needed for replay.

Parameters

record (dict) – Exported vault row.

Returns

(safe, reason).

Return type

tuple[bool, str | None]

is_row_fresh(record, now, stale_warning_age_days)

Check if an exported current row is inside the freshness warning window.

Parameters

• record (dict) – Exported current row.

• now (datetime.datetime) – Current naive UTC datetime.

• stale_warning_age_days (int) – Warning age in days.

Returns

True when the row is fresh.

Return type

bool

load_sticky_export_state(path, now)

Load sticky export state or create a new empty state.

Corrupt existing state raises instead of being reset, because the sticky qualification history is production state.

Parameters

• path (pathlib.Path) – State file path.

• now (datetime.datetime) – Current naive UTC datetime.

Returns

Loaded state mapping.

Return type

dict

main(data_dir=None, vault_db_path=None, parquet_path=None, output_path=None, core3_db_path=None, feed_db_path=None)

Main execution function for vault analysis and JSON export.

All four arguments are independently overridable. When a path argument is None, it is derived from data_dir so that a caller passing only data_dir reads and writes under that directory consistently — never a mix of data_dir for reads and ~/.tradingstrategy/vaults for writes.

Parameters

• data_dir (Optional[pathlib.Path]) – Pipeline data directory. When None, falls back to the DATA_DIR env var (default ~/.tradingstrategy/vaults). Acts as the anchor for both parquet_path and output_path defaults when those are also None.

• vault_db_path (Optional[pathlib.Path]) – Path to the vault metadata pickle. When None, VaultDatabase.read() uses eth_defi.vault.vaultdb.DEFAULT_VAULT_DATABASE.

• parquet_path (Optional[pathlib.Path]) – Path to the cleaned vault prices parquet. When None, defaults to data_dir / "cleaned-vault-prices-1h.parquet".

• output_path (Optional[pathlib.Path]) – Destination JSON path. When None, defaults to data_dir / DEFAULT_OUTPUT_FILENAME. The OUTPUT_JSON env var is honoured by _resolve_defaults_from_env() in the __main__ entrypoint, not by main() itself, so in-process callers get deterministic path anchoring with no env var surprises.

• core3_db_path (Optional[pathlib.Path]) – Path to the Core3 risk intelligence DuckDB database. When None, resolved from CORE3_DATABASE_PATH env var, then the default constant. The database is only opened if the resolved file exists on disk.

• feed_db_path (Optional[pathlib.Path]) – Path to the vault post feed DuckDB database. When None, resolved from FEED_DB_PATH env var, falling back to DB_PATH (used by the feed collector), then DEFAULT_VAULT_POST_DATABASE. The database is only opened if the resolved file exists on disk.

make_current_vault_row(row, now, stale_warning_age_days)

Normalise one metrics dataframe row for sticky processing.

Parameters

• row (pandas.Series) – Lifetime metrics dataframe row.

• now (datetime.datetime) – Current naive UTC datetime.

• stale_warning_age_days (int) – Warning age in days.

Returns

Normalised current row wrapper.

Return type

eth_defi.vault.top_vaults_json.CurrentVaultRow

make_empty_sticky_export_state(now)

Create an empty sticky export state document.

Parameters

now (datetime.datetime) – Current naive UTC datetime.

Returns

Empty state mapping.

Return type

dict

make_state_entry_from_current_row(current_row, now_text, threshold_tvl, existing_entry=None)

Create or update sticky state for an exported current row.

Parameters

• current_row (eth_defi.vault.top_vaults_json.CurrentVaultRow) – Current row wrapper.

• now_text (str) – Current timestamp formatted for state.

• threshold_tvl (float) – Active MIN_TVL threshold.

• existing_entry (Optional[dict]) – Previous state entry.

Returns

Updated state entry.

Return type

dict

make_vault_export_state_key(chain_id, address)

Create a canonical sticky export state key.

State, dataframe rows, exported rows, and deduplication all use the same chain_id-lowercase_address shape.

Parameters

• chain_id (Union[int, str]) – Numeric chain id.

• address (str) – EVM address, possibly checksummed.

Returns

Canonical state key.

Return type

str

make_vault_export_state_key_from_record(record)

Create a sticky export state key from an exported vault record.

Parameters

record (dict) – Exported vault JSON row.

Returns

Canonical state key.

Return type

str

mark_state_entry_suppressed(state, key, reason, now_text, current_row=None, threshold_tvl=None)

Persist structural suppression for a vault.

Parameters

• state (dict) – Sticky export state.

• key (str) – Canonical vault key.

• reason (str) – Suppression reason.

• now_text (str) – Current timestamp formatted for state.

• current_row (Optional[eth_defi.vault.top_vaults_json.CurrentVaultRow]) – Current row, if available.

• threshold_tvl (Optional[float]) – Active MIN_TVL threshold, if the current row qualified.

Return type

None

normalise_datetime_to_naive_utc(value)

Normalise timestamp-like values to naive UTC datetimes.

Tz-aware Pandas and Python timestamps are converted to UTC first and only then stripped of tzinfo. This avoids both TypeError when comparing aware and naive values, and silent wall-clock shifts from dropping tzinfo before conversion.

Parameters

value – datetime, pd.Timestamp, ISO string, or null-like value.

Returns

Naive UTC datetime, or None for null-like values.

Return type

Optional[datetime.datetime]

resolve_sticky_export_state_path(data_dir)

Resolve the sticky export state path.

VAULT_EXPORT_STATE_PATH is an explicit override. Otherwise all vault metric JSON outputs share one state file under the pipeline data directory.

Parameters

data_dir (pathlib.Path) – Pipeline data directory.

Returns

State file path.

Return type

pathlib.Path

save_sticky_export_state(state, path)

Atomically write sticky export state.

Parameters

• state (dict) – State mapping.

• path (pathlib.Path) – Destination path.

Return type

None

validate_sticky_export_state(state)

Validate sticky export state structure.

Parameters

state (dict) – Decoded state JSON.

Return type

None

validate_strict_json_serialisable(obj)

Validate JSON serialisability with strict NaN handling.

Parameters

obj (dict) – Object to validate.

Return type

None