vault.top_vaults_json

Documentation for eth_defi.vault.top_vaults_json Python module.

Multi-chain vault analysis + safe JSON export.

Features: - Performs lifetime metric analysis for all available chains. - Filters and formats results for the top-performing vaults. - Safely exports to JSON with NaN/Inf -> null sanitization. - Normalizes column keys into snake_case. - Uses column-wise .map(parse_value) to comply with modern pandas. - Uses allow_nan=False to guarantee strict JSON validity.

To test out:

OUTPUT_JSON=/tmp/top-vaults.json VAULT_EXPORT_STATE_PATH=/tmp/vault-export-state.json python -m eth_defi.vault.top_vaults_json

The legacy wrapper also works:

OUTPUT_JSON=/tmp/top-vaults.json VAULT_EXPORT_STATE_PATH=/tmp/vault-export-state.json python scripts/erc-4626/vault-analysis-json.py

To test out Pandas warning issues in calculate_lifetime_metrics(), enable strict warnings:

Module Attributes

`PROTOCOL_MIN_TVL_OVERRIDES`	Per-protocol peak-NAV export threshold overrides, keyed by protocol slug.
`DEFAULT_OUTPUT_FILENAME`	Default output filename when no `OUTPUT_JSON` override is supplied and no `output_path` is passed to `main()`.

Functions

`add_exported_vault`(vaults_by_key, key, ...)	Add a vault export row with deterministic deduplication.
`annotate_current_record`(record, state_entry, ...)	Annotate an exported current row.
`annotate_fallback_record`(record, state_entry)	Annotate a stale fallback export.
`annotate_sticky_record`(record, state_entry, ...)	Annotate a sticky current-row export.
`apply_sticky_export_state`(lifetime_data_df, ...)	Apply append-biased sticky export state to lifetime metrics.
`build_export_metadata`([version_info])	Build provenance metadata shared by all vault metrics JSON outputs.
`copy_export_record`(record)	Create a detached copy of an exported row.
`find_non_serializable_paths`(obj[, path, results])	Recursively traverses a Python object (dict or list) and collects paths to non-serializable values or invalid keys.
`format_state_timestamp`(ts)	Format a naive UTC timestamp for sticky export state.
`is_blacklisted_record`(record)	Check if an exported vault row is blacklisted.
`is_blacklisted_risk_value`(value)	Check if a row or exported record risk means hard blacklist.
`is_current_record_export_safe`(record)	Check if a current row can replace the stored export record.
`is_export_record_key_safe`(record)	Check if a stored exported row has the key fields needed for replay.
`is_row_fresh`(record, now, stale_warning_age_days)	Check if an exported current row is inside the freshness warning window.
`load_sticky_export_state`(path, now)	Load sticky export state or create a new empty state.
`main`([data_dir, vault_db_path, ...])	Main execution function for vault analysis and JSON export.
`make_current_vault_row`(row, now, ...)	Normalise one metrics dataframe row for sticky processing.
`make_empty_sticky_export_state`(now)	Create an empty sticky export state document.
`make_state_entry_from_current_row`(...[, ...])	Create or update sticky state for an exported current row.
`make_vault_export_state_key`(chain_id, address)	Create a canonical sticky export state key.
`make_vault_export_state_key_from_record`(record)	Create a sticky export state key from an exported vault record.
`mark_state_entry_suppressed`(state, key, ...)	Persist structural suppression for a vault.
`normalise_datetime_to_naive_utc`(value)	Normalise timestamp-like values to naive UTC datetimes.
`resolve_export_threshold_tvl`(record, ...)	Resolve the peak-NAV export threshold for one vault row.
`resolve_sticky_export_state_path`(data_dir)	Resolve the sticky export state path.
`save_sticky_export_state`(state, path)	Atomically write sticky export state.
`validate_sticky_export_state`(state)	Validate sticky export state structure.
`validate_strict_json_serialisable`(obj)	Validate JSON serialisability with strict NaN handling.

Classes

`CurrentVaultRow`	Normalised current dataframe row for sticky export processing.
`StickyExportResult`	Result of applying sticky vault export state.
`StickyExportStats`	Counters emitted by sticky vault export state processing.

PROTOCOL_MIN_TVL_OVERRIDES: dict[str, float] = {'apex': 499.0}

Per-protocol peak-NAV export threshold overrides, keyed by protocol slug.

A vault normally has to reach THRESHOLD_TVL peak NAV to enter the exported dataset. Some protocols warrant a different bar. The value here replaces THRESHOLD_TVL for matching rows.

ApeX Omni native vaults are small retail copy-trading leader accounts and, at launch, none have yet gathered meaningful TVL (the largest currently holds roughly USDT 14k, and only about two dozen clear the default USDT 5k export bar). With the default USDT 5k bar the ApeX protocol page is almost empty, so during the rollout period we deliberately lower the ApeX threshold to USDT 499 to give the page a usable set of vaults. Revisit and raise this back towards the default once ApeX vaults gain real traction.

resolve_export_threshold_tvl(record, default_threshold)

Resolve the peak-NAV export threshold for one vault row.

Most vaults use the global THRESHOLD_TVL, but protocols listed in PROTOCOL_MIN_TVL_OVERRIDES use their own bar. The lookup is by exported protocol_slug so it does not depend on any protocol package.

Parameters

record (dict) – Exported vault row, expected to carry protocol_slug.
default_threshold (float) – Fallback threshold applied when no protocol override matches.

Returns

Peak-NAV threshold in denomination units for this row.

Return type

float

DEFAULT_OUTPUT_FILENAME = 'stablecoin-vault-metrics.json': Default output filename when no OUTPUT_JSON override is supplied and no output_path is passed to main().

class StickyExportStats

Bases: object

Counters emitted by sticky vault export state processing.

The counters are intentionally small and line-oriented, because this script is often run from cron and pipeline logs are the first place operators diagnose data gaps.

loaded_state_entries: int: State entries read before this run.

current_filter_passed: int: Rows passing the current peak TVL export filter.

sticky_additions: int: Current-filter rows that created a new sticky state entry.

sticky_fallback_exports: int: Rows replayed from last_exported_record.

current_row_structural_fallbacks: int: Current rows that could not replace the stored fallback record.

structurally_suppressed_vaults: int: Rows skipped because of structural suppression.

stale_warning_vaults: int: Rows carrying stale warning annotations.

missing_protocol_slugs: int: Exported sticky rows whose protocol slug no longer resolves in Core3.

missing_curator_slugs: int: Exported sticky rows whose curator slug no longer resolves in curator export.

previous_current_filter_count: Optional[int]: Previous successful run’s current-filter row count.

__init__(loaded_state_entries=0, current_filter_passed=0, sticky_additions=0, sticky_fallback_exports=0, current_row_structural_fallbacks=0, structurally_suppressed_vaults=0, stale_warning_vaults=0, missing_protocol_slugs=0, missing_curator_slugs=0, previous_current_filter_count=None)

Parameters

loaded_state_entries (int) –
current_filter_passed (int) –
sticky_additions (int) –
sticky_fallback_exports (int) –
current_row_structural_fallbacks (int) –
structurally_suppressed_vaults (int) –
stale_warning_vaults (int) –
missing_protocol_slugs (int) –
missing_curator_slugs (int) –
previous_current_filter_count (Optional[int]) –

Return type

None

class StickyExportResult

Bases: object

Result of applying sticky vault export state.

vaults: list[dict]: Final exported vault rows.

state: dict: Mutated state ready to be written after output JSON succeeds.

stats: eth_defi.vault.top_vaults_json.StickyExportStats: Operator-facing counters.

__init__(vaults, state, stats)

Parameters

vaults (list[dict]) –
state (dict) –
stats (eth_defi.vault.top_vaults_json.StickyExportStats) –

Return type

None

class CurrentVaultRow

Bases: object

Normalised current dataframe row for sticky export processing.

key: str: Canonical state key, chain_id-lowercase_address.

record: dict: JSON export row produced by export_lifetime_row().

export_safe: bool: Whether this current row can safely replace last_exported_record.

fresh: bool: Whether the row timestamp is inside the freshness warning window.

__init__(key, record, export_safe, fresh)

Parameters

key (str) –
record (dict) –
export_safe (bool) –
fresh (bool) –

Return type

None

format_state_timestamp(ts)

Format a naive UTC timestamp for sticky export state.

State files use Z suffixes for readability, while comparisons use naive UTC datetimes internally.

Parameters: ts (datetime.datetime) – Naive UTC datetime.
Returns: ISO 8601 string with Z suffix.
Return type: str

normalise_datetime_to_naive_utc(value)

Normalise timestamp-like values to naive UTC datetimes.

Tz-aware Pandas and Python timestamps are converted to UTC first and only then stripped of tzinfo. This avoids both TypeError when comparing aware and naive values, and silent wall-clock shifts from dropping tzinfo before conversion.

Parameters: value – datetime, pd.Timestamp, ISO string, or null-like value.
Returns: Naive UTC datetime, or None for null-like values.
Return type: Optional[datetime.datetime]

make_vault_export_state_key(chain_id, address)

Create a canonical sticky export state key.

State, dataframe rows, exported rows, and deduplication all use the same chain_id-lowercase_address shape.

Parameters

chain_id (Union[int, str]) – Numeric chain id.
address (str) – EVM address, possibly checksummed.

Returns

Canonical state key.

Return type

str

make_vault_export_state_key_from_record(record)

Create a sticky export state key from an exported vault record.

Parameters: record (dict) – Exported vault JSON row.
Returns: Canonical state key.
Return type: str

resolve_sticky_export_state_path(data_dir)

Resolve the sticky export state path.

VAULT_EXPORT_STATE_PATH is an explicit override. Otherwise all vault metric JSON outputs share one state file under the pipeline data directory.

Parameters: data_dir (pathlib.Path) – Pipeline data directory.
Returns: State file path.
Return type: pathlib.Path

make_empty_sticky_export_state(now)

Create an empty sticky export state document.

Parameters: now (datetime.datetime) – Current naive UTC datetime.
Returns: Empty state mapping.
Return type: dict

validate_sticky_export_state(state)

Validate sticky export state structure.

Parameters: state (dict) – Decoded state JSON.
Return type: None

load_sticky_export_state(path, now)

Load sticky export state or create a new empty state.

Corrupt existing state raises instead of being reset, because the sticky qualification history is production state.

Parameters

path (pathlib.Path) – State file path.
now (datetime.datetime) – Current naive UTC datetime.

Returns

Loaded state mapping.

Return type

dict

save_sticky_export_state(state, path)

Atomically write sticky export state.

Parameters

state (dict) – State mapping.
path (pathlib.Path) – Destination path.

Return type

None

build_export_metadata(version_info=None)

Build provenance metadata shared by all vault metrics JSON outputs.

Parameters: version_info (Optional[eth_defi.version_info.VersionInfo]) – Optional Docker image version stamp. When omitted, read the stamp from the running image.
Returns: JSON-serialisable metadata containing the exporter version information.
Return type: dict

find_non_serializable_paths(obj, path=None, results=None)

Recursively traverses a Python object (dict or list) and collects paths to non-serializable values or invalid keys.

Args:: obj: The object to check (dict, list, or nested combination). path: Current path (list of keys/indices; internal use). results: List to collect issues (internal use).
Returns:: List of tuples: (path_list, issue_description) for each problem found. Empty list if everything is serializable.

is_blacklisted_risk_value(value)

Check if a row or exported record risk means hard blacklist.

Exported records serialise VaultTechnicalRisk.blacklisted as "Blacklisted" through get_risk_level_name(). Current dataframe rows may still carry the enum before export.

Parameters: value – Risk enum or serialised risk value.
Returns: True if the value is the exact blacklist marker.
Return type: bool

is_blacklisted_record(record)

Check if an exported vault row is blacklisted.

Parameters: record (dict) – Exported vault row.
Returns: True when the row carries the hard blacklist risk.
Return type: bool

is_export_record_key_safe(record)

Check if a stored exported row has the key fields needed for replay.

Parameters: record (dict) – Exported vault row.
Returns: (safe, reason).
Return type: tuple[bool, str | None]

is_current_record_export_safe(record)

Check if a current row can replace the stored export record.

Missing non-key fields are treated as a transient metadata gap for sticky vaults. The caller can still replay a previous last_exported_record.

Parameters: record (dict) – Exported vault row.
Returns: (safe, reason).
Return type: tuple[bool, str | None]

is_row_fresh(record, now, stale_warning_age_days)

Check if an exported current row is inside the freshness warning window.

Parameters

record (dict) – Exported current row.
now (datetime.datetime) – Current naive UTC datetime.
stale_warning_age_days (int) – Warning age in days.

Returns

True when the row is fresh.

Return type

bool

copy_export_record(record)

Create a detached copy of an exported row.

Parameters: record (dict) – Exported row.
Returns: JSON-round-tripped copy.
Return type: dict

make_current_vault_row(row, now, stale_warning_age_days)

Normalise one metrics dataframe row for sticky processing.

Parameters

row (pandas.Series) – Lifetime metrics dataframe row.
now (datetime.datetime) – Current naive UTC datetime.
stale_warning_age_days (int) – Warning age in days.

Returns

Normalised current row wrapper.

Return type

eth_defi.vault.top_vaults_json.CurrentVaultRow

annotate_current_record(record, state_entry, fresh)

Annotate an exported current row.

Parameters

record (dict) – Exported row.
state_entry (Optional[dict]) – Previous or current state entry.
fresh (bool) – Whether the row is fresh.

Returns

Annotated copy.

Return type

dict

annotate_sticky_record(record, state_entry, fresh)

Annotate a sticky current-row export.

Parameters

record (dict) – Exported current row.
state_entry (dict) – Sticky state entry.
fresh (bool) – Whether the row is fresh.

Returns

Annotated copy.

Return type

dict

annotate_fallback_record(record, state_entry, fallback_reason=None)

Annotate a stale fallback export.

Parameters

record (dict) – Stored last_exported_record.
state_entry (dict) – Sticky state entry.
fallback_reason (Optional[str]) – Optional reason for fallback.

Returns

Annotated copy.

Return type

dict

make_state_entry_from_current_row(current_row, now_text, threshold_tvl, existing_entry=None)

Create or update sticky state for an exported current row.

Parameters

current_row (eth_defi.vault.top_vaults_json.CurrentVaultRow) – Current row wrapper.
now_text (str) – Current timestamp formatted for state.
threshold_tvl (float) – Active MIN_TVL threshold.
existing_entry (Optional[dict]) – Previous state entry.

Returns

Updated state entry.

Return type

dict

mark_state_entry_suppressed(state, key, reason, now_text, current_row=None, threshold_tvl=None)

Persist structural suppression for a vault.

Parameters

state (dict) – Sticky export state.
key (str) – Canonical vault key.
reason (str) – Suppression reason.
now_text (str) – Current timestamp formatted for state.
current_row (Optional[eth_defi.vault.top_vaults_json.CurrentVaultRow]) – Current row, if available.
threshold_tvl (Optional[float]) – Active MIN_TVL threshold, if the current row qualified.

Return type

None

add_exported_vault(vaults_by_key, key, priority, record)

Add a vault export row with deterministic deduplication.

Higher priority wins. The priority rule is a safety net for malformed duplicate inputs, because normal current/sticky paths are intended to be mutually exclusive.

Parameters

vaults_by_key (dict[str, tuple[int, dict]]) – Accumulator keyed by canonical vault key.
key (str) – Canonical vault key.
priority (int) – Priority number, higher wins.
record (dict) – Export row.

Return type

None

apply_sticky_export_state(lifetime_data_df, state, *, now, threshold_tvl, stale_warning_age_days)

Apply append-biased sticky export state to lifetime metrics.

Current rows passing the peak TVL filter qualify a vault forever. Later missing, stale, below-threshold, or structurally incomplete rows do not make a previously qualified vault disappear unless exact invalid-fallback suppression applies.

Parameters

lifetime_data_df (pandas.DataFrame) – Calculated lifetime metrics.
state (dict) – Mutable sticky export state.
now (datetime.datetime) – Current naive UTC datetime.
threshold_tvl (float) – Active default peak TVL export threshold. Per-protocol overrides may apply per row via resolve_export_threshold_tvl().
stale_warning_age_days (int) – Warning age in days.

Returns

Final vault rows, mutated state, and counters.

Return type

eth_defi.vault.top_vaults_json.StickyExportResult

validate_strict_json_serialisable(obj)

Validate JSON serialisability with strict NaN handling.

Parameters: obj (dict) – Object to validate.
Return type: None

main(data_dir=None, vault_db_path=None, parquet_path=None, output_path=None, core3_db_path=None, xerberus_db_path=None, feed_db_path=None)

Main execution function for vault analysis and JSON export.

All four arguments are independently overridable. When a path argument is None, it is derived from data_dir so that a caller passing only data_dir reads and writes under that directory consistently — never a mix of data_dir for reads and ~/.tradingstrategy/vaults for writes.

Parameters

data_dir (Optional[pathlib.Path]) – Pipeline data directory. When None, falls back to the DATA_DIR env var (default ~/.tradingstrategy/vaults). Acts as the anchor for both parquet_path and output_path defaults when those are also None.
vault_db_path (Optional[pathlib.Path]) – Path to the vault metadata pickle. When None, VaultDatabase.read() uses eth_defi.vault.vaultdb.DEFAULT_VAULT_DATABASE.
parquet_path (Optional[pathlib.Path]) – Path to the cleaned vault prices parquet. When None, defaults to data_dir / "cleaned-vault-prices-1h.parquet".
output_path (Optional[pathlib.Path]) – Destination JSON path. When None, defaults to data_dir / DEFAULT_OUTPUT_FILENAME. The OUTPUT_JSON env var is honoured by _resolve_defaults_from_env() in the __main__ entrypoint, not by main() itself, so in-process callers get deterministic path anchoring with no env var surprises.
core3_db_path (Optional[pathlib.Path]) – Path to the Core3 risk intelligence DuckDB database. When None, resolved from CORE3_DATABASE_PATH env var, then the default constant. The database is only opened if the resolved file exists on disk.
xerberus_db_path (Optional[pathlib.Path]) – Path to the Xerberus risk DuckDB database. When None, resolved from XERBERUS_DATABASE_PATH env var, then the default constant. Opened read-only when the file exists.
feed_db_path (Optional[pathlib.Path]) – Path to the vault post feed DuckDB database. When None, resolved from FEED_DB_PATH env var, falling back to DB_PATH (used by the feed collector), then DEFAULT_VAULT_POST_DATABASE. The database is only opened if the resolved file exists on disk.

Return type

eth_defi.research.vault_metrics.VaultMetricsExport