vault.crypto_vaults

Documentation for eth_defi.vault.crypto_vaults Python module.

Build isolated metadata for the private crypto-vaults bundle.

The module uses the shared metric calculations and JSON serialisation helpers with its own input selection and persistent state. Public top-vault selection and crypto-bundle admission remain separate.

Module Attributes

`CRYPTO_VAULTS_BUNDLE_NAME`	Bundle identifier stored in crypto metadata and manifest documents.
`CRYPTO_VAULTS_SCHEMA_VERSION`	Stable schema version for crypto metadata, sticky state and manifests.
`CRYPTO_CLEANED_PRICE_FILENAME`	Private daily Parquet filename for the isolated crypto-vaults bundle.
`USD_RATE_FORWARD_FILL_DAYS`	Maximum provider-day gap filled when building the effective USD rate curve.
`USD_RATE_BOUNDS`	Broad plausibility ranges for inverted USD-per-native exchange rates.
`CRYPTO_NATIVE_MIN_PEAK_TOTAL_ASSETS`	Hard native-unit admission thresholds for the private crypto bundle.
`NATIVE_METRIC_COLUMNS`	Columns read by `calculate_vault_record` on the native fast path.

Functions

`build_crypto_usd_conversion_context`(...)	Build effective USD rate curves and provenance for crypto metrics.
`build_crypto_vault_metadata`(*, ...[, ...])	Calculate and atomically persist private crypto-vault metadata.
`build_crypto_vault_prices`(*, vault_db_path, ...)	Create the isolated daily stablecoin/ETH/BTC price Parquet.
`build_crypto_vault_record`(record, vault_row, ...)	Convert one common metric record to the crypto bundle schema.
`calculate_crypto_native_admission`(vault_db, ...)	Calculate hard ETH/BTC admission from native lifetime peak assets.
`resolve_crypto_vault_paths`(data_dir[, directory])	Resolve crypto bundle paths under one pipeline data directory.

Classes

`CryptoNativeAdmission`	Summarise native-unit admission for one cleaned price frame.
`CryptoVaultPaths`	Explicit local paths for one crypto-vaults build.
`USDExchangeRateSeriesBuild`	Validated effective USD-rate series for one denomination family.

CRYPTO_VAULTS_BUNDLE_NAME = 'crypto-vaults': Bundle identifier stored in crypto metadata and manifest documents.

CRYPTO_VAULTS_SCHEMA_VERSION = 2: Stable schema version for crypto metadata, sticky state and manifests.

CRYPTO_CLEANED_PRICE_FILENAME = 'crypto-cleaned-vault-prices-1d.parquet': Private daily Parquet filename for the isolated crypto-vaults bundle.

USD_RATE_FORWARD_FILL_DAYS = 3: Maximum provider-day gap filled when building the effective USD rate curve. A longer gap starts a new metric segment rather than inventing a price.

USD_RATE_BOUNDS = {'btc': (100.0, 1000000.0), 'eth': (10.0, 100000.0)}: Broad plausibility ranges for inverted USD-per-native exchange rates. They reject malformed provider values without turning a normal market move into an unsupported denomination failure.

CRYPTO_NATIVE_MIN_PEAK_TOTAL_ASSETS = mappingproxy({<DenominationFamily.eth: 'eth'>: Decimal('2.5'), <DenominationFamily.btc: 'btc'>: Decimal('0.1')})

Hard native-unit admission thresholds for the private crypto bundle.

These values deliberately do not use a live USD conversion. The source denomination family is the unit used by total_assets in the cleaned price data, so the policy cannot drift as exchange rates change.

NATIVE_METRIC_COLUMNS = ('id', 'chain', 'event_count', 'block_number', 'share_price', 'total_assets', 'vault_poll_frequency', 'available_liquidity', 'utilisation', 'leader_fraction', 'leader_commission', 'account_pnl', 'follower_count', 'cumulative_volume', 'perp_position_data_status', 'perp_long_notional', 'perp_short_notional', 'perp_largest_position_notional', 'perp_open_position_count', 'perp_metrics_observed_at', 'perp_quote_asset')

Columns read by calculate_vault_record on the native fast path.

Keeping this projection explicit prevents large descriptive/object columns from being copied into every per-vault group. Optional columns retain the protocol-specific fields that the common record builder can export.

class USDExchangeRateSeriesBuild

Bases: object

Validated effective USD-rate series for one denomination family.

The result keeps the optional rate curve, its export provenance and a stable error together, avoiding positional tuple handling at the caller.

rates: Optional[pandas.Series]: Effective USD-per-underlying daily rate curve when validation succeeds.

coverage: Optional[dict[str, Any]]: JSON-serialisable source coverage information when source rows exist.

error_reason: Optional[str]: Stable reason USD metrics cannot be calculated for this family.

__init__(rates, coverage, error_reason)

Parameters

rates (Optional[pandas.Series]) –
coverage (Optional[dict[str, Any]]) –
error_reason (Optional[str]) –

Return type

None

class CryptoNativeAdmission

Bases: object

Summarise native-unit admission for one cleaned price frame.

The summary is calculated from the lifetime maximum finite total_assets value for each ETH/BTC vault. It is also used to filter the cleaned Parquet before the expensive metadata calculation.

native_ids: frozenset[str]: Native ETH/BTC vault IDs represented by the input frame.

qualifying_ids: frozenset[str]: IDs whose lifetime peak reaches the configured family threshold.

native_row_count: int: Number of ETH/BTC source rows represented by the input frame.

qualifying_row_count: int: Number of source rows belonging to qualifying IDs.

family_counts: dict[str, dict[str, int]]: Per-family vault and row counts for audit logging.

property skipped_ids: frozenset[str]

Return native IDs rejected by the hard threshold.

The difference is computed from immutable ID sets, so callers cannot accidentally alter the admission decision.

Returns: Rejected native vault IDs.

property skipped_row_count: int

Return the number of native rows belonging to rejected IDs.

This count is derived from the source and qualifying row totals used by the admission log.

Returns: Number of rejected source rows.

__init__(native_ids, qualifying_ids, native_row_count, qualifying_row_count, family_counts)

Parameters

native_ids (frozenset[str]) –
qualifying_ids (frozenset[str]) –
native_row_count (int) –
qualifying_row_count (int) –
family_counts (dict[str, dict[str, int]]) –

Return type

None

class CryptoVaultPaths

Bases: object

Explicit local paths for one crypto-vaults build.

Instances are constructed by resolve_crypto_vault_paths(), which derives every payload from one bundle directory.

directory: pathlib.Path: Bundle directory below the pipeline data directory.

cleaned_price_path: pathlib.Path: Daily observation-preserving Parquet file.

metadata_path: pathlib.Path: JSON metadata output.

compressed_metadata_path: pathlib.Path: Brotli-compressed JSON metadata output.

sticky_state_path: pathlib.Path: Private sticky qualification state.

manifest_path: pathlib.Path: Current bundle manifest, created during R2 publication.

__init__(directory, cleaned_price_path, metadata_path, compressed_metadata_path, sticky_state_path, manifest_path)

Parameters

directory (pathlib.Path) –
cleaned_price_path (pathlib.Path) –
metadata_path (pathlib.Path) –
compressed_metadata_path (pathlib.Path) –
sticky_state_path (pathlib.Path) –
manifest_path (pathlib.Path) –

Return type

None

resolve_crypto_vault_paths(data_dir, directory=None)

Resolve crypto bundle paths under one pipeline data directory.

Every private bundle artefact is derived here so cleaning, metadata generation and publication cannot silently choose different locations.

Parameters

data_dir (pathlib.Path) – Root pipeline data directory.
directory (Optional[pathlib.Path]) – Optional explicit crypto bundle directory.

Returns

Explicit bundle paths using backup-safe unique basenames.

Return type

eth_defi.vault.crypto_vaults.CryptoVaultPaths

calculate_crypto_native_admission(vault_db, prices_df)

Calculate hard ETH/BTC admission from native lifetime peak assets.

The helper reads only id and total_assets plus the denomination family from the vault database. Invalid, negative and non-finite asset observations cannot qualify a vault. Threshold comparisons use the float64 representation actually stored in the Parquet, with the Decimal constants converted once at the policy boundary.

Parameters

vault_db (eth_defi.vault.vaultdb.VaultDatabase) – Vault metadata database used to classify each price row.
prices_df (pandas.DataFrame) – Cleaned price rows with id and total_assets columns.

Returns

Native admission summary including qualifying IDs and audit counts.

Raises

ValueError – If the input frame lacks the columns required for native admission.

Return type

eth_defi.vault.crypto_vaults.CryptoNativeAdmission

build_crypto_vault_prices(*, vault_db_path, uncleaned_path, cleaned_path, cleaned_stablecoin_path, cleaned_stablecoin_daily_path=None, settlement_db_path=None)

Create the isolated daily stablecoin/ETH/BTC price Parquet.

Stablecoin rows are derived from the existing standard cleaned Parquet; ETH/BTC rows are cleaned from raw data. The result retains the final valid price observation for each vault and UTC day. It also retains each vault’s first valid observation before its first daily close, preserving the lifetime-return basis and same-day chart eligibility. It does not forward fill the exported rows; the shared lifetime-metrics calculation forward fills only its internal calendar-day series.

Parameters

vault_db_path (pathlib.Path) – Common scanner vault-metadata pickle.
uncleaned_path (pathlib.Path) – Shared raw vault-price Parquet source.
cleaned_path (pathlib.Path) – Isolated daily crypto Parquet destination.
cleaned_stablecoin_path (pathlib.Path) – Existing stablecoin-only cleaned Parquet from the standard cleaner.
cleaned_stablecoin_daily_path (Optional[pathlib.Path]) – Optional daily sidecar written by the standard cleaner while its hourly frame is resident. If absent, the hourly source is read and materialised as before.
settlement_db_path (Optional[pathlib.Path]) – Optional vault-settlement DuckDB database.

Returns

None. Raises if price cleaning cannot safely complete.

Return type

None

Baseline (2026-09-22 production log): the crypto phase reread and materialised 10,116,623 hourly stablecoin rows and took about 1m30s. With the in-memory daily sidecar, the production-shaped bundle builder took 23.36 seconds for 2,409,598 output rows; the primary cleaner’s sidecar write took 5.91 seconds, for 29.27 seconds of integrated extra work. The production inputs were not retained as an immutable before/after pair, so no like-for-like speed-up is claimed. Peak RSS was 7.1 GiB for the builder process; the old phase did not record RSS. The sidecar is used only when its observation-policy version matches and its modification time is at least as new as the hourly source. The mtime comparison is a best-effort stale-file guard for restored artefacts, not a cryptographic generation identifier; the hourly file remains the fallback authority.

build_crypto_vault_record(record, vault_row, threshold_usd)

Convert one common metric record to the crypto bundle schema.

The transformation preserves the observed denomination while replacing mixed-unit rankings and choosing the applicable stablecoin or native qualification policy.

Parameters

record (dict[str, Any]) – Serialised common metric record.
vault_row (eth_defi.vault.vaultdb.VaultRow) – Source vault database row.
threshold_usd (decimal.Decimal) – Fixed USD qualification guideline for stablecoin records. ETH/BTC records use CRYPTO_NATIVE_MIN_PEAK_TOTAL_ASSETS instead.

Returns

Native-unit crypto record and its resolved qualification threshold.

Return type

tuple[dict[str, Any], decimal.Decimal]

build_crypto_usd_conversion_context(vault_db, prices_df, exchange_rate_parquet_path)

Build effective USD rate curves and provenance for crypto metrics.

Currency API values are stored as quote units per USD. The provider label for day D represents the value fetched around 00:00 UTC, so each raw USD→ETH/BTC value becomes an effective USD price for D - 1. Bounded forward fill only bridges short provider gaps; metric calculation later isolates the newest remaining contiguous segment.

Parameters

vault_db (eth_defi.vault.vaultdb.VaultDatabase) – Common vault metadata used to map vault IDs to denomination families.
prices_df (pandas.DataFrame) – Crypto cleaned prices with a naive UTC DatetimeIndex.
exchange_rate_parquet_path (pathlib.Path) – The verified cleaned exchange-rate snapshot used by this export.

Returns

Shared calculation context and JSON-serialisable provenance.

Raises

FileNotFoundError – If the shared exchange-rate snapshot is absent.
ValueError – If its schema does not support a reliable USD conversion.

Return type

tuple[eth_defi.research.vault_metrics.CryptoUSDConversionContext, dict[str, Any]]

build_crypto_vault_metadata(*, vault_db_path, cleaned_price_path, metadata_path, sticky_state_path, exchange_rate_parquet_path=None, threshold_usd=None)

Calculate and atomically persist private crypto-vault metadata.

The function follows the existing top-vaults commit order: metadata JSON is written first and sticky state immediately afterwards. A later R2 failure leaves the local state advanced, ready for the next successful publication.

Parameters

vault_db_path (pathlib.Path) – Common vault metadata pickle.
cleaned_price_path (pathlib.Path) – Crypto daily cleaned Parquet path.
metadata_path (pathlib.Path) – Crypto metadata JSON destination.
sticky_state_path (pathlib.Path) – Crypto sticky-state JSON destination.
exchange_rate_parquet_path (Optional[pathlib.Path]) – Optional verified snapshot used to add USD period metrics to ETH/BTC vaults. Stablecoin records deliberately do not get a duplicate USD metric view.
threshold_usd (Optional[decimal.Decimal]) – Optional fixed USD guideline for stablecoin sticky admission; defaults to environment/config value. ETH/BTC admission is native-unit based.

Returns

JSON-serialisable metadata document.

Return type

dict[str, Any]