vault.scan_manifest

Documentation for eth_defi.vault.scan_manifest Python module.

Publish the small vault scan receipt consumed by live readiness checks.

The scanner writes the manifest only after the cleaned price parquet has been uploaded. It contains per-chain scan provenance and the maximum timestamp from that exact cleaned snapshot, allowing a live executor to poll a small JSON file without repeatedly downloading hundreds of megabytes of parquet data.

The object is published to the existing private alternative-vault bucket named by R2_ALTERNATIVE_VAULT_METADATA_BUCKET_NAME. Its key is {UPLOAD_PREFIX}vault-scan-manifest.json alongside the cleaned price object; the upload explicitly sets application/json and Cache-Control: no-store. The serving worker and CDN must also honour that policy; R2 metadata alone cannot prove that a readiness poll will bypass every intermediate cache.

See Cloudflare R2 object metadata for the S3-compatible HEAD and upload operations used here. Operational details and the consumer contract are documented in docs/README-vault-scan-manifest.md.

Functions

build_vault_scan_manifest(...)	Build a v1 manifest from the exact cleaned parquet being published.
publish_hypercore_permission_manifest(...)	Publish an opt-in v2 manifest binding immutable prices and permissions.
publish_vault_scan_manifest(...[, published_at])	Upload the manifest to the private vault-data R2 bucket.

Classes

VaultChainScanManifest	Separate successful collection from the newest published observation.
VaultPriceFileManifest	Bind freshness claims to the private price object the consumer downloads.
VaultScanManifest	Wire contract polled before a live strategy downloads its universe.

class VaultPriceFileManifest

Bases: TypedDict

Bind freshness claims to the private price object the consumer downloads.

Filled by build_vault_scan_manifest() from the publisher’s R2 HEAD response. Consumers compare the opaque ETag, including multipart suffixes, with the download response; it is not a content hash they must recompute.

key: str

Exact private R2 object key, retained for audit only.

etag: str

Strong opaque ETag without HTTP quote characters.

__init__(*args, **kwargs)

__new__(**kwargs)

clear()

Remove all items from the dict.

copy()

Return a shallow copy of the dict.

fromkeys(value=None, /)

Create a new dictionary with keys from iterable and values set to value.

get(key, default=None, /)

Return the value for key if key is in the dictionary, else default.

items()

Return a set-like object providing a view on the dict’s items.

keys()

Return a set-like object providing a view on the dict’s keys.

pop(k[, d]) → v, remove specified key and return the corresponding value.

If the key is not found, return the default if given; otherwise, raise a KeyError.

popitem()

Remove and return a (key, value) pair as a 2-tuple.

Pairs are returned in LIFO (last-in, first-out) order. Raises KeyError if the dict is empty.

setdefault(key, default=None, /)

Insert key with a value of default if key is not in the dictionary.

Return the value for key if key is in the dictionary, else default.

update([E, ]**F) → None.  Update D from mapping/iterable E and F.

If E is present and has a .keys() method, then does: for k in E.keys(): D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

values()

Return an object providing a view on the dict’s values.

class VaultChainScanManifest

Bases: TypedDict

Separate successful collection from the newest published observation.

The builder combines scanner provenance with cleaned parquet maxima. A metadata-only scan cannot advance collection time, and a completed price scan need not have received new candles. Both fields are UTC ISO-8601 strings ending in Z, or None when unknown.

name: str

Human-readable label; consumers select chains by numeric key.

last_successful_price_scan_ended_at: Optional[str]

Completion time of the last successful price fetch, or None if unknown.

last_candle_at: Optional[str]

Maximum timestamp in the published cleaned file, including hourly bucket labels from sparse observations. This is not per-vault completeness.

__init__(*args, **kwargs)

__new__(**kwargs)

clear()

Remove all items from the dict.

copy()

Return a shallow copy of the dict.

fromkeys(value=None, /)

Create a new dictionary with keys from iterable and values set to value.

get(key, default=None, /)

Return the value for key if key is in the dictionary, else default.

items()

Return a set-like object providing a view on the dict’s items.

keys()

Return a set-like object providing a view on the dict’s keys.

pop(k[, d]) → v, remove specified key and return the corresponding value.

If the key is not found, return the default if given; otherwise, raise a KeyError.

popitem()

Remove and return a (key, value) pair as a 2-tuple.

Pairs are returned in LIFO (last-in, first-out) order. Raises KeyError if the dict is empty.

setdefault(key, default=None, /)

Insert key with a value of default if key is not in the dictionary.

Return the value for key if key is in the dictionary, else default.

update([E, ]**F) → None.  Update D from mapping/iterable E and F.

If E is present and has a .keys() method, then does: for k in E.keys(): D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

values()

Return an object providing a view on the dict’s values.

class VaultScanManifest

Bases: TypedDict

Wire contract polled before a live strategy downloads its universe.

publish_vault_scan_manifest() serialises this mapping as JSON after private price export. The matching consumer is tradingstrategy.vault_scan_manifest.VaultScanManifest. Chain-wide freshness is a scheduling hint, not a guarantee that every vault has data.

schema_version: Literal[1]

Wire schema version; the consumer currently supports version 1 only.

published_at: str

UTC publication time after the referenced price upload.

price_file: eth_defi.vault.scan_manifest.VaultPriceFileManifest

Identity and source version of the referenced cleaned price object.

chains: dict[str, eth_defi.vault.scan_manifest.VaultChainScanManifest]

Decimal chain-ID keys and their published freshness metadata.

__init__(*args, **kwargs)

__new__(**kwargs)

clear()

Remove all items from the dict.

copy()

Return a shallow copy of the dict.

fromkeys(value=None, /)

Create a new dictionary with keys from iterable and values set to value.

get(key, default=None, /)

Return the value for key if key is in the dictionary, else default.

items()

Return a set-like object providing a view on the dict’s items.

keys()

Return a set-like object providing a view on the dict’s keys.

pop(k[, d]) → v, remove specified key and return the corresponding value.

If the key is not found, return the default if given; otherwise, raise a KeyError.

popitem()

Remove and return a (key, value) pair as a 2-tuple.

Pairs are returned in LIFO (last-in, first-out) order. Raises KeyError if the dict is empty.

setdefault(key, default=None, /)

Insert key with a value of default if key is not in the dictionary.

Return the value for key if key is in the dictionary, else default.

update([E, ]**F) → None.  Update D from mapping/iterable E and F.

If E is present and has a .keys() method, then does: for k in E.keys(): D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

values()

Return an object providing a view on the dict’s values.

build_vault_scan_manifest(cleaned_price_path, price_scan_state_path, price_object_key, price_etag, published_at)

Build a v1 manifest from the exact cleaned parquet being published.

Called by publish_vault_scan_manifest() after private price export. Only chain (numeric chain ID) and timestamp (Arrow timestamp) are loaded; aggregation stays in Arrow to avoid materialising millions of historical rows as Python objects. Sparse four-hour observations remain valid inputs: no artificial hourly completeness requirement is applied.

Parameters

• cleaned_price_path (pathlib.Path) – Cleaned parquet snapshot that was uploaded before this call.

• price_scan_state_path (pathlib.Path) – Local price-only provenance state written after successful scans.

• price_object_key (str) – R2 key of the uploaded parquet.

• price_etag (str) – Strong ETag read from the uploaded object.

• published_at (datetime.datetime) – UTC time immediately before manifest publication.

Returns

JSON-serialisable manifest mapping.

Return type

eth_defi.vault.scan_manifest.VaultScanManifest

publish_vault_scan_manifest(cleaned_price_path, price_scan_state_path, *, published_at=None)

Upload the manifest to the private vault-data R2 bucket.

The price object must already be uploaded. The manifest uses no-store so the authenticated JSON endpoint can be polled without a stale edge or client cache. Existing unrelated data-file cache policy is unchanged. Called by eth_defi.vault.post_processing.run_post_processing() after successful export, under the scanner’s existing single-writer lock.

Parameters

• cleaned_price_path (pathlib.Path) – Exact cleaned parquet snapshot that was uploaded.

• price_scan_state_path (pathlib.Path) – Local price-only scan provenance written by the scanner.

• published_at (Optional[datetime.datetime]) – Optional publication timestamp; defaults to the current UTC time.

Returns

True after publication, False when the local price file is absent. Missing remote objects or configuration raise an error.

Return type

bool

publish_hypercore_permission_manifest(cleaned_price_path, permission_path, price_scan_state_path)

Publish an opt-in v2 manifest binding immutable prices and permissions.

This separate manifest leaves v1 clients on their existing endpoint. Enable HYPERCORE_PERMISSION_MANIFEST_V2=true only when the authenticated endpoint, client and executor all consume coherent sidecar snapshots and original receipt clocks. Content-addressed private copies prevent mixed generations.

Parameters

• cleaned_price_path (pathlib.Path) – Exact uploaded cleaned price snapshot.

• permission_path (pathlib.Path) – Independently clocked permission Parquet.

• price_scan_state_path (pathlib.Path) – Price collection provenance.

Returns

True after successful publication, False when disabled.

Return type

bool