vault.data_file_export

Documentation for eth_defi.vault.data_file_export Python module.

Export vault database files to Cloudflare R2.

Uploads price databases, metadata pickles, Core3 risk intelligence and exchange-rate files to the private vault-data bucket. Public R2 uploads are limited to explicitly generated sample files.

Functions

`get_data_file_paths`(base_path[, ...])	Build the data file list uploaded to R2.
`main`([exchange_rate_parquet_path, ...])	Run the data file export script.
`publish_exchange_rate_parquet_to_alternative_bucket`(...)	Publish one verified exchange-rate Parquet to the private data bucket.
`resolve_exchange_rate_database_path`([base_path])	Resolve the exchange-rate DuckDB database path.
`resolve_exchange_rate_parquet_path`(base_path)	Resolve the cleaned exchange-rate Parquet destination.
`upload_files_to_r2`(file_paths, bucket_name, ...)	Upload a list of files to R2 bucket, excluding tmp* files.

resolve_exchange_rate_database_path(base_path=None)

Resolve the exchange-rate DuckDB database path.

The exchange-rate bundle is produced by the currency API scanner. Use the same CURRENCY_API_DB_PATH / CURRENCY_API_DATABASE_PATH override contract as the all-chain scanner so exports upload the database that the scanner writes. Without an override, data exports use exchange-rates.duckdb under the active pipeline data directory.

Parameters: base_path (Optional[pathlib.Path]) – Pipeline data directory. None falls back to the standalone currency API default path.
Returns: Path to the exchange-rate DuckDB database.
Return type: pathlib.Path

resolve_exchange_rate_parquet_path(base_path)

Resolve the cleaned exchange-rate Parquet destination.

The currency API DuckDB database may be kept outside pipeline state through CURRENCY_API_DB_PATH. The flat exported Parquet always belongs below the selected pipeline data directory so its R2 key stays stable.

Parameters: base_path (pathlib.Path) – Active pipeline data directory.
Returns: Destination path for exchange-rates.parquet.
Return type: pathlib.Path

get_data_file_paths(base_path, core3_db_path=None, exchange_rate_db_path=None, xerberus_db_path=None, *, exchange_rate_parquet_path=None, include_exchange_rate_parquet=True)

Build the data file list uploaded to R2.

Parameters

base_path (pathlib.Path) – Pipeline data directory.
core3_db_path (Optional[pathlib.Path]) – Optional Core3 DuckDB path override.
exchange_rate_db_path (Optional[pathlib.Path]) – Optional exchange-rate DuckDB path override.
xerberus_db_path (Optional[pathlib.Path]) – Optional Xerberus DuckDB path override.
exchange_rate_parquet_path (Optional[pathlib.Path]) – Optional already materialised cleaned exchange-rate Parquet path.
include_exchange_rate_parquet (bool) – Include the cleaned Parquet when its materialisation succeeded for this run. Set false to avoid re-uploading a stale prior file after a failure.

Returns

Files to upload, including optional files that may be skipped later if they do not exist.

Return type

list[pathlib.Path]

upload_files_to_r2(file_paths, bucket_name, endpoint_url, access_key_id, secret_access_key, key_prefix='')

Upload a list of files to R2 bucket, excluding tmp* files.

Parameters

file_paths (list[pathlib.Path]) – List of file paths to upload.
bucket_name (str) – R2 bucket name.
endpoint_url (str) – R2 API endpoint URL.
access_key_id (str) – R2 access key ID.
secret_access_key (str) – R2 secret access key.
key_prefix (str) – Prefix for S3 keys, e.g. test- for test uploads.

Returns

Number of files uploaded.

Return type

int

publish_exchange_rate_parquet_to_alternative_bucket(parquet_path)

Publish one verified exchange-rate Parquet to the private data bucket.

The focused crypto exporter uses this before its metadata bundle is made current. It shares the flat key, upload prefix and daily-backup convention with the complete data-file export without uploading unrelated vault files.

Parameters

parquet_path (pathlib.Path) – Existing verified exchange-rates.parquet snapshot.

Returns

None after upload and optional daily backup.

Raises

FileNotFoundError – If the supplied local snapshot does not exist.
AssertionError – If private R2 configuration is incomplete.

Return type

None

main(exchange_rate_parquet_path=None, exchange_rate_parquet_error=None)

Run the data file export script.

Reads R2 configuration from environment variables, uploads vault data files to the private bucket, and creates daily backups there when enabled. A scheduled caller can supply the exact Parquet snapshot already used for crypto USD metrics. Standalone use materialises a fresh snapshot before upload.

Logging is configured by the standalone script, not here: the scanner calls this in-process, and reconfiguring would remove its log file handler.

Parameters

exchange_rate_parquet_path (Optional[pathlib.Path]) – Optional verified Parquet snapshot created earlier in this scan cycle.
exchange_rate_parquet_error (Optional[Exception]) – Earlier materialisation failure. Unrelated files still upload, after which this error is propagated and stale Parquet is omitted.

Returns

Per-database backup statuses, reported separately from successful price publication.

Return type

dict[str, str]