tokenised_fund.asseto.backfill

Documentation for eth_defi.tokenised_fund.asseto.backfill Python module.

Backfill historical Asseto vault data into the shared vault pipeline.

This is a targeted production tool for the EVM products currently published by Asseto’s public product registry. It does not rediscover whole chains and updates only the selected Asseto vault identifiers. Historical rows combine on-chain ERC-20 supply with Asseto’s verified Pricer NAV/share where available, or its public daily display-NAV history for other products.

Usage:

source .local-test.env
export JSON_RPC_ETHEREUM="https://your-archive-ethereum-rpc"
PROTOCOLS=asseto poetry run python scripts/backfill-tokenised-funds.py

Useful environment variables:

Variable	Description
`DRY_RUN`	If `true`, print and validate the planned work without database writes.
`NETWORKS`	Optional comma-separated chain ids or known names, e.g. `177,hashkey`.
`JSON_RPC_<CHAIN>`	Archive-capable RPC URL for each selected supported chain, e.g. `JSON_RPC_ETHEREUM`. Products are skipped when this is not set.
`HYPERSYNC_API_KEY`	Required for cached HyperSync block-timestamp reads during price backfill.
`PRODUCTS`	Optional comma-separated Asseto symbols, e.g. `AoABT`.
`ASSETO_SCAN_PRICES`	If `false`, update only metadata. Default: `true`.
`ASSETO_CLEAN_PRICES`	If `true`, replace only selected histories in cleaned prices. Default: `true`.
`MAX_WORKERS`	Historical multicall worker count. Default: `8`.
`FREQUENCY`	Historical price frequency. Asseto supports daily `1d` samples only.
`START_BLOCK` / `END_BLOCK`	Optional global scan range overrides.
`VAULT_DB_PATH` / `UNCLEANED_PRICE_DATABASE`	Optional production database path overrides.
`CLEANED_PRICE_DATABASE` / `READER_STATE_DATABASE`	Optional cleaned price and reader-state path overrides.
`CURRENCY_API_DB_PATH`	Exchange-rate DuckDB used to convert HKD products to USD. Run `scan-currencies` first to populate it.

The backfill removes stale reader state only for selected Asseto vaults. This is required because the targeted scanner replaces those rows from its explicit start block onwards; retained later state would otherwise skip the rewrite. Only chains registered by the shared project registry and supported by HyperSync are included. Asseto products on unsupported chains, such as HashKey or Pharos until they are added to the project, are reported and skipped.

Functions

`backfill_chain`(chain_id, products, *, ...)	Backfill one Asseto EVM chain.
`build_vaults`(web3, products, token_cache)	Build Asseto vault adapters and attach deployment block hints.
`create_asseto_detection`(product)	Create a synthetic shared scanner detection for an Asseto product.
`create_asseto_lead`(product)	Create a synthetic discovery lead for an Asseto product.
`create_runtime_product`(product, ...[, ...])	Convert one public registry entry to a temporary scanner product.
`fetch_contract_deployment_block`(web3, ...)	Find the first block containing runtime code for an Asseto token.
`get_asseto_rpc_env`(chain_id)	Return the normal RPC environment variable for a supported chain.
`get_chain_selector_names`(chain_id)	Return accepted `NETWORKS` selector names for a chain.
`is_supported_asseto_chain`(chain_id)	Check whether a chain is eligible for the shared Asseto backfill.
`iter_selected_products`()	Iterate current eligible Asseto registry products filtered by environment.
`load_usd_exchange_rates`(database_path, ...)	Load historical fiat conversion rates needed by selected products.
`main`()	Run the targeted Asseto historical backfill.
`parse_bool_env`(name, *[, default])	Parse a boolean environment variable.
`parse_csv_env`(name)	Parse a comma-separated environment variable.
`parse_optional_int_env`(name)	Parse an optional integer environment variable.
`parse_path_env`(name, default)	Parse an optional filesystem path environment variable.
`read_asseto_json_rpc_url`(chain_id)	Read the script-local Asseto JSON-RPC URL from its environment variable.
`read_reader_states`(path)	Read trusted local historical reader states.
`read_vault_database`(path)	Read or initialise the vault metadata database.
`resolve_active_asseto_product_ids`(...)	Resolve active products from both registry TVL and on-chain supply.
`resolve_asseto_denomination_symbol`(product)	Resolve the currency in which Asseto publishes a product's NAV.
`resolve_frequency`()	Resolve the daily historical sampling frequency for Asseto.
`resolve_price_scan_start_block`(products)	Resolve the earliest history start for selected Asseto products.
`select_cleanable_vault_ids`(products, rows)	Select active Asseto histories supported by the USD-only cleaner.
`validate_active_asseto_coverage`(chain_id, ...)	Reject a backfill that would leave an active Asseto fund out of live data.
`write_reader_states`(path, states)	Atomically write historical reader state.

parse_bool_env(name, *, default=False)

Parse a boolean environment variable.

Parameters

name (str) – Environment variable name.
default (bool) – Value returned when the variable is unset.

Returns

Parsed boolean value.

Return type

bool

parse_csv_env(name)

Parse a comma-separated environment variable.

Parameters: name (str) – Environment variable name.
Returns: Lowercase values, or None when unset.
Return type: Optional[set[str]]

parse_optional_int_env(name)

Parse an optional integer environment variable.

Parameters: name (str) – Environment variable name.
Returns: Integer value, or None when unset.
Return type: Optional[int]

parse_path_env(name, default)

Parse an optional filesystem path environment variable.

Parameters

name (str) – Environment variable name.
default (pathlib.Path) – Default production path.

Returns

Expanded selected path.

Return type

pathlib.Path

resolve_frequency()

Resolve the daily historical sampling frequency for Asseto.

Asseto’s public NAV history contains daily observations. The scanner must therefore not generate artificial hourly rows by carrying a daily value forward between price publications.

Returns: The required daily sampling interval.
Raises: ValueError – If an hourly or other unsupported frequency is requested.
Return type: Literal[‘1h’, ‘1d’]

get_chain_selector_names(chain_id)

Return accepted NETWORKS selector names for a chain.

Parameters: chain_id (int) – EVM chain id.
Returns: Numeric and configured textual selector values.
Return type: set[str]

is_supported_asseto_chain(chain_id)

Check whether a chain is eligible for the shared Asseto backfill.

Parameters: chain_id (int) – Asseto product chain id.
Returns: True when the project registers the chain and HyperSync supports it.
Return type: bool

get_asseto_rpc_env(chain_id)

Return the normal RPC environment variable for a supported chain.

Parameters: chain_id (int) – Asseto product chain id.
Returns: The project-standard JSON_RPC_<CHAIN> environment variable.
Return type: str

read_asseto_json_rpc_url(chain_id)

Read the script-local Asseto JSON-RPC URL from its environment variable.

Parameters: chain_id (int) – Asseto product chain id.
Returns: Configured archive-capable RPC URL for a supported chain.
Raises: ValueError – If the standard RPC variable is unset.
Return type: str

iter_selected_products()

Iterate current eligible Asseto registry products filtered by environment.

Returns: Unique registry products on supported chains with configured RPC URLs.
Return type: collections.abc.Iterable[eth_defi.tokenised_fund.asseto.offchain_api.AssetoOffchainProduct]

resolve_price_scan_start_block(products)

Resolve the earliest history start for selected Asseto products.

The normal scanner’s incremental reader state must not decide the start of a targeted rewrite. Begin at the earliest Asseto deployment unless the operator supplied START_BLOCK. HyperSync populates any missing block timestamps, so an existing local timestamp cache must never truncate the requested history.

Parameters: products (list[eth_defi.tokenised_fund.asseto.constants.AssetoProduct]) – Selected products on one EVM chain.
Returns: Explicit override or the earliest deployment block.
Return type: int

create_asseto_detection(product)

Create a synthetic shared scanner detection for an Asseto product.

Parameters: product (eth_defi.tokenised_fund.asseto.constants.AssetoProduct) – Asseto product metadata.
Returns: Detection compatible with metadata scan record generation.
Return type: eth_defi.erc_4626.core.ERC4262VaultDetection

create_asseto_lead(product)

Create a synthetic discovery lead for an Asseto product.

Parameters: product (eth_defi.tokenised_fund.asseto.constants.AssetoProduct) – Asseto product metadata.
Returns: Lead compatible with the vault metadata database.
Return type: eth_defi.erc_4626.discovery_base.PotentialVaultMatch

read_vault_database(path)

Read or initialise the vault metadata database.

Parameters: path (pathlib.Path) – Vault metadata database path.
Returns: Existing or empty database.
Return type: eth_defi.vault.vaultdb.VaultDatabase

read_reader_states(path)

Read trusted local historical reader states.

Parameters: path (pathlib.Path) – Reader-state pickle file path.
Returns: Existing states, or an empty mapping when the file does not exist.
Return type: dict[eth_defi.vault.base.VaultSpec, dict]

write_reader_states(path, states)

Atomically write historical reader state.

Parameters

path (pathlib.Path) – Reader-state pickle file path.
states (dict[eth_defi.vault.base.VaultSpec, dict]) – Reader state mapping returned by the historical scanner.

Return type

None

build_vaults(web3, products, token_cache)

Build Asseto vault adapters and attach deployment block hints.

Parameters

web3 (web3.main.Web3) – Connected chain Web3 instance.
products (list[eth_defi.tokenised_fund.asseto.constants.AssetoProduct]) – Selected products on this chain.
token_cache (eth_defi.token.TokenDiskCache) – Shared token metadata cache.

Returns

Configured Asseto vault adapters.

Return type

list[eth_defi.vault.base.VaultBase]

select_cleanable_vault_ids(products, rows)

Select active Asseto histories supported by the USD-only cleaner.

HKD histories are converted to USD by the adapter before reaching this point. Inactive zero-supply products remain mapped in metadata and may keep raw observations, but are not passed to the strict replacement helper because normal cleaning intentionally removes every zero-NAV row.

Parameters

products (collections.abc.Iterable[eth_defi.tokenised_fund.asseto.constants.AssetoProduct]) – Asseto products whose raw histories were scanned.
rows (dict[eth_defi.vault.base.VaultSpec, dict]) – Fresh vault metadata rows keyed by vault specification.

Returns

Canonical vault identifiers eligible for cleaned-history replacement.

Return type

set[str]

resolve_active_asseto_product_ids(registry_products, runtime_products, rows)

Resolve active products from both registry TVL and on-chain supply.

Using both sources prevents a missing or stale registry TVL field from hiding a positive-supply product from live-feed coverage validation.

Parameters

registry_products (collections.abc.Iterable[eth_defi.tokenised_fund.asseto.offchain_api.AssetoOffchainProduct]) – Current public Asseto registry entries.
runtime_products (collections.abc.Iterable[eth_defi.tokenised_fund.asseto.constants.AssetoProduct]) – Corresponding products with on-chain deployment metadata.
rows (dict[eth_defi.vault.base.VaultSpec, dict]) – Fresh current vault metadata rows.

Returns

Lower-case token addresses considered active.

Return type

set[str]

validate_active_asseto_coverage(chain_id, active_product_ids, runtime_products, rows, price_history_ids)

Reject a backfill that would leave an active Asseto fund out of live data.

Parameters

chain_id (int) – EVM chain being validated.
active_product_ids (set[str]) – Lower-case active token addresses.
runtime_products (collections.abc.Iterable[eth_defi.tokenised_fund.asseto.constants.AssetoProduct]) – Products represented by the current metadata scan.
rows (dict[eth_defi.vault.base.VaultSpec, dict]) – Fresh metadata rows.
price_history_ids (set[str]) – Lower-case addresses whose adapters found NAV history.

Raises

RuntimeError – If an active product lacks price history or positive USD-compatible live metadata.

Return type

None

resolve_asseto_denomination_symbol(product)

Resolve the currency in which Asseto publishes a product’s NAV.

Asseto omits tokenSymbol for its stoken products even though their product pages and NAV series use United States dollars. Treat this registry category as synthetic USD accounting, while retaining explicit symbols for UDA products such as USDC, USDT and HKD.

Parameters: product (eth_defi.tokenised_fund.asseto.offchain_api.AssetoOffchainProduct) – Public Asseto registry entry.
Returns: Upper-case accounting denomination, or None if Asseto publishes neither a symbol nor a recognised product category.
Return type: Optional[str]

load_usd_exchange_rates(database_path, denomination_symbols)

Load historical fiat conversion rates needed by selected products.

Stored rates are units of quote currency per one USD. Asseto NAV values in a non-USD currency are consequently divided by the matching rate before they enter the shared USD-denominated cleaned history.

Parameters

database_path (pathlib.Path) – Currency API DuckDB produced by scan-currencies.
denomination_symbols (collections.abc.Iterable[Optional[str]]) – Asseto accounting currencies selected for this run.

Returns

Rates keyed by upper-case quote currency, each ordered by UTC day.

Raises

RuntimeError – If a required database or currency history is missing.

Return type

dict[str, tuple[tuple[int, decimal.Decimal], …]]

fetch_contract_deployment_block(web3, address, end_block)

Find the first block containing runtime code for an Asseto token.

Parameters

web3 (web3.main.Web3) – Archive-capable connection for the product chain.
address (eth_typing.evm.HexAddress) – Asseto ERC-20 token address.
end_block (int) – Highest block that may be checked.

Returns

First block containing contract code.

Raises

ValueError – If the public registry address has no contract code on this chain.

Return type

int

create_runtime_product(product, deployment_block, first_seen_at, usd_exchange_rates=())

Convert one public registry entry to a temporary scanner product.

Public registry products do not all expose Asseto’s request/claim manager and Pricer contracts. The generic adapter therefore uses their published daily NAV history, while preserving the exact token identity and deployment information read from the configured archive RPC.

Parameters

product (eth_defi.tokenised_fund.asseto.offchain_api.AssetoOffchainProduct) – Asseto public EVM product entry.
deployment_block (int) – First token-code block found on-chain.
first_seen_at (datetime.datetime) – Naive UTC deployment timestamp.
usd_exchange_rates (tuple[tuple[int, decimal.Decimal], ...]) – Historical units of the product denomination per USD. Empty for products already denominated in USD or a USD stablecoin.

Returns

Runtime Asseto adapter product metadata.

Return type

eth_defi.tokenised_fund.asseto.constants.AssetoProduct

backfill_chain(chain_id, products, *, dry_run, scan_prices, clean_prices, frequency, vault_db, vault_db_path, price_database_path, cleaned_price_database_path, reader_state_database_path, token_cache, usd_exchange_rates_by_symbol)

Backfill one Asseto EVM chain.

Metadata and historical writes use the same shared database and scanner paths as the normal vault pipeline, but both are limited to selected Asseto identifiers.

Parameters

chain_id (int) – EVM chain id.
products (list[eth_defi.tokenised_fund.asseto.offchain_api.AssetoOffchainProduct]) – Selected Asseto products on this chain.
dry_run (bool) – Whether filesystem/database writes are disabled.
scan_prices (bool) – Whether to scan supply, NAV, and TVL history.
clean_prices (bool) – Whether to replace selected cleaned price histories.
frequency (Literal['1h', '1d']) – Historical sampling interval.
vault_db (eth_defi.vault.vaultdb.VaultDatabase) – In-memory metadata database.
vault_db_path (pathlib.Path) – Metadata database output path.
price_database_path (pathlib.Path) – Uncleaned historical price parquet path.
cleaned_price_database_path (pathlib.Path) – Cleaned historical price parquet path.
reader_state_database_path (pathlib.Path) – Reader-state pickle path.
token_cache (eth_defi.token.TokenDiskCache) – Shared token metadata cache.
usd_exchange_rates_by_symbol (dict[str, tuple[tuple[int, decimal.Decimal], ...]]) – Historical fiat conversion rates for non-USD product denominations.

Returns

Summary row for operator output.

Return type

dict[str, object]

main()

Run the targeted Asseto historical backfill.

Return type: None