tokenised_fund.securitize.backfill

Documentation for eth_defi.tokenised_fund.securitize.backfill Python module.

Backfill all reviewed Securitize leads and supported price history.

This migration preserves unrelated vault database rows, reader states and Parquet histories. It upserts the reviewed products in SECURITIZE_PRODUCTS and rewrites price history only for products with an explicit adapter estimate or reviewed on-chain RedStone NAV feed. Products without an authoritative NAV source are registered as leads but excluded from the price scan.

Important operational note: RedStone push feeds do not necessarily contain the oldest available Securitize NAV observations.

This script intentionally uses reviewed on-chain RedStone push feeds for variable-NAV products because archive-state calls to latestRoundData() are reproducible. They can be re-run months later from any archive node and do not depend on RedStone’s short-lived HTTP cache. See the RedStone feed catalogue:

https://app.redstone.finance/app/feeds/

However, the push-feed deployment date can be later than the first NAV value RedStone collected off-chain from Securitize. The BCAP integration is the canonical example and should be treated as a warning for future Securitize products:

  • Product: Blockchain Capital III Digital Liquid Venture Fund (BCAP).

  • DSToken: 0x1f41e42d0a9e3c0dd3ba15b527342783b43200a9.

  • RedStone feed id: BCAP_FUNDAMENTAL.

  • On-chain push feed: 0x46f1b5f29a2dc1a730508a1b41a8b5b93e316eb2.

  • Feed contract first has code at Ethereum block 25_493_848 (2026-07-09 08:32:11 UTC).

  • latestRoundData() first returns a valid value at block 25_494_164 (2026-07-09 09:35:35 UTC), value 106.63.

  • eth_getLogs from deployment onward only returned setup/proxy logs (Upgraded(address), Initialized(uint8) and one admin/setup event) during the 2026-07-17 investigation. There were no Chainlink-style AnswerUpdated or NewRound NAV events to replay.

  • Before block 25_494_164 archive calls either see no feed contract code or latestRoundData() reverts, so this script cannot reconstruct older BCAP NAV from Ethereum chain state alone.

At the time of the 2026-07-17 investigation, RedStone’s public HTTP cache did contain older BCAP_FUNDAMENTAL observations sourced from securitize-api and securitize-tsso. Example endpoint:

https://api.redstone.finance/prices?symbol=BCAP_FUNDAMENTAL&provider=redstone

Observed older cache points included:

  • 2026-06-17 12:59 UTC: 107.07.

  • 2026-06-22 23:59 UTC: 107.03.

  • 2026-06-29 23:59 UTC: 106.97.

  • 2026-07-06 23:59 UTC: 106.63.

The same HTTP API rejected older requests with {"error":"Data older than 30 days is not available"}, which is why the production backfill in this script does not depend on it. This means a newly registered Securitize feed may have a one-time recoverable pre-push-feed REST window that will disappear if it is not imported quickly. For BCAP, a pure PROTOCOLS=securitize scripts/backfill-tokenised-funds.py run starts around 2026-07-09/10 because the script only trusts the archive-readable push feed.

If future operators need maximum history for a new Securitize RedStone feed, run a separate, explicitly documented one-off importer for the RedStone REST *_FUNDAMENTAL symbol before the 30-day cache expires, then let this script take over from the push feed’s first valid block. Do not silently extend this script to depend on REST history for normal production backfills unless the retention, failure and provenance semantics are made explicit. The PRs that introduced the current behaviour are:

https://github.com/tradingstrategy-ai/web3-ethereum-defi/pull/1299 https://github.com/tradingstrategy-ai/web3-ethereum-defi/pull/1306

Run with:

source .local-test.env && PROTOCOLS=securitize poetry run python scripts/backfill-tokenised-funds.py

Environment variables:

DRY_RUN

Show the plan without writing data. Default: false.

SECURITIZE_SCAN_PRICES

Scan DSTokens with a reviewed estimated or external NAV source. Default: true.

SECURITIZE_PRODUCTS

Optional comma-separated DSToken addresses to backfill.

SECURITIZE_CLEAN_PRICES

Rebuild selected cleaned histories after the raw scan. Default: true.

FREQUENCY

Historical sample interval, 1h or 1d. Default: 1d.

START_BLOCK / END_BLOCK

Optional inclusive block boundaries for a carefully scoped run.

MAX_WORKERS

Historical multicall worker count. Default: 8.

VAULT_DB_PATH, UNCLEANED_PRICE_DATABASE, CLEANED_PRICE_DATABASE and READER_STATE_DATABASE

Optional paths for isolated tests or production-data overrides.

The script reads archive-capable RPC URLs from JSON_RPC_<CHAIN_NAME>.

Functions

backfill_chain(chain_id, products, *, ...)

Upsert one chain's DSToken leads and rewrite only priced histories.

build_priced_vaults(web3, products, token_cache)

Build adapters only for DSTokens with a configured historical NAV.

create_detection(product, deployment_block, ...)

Create a scanner-compatible detection for a reviewed DSToken.

create_lead(product, deployment_block, ...)

Create a lead row without discovering the whole chain again.

create_price_row_report(products, ...)

Create and validate the per-product historical row report.

fetch_contract_deployment_block(web3, ...)

Find the first block where an address has deployed contract code.

fetch_product_first_seen_at(web3, ...)

Read the naive UTC timestamp for a product deployment block.

has_historical_price(product)

Check whether the scanner can produce price history for a product.

iter_products()

Iterate every reviewed Securitize product exactly once.

main()

Run the targeted Securitize lead and price-history migration.

parse_bool_env(name, *[, default])

Parse a conventional boolean environment setting.

parse_optional_int_env(name)

Read an optional integer block boundary from the environment.

parse_path_env(name, default)

Read a filesystem override while retaining production defaults.

read_reader_states(path)

Load saved price-reader states from a trusted local pickle.

read_vault_database(path)

Load an existing metadata database without replacing its entries.

resolve_frequency()

Resolve the requested historical sampling frequency.

resolve_price_scan_start_block(deployment_blocks)

Choose the earliest history start block for selected products.

write_reader_states(path, states)

Persist reader state atomically after the scoped history scan.
parse_bool_env(name, *, default=False)

Parse a conventional boolean environment setting.

The migration is operated through environment variables so it can run from the same production wrapper as other vault scripts without another command line parser.

Parameters

  • name (str) – Environment variable to parse.

  • default (bool) – Value used when the variable is absent.

Returns

Parsed boolean value.

Return type

bool

parse_optional_int_env(name)

Read an optional integer block boundary from the environment.

Operators can use this only to narrow a controlled rerun. The normal path discovers the first contract-code block automatically.

Parameters

name (str) – Environment variable to parse.

Returns

Parsed integer, or None when it is unset.

Return type

Optional[int]

parse_path_env(name, default)

Read a filesystem override while retaining production defaults.

Isolated test runs use this to keep state files out of the normal vault pipeline directory.

Parameters

  • name (str) – Environment variable holding a path.

  • default (pathlib.Path) – Default production path.

Returns

Expanded configured path.

Return type

pathlib.Path

resolve_frequency()

Resolve the requested historical sampling frequency.

The execution plan and the price scanner use the same validated value so a displayed daily run cannot silently perform hourly work.

Returns

One-hour or one-day scan interval.

Raises

ValueError – If the requested interval is unsupported.

Return type

Literal[‘1h’, ‘1d’]

iter_products()

Iterate every reviewed Securitize product exactly once.

The registry is keyed for direct chain-and-address lookup, so guard against an accidental duplicate product when future aliases are added.

Returns

Unique registry products.

Return type

collections.abc.Iterable[eth_defi.tokenised_fund.securitize.description.SecuritizeProduct]

has_historical_price(product)

Check whether the scanner can produce price history for a product.

Descriptive source labels are not executable configuration. A product is price-capable only when it has a reviewed static estimate or an actual RedStone feed registry entry.

Parameters

product (eth_defi.tokenised_fund.securitize.description.SecuritizeProduct) – Reviewed Securitize product.

Returns

True when the historical reader can calculate NAV and TVL.

Return type

bool

create_price_row_report(products, scan_result, *, scan_enabled)

Create and validate the per-product historical row report.

Counts come from the scanner’s existing in-memory export pass. The report therefore does not reread the output Parquet file. A completed scan must produce at least one non-null share-price row for every product whose static estimate or RedStone feed is available in the scanned block range.

Parameters
Returns

Tabular rows covering priced and metadata-only products.

Raises

RuntimeError – If an expected priced product emitted no non-null share-price rows.

Return type

list[dict[str, object]]

fetch_contract_deployment_block(web3, address, end_block)

Find the first block where an address has deployed contract code.

The reviewed DSToken registry does not duplicate a manually maintained creation-block list. An archive-node binary search yields the first code block in logarithmic RPC calls and remains correct when a new product is added to the registry.

Parameters

  • web3 (web3.main.Web3) – Archive-capable connection for the product chain.

  • address (eth_typing.evm.HexAddress) – DSToken proxy address.

  • end_block (int) – Highest block that may be inspected.

Returns

First block with non-empty runtime code.

Raises

ValueError – If the address has no contract code at end_block.

Return type

int

fetch_product_first_seen_at(web3, deployment_block)

Read the naive UTC timestamp for a product deployment block.

Shared lead records carry both the discovery block and its timestamp. The timestamp comes from the same archive endpoint used to locate deployment.

Parameters

  • web3 (web3.main.Web3) – Archive-capable connection for the product chain.

  • deployment_block (int) – First block with the product’s runtime code.

Returns

Naive UTC deployment timestamp.

Return type

datetime.datetime

create_detection(product, deployment_block, first_seen_at)

Create a scanner-compatible detection for a reviewed DSToken.

A migration uses the product registry as its authoritative lead source, rather than reprocessing unrelated discovery events from the chain.

Parameters
Returns

Detection record accepted by metadata extraction.

Return type

eth_defi.erc_4626.core.ERC4262VaultDetection

create_lead(product, deployment_block, first_seen_at)

Create a lead row without discovering the whole chain again.

The lead carries the same initial position as its synthetic detection so downstream metadata and history readers retain the expected ordering.

Parameters
Returns

Lead record for the shared vault database.

Return type

eth_defi.erc_4626.discovery_base.PotentialVaultMatch

read_vault_database(path)

Load an existing metadata database without replacing its entries.

A missing path starts an empty database for isolated use. Existing files are read intact and later updated only for selected Securitize rows.

Parameters

path (pathlib.Path) – Metadata database pickle path.

Returns

Existing or empty vault database.

Return type

eth_defi.vault.vaultdb.VaultDatabase

read_reader_states(path)

Load saved price-reader states from a trusted local pickle.

Only selected Securitize states are removed before their histories are rewritten; all unrelated states are retained.

Parameters

path (pathlib.Path) – Reader-state pickle path.

Returns

Existing states, or an empty mapping when the file is absent.

Return type

dict[eth_defi.vault.base.VaultSpec, dict]

write_reader_states(path, states)

Persist reader state atomically after the scoped history scan.

Atomic replacement ensures that an interrupted process cannot leave a partially serialised reader-state file.

Parameters
Returns

None.

Return type

None

resolve_price_scan_start_block(deployment_blocks)

Choose the earliest history start block for selected products.

A targeted rewrite must begin at deployment, not at the ordinary scanner’s latest cursor. HyperSync fills missing block timestamps, so a partial local timestamp cache must never truncate the requested history.

Parameters

deployment_blocks (collections.abc.Iterable[int]) – Deployment blocks for products with a configured price source.

Returns

Explicit override or the earliest deployment block.

Return type

int

build_priced_vaults(web3, products, token_cache)

Build adapters only for DSTokens with a configured historical NAV.

Unpriced DSTokens must remain leads and metadata rows, but their history cannot be inferred from supply alone and therefore must not rewrite price parquet rows.

Parameters
Returns

Vault adapters with their individual history start blocks.

Return type

list[eth_defi.vault.base.VaultBase]

backfill_chain(chain_id, products, *, dry_run, scan_prices, clean_prices, frequency, vault_db, vault_db_path, price_database_path, cleaned_price_database_path, reader_state_database_path, token_cache)

Upsert one chain’s DSToken leads and rewrite only priced histories.

Product deployments are derived independently for the selected addresses. The shared metadata database receives only their rows, and the historical scanner receives only priced address filters.

Parameters
Returns

Tabular migration summary for the chain.

Return type

dict[str, object]

main()

Run the targeted Securitize lead and price-history migration.

The full registry is grouped by chain so future Securitize deployments can be added without expanding this operational entry point.

Returns

None.

Return type

None