currency_api.scanner

Documentation for eth_defi.currency_api.scanner Python module.

Incremental, gap-aware exchange rate scanner.

Drives fetch_rates_for_date() over a date window, fetching dates in parallel (threaded) and writing to a single shared DuckDB connection. Resume is completeness-driven: it computes the missing (date, quote) cells from the stored data and the gap table, so transient holes and newly added quote currencies are always backfilled.

Example:

from eth_defi.currency_api.scanner import run_incremental_scan
from eth_defi.currency_api.constants import CURRENCY_API_DATABASE

result = run_incremental_scan(db_path=CURRENCY_API_DATABASE)
print(result.db.row_count())
result.db.close()

Functions

run_incremental_scan(db_path[, ...])

Incrementally populate exchange rates into DuckDB.

Classes

ScanResult

Outcome of an incremental scan.
class ScanResult

Bases: object

Outcome of an incremental scan.

Variables

  • db – The (still open) database. The caller is responsible for closing it.

  • dates_requested – Number of distinct dates fetched over HTTP this run.

  • rows_upserted – Number of (date, quote) rate rows written.

  • quotes_recorded – Individually missing quotes recorded as permanently unavailable.

  • quotes_pending – Individually missing quotes left pending (within grace, retried later).

  • dates_unavailable – Whole dates recorded as permanently unavailable (404 on both hosts).

  • dates_pending – Whole dates that 404’d but are within grace (retried later).

  • transient_failures – Dates that failed transiently this run and will be retried; the run should exit non-zero.

  • dates_given_up – Dates that exceeded max_transient_attempts consecutive transient failures and were recorded as permanent persistent_error gaps.

db: eth_defi.currency_api.database.CurrencyRateDatabase

The still-open database.

dates_requested: int

Distinct dates fetched over HTTP.

rows_upserted: int

Rate rows written.

quotes_recorded: int

Missing quotes recorded as permanently unavailable.

quotes_pending: int

Missing quotes left pending within grace.

dates_unavailable: int

Whole dates recorded as unavailable.

dates_pending: int

Whole dates pending within grace.

transient_failures: int

Transient failures requiring retry / non-zero exit.

dates_given_up: int

Dates given up on after exceeding the transient-attempt budget.

__init__(db, dates_requested=0, rows_upserted=0, quotes_recorded=0, quotes_pending=0, dates_unavailable=0, dates_pending=0, transient_failures=0, dates_given_up=0)
Parameters
Return type

None

run_incremental_scan(db_path, base_currency='usd', quote_currencies=('eur', 'gbp', 'jpy', 'aud', 'sgd', 'hkd', 'try', 'chf', 'cad', 'btc', 'eth'), start_date=None, end_date=None, source='fawazahmed0', max_workers=8, refetch_tail_days=3, unavailable_grace_days=2, max_transient_attempts=5, session=None)

Incrementally populate exchange rates into DuckDB.

Computes the missing (date, quote) work set from the stored data and the gap table, fetches the needed dates in parallel, and upserts the results. Permanent gaps are recorded; transient failures are left for retry and surfaced in the returned ScanResult.

Parameters

  • db_path (pathlib.Path) – DuckDB file path. Created if it does not exist.

  • base_currency (str) – Lower-cased base currency (e.g. usd).

  • quote_currencies (tuple[str, ...]) – Quote currencies to populate. Adding a currency later backfills its full history automatically (no schema change needed).

  • start_date (Optional[datetime.date]) – Lower bound (inclusive). Defaults to EARLIEST_AVAILABLE_DATE; clamped so it can never go below it.

  • end_date (Optional[datetime.date]) – Upper bound (inclusive). Defaults to today (UTC).

  • source (str) – Provider identifier written to the source column.

  • max_workers (int) – Number of threaded date fetchers.

  • refetch_tail_days (int) – Always re-fetch the most recent N days to pick up source corrections.

  • unavailable_grace_days (int) – A 404/missing-quote younger than this many days is treated as “not published yet” and retried, not recorded as permanent.

  • max_transient_attempts (int) – Maximum number of consecutive transient failures (non-404 errors such as 403/5xx/network) tolerated for a single date across runs. Once a date reaches this many failures it is given up on — recorded as a permanent persistent_error gap and no longer retried. The counter resets as soon as the date succeeds or is confirmed unavailable.

  • session (Optional[requests.sessions.Session]) – Optional pre-built session; one is created if omitted.

Returns

A ScanResult carrying the open db and per-run counts.

Return type

eth_defi.currency_api.scanner.ScanResult