run_incremental_scan

Documentation for eth_defi.currency_api.scanner.run_incremental_scan function.

run_incremental_scan(db_path, base_currency='usd', quote_currencies=('eur', 'gbp', 'jpy', 'aud', 'sgd', 'hkd', 'try', 'chf', 'cad', 'btc', 'eth'), start_date=None, end_date=None, source='fawazahmed0', max_workers=8, refetch_tail_days=3, unavailable_grace_days=2, max_transient_attempts=5, session=None)

Incrementally populate exchange rates into DuckDB.

Computes the missing (date, quote) work set from the stored data and the gap table, fetches the needed dates in parallel, and upserts the results. Permanent gaps are recorded; transient failures are left for retry and surfaced in the returned ScanResult.

Parameters

  • db_path (pathlib.Path) – DuckDB file path. Created if it does not exist.

  • base_currency (str) – Lower-cased base currency (e.g. usd).

  • quote_currencies (tuple[str, ...]) – Quote currencies to populate. Adding a currency later backfills its full history automatically (no schema change needed).

  • start_date (Optional[datetime.date]) – Lower bound (inclusive). Defaults to EARLIEST_AVAILABLE_DATE; clamped so it can never go below it.

  • end_date (Optional[datetime.date]) – Upper bound (inclusive). Defaults to today (UTC).

  • source (str) – Provider identifier written to the source column.

  • max_workers (int) – Number of threaded date fetchers.

  • refetch_tail_days (int) – Always re-fetch the most recent N days to pick up source corrections.

  • unavailable_grace_days (int) – A 404/missing-quote younger than this many days is treated as “not published yet” and retried, not recorded as permanent.

  • max_transient_attempts (int) – Maximum number of consecutive transient failures (non-404 errors such as 403/5xx/network) tolerated for a single date across runs. Once a date reaches this many failures it is given up on — recorded as a permanent persistent_error gap and no longer retried. The counter resets as soon as the date succeeds or is confirmed unavailable.

  • session (Optional[requests.sessions.Session]) – Optional pre-built session; one is created if omitted.

Returns

A ScanResult carrying the open db and per-run counts.

Return type

eth_defi.currency_api.scanner.ScanResult