currency_api.cleaning

Documentation for eth_defi.currency_api.cleaning Python module.

Cleaning step for currency_api exchange rate data.

Ingestion (eth_defi.currency_api.scanner) stores raw source values verbatim and never fabricates or “corrects” data. This module is the separate, explicit cleaning pass: it drops a hardcoded allowlist of known-bad datapoints — upstream source glitches found by manual data review — without mutating the stored raw data. Apply it on read/export when you want a sanitised view.

Keeping cleaning separate means the DuckDB file remains a faithful mirror of the source (auditable, re-cleanable), while consumers that want trustworthy series call filter_known_bad_rates().

Module Attributes

KNOWN_BAD_RATES

Hardcoded allowlist of known-bad (date, base_currency, quote_currency, source) cells from the fawazahmed0 Exchange API, identified by manual data review.

Functions

filter_known_bad_rates(df)

Drop known-bad exchange rate rows from a rates DataFrame.

KNOWN_BAD_RATES: tuple[tuple[datetime.date, str, str, str], ...] = ((datetime.date(2025, 12, 6), 'usd', 'btc', 'fawazahmed0'), (datetime.date(2025, 12, 6), 'usd', 'eth', 'fawazahmed0'))

Hardcoded allowlist of known-bad (date, base_currency, quote_currency, source) cells from the fawazahmed0 Exchange API, identified by manual data review.

Each entry documents the observed bad value and why it is dropped. Keep this list small and evidence-based — only add a cell after confirming the bad value is present at the source (i.e. the raw API returns it), not a local bug.

filter_known_bad_rates(df)

Drop known-bad exchange rate rows from a rates DataFrame.

Removes the rows whose (date, base_currency, quote_currency, source) match an entry in KNOWN_BAD_RATES. The input is not mutated; a filtered copy is returned. This is a separate cleaning step — the stored DuckDB data is left untouched (raw).

Parameters: df (pandas.DataFrame) – Rates DataFrame as returned by eth_defi.currency_api.database.CurrencyRateDatabase.get_rates_dataframe(), i.e. with columns date (datetime.date or pandas Timestamp), base_currency, quote_currency, rate, source and written_at.
Returns: A new DataFrame with the known-bad rows removed and the index reset. If the input is empty it is returned unchanged (a copy).
Return type: pandas.DataFrame