core3.scanner

Documentation for eth_defi.core3.scanner Python module.

Core3 project scanner with DuckDB storage.

Orchestrates fetching data from the Core3 API and storing it in a Core3Database. Supports parallel fetching with joblib.Parallel and incremental sync using watermarks.

Example usage:

from pathlib import Path
from eth_defi.core3.session import create_core3_session
from eth_defi.core3.scanner import scan_projects

session = create_core3_session()
db = scan_projects(session=session, limit=10)
try:
    df = db.get_latest_project_snapshots()
    print(df)
finally:
    db.close()

Functions

scan_projects(session[, db_path, ...])

Scan all Core3 projects and store snapshots in DuckDB.
scan_projects(session, db_path=PosixPath('/home/runner/.tradingstrategy/vaults/core3/core3.duckdb'), fetch_sections=False, fetch_pol_history=True, fetch_category_history=True, fetch_index_pol=True, limit=None, max_workers=8, timeout=60.0)

Scan all Core3 projects and store snapshots in DuckDB.

This function:

  1. Fetches the project list from /v1/list to get all slugs

  2. For each slug (parallelised with joblib):

    1. Fetches /v1/{slug} and inserts into project_snapshots

    2. Optionally fetches PoL history and inserts into pol_daily

    3. Optionally fetches category history and inserts into pol_category_daily

    4. Optionally fetches section endpoints and inserts into section_snapshots

  3. Optionally fetches index-level PoL history

Parameters

  • session (eth_defi.core3.session.Core3Session) – Core3 API session. Use create_core3_session() to create one.

  • db_path (pathlib.Path) – Path to the DuckDB database file.

  • fetch_sections (bool) – If True, also fetch section detail endpoints (security, financial, etc.). This is slower (5 extra API calls per project).

  • fetch_pol_history (bool) – If True, fetch PoL daily history for each project.

  • fetch_category_history (bool) – If True, fetch category PoL breakdown history for each project.

  • fetch_index_pol (bool) – If True, fetch the aggregate index-level PoL history.

  • limit (Optional[int]) – Limit the number of projects to scan. For testing only.

  • max_workers (int) – Maximum number of parallel workers for fetching project data.

  • timeout (float) – HTTP request timeout in seconds.

Returns

Core3Database instance with the newly inserted data. Caller must call close() when done.

Return type

eth_defi.core3.database.Core3Database