collect_posts

Documentation for eth_defi.feed.collector.collect_posts function.

collect_posts(db, sources, *, max_posts_per_source=20, max_workers=8, request_timeout=20.0, request_delay_seconds=1.0, twitter_rss_base_urls=None, twitter_url_templates=None, linkedin_url_templates=None, proxy_rotator=None, max_proxy_rotations=3, twitter_bearer_token=None, twitter_user_cache=None, label='')

Collect posts for all configured sources and persist them in DuckDB.

Parameters

• db (eth_defi.feed.database.VaultPostDatabase) –

• sources (Sequence[eth_defi.feed.sources.TrackedPostSource]) –

• max_posts_per_source (int) –

• max_workers (int) –

• request_timeout (float) –

• request_delay_seconds (float) –

• twitter_rss_base_urls (Optional[Sequence[str]]) –

• twitter_url_templates (Optional[Sequence[str]]) –

• linkedin_url_templates (Optional[Sequence[str]]) –

• proxy_rotator (Optional[eth_defi.event_reader.webshare.ProxyRotator]) –

• max_proxy_rotations (int) –

• twitter_bearer_token (Optional[str]) –

• twitter_user_cache (Optional[eth_defi.feed.twitter_api.TwitterUserCache]) –

• label (str) –

Return type

eth_defi.feed.collector.CollectorRunSummary