feed.collector

Documentation for eth_defi.feed.collector Python module.

Vault post collection and feed normalisation.

Module Attributes

DEFAULT_TWITTER_URL_TEMPLATES	Default Twitter/X RSS bridge URL templates used when no TWITTER_FEED_URL_TEMPLATES environment variable is set.
DEFAULT_LINKEDIN_URL_TEMPLATES	Default LinkedIn RSS bridge URL templates used when no LINKEDIN_FEED_URL_TEMPLATES environment variable is set.

Functions

build_linkedin_rss_feed_urls(company_id, ...)	Build live feed URLs for a LinkedIn company id.
build_twitter_rss_feed_urls(handle, base_urls, *)	Build live feed URLs for a Twitter handle.
collect_posts(db, sources, *[, ...])	Collect posts for all configured sources and persist them in DuckDB.
collect_posts_for_source(source, *, ...[, ...])	Collect posts for one tracked source.
collect_twitter_list_posts(db, sources, *, ...)	Collect Twitter/X posts through a single X list timeline read.
fetch_feed_proxy_rotator()	Fetch an optional Webshare proxy rotator for feed fetching.
load_feed_proxy_rotator()	Backwards-compatible alias for fetch_feed_proxy_rotator().

Classes

CollectedSourceResult	Detailed collection result for one tracked source.
CollectorRunSummary	Summary counters for one collector run.

Exceptions

AllBridgesFailedError

Raised when every bridge URL for a social feed source fails.

exception AllBridgesFailedError

Bases: RuntimeError

Raised when every bridge URL for a social feed source fails.

Parameters

• source_label – Human-readable source type label for error messages.

• canonical_url – Canonical source URL for diagnostics.

• bridge_errors – List of (url, http_status_or_none) for each attempt. None for the status code indicates a non-HTTP failure such as a timeout.

__init__(source_label, canonical_url, bridge_errors)

Parameters

• source_label (str) –

• canonical_url (str) –

• bridge_errors (list[tuple[str, int | None]]) –

property indicates_auth_block: bool

Return True when at least one bridge returned HTTP 503 (LinkedIn auth barrier).

When all bridges fail and at least one specifically returns 503, LinkedIn is most likely redirecting unauthenticated requests to the login page for this company. Bridges that are simply down (502 or connection error) do not indicate anything about LinkedIn’s stance on the company page, so they are not required to return 503.

__new__(**kwargs)

add_note(note, /)

Add a note to the exception

with_traceback(tb, /)

Set self.__traceback__ to tb and return self.

DEFAULT_TWITTER_URL_TEMPLATES: list[str] = ['https://xcancel.com/{handle}/rss', 'https://rss.xcancel.com/{handle}/rss']

Default Twitter/X RSS bridge URL templates used when no TWITTER_FEED_URL_TEMPLATES environment variable is set. Each template must contain a {handle} placeholder. Verified working as of 2026-04-03.

DEFAULT_LINKEDIN_URL_TEMPLATES: list[str] = ['https://rsshub.pseudoyu.com/linkedin/company/{company_id}/posts', 'https://rss.owo.nz/linkedin/company/{company_id}/posts', 'https://rsshub.umzzz.com/linkedin/company/{company_id}/posts']

Default LinkedIn RSS bridge URL templates used when no LINKEDIN_FEED_URL_TEMPLATES environment variable is set. Each template must contain a {company_id} placeholder. Verified working as of 2026-04-03.

Note

RSSHub bridges rely on unauthenticated LinkedIn access. LinkedIn serves public company post feeds only for large or verified organisations. Smaller companies redirect unauthenticated scrapers to the login page, causing every bridge to return 503 regardless of which instance is used. When the standard scan detects this pattern it writes linkedin-rss-hub-disabled-at to the feeder YAML so future runs skip the source without retrying.

class CollectorRunSummary

Bases: object

Summary counters for one collector run.

sources_loaded: int

Number of configured sources seen at the start of the run.

sources_succeeded: int

Number of sources collected successfully.

sources_failed: int

Number of sources that failed or were skipped with an error.

feeders_skipped: int

Number of feeder YAML files where all sources were disabled.

posts_fetched: int

Total number of parsed posts returned by all successful source reads.

posts_inserted: int

Number of newly inserted posts after deduplication.

source_results: Optional[list[eth_defi.feed.collector.CollectedSourceResult]]

Per-source collection results for dashboard rendering.

twitter_method: Optional[str]

Twitter collection method used: “list”, “rss-bridge”, or None if no Twitter sources.

rss_duration_seconds: Optional[float]

Duration of the RSS phase in seconds.

linkedin_duration_seconds: Optional[float]

Duration of the LinkedIn phase in seconds.

twitter_duration_seconds: Optional[float]

Duration of the Twitter phase in seconds.

total_duration_seconds: Optional[float]

Total scan duration in seconds.

stablecoin_rate_status: Optional[str]

Stablecoin rate side-job status: disabled, skipped_recent, succeeded, or failed.

stablecoin_rate_summary: Optional[Any]

Stablecoin rate side-job summary when a refresh ran successfully.

stablecoin_rate_error: Optional[str]

Stablecoin rate side-job error when an unexpected refresh exception was caught.

__init__(sources_loaded=0, sources_succeeded=0, sources_failed=0, feeders_skipped=0, posts_fetched=0, posts_inserted=0, source_results=None, twitter_method=None, rss_duration_seconds=None, linkedin_duration_seconds=None, twitter_duration_seconds=None, total_duration_seconds=None, stablecoin_rate_status=None, stablecoin_rate_summary=None, stablecoin_rate_error=None)

Parameters

• sources_loaded (int) –

• sources_succeeded (int) –

• sources_failed (int) –

• feeders_skipped (int) –

• posts_fetched (int) –

• posts_inserted (int) –

• source_results (Optional[list[eth_defi.feed.collector.CollectedSourceResult]]) –

• twitter_method (Optional[str]) –

• rss_duration_seconds (Optional[float]) –

• linkedin_duration_seconds (Optional[float]) –

• twitter_duration_seconds (Optional[float]) –

• total_duration_seconds (Optional[float]) –

• stablecoin_rate_status (Optional[str]) –

• stablecoin_rate_summary (Optional[Any]) –

• stablecoin_rate_error (Optional[str]) –

Return type

None

class CollectedSourceResult

Bases: object

Detailed collection result for one tracked source.

feeder_id: str

Canonical feeder slug for the collected source.

name: str

Human-readable feeder name for diagnostics.

role: str

Feeder role such as protocol, curator, or vault.

source_type: str

Source transport type such as rss, twitter, or linkedin.

status: str

Final status for this source, such as success or failed.

posts_fetched: int

Number of parsed posts fetched from this source.

posts_inserted: int

Number of inserted posts after deduplication.

last_post_published_at: Optional[datetime.datetime]

Last published timestamp seen in this source, if any.

error: Optional[str]

Error message when the source failed or was skipped.

auth_blocked: bool

True when all bridge attempts failed and at least one returned HTTP 503 (LinkedIn auth barrier).

__init__(feeder_id, name, role, source_type, status, posts_fetched=0, posts_inserted=0, last_post_published_at=None, error=None, auth_blocked=False)

Parameters

• feeder_id (str) –

• name (str) –

• role (str) –

• source_type (str) –

• status (str) –

• posts_fetched (int) –

• posts_inserted (int) –

• last_post_published_at (Optional[datetime.datetime]) –

• error (Optional[str]) –

• auth_blocked (bool) –

Return type

None

build_twitter_rss_feed_urls(handle, base_urls, *, url_templates=None)

Build live feed URLs for a Twitter handle.

Parameters

• handle (str) –

• base_urls (Sequence[str]) –

• url_templates (Optional[Sequence[str]]) –

Return type

list[str]

build_linkedin_rss_feed_urls(company_id, url_templates)

Build live feed URLs for a LinkedIn company id.

Parameters

• company_id (str) –

• url_templates (Sequence[str]) –

Return type

list[str]

fetch_feed_proxy_rotator()

Fetch an optional Webshare proxy rotator for feed fetching.

Return type

Optional[eth_defi.event_reader.webshare.ProxyRotator]

load_feed_proxy_rotator()

Backwards-compatible alias for fetch_feed_proxy_rotator().

Return type

Optional[eth_defi.event_reader.webshare.ProxyRotator]

collect_posts_for_source(source, *, max_posts_per_source, request_timeout, twitter_rss_base_urls, twitter_url_templates=None, linkedin_url_templates=None, proxy_rotator=None, max_proxy_rotations=3, twitter_bearer_token=None, twitter_user_cache=None)

Collect posts for one tracked source.

Parameters

• source (eth_defi.feed.sources.TrackedPostSource) –

• max_posts_per_source (int) –

• request_timeout (float) –

• twitter_rss_base_urls (Sequence[str]) –

• twitter_url_templates (Optional[Sequence[str]]) –

• linkedin_url_templates (Optional[Sequence[str]]) –

• proxy_rotator (Optional[eth_defi.event_reader.webshare.ProxyRotator]) –

• max_proxy_rotations (int) –

• twitter_bearer_token (Optional[str]) –

• twitter_user_cache (Optional[eth_defi.feed.twitter_api.TwitterUserCache]) –

Return type

list[eth_defi.feed.database.CollectedPost]

collect_posts(db, sources, *, max_posts_per_source=20, max_workers=8, request_timeout=20.0, request_delay_seconds=1.0, twitter_rss_base_urls=None, twitter_url_templates=None, linkedin_url_templates=None, proxy_rotator=None, max_proxy_rotations=3, twitter_bearer_token=None, twitter_user_cache=None, label='')

Collect posts for all configured sources and persist them in DuckDB.

Parameters

• db (eth_defi.feed.database.VaultPostDatabase) –

• sources (Sequence[eth_defi.feed.sources.TrackedPostSource]) –

• max_posts_per_source (int) –

• max_workers (int) –

• request_timeout (float) –

• request_delay_seconds (float) –

• twitter_rss_base_urls (Optional[Sequence[str]]) –

• twitter_url_templates (Optional[Sequence[str]]) –

• linkedin_url_templates (Optional[Sequence[str]]) –

• proxy_rotator (Optional[eth_defi.event_reader.webshare.ProxyRotator]) –

• max_proxy_rotations (int) –

• twitter_bearer_token (Optional[str]) –

• twitter_user_cache (Optional[eth_defi.feed.twitter_api.TwitterUserCache]) –

• label (str) –

Return type

eth_defi.feed.collector.CollectorRunSummary

collect_twitter_list_posts(db, sources, *, list_id, bearer_token, twitter_user_cache, max_tweets, fallback_max_tweets=5, label='Twitter list')

Collect Twitter/X posts through a single X list timeline read.

The list timeline API returns tweets across all list members in reverse chronological order. This lets production collection avoid one API call per tracked account while still storing posts under the account-specific tracked source rows.

When a handle has no tweets in the list timeline and has no stored last_post_published_at (i.e. it is a brand-new handle whose first scan returned nothing), the collector falls back to a single individual timeline read. This seeds the timestamp and stores a small number of recent posts without firing per-account API calls on steady-state runs where the list stopped early because all recent tweets were already known.

Parameters

• db (eth_defi.feed.database.VaultPostDatabase) – Vault post database.

• sources (Sequence[eth_defi.feed.sources.TrackedPostSource]) – Twitter tracked sources whose handles are represented in the X list.

• list_id (str) – Numeric X list ID.

• bearer_token (str) – X API bearer token used for list timeline reads.

• twitter_user_cache (eth_defi.feed.twitter_api.TwitterUserCache) – Cache containing handle-to-user-ID mappings.

• max_tweets (int) – Maximum tweets to read from the list timeline.

• fallback_max_tweets (int) – Maximum tweets to fetch per account when the list timeline returns zero results for that handle. Used to populate last_post_published_at for inactive accounts. Defaults to 5.

• label (str) – Dashboard label for this collection phase.

Returns

Collector run summary with per-source insert counters.

Return type

eth_defi.feed.collector.CollectorRunSummary