middleware

Documentation for eth_defi.middleware Python module.

Web3 middleware.

Most are for dealing with JSON-RPC unreliability issues with retries.

Taken from exception_retry_request.py from Web3.py
Modified to support sleep and throttling
Logs warnings to Python logging subsystem in the case there is need to retry
See also eth_defi.provider.broken_provider.

Module Attributes

`DEFAULT_RETRYABLE_EXCEPTIONS`	List of Web3 exceptions we know we should retry after some timeout
`DEFAULT_RETRYABLE_HTTP_STATUS_CODES`	List of HTTP status codes we know we might want to retry after a timeout
`AUTH_WARNING_HTTP_STATUS_CODES`	HTTP status codes that indicate potential API key/authentication issues We retry these but log a warning to alert the user
`DEFAULT_RETRYABLE_RPC_ERROR_CODES`	List of ValueError status codes we know we might want to retry after a timeout
`DEFAULT_RETRYABLE_RPC_ERROR_MESSAGES`	Because Ethreum JSON-RPC API is horribly broken, we also need to check for error messages besides error codes.
`STATIC_CALL_LIST`	Ethereum JSON-RPC calls where the value never changes
`ANVIL_SYNC_SEND_TIMEOUT_MS`	How long, in milliseconds, a send through `AnvilSyncSendMiddleware` waits for Anvil to mine the transaction before returning the hash anyway.
`ANVIL_SYNC_SEND_POLL_SECONDS`	Receipt poll interval, in seconds, after an `eth_sendTransaction` in `AnvilSyncSendMiddleware`.
`EIP_7966_TIMEOUT_ERROR_CODE`	EIP-7966 error code for a transaction accepted but not mined within the timeout.

Functions

`configure_provider_retry`(provider[, ...])	Configure provider retry settings for web3.py v7+.
`construct_sign_and_send_raw_middleware_anvil`(...)	Capture transactions sign and send as raw transactions - v6/v7 compatible.
`exception_retry_middleware`(make_request, ...)	Creates middleware that retries failed HTTP requests.
`http_retry_request_with_sleep_middleware`(...)	A HTTP retry middleware with sleep and backoff.
`is_retryable_http_exception`(exc[, ...])	Helper to check retryable errors from JSON-RPC calls.
`raise_on_revert_middleware`(make_request, web3)	Automatically show the transaction revert reason in Python traceback.

Classes

`AnvilSyncSendMiddleware`	Return from a transaction send only once Anvil has mined the transaction.
`StaticCallCacheMiddleware`	Cache JSON-RPC call values that never change.
`static_call_cache_middleware`	alias of `eth_defi.middleware.StaticCallCacheMiddleware`

Exceptions

`ProbablyNodeHasNoBlock`	A special exception raised when we suspect JSON-RPC node does not yet have data for a block we asked.
`SomeCrappyRPCProviderException`	Deal with non-standard RPC providers and whatever shitty logic they have invented for error codes

exception SomeCrappyRPCProviderException

Bases: Exception

Deal with non-standard RPC providers and whatever shitty logic they have invented for error codes

__init__(*args, **kwargs)

__new__(**kwargs)

add_note(note, /): Add a note to the exception

with_traceback(tb, /): Set self.__traceback__ to tb and return self.

DEFAULT_RETRYABLE_EXCEPTIONS: Tuple[BaseException] = (<class 'requests.exceptions.ConnectionError'>, <class 'requests.exceptions.HTTPError'>, <class 'requests.exceptions.Timeout'>, <class 'requests.exceptions.TooManyRedirects'>, <class 'web3.exceptions.BlockNotFound'>, <class 'requests.exceptions.ChunkedEncodingError'>, <class 'http.client.RemoteDisconnected'>, <class 'eth_defi.middleware.SomeCrappyRPCProviderException'>, <class 'requests.exceptions.ContentDecodingError'>)

List of Web3 exceptions we know we should retry after some timeout

For BlockNotFound see also eth_defi.rpc.broken_provider.

DEFAULT_RETRYABLE_HTTP_STATUS_CODES = (429, 500, 502, 503, 504, 525, 520, 410, 403, 400, 401)

List of HTTP status codes we know we might want to retry after a timeout

Taken from https://stackoverflow.com/a/72302017/315168

AUTH_WARNING_HTTP_STATUS_CODES = (401, 403): HTTP status codes that indicate potential API key/authentication issues We retry these but log a warning to alert the user

DEFAULT_RETRYABLE_RPC_ERROR_CODES = (-32003, -32043, -32005, -32701, 42903, -32002, -32603, -32052, -32601)

List of ValueError status codes we know we might want to retry after a timeout

This is a self-managed list curated by pain.

JSON-RPC error might be mapped to ValueError if nothing else is available.

Example from Pokt Network:

ValueError: {‘message’: ‘Internal JSON-RPC error.’, ‘code’: -32603}

We assume this is a broken RPC node and Pokt will reroute the the next retried request to some other node.

See GoEthereum error codes https://github.com/ethereum/go-ethereum/blob/master/rpc/errors.go

DEFAULT_RETRYABLE_RPC_ERROR_MESSAGES = {'Internal JSON-RPC error', 'Parse error', 'Unexpected error (code=40000)', 'empty reader set', 'execution aborted (timeout = 5s)', 'header not found', 'missing trie node', 'nonce too low', 'pruned history unavailable'}

Because Ethreum JSON-RPC API is horribly broken, we also need to check for error messages besides error codes.

See DEFAULT_RETRYABLE_RPC_ERROR_CODES.

STATIC_CALL_LIST = ('eth_chainId',): Ethereum JSON-RPC calls where the value never changes

exception ProbablyNodeHasNoBlock

Bases: Exception

A special exception raised when we suspect JSON-RPC node does not yet have data for a block we asked.

Calling a contract on a block before contract was deployed
Calling a contract on a block where the node does not have the block data yet

See eth_defi.provider.fallback for details.

__init__(*args, **kwargs)

__new__(**kwargs)

add_note(note, /): Add a note to the exception

with_traceback(tb, /): Set self.__traceback__ to tb and return self.

is_retryable_http_exception(exc, retryable_exceptions=(<class 'requests.exceptions.ConnectionError'>, <class 'requests.exceptions.HTTPError'>, <class 'requests.exceptions.Timeout'>, <class 'requests.exceptions.TooManyRedirects'>, <class 'web3.exceptions.BlockNotFound'>, <class 'requests.exceptions.ChunkedEncodingError'>, <class 'http.client.RemoteDisconnected'>, <class 'eth_defi.middleware.SomeCrappyRPCProviderException'>, <class 'requests.exceptions.ContentDecodingError'>), retryable_status_codes=(429, 500, 502, 503, 504, 525, 520, 410, 403, 400, 401), retryable_rpc_error_codes=(-32003, -32043, -32005, -32701, 42903, -32002, -32603, -32052, -32601), retryable_rpc_error_messages={'Internal JSON-RPC error', 'Parse error', 'Unexpected error (code=40000)', 'empty reader set', 'execution aborted (timeout = 5s)', 'header not found', 'missing trie node', 'nonce too low', 'pruned history unavailable'}, method=None, params=None)

Helper to check retryable errors from JSON-RPC calls.

Retryable reasons are connection timeouts, API throttling and such.

We support various kind of exceptions and HTTP status codes we know we can try.

Parameters

exc (Exception) – Exception raised by requests or Web3 machinery.
retryable_exceptions (Tuple[BaseException]) – Exception raised by requests or Web3 machinery.
retryable_status_codes (Collection[int]) – HTTP status codes we can retry. E.g. 429 Too Many requests.
retryable_rpc_error_messages (Collection[str]) – See DEFAULT_RETRYABLE_RPC_ERROR_MESSAGES.
method (Optional[str]) – JSON-RPC method name we called.
params (Optional[list]) – Method args.
retryable_rpc_error_codes (Collection[int]) –

exception_retry_middleware(make_request, web3, retryable_exceptions, retryable_status_codes, retryable_rpc_error_codes, retries=10, sleep=5.0, backoff=1.6)

Creates middleware that retries failed HTTP requests. Is a default middleware for HTTPProvider.

MIGRATED: Now uses compat version for v6/v7 compatibility.

See http_retry_request_with_sleep_middleware() for usage.

Parameters

make_request (Callable[[web3.types.RPCEndpoint, Any], web3.types.RPCResponse]) –
web3 (web3.main.Web3) –
retryable_exceptions (Tuple[BaseException]) –
retryable_status_codes (Collection[int]) –
retryable_rpc_error_codes (Collection[int]) –
retries (int) –
sleep (float) –
backoff (float) –

Return type

Callable[[web3.types.RPCEndpoint, Any], web3.types.RPCResponse]

http_retry_request_with_sleep_middleware(make_request, web3)

A HTTP retry middleware with sleep and backoff.

MIGRATED: In web3.py v7+, this function is deprecated in favor of ExceptionRetryConfiguration on the provider. However, for backwards compatibility, this function still works but may not be called if v7 provider retry configuration is used instead.

If you want to customise timeouts, supported exceptions and such you can directly create your own middleware using exception_retry_middleware().

Usage:

web3.middleware_onion.clear()
web3.middleware_onion.inject(http_retry_request_with_sleep_middleware, layer=0)

Parameters

make_request (Callable[[web3.types.RPCEndpoint, Any], Any]) – Part of middleware call signature
web3 (web3.main.Web3) – Part of middleware call signature

Returns

Web3.py middleware

Return type

Callable[[web3.types.RPCEndpoint, Any], Any]

configure_provider_retry(provider, retries=10, backoff_factor=0.5, retryable_exceptions=None)

Configure provider retry settings for web3.py v7+.

This is the recommended way to configure retries in v7+.

Parameters

provider – HTTPProvider or AsyncHTTPProvider instance
retries (int) – Number of retries to attempt (default 10)
backoff_factor (float) – Initial delay multiplier (default 0.5)
retryable_exceptions (tuple) – Tuple of exceptions to retry on

raise_on_revert_middleware(make_request, web3)

Automatically show the transaction revert reason in Python traceback.

Designed to make writing unit tests more productive
Transaction will already revert in eth_estimateGas call unless you have manually set the gas limit for your transaction
If a transaction fails, this middleware display its revert reason in Python exception message
Tested with Anvil testing backend
May interfere with http_retry_request_with_sleep_middleware(), others, so don’t use in production

from eth_defi.middleware import revert_reason_middleware

# Fix the web3.py stock gas estimate middlware with smarted one
web3.middleware_onion.replace("gas_estimate", revert_reason_aware_buffered_gas_estimate_middleware)

# Now you check the revert reason as the following

Parameters

make_request (Callable[[web3.types.RPCEndpoint, Any], Any]) –
web3 (web3.main.Web3) –

Return type

Callable[[web3.types.RPCEndpoint, Any], Any]

construct_sign_and_send_raw_middleware_anvil(private_key_or_account)

Capture transactions sign and send as raw transactions - v6/v7 compatible.

Return type: Type[web3.middleware.base.Web3Middleware]

class StaticCallCacheMiddleware

Bases: web3.middleware.base.Web3Middleware

Cache JSON-RPC call values that never change.

The cache is stored on the web3 instance itself, to allow sharing the cache between different JSON-RPC providers.

__init__(w3)

static_call_cache_middleware: alias of eth_defi.middleware.StaticCallCacheMiddleware

ANVIL_SYNC_SEND_TIMEOUT_MS = 1000

How long, in milliseconds, a send through AnvilSyncSendMiddleware waits for Anvil to mine the transaction before returning the hash anyway.

An automined transaction is mined in about 15 ms. The bound only matters when the transaction cannot be mined at once: automining is off, the fork is on interval mining, there is a nonce gap, or a cold fork read is slow. The caller then gets the hash as with a normal send and polls for the receipt as before.

ANVIL_SYNC_SEND_POLL_SECONDS = 0.005: Receipt poll interval, in seconds, after an eth_sendTransaction in AnvilSyncSendMiddleware.

EIP_7966_TIMEOUT_ERROR_CODE = 4: EIP-7966 error code for a transaction accepted but not mined within the timeout. The error data carries the transaction hash.

class AnvilSyncSendMiddleware

Bases: web3.middleware.base.Web3Middleware

Return from a transaction send only once Anvil has mined the transaction.

Anvil 1.8 automines asynchronously: eth_sendRawTransaction and eth_sendTransaction return the hash before the block exists, so the first receipt poll misses and every receipt wait sleeps one poll interval, one second in our confirmation helpers.

eth_sendRawTransaction goes through the EIP-7966 eth_sendRawTransactionSync method, which Anvil answers with the receipt once the transaction is mined. The middleware returns the hash, as the plain method would.
eth_sendTransaction is sent as is, because Anvil’s eth_sendTransactionSync takes no timeout and waits 30 s for a transaction that cannot be mined. The middleware then polls the receipt every ANVIL_SYNC_SEND_POLL_SECONDS before returning.
Both wait at most ANVIL_SYNC_SEND_TIMEOUT_MS, then return the hash, so tests that mine on demand keep working.
On an Anvil without the sync method, raw sends fall back to the plain method for the rest of the connection’s life.
Other errors, such as a nonce or gas error, are passed through unchanged.

Callers see no API change; their first receipt read succeeds.

Installed by eth_defi.provider.multi_provider.create_multi_provider_web3() for Anvil connections only. It must be the innermost middleware, so that the signing middleware’s eth_sendRawTransaction also goes through it.

__init__(w3)

Parameters: w3 (web3.main.Web3) –