Requesters#

Overview#

The core-https library provides a unified interface for making HTTP requests across different HTTP client libraries. All requesters implement the IRequester interface, providing consistent behavior regardless of the underlying HTTP library.

Available Implementations:

RequestsRequester - Synchronous HTTP client using the popular requests library
Urllib3Requester - Synchronous HTTP client using urllib3 with connection pooling
AioHttpRequester - Asynchronous HTTP client using aiohttp
AioHttpThrottleRequester - Async client with concurrency throttling
AioHttpRateLimitRequester - Async client with time-based rate limiting

Choosing the Right Requester#

Requester	Sync/Async	Use Case	Special Features
RequestsRequester	Synchronous	Simple HTTP requests	Easy to use
Urllib3Requester	Synchronous	Connection pooling	Low-level control
AioHttpRequester	Asynchronous	High concurrency	Fast, async/await
AioHttpThrottleRequester	Asynchronous	Limited resources	Concurrency limit
AioHttpRateLimitRequester	Asynchronous	API rate limits	Time-based limits

Common Features#

All requesters support:

Automatic retries with configurable retry counts
Exponential backoff between retry attempts
Timeout handling to prevent hanging requests
Status code validation with raise_for_status
Custom headers and request parameters
Connection pooling (where applicable)

Base Interface#

Abstract base class and shared utilities for all HTTP requester implementations.

class core_https.requesters.base.IRequester(encoding: str = 'utf-8', raise_for_status: bool = False, retries: Any | None = None, backoff_factor: float | None = None, connector_limit: int = 100, connector_limit_per_host: int = 30, timeout: int = 10)[source]#

Bases: IFactory, ABC

Abstract base interface for HTTP requesters with comprehensive configuration and error handling.

This abstract class defines the standard interface for all HTTP requester implementations in the core_https library. It provides a factory pattern for creating requester instances, comprehensive parameter validation, encoding detection, and standardized exception mapping.

All concrete implementations must inherit from this class and implement the abstract methods: engine() and request(). The class supports various HTTP libraries through a pluggable architecture while maintaining consistent behavior and error handling.

Features:

Factory pattern for requester instantiation
Comprehensive parameter validation with clear error messages
Automatic encoding detection from HTTP response headers
Standardized HTTP exception mapping (401→AuthenticationException, etc.)
Configurable connection pooling and timeouts
Support for retry strategies and backoff factors

Parameters:

encoding – Character encoding for response decoding. Defaults to “utf-8”. Must be a non-empty string after stripping whitespace.
raise_for_status – Whether to automatically raise exceptions for HTTP error status codes. When True, 4xx and 5xx responses will raise appropriate exceptions.
retries – Retry strategy configuration. Can be an integer for simple retry counts or a library-specific retry object (e.g., urllib3.util.retry.Retry). Different implementations may handle this parameter differently.
backoff_factor – Exponential backoff multiplier between retry attempts. Must be non-negative. The actual delay is typically: backoff_factor * attempt_number.
connector_limit – Maximum number of connections in the connection pool. Must be positive and greater than or equal to connector_limit_per_host.
connector_limit_per_host – Maximum connections per individual host. Must be positive and cannot exceed connector_limit.
timeout – Request timeout in seconds. Must be positive and should not exceed 3600 seconds. Individual requests may override this value based on implementation.

Synchronous Requesters#

RequestsRequester#

Uses the requests library for synchronous HTTP requests. Best for simple use cases and scripts.

Key Features:

Simple, intuitive API
Excellent documentation and community support
Built-in SSL verification
Session management

Example:

from core_https.requesters.requests_ import RequestsRequester
from core_https.requesters.base import HTTPMethod

requester = RequestsRequester(timeout=30)
response = requester.request(url="https://api.example.com/data", method=HTTPMethod.GET)

Synchronous HTTP requester implementation backed by the requests library.

class core_https.requesters.requests_.RequestsRequester(session: Session | None = None, retries: Retry | None = None, backoff_factor: float | None = None, **kwargs)[source]#

Bases: IRequester

It uses requests to make the requests.

from core_https.requesters.requests_ import RequestsRequester
from core_https.utils import HTTPMethod

requester: RequestsRequester = RequestsRequester()

response = requester.request(
    method=HTTPMethod.GET,
    url=url,
    params={
        "x-api-key": "..."
    })

print(response.json())

__init__(session: Session | None = None, retries: Retry | None = None, backoff_factor: float | None = None, **kwargs) → None[source]#

Parameters:

session – Session to use for the requests.
retries – Retry strategy to apply. Pass zero (0) to avoid retries.

classmethod engine() → str[source]#

Return the unique engine identifier for this requester implementation.

Returns:: Unique engine identifier (e.g., “aiohttp”, “requests”, “urllib3”).
Return type:: str

_abc_impl = <_abc._abc_data object>#

It makes the request using the session (externally provided or created if required) and return the response…

Returns:: requests.Response object.

Urllib3Requester#

Uses urllib3 for low-level HTTP operations with connection pooling.

Key Features:

Efficient connection pooling
Lower-level control
Thread-safe
Lighter weight than requests

Example:

from core_https.requesters.urllib3_ import Urllib3Requester
from core_https.requesters.base import HTTPMethod

requester = Urllib3Requester(timeout=30)
response = requester.request(url="https://api.example.com/data", method=HTTPMethod.GET)

Low-level synchronous HTTP requester implementation backed by urllib3.

class core_https.requesters.urllib3_.Urllib3Requester(pool_manager: PoolManager | None = None, retries: Retry | None = None, **kwargs)[source]#

Bases: IRequester

It uses urllib3 to make the requests.

from core_https.requesters.urllib3_ import Urllib3Requester
from core_https.utils import HTTPMethod

requester: Urllib3Requester = IRequester.get_class(Urllib3Requester.engine())()
response = requester.request(method=HTTPMethod.GET, url="https://google.com")
print(response.data.decode())

__init__(pool_manager: PoolManager | None = None, retries: Retry | None = None, **kwargs) → None[source]#

Parameters:

pool_manager – The pool manager to use or one will be created.
retries – Retry strategy to apply. Pass zero (0) to avoid retries.

classmethod engine()[source]#

Return the unique engine identifier for this requester implementation.

Returns:: Unique engine identifier (e.g., “aiohttp”, “requests”, “urllib3”).
Return type:: str

request(url: str, method: HTTPMethod = <HTTPMethod.GET>, headers: Dict[str, str] | None=None, retries: Retry | None = None, backoff_factor: float | None = None, fields: Dict | None = None, timeout: float | None = None, **kwargs) → BaseHTTPResponse[source]#

Raises:: ServiceException for other 4XX status_code.
Raises:: AuthenticationException for status_code == 401.
Raises:: AuthorizationException for status_code == 403.
Raises:: InternalServerError for status_code >= 500.
Returns:: HTTPResponse object.

_abc_impl = <_abc._abc_data object>#

Asynchronous Requesters#

AioHttpRequester#

Async HTTP client using aiohttp for high-performance concurrent requests.

Key Features:

True asynchronous I/O
Excellent for high concurrency
Async context manager support
WebSocket support (via aiohttp)

Example:

import asyncio
from core_https.requesters.aiohttp_ import AioHttpRequester

async def fetch():
    async with AioHttpRequester() as requester:
        response = await requester.request(url="https://api.example.com/data")
        return await response.json()

result = asyncio.run(fetch())

Asynchronous HTTP requester implementation backed by aiohttp.

class core_https.requesters.aiohttp_.AioHttpRequester(session: ClientSession | None = None, retries: int | None = 3, **kwargs)[source]#

Bases: IRequester

Asynchronous HTTP requester implementation using aiohttp.

This class provides an async HTTP client interface using the aiohttp library. It supports automatic session management, configurable retry logic with exponential backoff, connection pooling, and comprehensive error handling.

The requester can work with externally provided ClientSession objects or create and manage its own session internally. It implements the async context manager protocol for convenient resource cleanup.

Features:

Automatic session creation and management
Configurable retry logic with exponential backoff
Connection pooling with configurable limits
Comprehensive HTTP exception mapping
Support for custom timeouts per request
Context manager support for resource cleanup

import aiohttp
from core_https.requesters.aiohttp_ import AioHttpRequester
from core_https.utils import HTTPMethod

requester: AioHttpRequester = AioHttpRequester(raise_for_status=True)

async def get():
    # This is optional as the client creates one session for you if not provided.
    session = aiohttp.ClientSession()

    try:
        response = await requester.request(
            method=HTTPMethod.GET,
            session=session,
            url=url,
            params={
                "x-api-key": "..."
            })

        return await response.text()

    except Exception as error:
        pass

    finally:
        await session.close()

res = asyncio.run(get())
print(res)

__init__(session: ClientSession | None = None, retries: int | None = 3, **kwargs) → None[source]#

Initialize the AioHttpRequester.

Parameters:

session – Optional pre-configured aiohttp ClientSession to use for requests. If not provided, a new session will be created automatically with the configured timeout and connection limits. When providing a custom session, you are responsible for closing it.
retries – Number of retry attempts for failed requests. Defaults to 3. Set to 0 to disable retries completely. Only applies to retryable errors like network timeouts and server errors (5xx status codes).
**kwargs – Additional arguments passed to the base IRequester class, including encoding, raise_for_status, backoff_factor, timeout, connector_limit, and connector_limit_per_host.

Note

The timeout specified in kwargs becomes the default session timeout. Individual requests can override this using the timeout parameter in the request() method.

classmethod engine() → str[source]#

Return the unique engine identifier for this requester implementation.

Returns:: Unique engine identifier (e.g., “aiohttp”, “requests”, “urllib3”).
Return type:: str

async _ensure_session() → ClientSession[source]#

Ensure a ClientSession exists, creating one if necessary.

This method implements a thread-safe lazy initialization pattern using double-checked locking to ensure only one session is created even when called concurrently from multiple coroutines.

Returns:: The active aiohttp ClientSession instance.
Return type:: ClientSession

Note

If a session was provided in the constructor, it will be returned as-is. If no session was provided, a new one will be created with the configured timeout and connection limits.

Thread Safety:: This method is safe to call concurrently from multiple coroutines. The double-check locking pattern ensures only one session is created.

async request(url: str, method: HTTPMethod = <HTTPMethod.GET>, headers: Dict[str, ~typing.Any] | None=None, retries: int | None = None, backoff_factor: float | None = None, session: ClientSession | None = None, params: Dict[str, ~typing.Any] | None=None, timeout: float | None = None, **kwargs) → ClientResponse[source]#

Make an asynchronous HTTP request with retry logic.

This method performs HTTP requests with automatic retry functionality, exponential backoff, and comprehensive error handling. Failed requests are retried based on the configured retry policy.

Parameters:

url – The target URL for the HTTP request.
method – HTTP method to use. Defaults to GET. Supports all standard HTTP methods (GET, POST, PUT, DELETE, etc.).
headers – Optional HTTP headers to include in the request. These will be merged with any session-level headers.
retries – Number of retry attempts for this specific request. If not provided, uses the instance-level retry setting. Set to 0 to disable retries for this request.
backoff_factor – Multiplier for exponential backoff between retries. The actual delay is calculated as: backoff_factor * attempt_number. If not provided, uses the instance-level setting or defaults to 0.5.
session – Optional ClientSession to use for this request. If not provided, uses the requester’s session (creating one if necessary). Useful for per-request session customization.
params – URL query parameters to include in the request. Will be properly URL-encoded and appended to the URL.
timeout – Request timeout in seconds for this specific request. Overrides the instance-level timeout setting. Set to None to use the default timeout.
**kwargs – Additional parameters passed to aiohttp’s request method. Common options include: json, data, cookies, ssl, proxy, etc. See aiohttp.ClientSession.request documentation for full list.

Returns:

The aiohttp response object. Use methods like: .json(), .text(), .read() to extract the response content.

Return type:

ClientResponse

Raises:

AuthenticationException – For 401 Unauthorized responses.
AuthorizationException – For 403 Forbidden responses.
RateLimitException – For 429 Too Many Requests (when retries exhausted).
RetryableException – For 502/503/504 server errors (when retries exhausted).
ServiceException – For other 4xx client errors.
InternalServerError – For 5xx server errors (when retries exhausted).

Note

The retry logic only applies to network errors and server errors (5xx). Client errors (4xx) are not retried, except for 429 (rate limit) which may be retried based on the configured policy.

async close() → None[source]#

Close the internal session if it was created by this requester.

This method performs cleanup of the internal ClientSession, but only if the session was created internally. If a custom session was provided in the constructor, it will not be closed as the caller is responsible for managing its lifecycle.

The session reference is cleared only when the session was created internally. External sessions retain their reference so the requester remains usable after close() without silently creating a new session.

Note

This method is automatically called when using the async context manager protocol (__aexit__). Manual calling is only necessary when not using the context manager pattern.

_abc_impl = <_abc._abc_data object>#

AioHttpThrottleRequester#

Extends AioHttpRequester with concurrency throttling to limit simultaneous requests.

When to Use:

Prevent overwhelming your client with too many concurrent connections
Limit resource usage (memory, CPU, network)
Protect downstream services from excessive load
No time-based restrictions needed

How It Works:

Uses asyncio.Semaphore to limit the number of requests executing simultaneously. If the limit is reached, additional requests wait until a slot becomes available.

Example:

import asyncio
from core_https.requesters.aiohttp_throttle import AioHttpThrottleRequester

async def fetch_many():
    # Only 5 requests will execute concurrently
    requester = AioHttpThrottleRequester(max_concurrency=5)

    try:
        tasks = [
            requester.request(url=f"https://api.example.com/item/{i}")
            for i in range(100)
        ]
        # Even with 100 tasks, only 5 run at once
        responses = await asyncio.gather(*tasks)
        return responses
    finally:
        await requester.close()

asyncio.run(fetch_many())

Parameters:

max_concurrency: Maximum number of simultaneous requests

Concurrency-throttled aiohttp requester using asyncio.Semaphore.

class core_https.requesters.aiohttp_throttle.AioHttpThrottleRequester(max_concurrency: int, **kwargs)[source]#

Bases: AioHttpRequester

An AioHttpRequester implementation that limits the number of concurrent in-flight HTTP requests using an asyncio.Semaphore. This throttler enforces max_concurrency at the coroutine level within a single event loop. Each call to request() must acquire a semaphore permit before dispatching the actual HTTP request.

Notes:

The throttling mechanism limits only concurrent coroutine execution. It does not enforce rate limiting (requests per second).
If you override request() in a subclass, be aware that await super().request(…) calls the parent implementation (so the parent semaphore will be used for the actual HTTP call). However, any code you run before or after that super() call executes outside the parent’s semaphore (and therefore is not throttled).

__init__(max_concurrency: int, **kwargs) → None[source]#

Parameters:

max_concurrency – Maximum number of concurrent requests allowed.
kwargs – Passed through to AioHttpRequester.

classmethod engine() → str[source]#

Return the unique engine identifier for this requester implementation.

Returns:: Unique engine identifier (e.g., “aiohttp”, “requests”, “urllib3”).
Return type:: str

async request(*args, **kwargs) → ClientResponse[source]#

Execute an HTTP request with concurrency throttling. It acquires a semaphore permit before delegating the actual request to the underlying AioHttpRequester implementation.

Returns:: The aiohttp response object.
Raises:: Any exception raised by the underlying session.

async close() → None[source]#

Close the internal session if it was created by this requester.

Note

This method is automatically called when using the async context manager protocol (__aexit__). Manual calling is only necessary when not using the context manager pattern.

AioHttpRateLimitRequester#

Enforces a time-based rate limit on top of AioHttpRequester.

When to Use:

API has rate limits (e.g., 1000 requests per hour)
Need to comply with service quotas
Prevent getting banned or throttled by external APIs

How It Works:

Uses aiolimiter.AsyncLimiter (token bucket algorithm) to control how many requests may start within a given time window. Multiple requests can still run concurrently if they acquire a token within the same window.

Example:

import asyncio
from core_https.requesters.aiohttp_rate_limit import AioHttpRateLimitRequester

async def fetch_with_rate_limit():
    # Max 100 requests per minute
    requester = AioHttpRateLimitRequester(
        max_rate=100,
        time_period=60  # seconds
    )

    try:
        tasks = [
            requester.request(url=f"https://api.example.com/item/{i}")
            for i in range(500)
        ]
        # Requests are automatically rate-limited
        responses = await asyncio.gather(*tasks)
        return responses
    finally:
        await requester.close()

asyncio.run(fetch_with_rate_limit())

Parameters:

max_rate: Maximum number of requests allowed in the time window
time_period: Duration in seconds for the rate limit window

Rate Limit Examples:

# GitHub API: 5000 requests per hour
AioHttpRateLimitRequester(max_rate=5000, time_period=3600)

# Twitter API: 15 requests per 15 minutes
AioHttpRateLimitRequester(max_rate=15, time_period=900)

# Conservative: 1 request per second
AioHttpRateLimitRequester(max_rate=1, time_period=1)

Throttle vs Rate Limit Comparison#

Understanding the Difference#

Feature	Throttle	Rate Limit
What it limits	Concurrent execution	Requests per time period
Time-based	No	Yes
Use case	Resource protection	API quota compliance
Example	Max 10 at once	Max 100 per minute
Mechanism	Semaphore	Token bucket
Dependencies	Built-in asyncio	External aiolimiter

Visual Example:

Throttle (max_concurrency=3):
Timeline: [Request1][Request2][Request3] <- only 3 running
          [waiting...][waiting...][waiting...]

Rate Limit (max_rate=10, time_period=1):
Timeline: [10 requests] <- can all run concurrently
          [wait 1 second]
          [10 more requests]

When to Combine Both#

Rate limiting and concurrency throttling are independent concerns. To apply both simultaneously, compose the two classes via multiple inheritance:

import asyncio
from core_https.requesters.aiohttp_rate_limit import AioHttpRateLimitRequester
from core_https.requesters.aiohttp_throttle import AioHttpThrottleRequester

class RateLimitedThrottleClient(AioHttpRateLimitRequester, AioHttpThrottleRequester):
    @classmethod
    def engine(cls) -> str:
        return "aiohttp_rate_limit_throttle"

async def fetch():
    # Respects API quota AND caps local concurrency
    async with RateLimitedThrottleClient(
        max_rate=1000,        # API allows 1000/hour
        time_period=3600,     # One hour
        max_concurrency=10,   # Never more than 10 in-flight
    ) as client:
        response = await client.request(url="https://api.example.com/data")
        return await response.json()

result = asyncio.run(fetch())

The MRO ensures each request flows through the rate limiter first, then the semaphore, then the actual HTTP call: RateLimitedThrottleClient → AioHttpRateLimitRequester → AioHttpThrottleRequester → AioHttpRequester

Requesters#

Overview#

Choosing the Right Requester#

Common Features#

Base Interface#

Synchronous Requesters#

RequestsRequester#

Urllib3Requester#

Asynchronous Requesters#

AioHttpRequester#

AioHttpThrottleRequester#

AioHttpRateLimitRequester#

Throttle vs Rate Limit Comparison#

Understanding the Difference#

When to Combine Both#

This Page