.. module:: Requesters


Requesters
=========================

Overview
*************************

The ``core-https`` library provides a unified interface for making HTTP requests across different HTTP client libraries.
All requesters implement the ``IRequester`` interface, providing consistent behavior regardless of the underlying HTTP library.

**Available Implementations:**

* **RequestsRequester** - Synchronous HTTP client using the popular ``requests`` library
* **Urllib3Requester** - Synchronous HTTP client using ``urllib3`` with connection pooling
* **AioHttpRequester** - Asynchronous HTTP client using ``aiohttp``
* **AioHttpThrottleRequester** - Async client with concurrency throttling
* **AioHttpRateLimitRequester** - Async client with time-based rate limiting

Choosing the Right Requester
***************************************************

+---------------------------+------------------+----------------------+-------------------+
| Requester                 | Sync/Async       | Use Case             | Special Features  |
+===========================+==================+======================+===================+
| RequestsRequester         | Synchronous      | Simple HTTP requests | Easy to use       |
+---------------------------+------------------+----------------------+-------------------+
| Urllib3Requester          | Synchronous      | Connection pooling   | Low-level control |
+---------------------------+------------------+----------------------+-------------------+
| AioHttpRequester          | Asynchronous     | High concurrency     | Fast, async/await |
+---------------------------+------------------+----------------------+-------------------+
| AioHttpThrottleRequester  | Asynchronous     | Limited resources    | Concurrency limit |
+---------------------------+------------------+----------------------+-------------------+
| AioHttpRateLimitRequester | Asynchronous     | API rate limits      | Time-based limits |
+---------------------------+------------------+----------------------+-------------------+

Common Features
*************************

All requesters support:

- **Automatic retries** with configurable retry counts
- **Exponential backoff** between retry attempts
- **Timeout handling** to prevent hanging requests
- **Status code validation** with ``raise_for_status``
- **Custom headers** and request parameters
- **Connection pooling** (where applicable)

Base Interface
*************************

.. automodule:: core_https.requesters.base
   :members:
   :undoc-members:
   :show-inheritance:
   :private-members:

Synchronous Requesters
*************************

RequestsRequester
-----------------------------------------------

Uses the ``requests`` library for synchronous HTTP requests. Best for simple use cases and scripts.

**Key Features:**

- Simple, intuitive API
- Excellent documentation and community support
- Built-in SSL verification
- Session management

**Example:**

.. code-block:: python

    from core_https.requesters.requests_ import RequestsRequester
    from core_https.requesters.base import HTTPMethod

    requester = RequestsRequester(timeout=30)
    response = requester.request(url="https://api.example.com/data", method=HTTPMethod.GET)

.. automodule:: core_https.requesters.requests_
   :members:
   :undoc-members:
   :show-inheritance:
   :private-members:

Urllib3Requester
-----------------------------------------------

Uses ``urllib3`` for low-level HTTP operations with connection pooling.

**Key Features:**

- Efficient connection pooling
- Lower-level control
- Thread-safe
- Lighter weight than requests

**Example:**

.. code-block:: python

    from core_https.requesters.urllib3_ import Urllib3Requester
    from core_https.requesters.base import HTTPMethod

    requester = Urllib3Requester(timeout=30)
    response = requester.request(url="https://api.example.com/data", method=HTTPMethod.GET)

.. automodule:: core_https.requesters.urllib3_
   :members:
   :undoc-members:
   :show-inheritance:
   :private-members:

Asynchronous Requesters
*************************

AioHttpRequester
-----------------------------------------------

Async HTTP client using ``aiohttp`` for high-performance concurrent requests.

**Key Features:**

- True asynchronous I/O
- Excellent for high concurrency
- Async context manager support
- WebSocket support (via aiohttp)

**Example:**

.. code-block:: python

    import asyncio
    from core_https.requesters.aiohttp_ import AioHttpRequester

    async def fetch():
        async with AioHttpRequester() as requester:
            response = await requester.request(url="https://api.example.com/data")
            return await response.json()

    result = asyncio.run(fetch())

.. automodule:: core_https.requesters.aiohttp_
   :members:
   :undoc-members:
   :show-inheritance:
   :private-members:

AioHttpThrottleRequester
-----------------------------------------------

Extends ``AioHttpRequester`` with **concurrency throttling** to limit simultaneous requests.

**When to Use:**

- Prevent overwhelming your client with too many concurrent connections
- Limit resource usage (memory, CPU, network)
- Protect downstream services from excessive load
- No time-based restrictions needed

**How It Works:**

Uses ``asyncio.Semaphore`` to limit the number of requests executing simultaneously.
If the limit is reached, additional requests wait until a slot becomes available.

**Example:**

.. code-block:: python

    import asyncio
    from core_https.requesters.aiohttp_throttle import AioHttpThrottleRequester

    async def fetch_many():
        # Only 5 requests will execute concurrently
        requester = AioHttpThrottleRequester(max_concurrency=5)

        try:
            tasks = [
                requester.request(url=f"https://api.example.com/item/{i}")
                for i in range(100)
            ]
            # Even with 100 tasks, only 5 run at once
            responses = await asyncio.gather(*tasks)
            return responses
        finally:
            await requester.close()

    asyncio.run(fetch_many())

**Parameters:**

- ``max_concurrency``: Maximum number of simultaneous requests

.. automodule:: core_https.requesters.aiohttp_throttle
   :members:
   :undoc-members:
   :show-inheritance:

AioHttpRateLimitRequester
-----------------------------------------------

Enforces a **time-based rate limit** on top of ``AioHttpRequester``.

**When to Use:**

- API has rate limits (e.g., 1000 requests per hour)
- Need to comply with service quotas
- Prevent getting banned or throttled by external APIs

**How It Works:**

Uses ``aiolimiter.AsyncLimiter`` (token bucket algorithm) to control how many
requests may *start* within a given time window. Multiple requests can still
run concurrently if they acquire a token within the same window.

**Example:**

.. code-block:: python

    import asyncio
    from core_https.requesters.aiohttp_rate_limit import AioHttpRateLimitRequester

    async def fetch_with_rate_limit():
        # Max 100 requests per minute
        requester = AioHttpRateLimitRequester(
            max_rate=100,
            time_period=60  # seconds
        )

        try:
            tasks = [
                requester.request(url=f"https://api.example.com/item/{i}")
                for i in range(500)
            ]
            # Requests are automatically rate-limited
            responses = await asyncio.gather(*tasks)
            return responses
        finally:
            await requester.close()

    asyncio.run(fetch_with_rate_limit())

**Parameters:**

- ``max_rate``: Maximum number of requests allowed in the time window
- ``time_period``: Duration in seconds for the rate limit window

**Rate Limit Examples:**

.. code-block:: python

    # GitHub API: 5000 requests per hour
    AioHttpRateLimitRequester(max_rate=5000, time_period=3600)

    # Twitter API: 15 requests per 15 minutes
    AioHttpRateLimitRequester(max_rate=15, time_period=900)

    # Conservative: 1 request per second
    AioHttpRateLimitRequester(max_rate=1, time_period=1)

.. automodule:: core_https.requesters.aiohttp_rate_limit
   :members:
   :undoc-members:
   :show-inheritance:

Throttle vs Rate Limit Comparison
***************************************************

Understanding the Difference
-----------------------------------------------

+----------------------+-------------------------+---------------------------+
| Feature              | Throttle                | Rate Limit                |
+======================+=========================+===========================+
| **What it limits**   | Concurrent execution    | Requests per time period  |
+----------------------+-------------------------+---------------------------+
| **Time-based**       | No                      | Yes                       |
+----------------------+-------------------------+---------------------------+
| **Use case**         | Resource protection     | API quota compliance      |
+----------------------+-------------------------+---------------------------+
| **Example**          | Max 10 at once          | Max 100 per minute        |
+----------------------+-------------------------+---------------------------+
| **Mechanism**        | Semaphore               | Token bucket              |
+----------------------+-------------------------+---------------------------+
| **Dependencies**     | Built-in asyncio        | External aiolimiter       |
+----------------------+-------------------------+---------------------------+

**Visual Example:**

.. code-block:: text

    Throttle (max_concurrency=3):
    Timeline: [Request1][Request2][Request3] <- only 3 running
              [waiting...][waiting...][waiting...]

    Rate Limit (max_rate=10, time_period=1):
    Timeline: [10 requests] <- can all run concurrently
              [wait 1 second]
              [10 more requests]

When to Combine Both
-----------------------------------------------

Rate limiting and concurrency throttling are **independent** concerns.
To apply both simultaneously, compose the two classes via multiple inheritance:

.. code-block:: python

    import asyncio
    from core_https.requesters.aiohttp_rate_limit import AioHttpRateLimitRequester
    from core_https.requesters.aiohttp_throttle import AioHttpThrottleRequester

    class RateLimitedThrottleClient(AioHttpRateLimitRequester, AioHttpThrottleRequester):
        @classmethod
        def engine(cls) -> str:
            return "aiohttp_rate_limit_throttle"

    async def fetch():
        # Respects API quota AND caps local concurrency
        async with RateLimitedThrottleClient(
            max_rate=1000,        # API allows 1000/hour
            time_period=3600,     # One hour
            max_concurrency=10,   # Never more than 10 in-flight
        ) as client:
            response = await client.request(url="https://api.example.com/data")
            return await response.json()

    result = asyncio.run(fetch())

The MRO ensures each request flows through the rate limiter first, then the
semaphore, then the actual HTTP call:
``RateLimitedThrottleClient → AioHttpRateLimitRequester → AioHttpThrottleRequester → AioHttpRequester``