Using Authenticated HTTP Proxies in aiohttp
aiohttp is a powerful async HTTP client/server framework for Python, built on asyncio. To scrape geo-restricted content or distribute requests across multiple IPs, you need authenticated HTTP proxies. This guide covers the two most reliable methods using a centralized config.py for your proxy string.
Prerequisites
- Python 3.7+ (
asyncio) pip install aiohttp aiohttp-proxy- A proxy with
login:password@host:portformat
Step 1: Centralize Proxy Credentials
Create a config.py alongside your scripts to store your proxy in one place:
# config.py
# Proxy string: login:password@host:port
PROXY = "modeler_pD5RQk:[email protected]:12203"Method 1: Using proxy_auth with BasicAuth
aiohttp’s built‑in BasicAuth works seamlessly for proxy authentication:
import asyncio
from aiohttp import ClientSession, BasicAuth
from config import PROXY
async def main():
# Split login and hostport
creds, hostport = PROXY.split("@", 1)
username, password = creds.split(":", 1)
proxy_url = f"http://{hostport}"
proxy_auth = BasicAuth(username, password)
async with ClientSession() as session:
async with session.get(
"https://httpbin.org/ip",
proxy=proxy_url,
proxy_auth=proxy_auth,
timeout=10
) as resp:
data = await resp.json()
print(f"IP via proxy: {data['origin']}")
if __name__ == '__main__':
asyncio.run(main())Method 2: aiohttp-proxy Connector
For advanced proxy support (HTTP, HTTPS, SOCKS), use aiohttp-proxy:
import asyncio
from aiohttp import ClientSession
from aiohttp_proxy import ProxyConnector, ProxyType
from config import PROXY
async def main():
creds, hostport = PROXY.split("@", 1)
username, password = creds.split(":", 1)
host, port = hostport.split(":", 1)
connector = ProxyConnector(
proxy_type=ProxyType.HTTP,
host=host,
port=int(port),
username=username,
password=password
)
async with ClientSession(connector=connector) as session:
async with session.get("https://httpbin.org/ip", timeout=10) as resp:
data = await resp.json()
print(f"IP via connector proxy: {data['origin']}")
if __name__ == '__main__':
asyncio.run(main())Best Practices & SEO Tips
- Keywords: aiohttp proxy auth, aiohttp-proxy
- Store credentials securely (env vars or config file).
- Use timeouts to avoid hangs (
timeout=10). - Rotate proxies by relaunching sessions per proxy.
- Respect target site’s rate limits and
robots.txt.
Conclusion
You now have two battle‑tested methods to integrate authenticated HTTP proxies into your aiohttp workflows:BasicAuth for simplicity and aiohttp-proxy for full-feature support. Centralize your proxy string in config.py and build robust, anonymous, and reliable async clients.
