Proxy Integration With Python Requests

What Is Python Requests?

Requests is a third-party HTTP library for Python. It gives you a simple interface for sending GET, POST, PUT, DELETE, and other HTTP methods without dealing with the lower-level urllib module. A call that takes several lines in urllib takes one in Requests.

Developers use it for web scraping, API integration, automated testing, and any task that involves sending HTTP traffic. Because most of that traffic eventually hits rate limits or geo-restricted endpoints, knowing how to plug a proxy into Requests is a core skill for any serious project.

Why use a proxy with Python Requests?

A proxy sits between your script and the target server. Your outgoing request goes to the proxy first, the proxy forwards it, and the response comes back through the same path. The target server sees the proxy's IP, not yours. Let's take a look at some common reasons to use one:

• Web scraping at scale. A single IP gets blocked quickly. Proxies spread traffic across many IPs so one bad response does not break the whole job.
• Rate limits. Many APIs throttle per-IP. Multiple proxies multiply your effective throughput.
• Geo-targeting. If prices, search results, or content vary by country, proxies located in that country let you see the local version.
• IP reputation. Residential IPs are harder to flag than known data center ranges.
• Privacy. Your real IP stays off the target server's logs.

The two main categories are datacenter and residential. Datacenter proxies come from commercial hosting providers and are fast and cheap. Residential proxies use real consumer IPs assigned by ISPs and look like normal users. Rotating residential proxies change the exit IP automatically, either per request or on a fixed interval.

What you need before getting started

Install the Requests library

Requests is not part of the Python standard library. Install it with pip:

Verify the install:

Understand the proxy URL format

Every proxy URL follows the same pattern:

Each part matters:

• protocol: usually http, https, or socks5
• username and password: credentials from your provider, or you can skip them if you whitelisted your IP
• host: the IP address or hostname of the proxy server
• port: the port the proxy listens on

Requests uses this same URL structure whether the traffic is HTTP or HTTPS.

How to set a proxy in Python Requests

Requests accepts proxies through a dictionary passed to the proxies parameter. The dictionary maps each target protocol to the proxy URL that should handle it.

Basic proxy example with the proxies dictionary

You'll get the proxy's IP in your output.

The request goes out through the proxy at 45.43.134.135:43222. httpbin.org/ip returns the origin IP it sees, which should be the proxy's IP, not yours. That single call is enough to confirm the proxy works.

Add username and password authentication

Most commercial proxies require credentials. Embed them directly in the URL:

The output will show a result like the one below.

If the credentials are wrong or missing, you get this instead:

If your password contains special characters like @, :, or /, URL-encode them first:

Send both HTTP and HTTPS traffic through the proxy

The keys http and https refer to the protocol of the target URL, not the proxy itself. Both keys usually point to the same proxy. If you set only http and the target URL is HTTPS, Requests sends that call directly and leaks your real IP. Always set both keys, even when the values are identical.

How to use a proxy with requests.Session()

A Session object reuses the underlying TCP connection, persists cookies, and holds default headers and proxy settings across multiple requests. If you hit the same host many times, sessions are faster and cleaner than calling requests.get over and over.

You should see a result like the output below.

The first response confirms the proxy handled the call. The second confirms the custom User-Agent persisted across requests on the session.

Setting session.proxies once means every call on that session goes through the proxy automatically. Use sessions when you need login cookies, repeated calls to the same API, or consistent headers across a scraping job. Connection reuse also cuts handshake overhead, which adds up on long runs.

How to rotate proxies in Python Requests

Rotation spreads requests across multiple IPs, which lowers the chance of hitting rate limits or IP bans. There are two practical approaches.

Rotate from a list of proxies

If you have a pool of static proxies, pick one at random or in sequence for each request.

Example output across five calls.

Some IPs repeat, and that is expected. random.choice() picks a proxy at random, but it does not guarantee that each one will be used only once. For most scraping jobs, that is perfectly fine. If you need even traffic distribution, though, you will need a more controlled rotation strategy.

The bigger issue is maintenance. Once you manage your own proxy pool, you also have to monitor which proxies are dead, remove bad ones, track bans by IP, and retry failed requests intelligently. That works, but it quickly turns into extra infrastructure you have to build and maintain.

Use a rotating proxy endpoint

A rotating endpoint is a single hostname that assigns a different exit IP on every request or every session. You connect to one address and the provider handles the IP changes on their side.

Example output across five calls.

Conclusion

Now you should be able to route any Python Requests script through a proxy with confidence. You know how to build the proxy URL, add authentication, keep settings consistent across calls with requests.Session(), and rotate IPs either from your own list or through a single rotating endpoint.

If you run into any problems during the setup process or have trouble selecting a suitable proxy, our customer service team is always available to assist you with it. And if you want to see more tutorials like this one or with other third-party tools, you can check our integrations page.