Libraries and frameworks
Python urllib proxy setup
Python urllib works with any HTTP proxy, and DataImpulse proxies are HTTP proxies with username and password authentication. urllib ships with the Python standard library and needs no extra packages. Follow the steps to connect them, then use the checklist to make sure the connection does what you expect.
You need an active DataImpulse proxy first. Create an account, add funds from US$1, and order a proxy in one of the 62 countries we stock. The dashboard then shows the host, port, username and password. If you have not done that yet, start with the quick-start guides.
- Tool
- Python urllib
- Category
- Libraries and frameworks
- Proxy protocol
- HTTP with username and password
- Example host
- 203.0.113.10
- Countries in stock
- 62
What you need from your dashboard
Every DataImpulse proxy has the same four values, and each proxy has its own. The example values below are placeholders, so always copy the real ones from your dashboard. There is nothing to add to the username: the country was chosen when you ordered.
| Setting | Example | What it is |
|---|---|---|
| Host | 203.0.113.10 | The proxy's address, usually an IP, shown with the proxy in the dashboard |
| Port | 8080 | The proxy's port, which differs from proxy to proxy |
| Username | username | The proxy username, used exactly as shown |
| Password | password | The proxy password. Treat it like any other secret |
| Protocol | HTTP | Use an http:// proxy URL, even for HTTPS sites |
Set up Python urllib with a DataImpulse proxy
Python urllib is a library. The example below is complete and runs as it is once you replace the host, port, username and password with your own values.
- Make sure Python urllib is installed and up to date.
- Build the proxy URL from your dashboard values: http://USERNAME:PASSWORD@HOST:PORT.
- Pass the proxy to Python urllib as shown in the example.
- Request an IP echo page such as https://httpbin.org/ip and read the address it returns.
- Add a timeout and a retry limit before you run a larger job.
import urllib.request
proxy = "http://username:[email protected]:8080"
handler = urllib.request.ProxyHandler({"http": proxy, "https": proxy})
opener = urllib.request.build_opener(handler)
print(opener.open("https://httpbin.org/ip", timeout=30).read().decode())Countries and addresses
You choose the exit country when you order, and every proxy in the order is an IP registered in that country. There is nothing to add to the username in Python urllib: enter the credentials exactly as the dashboard shows them.
Each proxy keeps the same address for the whole rental period, so a session in Python urllib stays on one address by default. To work from another country, or to spread work across several addresses, order more proxies and give each one its own profile, connection or entry in Python urllib.
Check that Python urllib is really using the proxy
Do not assume. Verify the connection before your real job starts, and again after any change to the settings.
- Request https://httpbin.org/ip or any IP lookup page with Python urllib.
- Compare the address with your own. It must be different.
- Check the country in two lookup tools and compare it with the country you ordered.
- Repeat the request once. The address should stay the same, because each proxy keeps its address for the rental period.
Fixing common Python urllib proxy problems
These are the problems we see most often, with the fix that usually works.
- 407 Proxy Authentication Required. The username or password is wrong, or the tool did not send them. Copy both from the dashboard again, from the same proxy as the host and port.
- Connection timed out. The host or port is wrong, or a firewall blocks the port. Try curl first to separate a tool problem from a network problem.
- The exit country is not the one you expected. Look the address up in two lookup tools, because databases do not always agree, and check that you used the proxy from the order for that country.
- TLS or certificate errors on HTTPS pages. Use an HTTP proxy URL (http://...) for the proxy itself, even when the website is HTTPS. The proxy opens a tunnel and does not inspect the traffic.
- Pages load slowly. Lower your concurrency, reuse connections and try a proxy in a country closer to the target.
- The site blocks you. Slow down, send realistic headers, and consider an ISP proxy if the site is strict about datacenter ranges.
- Your balance ran out. Check the dashboard. Orders are paid from your balance, and an expired rental stops working.
Good habits when you use Python urllib with a proxy
A proxy only changes the address your traffic comes from. These habits make the difference between a setup that lasts and one that does not.
- Set a timeout on every request. A request without a timeout can hang for minutes and block a whole worker.
- Retry with backoff, not in a tight loop. Wait a little longer after each failure and stop after a few attempts.
- Limit concurrency per target site. A handful of parallel requests is usually enough, and hundreds from one address is how blocks start.
- Send realistic headers and respect robots rules and rate limits on the sites you read.
- Log the exit country and the status code of every request so you can see problems early.
urllib is fine for small scripts. For retries and sessions, use Requests or HTTPX.
Using proxies responsibly in any country
A proxy is a networking tool and it is lawful in most places, but what you do with it still has to be lawful and fair. Follow the laws that apply to you and to the place you send traffic to, read the terms of the sites you visit, honour robots rules and rate limits, and never use an address to access accounts or data that are not yours. DataImpulse cannot give legal advice, and our acceptable use policy applies to every order.
In practice that means reading public pages, checking your own sites and ads, and testing your own products. If a task would be unwelcome to the site owner when done by hand, it is unwelcome when done through a proxy too.
Frequently asked questions
Does Python urllib work with DataImpulse proxies?
Yes. DataImpulse proxies are standard HTTP proxies with username and password authentication, and Python urllib supports that kind of proxy. Use the setup above and verify the address with an IP lookup.
What proxy settings do I enter in Python urllib?
Enter the host, the port, the username and the password of your proxy from the DataImpulse dashboard. Use HTTP as the proxy type.
Why does Python urllib show a 407 error?
A 407 means the proxy did not accept the credentials. Copy the username and password again, check for extra spaces, and make sure they belong to the same proxy as the host and port. Also check that the rental period has not ended.
Can I choose the country in Python urllib?
You choose it when you order. Every proxy in the order is registered in that country, so use the proxy for the country you need and check it with an IP lookup.
How do I keep the same IP in Python urllib?
Keep using the same proxy. Each proxy keeps its address for the whole rental period. For a different address, use another proxy from your order.
Is it safe to put my proxy password in Python urllib?
It is as safe as any other password in that tool. Do not share exported settings or projects that contain it, do not commit it to a repository and tell support right away if it leaks.
Ready to try it?
Create an account, add funds from US$1 and pick your country and proxy type.