Scrapy's built-in proxy support is made for HTTP proxies, not SOCKS5. Setting meta['proxy'] to a socks5:// link does not work with the default download handler. The reliable fix is a small local forwarder: it logs in to your SOCKS5 proxy, and Scrapy talks to it as a plain HTTP proxy.
Elusive proxies are SOCKS5 with a username and password. There are no HTTP proxies and no IP-whitelist login. So a Scrapy project with Elusive needs the forwarder below.
Key takeaways
- Scrapy's
HttpProxyMiddlewareand default handler are meant for HTTP and HTTPS proxies. - The simple fix is a local forwarder (
gostor Privoxy). Then usemeta['proxy'] = 'http://127.0.0.1:8080'. - We found no healthy third-party SOCKS middleware for current Scrapy. Use a forwarder.
- Scrapy 2.15 added an experimental httpx handler with SOCKS5. Its docs say not to use it in production yet.
- For JavaScript pages,
scrapy-playwrightworks with the same forwarder.
What Scrapy does with a proxy
The Scrapy docs say the proxy for a request is read from request.meta['proxy']. If you set nothing, the middleware reads the http_proxy, https_proxy and no_proxy environment variables, like Python's urllib. A value in meta wins over the environment.
The docs describe proxy URLs as http:// (and less often https://). The actual work happens in the download handler, not the middleware. The default HTTP/1.1 handler has no SOCKS code. The docs' comparison table lists SOCKS proxies as supported only by the new experimental HttpxDownloadHandler, which they call "not yet recommended for production environments." It was added in Scrapy 2.15.0.
So with default settings, SOCKS5 in a Scrapy proxy link will not work, and even a SOCKS-capable handler would be a risk for a real project.
Option A: a local forwarder
Run a program on your machine that listens as an HTTP proxy and forwards to your SOCKS5 proxy with the login. Scrapy then sees an ordinary HTTP proxy.
gost
gost is one Go binary. This is the v3 syntax from the gost docs, with -L for the local listener and -F for the upstream:
gost -L http://127.0.0.1:8080 -F socks5://USER:PASS@HOST:PORTKeep it bound to 127.0.0.1 so only your machine can use it.
Privoxy
Privoxy is an older HTTP proxy that can forward to SOCKS. Put this line in its config file:
forward-socks5 / USER:PASS@HOST:PORT .The Privoxy manual shows the form forward-socks5 target_pattern [user:pass@]socks_proxy[:port] http_parent[:port], and says user and password can be used for SOCKS5 authentication. The manual does not say which version added that, so use a recent release (the current stable is 4.2.0). By default Privoxy listens on 127.0.0.1:8118.
With forward-socks5, the manual says name resolution happens on the SOCKS server, which is what you want. The forward-socks5t form works the same and adds Tor-specific extras you don't need.
We use gost below because it is a single command.
A minimal spider
Per-request proxy, using the current start() method (Scrapy 2.13 and newer):
import scrapy
class IpSpider(scrapy.Spider):
name = "ip"
custom_settings = {"DOWNLOAD_DELAY": 1, "ROBOTSTXT_OBEY": True}
async def start(self):
yield scrapy.Request(
"https://example.com",
meta={"proxy": "http://127.0.0.1:8080"},
)
def parse(self, response):
yield {"url": response.url, "status": response.status}No username or password goes in the link. The forwarder holds it.
If you want every request to use the proxy, skip meta and set the variables before you start:
export http_proxy=http://127.0.0.1:8080
export https_proxy=http://127.0.0.1:8080
scrapy crawl ipSet both: Scrapy reads them by the scheme of each target URL.
Please respect each site's terms and robots.txt, and keep your rate low. The snippet sets a one second delay.
Rotating and Sticky
A forwarder talks to one upstream, so plan it by product.
- Sticky: run one forwarder per proxy, each on its own local port (8080, 8081, ...). In the spider, choose the port per request in
meta['proxy']. Sticky keeps an IP up to 24 hours on a real device, so be ready to replace it. - Static ISP: same setup, but the IP stays the same for 7 to 90 days.
- Rotating: one forwarder is enough. The IP changes on every request behind it, and billing is per GB, so don't download more than you need.
Browse the options in the catalog. For which type fits which job, see which proxy for which job.
Option B: third-party SOCKS packages
We looked for a maintained SOCKS add-on for Scrapy and did not find a good one.
- There is no
scrapy-sockspackage on PyPI under that name. scrapy-proxiesis a random-proxy picker for lists of proxies. Its last release was in 2017. It does not add SOCKS.scrapy-rotating-proxieslast released in 2019.
We would not build a project on these. Some forks and snippets exist on GitHub, but we can't vouch for them with current Scrapy. Use the forwarder.
Option C: scrapy-playwright
For pages that need a real browser, scrapy-playwright (0.0.48, Python 3.10+) drives Playwright from Scrapy. Playwright can't send a SOCKS5 password either, so the same forwarder applies. See Playwright SOCKS5 authentication for why.
The scrapy-playwright docs set a proxy at browser level through PLAYWRIGHT_LAUNCH_OPTIONS:
import scrapy
class PlaywrightSpider(scrapy.Spider):
name = "pw"
custom_settings = {
"DOWNLOAD_HANDLERS": {
"https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
},
"TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
"PLAYWRIGHT_LAUNCH_OPTIONS": {"proxy": {"server": "http://127.0.0.1:8080"}},
}
async def start(self):
yield scrapy.Request("https://example.com", meta={"playwright": True})
def parse(self, response):
yield {"title": response.css("title::text").get()}Note that meta['proxy'] is for the normal handler. Requests sent with meta={"playwright": True} use the browser's proxy setting instead. The docs also show per-context proxies through PLAYWRIGHT_CONTEXTS, if you want a different proxy for each context.
Check that DNS goes through the proxy
With the forwarder, Scrapy sends the site name to it, not an IP. Privoxy's manual says SOCKS5 lookups happen at the SOCKS server. In gost's source, names go to the SOCKS5 server as domain names, though its docs don't promise it, so test it. Elusive proxies accept domain names. See SOCKS5 vs SOCKS5h for the background, and use the DNS leak check steps from the machine that runs the crawl. For the same idea in plain code, see SOCKS5 proxy in Python, Node and Go.
Common errors
**Requests fail or time out with a socks5:// link in meta['proxy'].** The default handler does not speak SOCKS. Point meta['proxy'] at the forwarder instead.
**Connection refused on 127.0.0.1.** The forwarder is not running, or the port in meta is wrong.
**Proxy works for http:// sites but not https://.** You only set http_proxy. Set https_proxy too, or use meta['proxy'].
A login in the link makes it fail. Don't put the SOCKS5 login in the Scrapy link. Only the forwarder gets it.
Everything stopped after some hours. A Sticky IP may have gone offline. Replace the proxy, then restart the forwarder.
FAQ
Does Scrapy support SOCKS5 proxies?
Not in the default setup. The built-in middleware and handler are for HTTP and HTTPS proxies. Scrapy 2.15 added an experimental httpx handler that lists SOCKS5, but its docs say it is not yet recommended for production.
How do I use a SOCKS5 proxy with a username and password in Scrapy?
Run a local forwarder such as gost: gost -L http://127.0.0.1:8080 -F socks5://USER:PASS@HOST:PORT. Then set meta['proxy'] to http://127.0.0.1:8080.
Is there a maintained Scrapy SOCKS middleware?
We did not find one that works with current Scrapy. The common names are old or not on PyPI. A forwarder is simpler and does not depend on Scrapy versions.
Can I set the proxy for the whole project?
Yes. Set the http_proxy and https_proxy environment variables, or add the meta['proxy'] value in a small downloader middleware.
Does scrapy-playwright work with SOCKS5?
Not with a password. Playwright does not support SOCKS5 login, so use the forwarder and give Playwright its HTTP address.