How to Download Files With cURL: Flags, Proxies, Fixes

Four flags handle most production download work.

Follow redirects with -L

Signed S3 URLs, CDN edges, and most modern download links return a 301 or 302 first. Without -L, cURL writes the redirect response to disk and that's where it stops:

Resume an interrupted download with -C -

For large files on flaky networks, never restart from zero. The -C - flag reads the local file size and asks the server for the rest:

The dash after -C is doing real work. It tells cURL to figure out the resume offset itself. If the server doesn't honor range requests, you get an error instead of a silent restart from byte zero, which is what you want.

Authenticated downloads with -u and bearer tokens

Basic auth uses -u:

Modern APIs prefer bearer tokens, which go through the -H flag:

For header-based auth and debugging the headers cURL is actually sending, see the guide on sending HTTP headers with cURL.

Bulk downloads with {} and [1-N] expansion

cURL expands brace and bracket patterns into multiple requests in a single command:

This works for any URL where the variable part follows a clean pattern. For lists that do not fit a pattern, jump to the bulk rotation section below.

Direct downloads work fine until the server starts caring who you are. Three situations push you toward a proxy:

• The target enforces per-IP rate limits and the job is pulling more than fits inside one window.
• The file is geo-restricted and the server is in the wrong region.
• The same IP has been hitting the endpoint long enough to pick up reputation flags, and URLs that used to work are now coming back 403.

-x syntax for HTTP, HTTPS, and SOCKS5

The -x flag accepts any supported proxy URL:

The socks5h:// scheme resolves DNS through the proxy and prevents leaks. Plain socks5:// resolves locally, which can give your real location away.

Matching proxy type to download job

Not every proxy fits every download:

• Datacenter proxies move data fast on permissive targets where IP reputation is not the bottleneck.
• ISP proxies hold a session IP stable across multi-step downloads behind a login wall.
• Rotating residential or mobile proxies handle hardened targets that block datacenter ranges outright.
• SOCKS5 proxies cover non-HTTP downloads (FTP, SCP, custom protocols) where HTTP-only proxies fail.

Bulk Downloads With IP Rotation

One proxy IP buys you a few requests. After that, things break in a predictable order. The rate limiter returns 429. The WAF notices the traffic concentration and issues a temporary block. The IP lands on a reputation list shared across CDNs and stays there for days. Every retry pushes you further down that list.

A backconnect proxy endpoint avoids the whole sequence by handing out a different upstream IP on every connection. Pair it with a simple bash loop:

The loop reads one URL per line and opens a fresh proxy connection on each request, which is what makes the IP swap. The two extra flags cover the obvious edge cases: a per-request timeout so a single slow file doesn't hang the queue, and a small retry budget for flaky network errors.

For ongoing bulk work, point the loop at a Verifying Downloads and Catching Silent Failures sized to your concurrency. You want rotation on every request. Less than that and you're back to the static-proxy version above, just with extra steps.

A successful cURL exit code does not always mean you saved the file you expected. It might be an HTML block page that came back as a 200. It might be half an archive. It might be a login page saved as report.csv because the session expired two URLs ago. A few checks catch most of it.

Capture the HTTP status with --write-out

This catches 403, 404, 429, and 5xx responses that would otherwise leave a junk file behind.

Compare file size against Content-Length

For files larger than a few MB, compare the size on disk to the size the server reported:

This check works only when the final server response includes a reliable Content-Length header.

For a deeper look at reading server responses, the guide on HTTP response headers in cURL walks through the rest of the useful header values.

SHA256 check when integrity matters

If the vendor publishes a checksum, verify it before using the file. This matters for software releases, production datasets, backups, and any download where silent corruption breaks something downstream.

HashiCorp Terraform is a clean example because every release ships with a SHA256SUMS file next to the archives:

Expected output:

On macOS, swap sha256sum -c for shasum -a 256 -c:

403 Forbidden

The server accepted the connection and refused the request. Three causes dominate. The IP is rate-limited or on a blocklist. The User-Agent is missing or flagged. The auth header is wrong or expired. Set a browser-like User-Agent with -A "Mozilla/5.0", route through a residential or mobile proxy, and re-check your credentials.

416 Requested Range Not Satisfiable

-C - failed because the server's file changed since your partial download started. The local byte offset no longer matches the remote file. Delete the partial file and start fresh.

SSL handshake failures

The most common cause is a corporate proxy intercepting TLS, or a server using an outdated cipher. The wrong fix is -k. The right fix is to update your CA bundle or stop using a proxy that man-in-the-middles TLS. Full breakdown in the guide on ignoring SSL certificate errors in cURL.

Empty file with exit code 0

If the server returned 200 with an empty body, the response was probably gzipped and the client didn't decompress it. Add --compressed and re-run the status-code check.

A few flags do most of the work in cURL, and once you know them, downloading files isn't really a thing you have to think about. The same small set handles a one-off pull and a script churning through thousands of URLs overnight. The bit nobody covers is verification, which is what keeps a working script from sitting broken for a month before anyone notices.

When a download fails, it's almost never cURL. It's the server pushing back, usually through rate limits or IP reputation. That's why you need to match the proxy type to what block you're hitting, rotate the IP at volume, and most of those failures will go away.

If downloads end up inside a bigger pipeline, the cURL with Python guide is where to go next.