Common HTTP Headers Explained: Reading Them in curl, Python and PHP

Every HTTP request and response carries headers: name-value pairs that describe the message and control how clients, servers, caches and proxies treat it. When an API call fails with no obvious reason, the answer is usually in a header: a missing `Content-Type`, a CORS header that was never sent, a cache that served a stale copy, or an `Authorization` header a proxy stripped. Below are the rules code gets wrong when it reads headers, the request and response headers worth checking first, and how to see what was really sent from curl, Python, PHP and the browser.

What raw headers look like

HTTP/1.1 200 OK
Date: Mon, 28 Sep 2026 10:15:00 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 348
Cache-Control: private, max-age=0
ETag: "5f2a-9c1"
Set-Cookie: session=abc123; Path=/; Secure; HttpOnly; SameSite=Lax
Set-Cookie: theme=dark; Path=/; Max-Age=31536000
Access-Control-Allow-Origin: https://app.example.com
Strict-Transport-Security: max-age=31536000; includeSubDomains

Each line is a field name, a colon and a value. Headers end at the first empty line, after which the body starts. In HTTP/2 and HTTP/3 headers travel in a compressed binary form, but tools show them in this same textual shape.

Parsing rules that trip people up

Names are case-insensitive, and each library handles that in its own way. Node.js lowercases incoming names in req.headers, fetch's Headers object matches any spelling, Python's requests keeps r.headers in a case-insensitive dictionary, and PHP puts request headers into $_SERVER in upper case with an HTTP_ prefix and underscores for dashes, so X-Request-ID becomes $_SERVER['HTTP_X_REQUEST_ID']. Bugs appear when code copies headers into a plain dictionary or array and then looks up Content-Type in a map that holds content-type.

Most headers can repeat, and repeated values are equivalent to one comma-separated value: two Accept lines mean the same as Accept: a, b. Set-Cookie is the exception. Its values can contain commas, in the Expires date, so each cookie must stay a separate header and must never be joined. Code that stores headers in a plain dictionary silently keeps only the last cookie; use headers.getSetCookie() in modern JavaScript or response.raw.headers.getlist('Set-Cookie') in Python's requests.

Leading and trailing whitespace around values is not significant. Old multi-line "folded" headers, continued on a line starting with a space, are obsolete: RFC 9112 says servers should reject them or replace the fold with a space.

Headers have size limits. Nginx's default buffers accept 8 KB per header line, and Node.js rejects total request headers over 16 KB by default. Oversized cookies or JWTs cause 431 Request Header Fields Too Large or a bare 400.

Request headers that matter for APIs

Content-Type says what the body is. Sending JSON without Content-Type: application/json is the most common cause of a request body arriving empty or as a form. Accept says what the client wants back; some APIs return HTML error pages unless it asks for JSON. Authorization carries credentials, such as Bearer <token> or Basic <base64>. User-Agent identifies the client, and some APIs require one.

Idempotency-Key makes retries safe on payment and order APIs: the server returns the original result when the same key arrives twice. If-None-Match and If-Modified-Since make conditional requests that return 304 Not Modified when nothing changed. X-Request-ID or traceparent (W3C Trace Context) correlate a request across services and logs.

Response headers that explain odd behaviour

Caching problems usually show up in headers your application never set. Age: 3500 means a shared cache served a copy it has held for almost an hour, and CDN headers such as X-Cache or Cloudflare's CF-Cache-Status say whether the response was a hit. Vary lists request headers that change the response, such as Accept-Encoding or Origin; a missing Vary: Origin on a CORS response can make a CDN serve one origin's headers to another. When users report stale pages after a deploy, compare Cache-Control, Age and ETag on the HTML itself before blaming the browser.

CORS: the browser only exposes a cross-origin response to JavaScript if it carries Access-Control-Allow-Origin matching the calling origin. For requests with credentials, the value cannot be * and Access-Control-Allow-Credentials: true is also needed. Preflight OPTIONS requests need Access-Control-Allow-Methods and Access-Control-Allow-Headers. CORS errors appear only in the browser console; curl never shows them, because CORS is enforced by browsers, not servers.

Security headers are often added by the web server or the CDN rather than by the application, so audit them against the production URL, not a local build. curl -sI https://example.com | grep -iE 'strict-transport|content-security|x-content-type' tells you in a second which ones are missing, and running it against staging and production side by side catches the config that was never copied over.

Seeing exactly what was sent: curl, code and the browser

curl -sI https://api.example.com/health            # response headers only (HEAD request)
curl -s -D - -o /dev/null https://api.example.com  # response headers of a real GET
curl -v https://api.example.com/orders             # request (>) and response (<) headers
curl -v -H 'Origin: https://app.example.com' \
     -X OPTIONS https://api.example.com/orders     # simulate a CORS preflight

In code, the request that actually left your program is usually one attribute away. With Python's requests, r.headers holds the response headers and r.request.headers the ones that were sent, including defaults such as User-Agent. In Node, res.headers gives the response headers with lowercase names. In PHP, getallheaders() returns the incoming request headers, headers_list() the response headers your script has queued so far, and get_headers($url, true) fetches a remote URL's response headers as an array.

In the browser, the Network tab shows request and response headers for each call. Enable "Preserve log" to keep them across redirects. The headers the browser shows as sent are not always what reached your application, because proxies, CDNs and load balancers add, remove and rewrite headers. When in doubt, log the incoming headers in the application itself, with Authorization and Cookie redacted, and compare.

Two quick checks solve many API mysteries: whether Content-Type on the request matches the body you think you sent, and whether a reverse proxy passes Authorization and X-Forwarded-* headers through to the application.

Custom headers

Name custom headers clearly and document them. The old X- prefix convention was deprecated by RFC 6648, because prefixed headers that became standard were stuck with the prefix. A plain descriptive name such as Request-Id is fine, and many existing X- headers remain in use. For browser clients, remember that custom request headers trigger a CORS preflight, and custom response headers are hidden from JavaScript unless listed in Access-Control-Expose-Headers.

Sources and further reading

RFC 9110 defines HTTP header semantics, including case-insensitivity and field combination, and RFC 9112 covers HTTP/1.1 message syntax and obsolete line folding. RFC 6265 defines Set-Cookie, RFC 9111 defines caching, and the WHATWG Fetch Standard defines CORS. RFC 6648 deprecates the X- prefix. MDN documents each header, and the nginx and Node.js documentation describe their header size limits.

Frequently asked questions

How do I read request and response headers in Python?

With requests, `r.headers` holds the response headers in a case-insensitive dictionary and `r.request.headers` the headers that were actually sent. Repeated headers are joined into one comma-separated string there, so read cookies from `r.cookies` rather than parsing `Set-Cookie` by hand.

How do I show response headers with curl?

`curl -sI URL` sends a HEAD request and prints only the response headers. Some servers answer HEAD differently from GET, so for the headers of a real request use `curl -s -D - -o /dev/null URL`, or `curl -v` to see request and response headers together.

Why do I only get one cookie when parsing Set-Cookie headers?

Your code stores headers in a dictionary that keeps the last value. `Set-Cookie` headers must stay separate; use `headers.getSetCookie()` in JavaScript or a multi-value accessor in your HTTP library.

Which headers do I need for a JSON API request?

Usually `Content-Type: application/json` when sending a body, `Accept: application/json`, and `Authorization` with your credentials. Some APIs also require a `User-Agent` or an `Idempotency-Key` for writes.

Why does my request work in curl but fail with a CORS error in the browser?

CORS is enforced by the browser, not the server. The response must include `Access-Control-Allow-Origin` for your origin, and preflighted requests need the allowed methods and headers.

What causes 431 Request Header Fields Too Large?

The request headers exceed the server's limit, usually because of large cookies or long JWTs. Reduce cookie size, avoid putting bulky data in tokens, or raise the limit in the server or proxy.