Why Half of Large Retail Homepages Refuse a Bot GET

Overview

Digitarise measured one HTTPS GET per homepage with no JavaScript executed. The retail result is the article’s starting point, not an industry generalisation: Retail chains (n=17 of 35 sampled, 51% refused an automated request, measured 2026-07-29): median server response 297 ms; median HTML 833 KB; skip link present on 41%; main landmark on 71%; median alt-text coverage 93%; median trackers 1; median third-party domains 11; consent mechanism on 59%. The adjacent commerce row is similar but not identical: DTC ecommerce (n=20 of 38 sampled, 47% refused an automated request, measured 2026-07-28): median server response 196 ms; median HTML 846 KB; skip link present on 55%; main landmark on 90%; median alt-text coverage 64%; median trackers 2; median third-party domains 16; consent mechanism on 65%.

The problem solved by refusing an automated request is narrower than the usual phrase “bot blocking” suggests. At the systems level it is a request-classification problem at the first HTTP boundary: decide whether a single unauthenticated GET for a high-value document should receive the normal HTML, a challenge page, a rate-limit response, or a refusal. The measured retail rows show that the choice is common in these samples, but they do not prove a sector-wide norm. They also show that refusal is not simply a symptom of slow origin infrastructure: in the same measurements, DTC ecommerce (n=20 of 38 sampled) had median server response 196 ms, while Retail chains (n=17 of 35 sampled) had median server response 297 ms. The gate can be fast because it often runs before the application renders the homepage.

How it works

A modern refusal path usually sits in front of the origin application: DNS resolves to an edge proxy, the client negotiates TLS, the proxy parses the HTTP request, and a policy engine decides whether to forward, challenge, throttle, or reject. The inputs are not a single “bot” bit. They can include the HTTP method, path, Host header, User-Agent field, Accept and Accept-Language fields, connection reuse, source network reputation, request rate, cookie state, and whether browser-only request metadata is present. Where browsers participate, Fetch Metadata request headers such as Sec-Fetch-Site, Sec-Fetch-Mode, Sec-Fetch-Dest, and Sec-Fetch-User can help distinguish a top-level navigation from a subresource fetch. A bare HTTPS GET issued by a script without JavaScript execution lacks many signals created during an interactive session.

That architecture explains the measured pattern. Retail homepages are heavy entry documents with many dependencies: Retail chains (n=17 of 35 sampled) had median HTML 833 KB and median third-party domains 11; DTC ecommerce (n=20 of 38 sampled) had median HTML 846 KB and median third-party domains 16. A static GET sees the initial document only, while the production browser path often expects cookies, consent state, client hints, JavaScript-created beacons, and subsequent resource requests. The refusal decision may therefore occur before any homepage component is selected. This is also why a refused request can look operationally healthy in timing data: the edge proxy can return a compact denial faster than an application can assemble personalized HTML.

What the standard says

HTTP does not define a standard test for whether a requester is automated. RFC 9110, HTTP Semantics, defines the GET method as a request for a current representation of the target resource and classifies GET as a safe method, meaning the client does not request state-changing semantics. Safe does not mean entitled to content. RFC 9110 defines 403 Forbidden as a status code for a request the server understood but refuses to fulfill. RFC 6585 defines 429 Too Many Requests for rate limiting and says the response representations SHOULD include details explaining the condition and MAY include a Retry-After header. If a denial varies by request fields, RFC 9110’s Vary header and RFC 9111’s caching rules matter, because a shared cache must not reuse a decision for a different request context unless the response is explicitly cacheable for that context.

The adjacent standards are easy to overstate. RFC 9309, Robots Exclusion Protocol, specifies robots.txt records such as User-agent, Allow, and Disallow, but it is a voluntary crawling convention, not an authorization protocol and not a browser requirement. The WHATWG Fetch Standard defines request modes, credentials modes, and the browser fetch model; Fetch Metadata Request Headers are specified by W3C Web Application Security work, but non-browser clients are not required to emit them. TLS 1.3 is specified in RFC 8446, but TLS fingerprinting is an observational classification technique, not a normative access-control signal. For page structure, the HTML Living Standard defines the main element, and WCAG 2.2 Success Criterion 2.4.1, Bypass Blocks, is the current accessibility criterion commonly satisfied by skip links. WCAG 2.2 Success Criterion 1.1.1, Non-text Content, is the relevant current criterion for meaningful image text alternatives.

Trade-offs

The most visible trade-off is observability versus protection. The cross-sector ordering supplied with the same rows places refusal at B2B SaaS 6, AI platforms 6, Developer tools 6, Healthcare 17, Fintech 20, DTC ecommerce 47, Retail chains 51, best first. Because the retail rows have high refusal rates, external monitors that use a simple HTTPS GET will see less of the real homepage. That affects uptime checks, archival capture, accessibility crawls, link preview agents, and standards audits. The measurement does not say the refusals are wrong; it says that a non-browser fetch is no longer a neutral proxy for “homepage availability” in these samples.

The second trade-off is that refusal can hide or distort other qualities. Retail chains (n=17 of 35 sampled) showed median alt-text coverage 93%, but skip link present on 41% and main landmark on 71%. DTC ecommerce (n=20 of 38 sampled) showed skip link present on 55%, main landmark on 90%, and median alt-text coverage 64%. Those figures are only for pages not refused in the measurement set. If the refusal response is an interstitial without a main landmark, without a bypass link, or with images lacking alternatives, the access-control layer becomes part of the user-facing document and must meet the same structural expectations. The consent mechanism figures add another trade-off: Retail chains (n=17 of 35 sampled) had consent mechanism on 59%, and DTC ecommerce (n=20 of 38 sampled) had consent mechanism on 65%, so a gate that depends on post-consent JavaScript can be invisible to a static GET while still shaping the browser path.

Failure modes

The common false positive is treating “no JavaScript executed” as equivalent to hostile automation. That catches simple scrapers, but it can also catch synthetic availability checks, accessibility scanners, security research tools, text-mode browsers, and internal regression tests. Diagnosis starts by separating transport, HTTP, and policy layers: confirm TLS negotiation, capture the HTTP version, method, status code, response headers, and body class, then correlate the edge log entry with the policy rule identifier. If the denial is 403, RFC 9110 frames it as a refusal rather than a missing page. If it is 429, RFC 6585 makes Retry-After the documented mechanism for communicating when the client may try again.

A second failure mode is cache amplification. If an edge proxy refuses a request based on User-Agent, cookies, Fetch Metadata headers, or source network properties, but emits cacheable HTML without an appropriate Vary or Cache-Control policy, a denial can be reused for clients that should have received the homepage. A third failure mode is semantic mismatch: robots.txt may allow a path while the HTTP gateway refuses it, or robots.txt may disallow a crawler while the gateway still serves the page. That is not a standards conflict, because RFC 9309 is advisory for crawlers and HTTP authorization remains separate, but it is an operational contradiction. A fourth failure mode is method inconsistency: HEAD should normally reveal metadata corresponding to GET semantics, but many gates forget to classify HEAD coherently, producing monitoring noise.

A minimal implementation

A correct minimal design is explicit, early, and reversible. Put the decision at the edge proxy before expensive homepage rendering. Define a small set of outcomes: allow normal GET and HEAD for the homepage, return 403 when a documented policy refuses the request, return 429 when a documented rate threshold is exceeded, and never use a JavaScript challenge as the only path for essential access. Log the decision inputs at a coarse level: timestamp, request path, method, response status, policy rule identifier, and whether browser request metadata was present. The implementation should not depend on a single spoofable header, and it should not classify the absence of Sec-Fetch-* headers as sufficient evidence by itself, because the Fetch Metadata specification describes browser behavior rather than a universal client obligation.

The minimal response also needs HTTP hygiene. A 403 or 429 body should be small, machine-readable where appropriate, and structurally valid HTML if it is shown to people. If the decision varies by fields, emit Vary for those fields or make the refusal uncacheable with Cache-Control; if retry is intended after throttling, emit Retry-After with 429. Keep robots.txt aligned with the published crawl policy under RFC 9309, while remembering that robots.txt cannot substitute for refusal logic. Finally, test the homepage through three paths: an ordinary browser navigation, a no-JavaScript browser session, and a single HTTPS GET matching the Digitarise measurement method. The point is not to make every automated request succeed; it is to ensure that refusal is a deliberate HTTP outcome, not an accidental side effect of a heavyweight retail frontend.