The State of AI Agent Readiness: 50 Y Combinator companies

Analysis

Why blocking scanners is an outdated security measure

9 of the 50 companies we tested (18%) refused our scanner outright. The scanner never saw a single page. That raises a bigger question: who else is getting turned away?

What "Blocked scan" means

When a site shows "Blocked scan", our request got an access-denied response (usually HTTP 403), a bot challenge page, or a JavaScript puzzle in place of the real site. This is the job of web application firewalls and bot-management tools. They were built to stop scrapers, credential stuffing, DDoS traffic and price harvesting, so they treat any automated visitor that isn't a browser as suspicious.

That made sense when nearly all useful automated traffic came from a handful of search engines. In 2026 it no longer holds. AI assistants like ChatGPT Search, Perplexity, Claude and Gemini read live pages to answer buyers' questions. Coding copilots read docs, and shopping and booking agents act for real customers. A blanket block treats all of them like attackers.

Why it's outdated

• You risk reduced visibility in AI answers. If an assistant can't read your pricing or docs, it may rely on older sources or cite a competitor it can read. Blocking one scanner does not prove that all AI assistants are blocked.
• Agents can't act for your customers. A buyer's agent that hits a challenge wall can't compare your plans, check whether you integrate with their tools, or start a signup.
• It doesn't stop determined scrapers. Serious scrapers rotate residential IPs and use headless browsers to get past challenges. Blanket rules mostly block the well-behaved bots that announce who they are.
• Public content needs proportionate protection. Marketing pages, pricing and docs are intended to be read. They still face scraping, abuse and availability risks; caching, selective access and rate limits can protect them without blanket exclusion.

How to restructure: selective agent access

The goal isn't to open the doors to every bot. It's to split your site into zones and give each one the right level of protection.

1. 1

   Open a dedicated machine-readable layer

   Publish /llms.txt, /agents.md, /.well-known/agent.json, /robots.txt and /sitemap.xml as static, cacheable files. Add a firewall rule that skips bot challenges for these exact paths only. They contain only what you choose to publish, run no database queries and are served from the CDN cache. They add almost no attack surface.

2. 2

   Allow verified AI crawlers instead of all bots

   Use your firewall's verified-bot list, or check reverse DNS and published IP ranges, to allow named crawlers such as OAI-SearchBot and PerplexityBot on public pages, where the provider offers a documented verification method. Do not trust user-agent strings alone. Google-Extended is a robots.txt control token, not a separate crawler identity or firewall allowlist entry.

3. 3

   Separate public content from the application

   Treat the marketing site, pricing and docs as a public zone. Keep strict bot protection, login walls and CAPTCHAs on the app itself: dashboards, account pages, checkout, signup forms and private APIs. Ideally put them on separate hosts or paths (for example app. vs the main domain) so the rules stay simple.

4. 4

   Rate-limit instead of hard-blocking

   On public pages, swap the blanket 403 for per-client rate limits and gradual slowdowns. A legitimate agent reading a few pages is never interrupted. A scraper pulling thousands of URLs a minute gets throttled and then blocked.

5. 5

   Serve real HTML, not an empty shell

   Make sure key pages render their content on the server and include JSON-LD structured data, so agents get the facts without having to run heavy JavaScript.

6. 6

   Control use with policy, not just firewalls

   Use robots.txt to state which AI crawlers may use which sections, for example allowing live AI search while opting out of model training. Monitor bot traffic in your logs and review the allowlist every quarter.

IF path IN (/llms.txt, /agents.md, /.well-known/agent.json,
            /robots.txt, /sitemap.xml)          → allow, cache
ELSE IF verified_bot AND zone = public       → allow, rate-limit
ELSE IF zone = app / login / checkout / api  → full bot protection
ELSE                                         → managed challenge

The bottom line

Security and discoverability can coexist. This benchmark measures discoverability signals, not security quality, and cannot establish which companies are more secure. Keep your firewall, preserve authentication and authorization, and give verified agents a controlled front door to intentionally public information.