AI agents face a new hurdle: getting websites to let them in

Personal AI agents designed to shop, book flights and make reservations on behalf of users are running into a persistent obstacle: many websites are not built to let them in, and often block them outright through anti-bot defenses.

The promise of these agents is straightforward. Rather than a person manually navigating a retailer's or airline's website, an AI agent handles the process autonomously, following instructions to complete tasks like finding deals or securing bookings. But websites frequently cannot distinguish between a legitimate AI agent acting on a real user's behalf and an automated bot attempting something harmful, such as scraping data or abusing a service.

Consumers caught in the middle

That ambiguity leaves everyday users stuck between the two outcomes: either the website blocks their AI agent as a precaution, defeating the purpose of using the agent, or sites remain exposed to the kinds of automated abuse that anti-bot defenses were designed to stop in the first place.

Deliberate blocks against automated traffic, often implemented to protect against fraud, scraping or denial-of-service activity, were generally built before the recent wave of consumer AI agents existed, meaning many of today's defenses were not designed with legitimate agent traffic in mind.

A new standard is being developed to address the problem, aiming to give websites a reliable way to recognize and permit legitimate AI agents while continuing to block harmful automated traffic. The approach reflects a broader industry effort to adapt web infrastructure to a growing category of software acting directly on behalf of individual users rather than browsing on a human's own input.