Making web access useful to Hermès
I use Hermès for tasks that involve finding and reading pages, including listings on French marketplaces. When an anti-bot protection intervenes, the agent has to decide whether to wait, use another engine or continue in a browser. I adapted Scrapling, Karim Shoair’s framework, to organise those decisions through its MCP interface.
The parser, retrieval engines and framework’s collection features belong to upstream. My work extends the layer agents use: progressive retrieval, presets, response states, interactions and operational tools. The agent remains responsible for its objective and for deciding how to use the information returned.
Moving from retrieval to interaction
The smart_fetch tool begins with a relatively inexpensive request, then moves to a dynamic browser or a stronger configuration when needed. It examines the response and returns an explicit state: useful content, a rate limit, a challenge, an error or a need for interaction. An agent can therefore distinguish a retrieved page from a blocking screen.
When automatic extraction is insufficient, the server keeps a page open and returns its identifiers. The agent can inspect an accessibility tree, click, type, wait or scroll before extracting relevant content. This handover avoids starting navigation again when a working session already exists. Agent actions extend the work prepared by the service.
Adapting attempts to each domain
I added operational memory per domain, called DomainBrain. During the server’s lifetime, it retains successful levels and manages concurrency, delays and pauses. Success can shorten the delay; rate limits or repeated challenges increase it. These decisions use rules and observed outcomes.
Profiles and cookies answer a separate need: recovering browsing context across workflows. Presets group recurring settings while preserving the priority of explicit arguments. I kept escalation policy in the MCP layer so the Python engines remain focused on retrieving pages. This provides a clear boundary between the existing framework and the orchestration added for agents.
Operating and maintaining web tools
The fork includes MCP contract checks, procedures for locked profiles and recovery of orphaned browsers. It supports a virtual display for browser configurations that need one on a server without a screen. Guides also describe the resource lifecycle: open, navigate, interact, extract and close.
I maintained these additions through upstream updates and documented component boundaries. Website protections change over time, so diagnostic information and explicit failure states are part of operating the tool. My adaptation’s repository is private; the original project and its documentation provide a public view of the framework it extends.