Various questions & feedback

[jakearchibald]

Noting this all down in one place ahead of a meeting. I only read through the spec and explainers on a plane, so apologies if this is stuff that's already been discussed. If anything isn't a quick answer, I can create a separate issue.

Also see #236.

---

Browsing context required: As tool calls are handled in JavaScript, a browser tab or a webview must be opened to provide a visible interface and browser context. In other words, there is no support for agents or assistive tools to call tools in a headless state.

Why? Headless browsers have a browsing context right?

---

WebMCP is only available in origin-isolated documents. This ensures that the document's origin remains stable throughout the tool's lifetime.

Why? Not saying it's wrong, just don't understand. postMessage doesn't have this limitation.

---

Can tools provide progress? Seems like a job for a stream/observable!

---

How does document.modelContext access work across origin for exposed tools? (document access will be blocked right?)

---

Why is inputSchema stringified? Seems a bit weird.

---

The browser agent, on the other hand, does not run JavaScript on the page.

I guess you're already on top of this, but the same should happen for executing tools.

---

Using JSON schema seems a bit… WebSQL. Is the format frozen? What if it's changed in future? I assume there's no actual casting/validation of the format?

---

Do we need an output schema? So, if the function returns an object, the agent can understand how the various keys can be used.

---

Should an observation be required before attempting to call a tool? In case things have changed.

[domfarolino]

Thanks for the great questions, and being willing to discuss a subset of them offline with us. Some answers below:

Why? Headless browsers have a browsing context right?

It looks like you got this from https://developer.chrome.com/docs/ai/webmcp#limitations, which I agree is a misrepresentation of the explainer. I think the first section of https://github.com/webmachinelearning/webmcp/blob/main/README.md#non-goals is a better source of truth, but yeah, I'll work to get that fixed in Google's own documentation.

Why? Not saying it's wrong, just don't understand. postMessage doesn't have this limitation.

I believe originally we were worried about a document's origin changing over time (via document.domain), and that being a security concern for cross-origin tool execution. But as I understand postMessage() doesn't have this concern because the targetOrigin checked on either end doesn't change (from the perspective of the same origin check) as a result of document.domain mutations. And WebMCP tools have a similar "same origin" check during execution. So maybe we don't need this restriction after all? Or maybe the concern is document.domain could let the tool owner get access to something it shouldn't have via document.domain, and communicate it to the tool caller in an unexpected way? Let me /cc @khushalsagar and @camillelamy to keep me honest.

Can tools provide progress? Seems like a job for a stream/observable!

Great question. This is being discussed in #196. I'd like to see something like what you suggest, provide a nice webby version of MCP's Progress.

How does document.modelContext access work across origin for exposed tools? (document access will be blocked right?)

Right, crossOriginWindow.document is indeed blocked. The answer to this is probably related more to #236, but as we discussed offline, how this works is you only ever consult your own document.modelContext.getTools() to get all of the tools that are exposed to your document's origin. Initially, the API had no notion of cross-origin tools, and everything was limited to your own document, so getTools() there made sense. We added cross-origin iframe support with permission policy, we reached for "getTools() should be able to access tools in other frames; origin seems like the right boundary to enforce". Will follow-up on whether we want to more align with postMessage() etc., over in #236.

Why is inputSchema stringified? Seems a bit weird.

Yeah I agree. So just to clarify, it's weird that ModelContextTool#inputSchema is an object, while RegisteredTool#inputSchema is a string. They should probably both be the same, we just haven't done that yet.

I guess you're already on top of this, but the same should happen for executing tools.

Can you clarify what you'd like to see here? Here's what I'd like to see:

- The browser agent, on the other hand, does not run JavaScript on the page.
- Instead, it obtains a view of the page’s tools and any other relevant context by getting an observation.
+ The browser agent, on the other hand, does not call JavaScript APIs like getTools() and executeTool() directly.
+ Instead, it obtains a view of the page’s tools and any other relevant context by getting an observation, and executes tools through a bespoke internal path.

Does something like that match your understanding of how this should be structured?

Using JSON schema seems a bit… WebSQL. Is the format frozen? What if it's changed in future? I assume there's no actual casting/validation of the format?

All good questions. We haven't answered them all yet, but discussion is in #92 which mostly has to do with schema validation (schema registration, tool calls filling out the schema, and output schema). In short, it's TBD how much validation we want to enforce at the browser layer. I have a draft write-up somewhere that I'll try and pull out and finish. But I'd love your opinions.

Do we need an output schema? So, if the function returns an object, the agent can understand how the various keys can be used.

See #9. Answer is probably yes, but I was hoping to make more progress on #92 before adding it, just so any new schemas we add can be layered on top of whatever validation solution we have.

Should an observation be required before attempting to call a tool? In case things have changed.

TLDR: In my opinion no, I'm trying to let "observations" be entirely implementation-dependent.

Longer answer:
The concept of an observation serves built-in/native agents, not the web platform. What we found is that these agents tend to not be listening to toolchange events and interacting with other JS-ey things, as we discussed. Furthermore, they're already equipped to actuate web sites (albeit poorly) without WebMCP, and they do this by taking "observation" snapshots of the page, which might include things like parts of the DOM serialized in a proprietary format, the native accessibility tree, some product-specific annotations and other metadata about the page, etc). That was the view that the model had of the site. This technique is universal, so to demonstrate how exactly WebMCP layers on top, and to describe how these agents are not expected to use the JS APIs directly, we spec'd this implementation-defined concept of an observation. We let browsers take one "whenever" (since observation frequency is thus far not controlled by the developer), and described how "tools" layer on top, including their origin and other metadata designed for the model's harness.

But just like an "in-page" agent built in JavaScript can call tools without having listened to any of the toolchange events, I think it makes sense to not control the observation frequency of non-web-platform agents. Hopefully this short novel cleared some of that up, but happy to discuss more.