Skip to content

feat(sandbox): read_sandbox_file returns text pages and images - #3120

Open
thomasbarrett wants to merge 2 commits into
kagent-dev:mainfrom
thomasbarrett:sandbox-tool-content
Open

thomasbarrett wants to merge 2 commits into
kagent-dev:mainfrom
thomasbarrett:sandbox-tool-content

Conversation

@thomasbarrett

@thomasbarrett thomasbarrett commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

read_sandbox_file returns base64, which models can only read by decoding it as output tokens. It now behaves like Claude Code's native Read tool:

  • Text comes back as numbered lines (N<tab>content), 2000 lines by default, paged with offset/limit. A page also stops at 32 KiB, and lines over 2000 bytes are cut.
  • Images (PNG, JPEG, GIF) up to 10 MiB, the chat attachment limit, come back as images; Claude Code and Codex resize them for their models. Larger images ask the agent to write a smaller copy.
  • Other files come back as their type.

The Claude and Codex harnesses now read output lines up to 16 MiB, up from 1 MiB, so image results fit.

Breaking: read_sandbox_file returns content instead of data_base64. It has only shipped in the 1.0.0 alphas.

🤖 Generated with Claude Code

@github-actions github-actions Bot added the bug Something isn't working label Oct 9, 2026
@thomasbarrett thomasbarrett changed the title fix(mcp): return sandbox text as text and images as images fix(mcp): return sandbox reads as text and images the model can read Oct 9, 2026
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 9, 2026
@thomasbarrett thomasbarrett changed the title fix(mcp): return sandbox reads as text and images the model can read fix(mcp): return sandbox reads as text and images Oct 9, 2026
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 9, 2026
@thomasbarrett
thomasbarrett force-pushed the sandbox-tool-content branch 2 times, most recently from 1b544a6 to bc6a734 Compare October 9, 2026 05:41
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 9, 2026
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 9, 2026
@thomasbarrett
thomasbarrett force-pushed the sandbox-tool-content branch 2 times, most recently from fe1f555 to ab39df8 Compare October 9, 2026 05:51
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 9, 2026
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 9, 2026
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 9, 2026
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 9, 2026
@thomasbarrett thomasbarrett changed the title fix(mcp): return sandbox reads as text and images fix: sandbox reads return text and images, not base64 Oct 9, 2026
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 9, 2026
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 9, 2026
read_sandbox_outputs and read_sandbox_file returned base64, which agents
decoded by retyping it as output tokens. They now return text and images
the model can read, and the Claude and Codex harnesses accept output
lines large enough to carry those images.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Thomas Barrett <thomas@fluidstack.io>
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 9, 2026
@thomasbarrett
thomasbarrett marked this pull request as ready for review October 9, 2026 18:49
@thomasbarrett

Copy link
Copy Markdown
Contributor Author

We can take this further and and other file-format specific tools like PDF or Jupyter notebook reading tools that the harness's native Read tools have so that they can work well in sandbox. Might also be worth tuning the sandbox read tool to more closely match the native read tool for each harness

@EItanya EItanya left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI-generated review.

Reviewed abdcd2f. Found two reproducible issues and one documentation improvement:

  1. Output polling can corrupt valid UTF-8. sandboxes.go:139 skips index zero when checking incomplete characters. If é arrives across two polls, the first poll consumes its leading byte; the combined output becomes ��. Preserve incomplete characters during polling, including at index zero, and handle incomplete terminal output separately.

  2. Image encoding doesn’t enforce the 500 KB limit. sandboxes.go:264–273 returns the final JPEG even when every quality setting exceeds the budget. A valid 2000×2000 noise PNG produced 1,384,698 bytes, defeating the stated goal of avoiding downstream re-encoding. Reduce dimensions and retry, or explicitly report that the limit cannot be met. Add a detailed-image test that asserts encoded size.

  3. Optional: correct the shared MCP instructions. prompts.go:18 still says all file transfers have a 1 MiB limit, potentially discouraging the larger reads this PR enables. Scope that statement to writes; the architecture docs also retain the obsolete base64 read contract.

MCP, Claude/Codex harness, and compiler tests passed. Both helper-level reproduction probes failed as described. Deployed E2E and live model behavior weren’t tested.

@thomasbarrett

thomasbarrett commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor Author
  1. I am dropping the changes to read_sandbox_outputs and focusing on read_sandbox_file tool.
  2. I am dropping downsampling of image files from read_sandbox_file tool and simply reporting an error on image files that are larger than ~10 MiB (same as attachment limit in feat: attach text files and images to chat messages #2936). Claude Code or Codex will both downsample images returned from MCP results themself while respecting vision model limits - this model-specific logic belongs in the harness itself.

read_sandbox_outputs goes back to base64; only read_sandbox_file changes.
PNG, JPEG and GIF images up to 10 MiB, the chat attachment limit, pass
through unchanged, and Claude Code and Codex resize them for their models.
Larger images ask the agent to write a smaller copy.

The harnesses read CLI output lines up to 16 MiB instead of 1 MiB, so
image results fit.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Thomas Barrett <thomas@fluidstack.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants