Skip to content

feat: activate cloud-run provider in sandbox benchmarks - #164

Merged
HeyGarrison merged 1 commit into
masterfrom
activate-cloud-run-benchmarks
Jul 8, 2026
Merged

feat: activate cloud-run provider in sandbox benchmarks#164
HeyGarrison merged 1 commit into
masterfrom
activate-cloud-run-benchmarks

Conversation

@HeyGarrison

Copy link
Copy Markdown
Collaborator

Activates the Cloud Run provider for sandbox benchmarks now that the gateway is deployed and secrets are configured.

Changes:

  • Uncommented cloudRun import and provider entry in src/sandbox/providers.ts
  • Added cloud-run to the benchmark matrix in .github/workflows/sandbox-benchmarks.yml
  • Uncommented CLOUD_RUN_SANDBOX_URL and CLOUD_RUN_SANDBOX_SECRET env vars in the workflow

Gateway deployed to main-498812 in us-east4 (Virginia). Secrets added to GitHub repo.

@open-cla

open-cla Bot commented Jul 2, 2026

Copy link
Copy Markdown

Contributor License Agreement

All contributors are covered by a CLA.

@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Sandbox Benchmark Results

Sequential

# Provider Score Median TTI P95 P99 Status
1 isorun 99.7 0.01s 0.06s 0.06s 10/10
2 northflank 98.6 0.10s 0.20s 0.20s 10/10
3 declaw 98.4 0.05s 0.32s 0.32s 10/10
4 archil 96.5 0.31s 0.40s 0.40s 10/10
5 upstash 96.4 0.27s 0.50s 0.50s 10/10
6 e2b 95.9 0.37s 0.47s 0.47s 10/10
7 vercel 95.8 0.38s 0.49s 0.49s 10/10
8 daytona 95.6 0.14s 0.89s 0.89s 10/10
9 cloud-run 95.0 0.46s 0.55s 0.55s 10/10
10 modal 94.0 0.49s 0.77s 0.77s 10/10
11 runloop 93.1 0.61s 0.80s 0.80s 10/10
12 blaxel 92.5 0.43s 1.21s 1.21s 10/10
13 tensorlake 91.8 0.55s 1.23s 1.23s 10/10
14 createos 90.0 0.80s 1.31s 1.31s 10/10
15 beam 89.3 0.45s 1.99s 1.99s 10/10
16 superserve 87.8 1.15s 1.33s 1.33s 10/10
17 hopx 83.8 1.46s 1.86s 1.86s 10/10
18 codesandbox 74.1 2.35s 2.95s 2.95s 10/10
19 cloudflare 0.0 0.00s 0.00s 0.00s 0/10

Staggered

# Provider Score Median TTI P95 P99 Status
1 isorun 99.8 0.01s 0.02s 0.02s 10/10
2 declaw 99.2 0.06s 0.12s 0.12s 10/10
3 northflank 98.8 0.11s 0.13s 0.13s 10/10
4 daytona 97.9 0.16s 0.30s 0.30s 10/10
5 archil 97.6 0.17s 0.35s 0.35s 10/10
6 upstash 96.6 0.28s 0.43s 0.43s 10/10
7 beam 95.9 0.39s 0.44s 0.44s 10/10
8 cloud-run 95.5 0.44s 0.46s 0.46s 10/10
9 e2b 95.5 0.41s 0.51s 0.51s 10/10
10 blaxel 95.3 0.45s 0.51s 0.51s 10/10
11 tensorlake 95.2 0.46s 0.52s 0.52s 10/10
12 vercel 94.9 0.37s 0.72s 0.72s 10/10
13 modal 94.5 0.48s 0.65s 0.65s 10/10
14 runloop 93.6 0.61s 0.67s 0.67s 10/10
15 createos 91.6 0.75s 0.98s 0.98s 10/10
16 superserve 88.5 1.09s 1.24s 1.24s 10/10
17 hopx 82.9 1.68s 1.76s 1.76s 10/10
18 codesandbox 75.7 2.27s 2.67s 2.67s 10/10
19 cloudflare 0.0 0.00s 0.00s 0.00s 0/10

Burst

# Provider Score Median TTI P95 P99 Status
1 isorun 99.8 0.03s 0.03s 0.03s 10/10
2 northflank 98.7 0.12s 0.15s 0.15s 10/10
3 declaw 97.9 0.19s 0.23s 0.23s 10/10
4 daytona 97.1 0.24s 0.36s 0.36s 10/10
5 archil 96.6 0.25s 0.47s 0.47s 10/10
6 beam 96.3 0.34s 0.41s 0.41s 10/10
7 upstash 95.4 0.41s 0.53s 0.53s 10/10
8 blaxel 95.2 0.47s 0.49s 0.49s 10/10
9 e2b 95.1 0.43s 0.58s 0.58s 10/10
10 tensorlake 95.0 0.49s 0.51s 0.51s 10/10
11 cloud-run 94.9 0.48s 0.55s 0.55s 10/10
12 vercel 94.4 0.39s 0.80s 0.80s 10/10
13 modal 93.5 0.55s 0.80s 0.80s 10/10
14 runloop 93.0 0.69s 0.72s 0.72s 10/10
15 createos 91.5 0.70s 1.07s 1.07s 10/10
16 superserve 84.9 1.50s 1.54s 1.54s 10/10
17 hopx 75.4 2.36s 2.61s 2.61s 10/10
18 codesandbox 70.8 2.66s 3.30s 3.30s 10/10
19 cloudflare 0.0 0.00s 0.00s 0.00s 0/10

View full run · SVGs available as build artifacts

@HeyGarrison
HeyGarrison force-pushed the activate-cloud-run-benchmarks branch from 2886c66 to 7232f4e Compare July 7, 2026 17:48
@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Storage Benchmark Results

1MB Files

# Provider Score Download Throughput Upload Status
1 Azure Blob Storage 96.8 0.02s 396.9 Mbps 0.06s 10/10
2 Tigris 96.0 0.03s 281.4 Mbps 0.25s 10/10
3 AWS S3 94.7 0.13s 63.2 Mbps 0.27s 10/10
4 Vercel Blob 94.7 0.10s 83.0 Mbps 0.28s 10/10
5 Cloudflare R2 94.1 0.21s 40.0 Mbps 0.38s 10/10

View full run · SVGs available as build artifacts

@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Snapshot/Fork Benchmark Results

small dataset

# Provider Score Snapshot create Fork (snapshot) Fork (live) First read Status
1 Tigris 99.8 0.06s 0.15s 0.10s 0.10s 1/1
2 Azure-blob 99.2 0.55s 0.47s 0.54s 0.02s 1/1
3 AWS S3 98.0 1.33s 1.32s 1.33s 0.09s 1/1
4 Cloudflare R2 94.8 2.68s 4.93s 1.85s 0.30s 1/1

View full run

@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Browser Benchmark Results

# Provider Score Create Connect Navigate Release Total Status
1 Kernel 96.6 0.04s 0.15s 0.17s 0.04s 0.41s 10/10
2 Browserbase 93.8 0.23s 0.15s 0.13s 0.13s 0.66s 10/10
3 Browseruse 91.4 0.61s 0.13s 0.06s 0.05s 0.88s 10/10
4 Notte 89.9 0.18s 0.51s 0.22s 0.10s 1.07s 10/10
5 Steel 81.6 0.67s 0.76s 0.13s 0.18s 1.94s 10/10
6 Hyperbrowser 64.6 2.49s 0.13s 0.15s 0.10s 2.87s 10/10

View full run · SVG available as build artifact

@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Browser Throughput Benchmark Results

# Provider Score APS (med) Task (med) Task (p95) Screenshot Status
1 Browseruse 67.6 3.14/s 3.19s 3.23s 341ms 3/3
2 Kernel 65.5 2.79/s 3.59s 3.64s 385ms 3/3
3 Hyperbrowser 65.0 2.80/s 3.57s 4.52s 457ms 3/3
4 Browserbase 60.8 2.14/s 4.67s 5.56s 227ms 3/3
5 Steel 54.3 1.42/s 7.05s 7.72s 632ms 3/3
6 Notte 52.0 1.26/s 7.92s 8.70s 1343ms 3/3

View full run · SVG available as build artifact

@HeyGarrison
HeyGarrison force-pushed the activate-cloud-run-benchmarks branch from eb028e6 to f6b0404 Compare July 7, 2026 18:46
@HeyGarrison
HeyGarrison force-pushed the activate-cloud-run-benchmarks branch 2 times, most recently from b34d706 to 77e8fe4 Compare July 7, 2026 20:10
@HeyGarrison
HeyGarrison merged commit 7700139 into master Jul 8, 2026
106 of 135 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant