feat(CLI): add tg batches commands for the batch API by blainekasten · Pull Request #525 · togethercomputer/together-py

blainekasten · 2026-08-13T13:03:48Z

Fixes DX-918.

Adds CLI commands for the Batch API:

tg batches submit FILE_ID_OR_PATH API_TYPE
tg batches ls
tg batches get|retrieve BATCH_ID
tg batches download BATCH_ID [--output PATH]
tg batches cancel BATCH_ID

Submit

FILE_ID_OR_PATH behaves like ft create: local paths are uploaded with purpose=batch-api, otherwise treated as a file ID
API_TYPE is chat.completions | audio.transcriptions | audio.translations, positional or --api
Model is taken from each JSONL request body, not from the CLI
Human output uses the same curated job view as retrieve (no X Model Id dump)
API warnings are Rich-escaped; a null job exits 1 (including --json)

Retrieve

Human output is a curated summary (created/completed time, API, model, output/error file IDs)
Status always prints; in-progress jobs also show a progress bar
Completed/failed jobs print a download hint
tg batches <batch-id> implicit retrieve
Error file ID and error share a single "An error occurred" header; API error text is escaped

Download

Writes output (and error) files to --output (file or directory)
Existing suffix-less paths like ./results are treated as files, not directories
Error files are written as <stem>.errors<suffix> so a shared server filename cannot clobber output
Omitting --output streams the output file to stdout
--json requires --output (does not embed file contents); success JSON lists saved paths
Early failures (--json included) print {"error": "..."} then exit 1
Shared helper also used by tg files download

Cancel

Human output uses the curated job view

Also supports --json and ls/get aliases.

Linear Issue: DX-918

broly-code-security-scanner · 2026-08-13T13:03:57Z

Broly Security Scan

Note

✅ Clean scan
No vulnerabilities detected in this PR.

Note

Re-scan this PR anytime with /broly scan — useful after /broly undismiss, or to refresh findings without a new push.

Broly — SAST (zai-org/GLM-5.2) · Secrets · SCA · IaC · GH Actions · Base Images · Supply Chain Threats · Exploit Chains · Adversarial Verification

We're continuously improving Broly's accuracy and finding quality — your feedback is valuable. False positives, missed findings, bugs, and feature requests all welcome.

blainekasten · 2026-08-13T19:39:42Z

@cursoragent fix tests and update the PR description for changes I made

cursor · 2026-08-13T19:39:50Z

Fixed the three CI failures and updated the PR description for your download/output changes.

tg batches <id> was broken: a duplicate "batches": UUID key overwrote the batch* matcher. Restored implicit retrieve.
Retrieve tests now match the curated output (no raw COMPLETED dump; in-progress shows In_progress + bar, not Progress:).

PR: #525 — 43 CLI tests passed.

zainhas · 2026-08-20T22:26:11Z

+    if output is not None:
+        saved: list[dict[str, str]] = []
+        directory_output = _is_directory_output(output)
+        error_file_id = job.error_file_id
+
+        if job.output_file_id:
+            out_path = await download_file_content(
+                config.client,
+                job.output_file_id,
+                output=output,
+                loading_message="Downloading batch output...",
+            )
+            assert isinstance(out_path, Path)
+            saved.append({"kind": "output", "id": job.output_file_id, "path": str(out_path)})
+        elif error_file_id and not directory_output:
+            # No output file — write the error file to the exact path the user asked for.
+            err_path = await download_file_content(
+                config.client,
+                error_file_id,
+                output=output,
+                loading_message="Downloading batch errors...",
+            )
+            assert isinstance(err_path, Path)
+            saved.append({"kind": "error", "id": error_file_id, "path": str(err_path)})
+            error_file_id = None
+
+        if error_file_id:
+            err_dest = output if directory_output else _error_output_path(output)
+            err_path = await download_file_content(
+                config.client,
+                error_file_id,
+                output=err_dest,
+                loading_message="Downloading batch errors...",
+            )
+            assert isinstance(err_path, Path)
+            saved.append({"kind": "error", "id": error_file_id, "path": str(err_path)})


When --output is a directory and the job has both an output and an error file, both downloads go through download_file_content(output=
), which names each file from whatever the Files API returns for it. If those two filenames match, the second write clobbers the first — and we still print both "Output saved to …" and "Errors saved to …" as if two files landed. I mocked both files returning filename: "batch.jsonl" and ended up with a single batch.jsonl containing only the error content.

_error_output_path() already solves this for the concrete-file case; the directory case needs the same guarantee (suffix on collision, or force .errors into the error filename).

zainhas · 2026-08-20T22:27:08Z

+    console.print(raw.decode("utf-8"))
+    if job.error_file_id:
+        console.print(f"\n[dim]Error file also available: tg batches download {id} --output ./out[/dim]")


The default (no --output) path is console.print(raw.decode("utf-8")), so Rich does two things to the payload: it parses markup, and it hard-wraps at the console width. With a completion containing [bold]…[/bold] and a line over 80 chars, the tags get eaten and newlines get injected mid-JSON — > results.jsonl produces invalid JSONL. Model output containing an unbalanced tag like [/close] raises MarkupError and kills the command outright.

Since this is the default mode of the command, I think it has to be sys.stdout.buffer.write(raw) (or at minimum a markup=False, soft_wrap=True console). Two related things in the same block:

No UnicodeDecodeError guard here, even though the --json branch right above has one — a non-UTF-8 byte traces back instead of falling back.

The "Error file also available: …" hint on line 139 goes to stdout, so it becomes the last line of a redirected results.jsonl. Hints belong on stderr.

zainhas · 2026-08-20T22:28:06Z

+    if job.status in _INCOMPLETE_STATUSES and job.progress is not None:
+        console.print(f"{format_progress(job.progress)} {format_status(job.status)}")


print_batch_detail only prints a status line when status in _INCOMPLETE_STATUSES and progress is not None. So on a CANCELLED job you get created-at, the API, the model, and nothing else — no indication it was cancelled. Same for EXPIRED, for FAILED with no error field, and for VALIDATING before progress is populated. STATUS_COLORS and format_status() already cover all six states, so the four terminal ones are effectively dead code. Can we just always print the status line?

zainhas · 2026-08-20T22:28:44Z

+    if job is None or not job.id:
+        console.print("[red]x[/red] Batch job was not created")
+        if response.warning:
+            console.print(response.warning)
+        return


If the API comes back with {"job": null, "warning": …} we print x Batch job was not created and then return, so the shell sees success. tg batches submit … && next-step will happily keep going against a batch that doesn't exist. Needs sys.exit(1).

zainhas · 2026-08-20T22:29:31Z

+def _is_directory_output(path: Path) -> bool:
+    return path.is_dir() or path.suffix == ""


--output ./results crashes when ./results already exists as a file: _is_directory_output() treats any suffix-less path as a directory, and since the validator here was loosened to file_okay=True (files download uses file_okay=False), that path now reaches output.mkdir(parents=True, exist_ok=True) inside download_file_content — and exist_ok does not tolerate an existing non-directory:

touch ./results && tg batches download --output ./results
→ Error: [Errno 17] File exists: '/…/results'

An explicit output.exists() and not output.is_dir() check before the mkdir would sort it, or restore file_okay=False.

zainhas · 2026-08-20T22:35:36Z

+    if not job.output_file_id:
+        console.print(
+            "[red]Batch job has no output file[/red]. "
+            "Use [primary]--output[/primary] to download the error file instead."
+        )
+        sys.exit(1)


All three early exits print Rich prose to stdout regardless of config.json, so tg batches download --json | jq fails on non-JSON input. I noticed download is the one command excluded from the new test_json_mode_pipeable_to_jq case (tests/cli/test_json_mode_pipeable_to_jq.py:100) — if that's why, I'd rather fix the output than skip the check.

zainhas · 2026-08-20T22:39:33Z

+        )
+        sys.exit(1)
+
+    raw = await download_file_content(


response.read() pulls the entire batch output into RAM, and in --json mode we then build a second full copy as a string (a third, base64, on decode failure). Batch outputs can be very large by design, and download_file_content already has a streaming write_to_file path for --output. tg batches download | head shouldn't need the whole body resident.

zainhas · 2026-08-20T22:41:25Z

+    if job.error_file_id:
+        console.print(f" - [red]An error occurred[/red]")
+        console.print(f" - Error file ID {job.error_file_id}")
+
+    if job.error:
+        console.print(f" - [red]An error occurred[/red]")


nit: A job with both error_file_id and error set renders the header line once per block:

An error occurred

Error file ID file-err

An error occurred
boom

zainhas · 2026-08-20T22:42:00Z

+    console.print(f"[green]√ Batch job submitted.[/green] [dim]({job.id})[/dim]")
+    if response.warning:
+        console.print(f"[yellow]{response.warning}[/yellow]")
+    print_model_dump(job, show_nulls=False)


BatchJob declares x_model_id = FieldInfo(alias="model_id"), and print_model_dump renders model_dump() (field names, not aliases), so submit shows a row literally labelled "X Model Id:". cancel is worse — the top-level parse keeps model_id as an extra field alongside x_model_id, so it prints both rows with the same value (I confirmed this against the real construct_type path, not just a plain model_validate). retrieve sidesteps the whole thing with its curated view; submit/cancel want either the same treatment or a by_alias dump.

zainhas · 2026-08-20T22:42:11Z

+        return
+
+    console.print("[green]√[/green] Cancelled batch job")
+    print_model_dump(response, show_nulls=False)


BatchJob declares x_model_id = FieldInfo(alias="model_id"), and print_model_dump renders model_dump() (field names, not aliases), so submit shows a row literally labelled "X Model Id:". cancel is worse — the top-level parse keeps model_id as an extra field alongside x_model_id, so it prints both rows with the same value (I confirmed this against the real construct_type path, not just a plain model_validate). retrieve sidesteps the whole thing with its curated view; submit/cancel want either the same treatment or a by_alias dump.

blainekasten · 2026-08-21T13:38:02Z

@cursoragent address all of Zain's feedback on this PR

cursor · 2026-08-21T13:38:11Z

Addressed all of Zain’s review comments on #525.

Submit exits 1 when job is null (human and --json). Warnings are Rich-escaped. Human output uses the curated job view, so no X Model Id.
Retrieve prints “An error occurred” once when both error and error_file_id are set, and escapes API error text.
Cancel uses the same curated view.
Download treats an existing suffix-less --output ./results as a file. --json errors are {"error": "..."}. stdout streams instead of buffering. --json without --output prints file IDs only.
Tests: 53 CLI tests + batches --json | jq against prism.

53 CLI tests passed locally; batches JSON-mode jq check passed against the mock server.

zainhas · 2026-08-21T17:27:19Z

some places where the current design diverges from the batch inference API:

currently batch inference functionality doesnt require you to specify the model in the API call -- the file requests actually contain the model info like this:
{"custom_id": "request-1", "body": {"model": "meta-llama/Llama-3.3-70B-Instruct-Turbo", "messages": [{"role": "user", "content": "Hello, world!"}], "max_tokens": 200}}
{"custom_id": "request-2", "body": {"model": "meta-llama/Llama-3.3-70B-Instruct-Turbo", "messages": [{"role": "user", "content": "Explain quantum computing."}], "max_tokens": 200}}

So you should take the model specification out of the CLI commands

You have a completion-window param which we should just drop - I don't think there is any param serverside that does anything with this number - batch inference is 24 hours best effort afaik
You default check file to False ( check=False) but I think this should be True (+ you need to make sure that you pass in purpose="batch-api" otherwise it defaults to a useless finetune check in this case) since it'll let the user know faster if something is wrong with their batch file. we can add a --no-check flag if a user want to turn this off
we could add something like a --wait or --watch functionality that allows monitoring a batch job similar to what we do for a DMI or DCI
- I think print_batch_detail should also specify the input_file_id since that can get lost if not saved and is the main way to trigger reruns etc.

Add submit, list/ls, get/retrieve, and cancel under `tg batches`, including local-file upload (like `ft create`), positional + `--api`/`--model`/`-M` flags, and JSON output. Co-authored-by: Blaine Kasten <blainekasten@gmail.com>

The UUID identifier overwrote the `batch*` prefix matcher, so `tg batches <batch-id>` stopped resolving to retrieve. Align retrieve assertions with the curated human output (no raw COMPLETED dump, progress bar uses In_progress). Co-authored-by: Blaine Kasten <blainekasten@gmail.com>

Fail submit when the API returns a null job, escape API text in Rich output, use the curated batch view on submit/cancel, stream download stdout, treat existing suffix-less --output paths as files, and emit JSON on download errors. Co-authored-by: Blaine Kasten <blainekasten@gmail.com>

Co-authored-by: Blaine Kasten <blainekasten@gmail.com>

zainhas · 2026-08-25T01:17:41Z

+            error_file_id = None
+
+        if error_file_id:
+            err_dest = output if directory_output else _error_output_path(output)


Still open from last round. When --output is a directory and the job has both files, err_dest = output, so both resolve to
/. With both mocked as batch.jsonl the output is silently clobbered — and we still print two "saved to" lines pointing at the same path, plus two --json files entries with identical path. The existing test only passes because it mocks two distinct filenames, which we don't control. Simplest fix is to disambiguate in the directory branch the way _error_output_path() already does for the file branch.

@cursoragent fix this

Directory --output no longer overwrites results when output and error files share a server filename.

Error files now land at <stem>.errors<suffix> next to the output — same rule as --output ./results.jsonl. A test with both files named batch.jsonl covers it.

zainhas · 2026-08-25T01:18:48Z

+        if job.error:
+            console.print(f"   {escape_rich_markup(job.error)}")
+
+    if job.status in _INCOMPLETE_STATUSES and job.progress is not None:


The status line is gated on _INCOMPLETE_STATUSES and non-null progress, so a CANCELLED job prints created-at, endpoint, model and file IDs, and the word "cancelled" appears nowhere. Same for EXPIRED, and for FAILED when the server didn't populate error. Status is the one field people run get for — I think it should always print, with the progress bar as the extra that's only shown for in-flight jobs.

@cursoragent fix this

Status now always prints on tg batches get. Progress bar still only shows for in-flight jobs with a progress value.

Cancelled / expired / failed jobs (even with no error payload) now show Cancelled, Expired, or Failed instead of just timestamps and file IDs.

zainhas · 2026-08-25T01:19:46Z

+            ),
+        )
+
+    if config.json:


--json with no --output downloads nothing and exits 0 - test freezing it in (test_download_stdout_json_is_metadata_only). tg batches download streams the results; adding --json makes it print {"batch_id":…,"output_file_id":…} and exit 0 having fetched nothing. A format flag shouldn't change what the command does. Either stream the content into a JSON field the way retrieve-content --stdout --json already does, or _fail() telling the user to pass --output. Downloading nothing while exiting 0 is the option that'll bite someone in a pipeline.

@cursoragent fix this

--json without --output now exits 1 with {"error": "Pass --output to download..."} instead of pretending it downloaded.

--json is a format flag, not a skip-the-download flag. File contents stay on disk via --output; stdout streaming is unchanged for the human path.

Status was only shown for in-flight jobs with progress, so cancelled, expired, and failed jobs omitted the field people run get for. Co-authored-by: Blaine Kasten <blainekasten@gmail.com>

+from unittest.mock import AsyncMock, patch
+
+import httpx
+import pytest


Directory --output used the server filename for both files, so identical names silently overwrote the results. Error files now get the same .errors suffix already used for file destinations. Co-authored-by: Blaine Kasten <blainekasten@gmail.com>

--json was printing file IDs and exiting 0 without fetching anything. Require --output so a format flag cannot silently skip the download. Co-authored-by: Blaine Kasten <blainekasten@gmail.com>

blainekasten · 2026-08-26T18:51:02Z

@cursoragent fix the CLI tests from test-others - don't worry about the failing integration tests

cursor · 2026-08-26T18:51:09Z

The three test-others failures were leftover --model / model_id assertions after submit stopped taking a model.

Submit tests now call tg batches submit FILE API only, and they assert the create payload has no model_id. All 10 submit tests passed locally.

Submit no longer takes a model argument (it's in the JSONL body). Co-authored-by: Blaine Kasten <blainekasten@gmail.com>

* feat(CLI): add tg batches commands for the batch API Add submit, list/ls, get/retrieve, and cancel under `tg batches`, including local-file upload (like `ft create`), positional + `--api`/`--model`/`-M` flags, and JSON output. Co-authored-by: Blaine Kasten <blainekasten@gmail.com> * add download and improve outputs * fix(CLI): restore batches implicit retrieve and match retrieve tests The UUID identifier overwrote the `batch*` prefix matcher, so `tg batches <batch-id>` stopped resolving to retrieve. Align retrieve assertions with the curated human output (no raw COMPLETED dump, progress bar uses In_progress). Co-authored-by: Blaine Kasten <blainekasten@gmail.com> * fix(CLI): address batches review feedback Fail submit when the API returns a null job, escape API text in Rich output, use the curated batch view on submit/cancel, stream download stdout, treat existing suffix-less --output paths as files, and emit JSON on download errors. Co-authored-by: Blaine Kasten <blainekasten@gmail.com> * test(CLI): avoid unused respx route in batches --json download test Co-authored-by: Blaine Kasten <blainekasten@gmail.com> * test(CLI): keep batches --json jq checks when prism omits a job Co-authored-by: Blaine Kasten <blainekasten@gmail.com> * prune unused cli parameters * fix(CLI): always print batch job status on retrieve Status was only shown for in-flight jobs with progress, so cancelled, expired, and failed jobs omitted the field people run get for. Co-authored-by: Blaine Kasten <blainekasten@gmail.com> * fix(CLI): don't clobber batch output when error file shares a name Directory --output used the server filename for both files, so identical names silently overwrote the results. Error files now get the same .errors suffix already used for file destinations. Co-authored-by: Blaine Kasten <blainekasten@gmail.com> * fix(CLI): reject batches download --json without --output --json was printing file IDs and exiting 0 without fetching anything. Require --output so a format flag cannot silently skip the download. Co-authored-by: Blaine Kasten <blainekasten@gmail.com> * test(CLI): drop model from batches submit tests Submit no longer takes a model argument (it's in the JSONL body). Co-authored-by: Blaine Kasten <blainekasten@gmail.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>

blainekasten marked this pull request as ready for review August 13, 2026 13:03

blainekasten requested review from Nutlope and zainhas August 13, 2026 19:41

zainhas requested changes Aug 20, 2026

View reviewed changes

cursoragent and others added 7 commits August 24, 2026 11:06

feat(CLI): add tg batches commands for the batch API

5e341e3

Add submit, list/ls, get/retrieve, and cancel under `tg batches`, including local-file upload (like `ft create`), positional + `--api`/`--model`/`-M` flags, and JSON output. Co-authored-by: Blaine Kasten <blainekasten@gmail.com>

add download and improve outputs

f44a69c

test(CLI): avoid unused respx route in batches --json download test

13199b7

Co-authored-by: Blaine Kasten <blainekasten@gmail.com>

test(CLI): keep batches --json jq checks when prism omits a job

dd5b617

Co-authored-by: Blaine Kasten <blainekasten@gmail.com>

prune unused cli parameters

68d4958

blainekasten force-pushed the cursor/cli-batches-commands-c96a branch from 1024dea to 68d4958 Compare August 24, 2026 16:06

blainekasten requested a review from zainhas August 24, 2026 16:07

zainhas approved these changes Aug 25, 2026

View reviewed changes

fix(CLI): always print batch job status on retrieve

56040c4

Status was only shown for in-flight jobs with progress, so cancelled, expired, and failed jobs omitted the field people run get for. Co-authored-by: Blaine Kasten <blainekasten@gmail.com>

github-advanced-security Bot found potential problems Aug 26, 2026

View reviewed changes

Comment thread tests/cli/test_batches.py

from unittest.mock import AsyncMock, patch

import httpx

import pytest

cursoragent and others added 2 commits August 26, 2026 18:38

fix(CLI): reject batches download --json without --output

a239688

--json was printing file IDs and exiting 0 without fetching anything. Require --output so a format flag cannot silently skip the download. Co-authored-by: Blaine Kasten <blainekasten@gmail.com>

github-advanced-security Bot found potential problems Aug 26, 2026

View reviewed changes

Comment thread src/together/lib/cli/api/batches/_utils.py Fixed

test(CLI): drop model from batches submit tests

806b8f5

Submit no longer takes a model argument (it's in the JSONL body). Co-authored-by: Blaine Kasten <blainekasten@gmail.com>

blainekasten merged commit f7d63e0 into main Aug 26, 2026
11 of 12 checks passed

github-actions Bot mentioned this pull request Aug 26, 2026

release: 2.32.0 #527

Merged

github-actions Bot mentioned this pull request Aug 26, 2026

release: 2.33.0 #548

Merged

		if job.status in _INCOMPLETE_STATUSES and job.progress is not None:
		console.print(f"{format_progress(job.progress)} {format_status(job.status)}")

		def _is_directory_output(path: Path) -> bool:
		return path.is_dir() or path.suffix == ""

Conversation

blainekasten commented Aug 13, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

broly-code-security-scanner Bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Broly Security Scan

Uh oh!

blainekasten commented Aug 13, 2026

Uh oh!

cursor Bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

blainekasten commented Aug 21, 2026

Uh oh!

cursor Bot commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zainhas commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

blainekasten commented Aug 26, 2026

Uh oh!

cursor Bot commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

blainekasten commented Aug 13, 2026 •

edited by cursor Bot

Loading

broly-code-security-scanner Bot commented Aug 13, 2026 •

edited

Loading

cursor Bot commented Aug 13, 2026 •

edited

Loading

cursor Bot commented Aug 21, 2026 •

edited

Loading

zainhas commented Aug 21, 2026 •

edited

Loading

cursor Bot Aug 26, 2026 •

edited

Loading

cursor Bot Aug 26, 2026 •

edited

Loading

cursor Bot Aug 26, 2026 •

edited

Loading

cursor Bot commented Aug 26, 2026 •

edited

Loading