feat: data-links - recover from expired signing credentials during large data-link uploads by georgi-seqera · Pull Request #652 · seqeralabs/tower-cli

georgi-seqera · 2026-07-30T09:10:01Z

Summary

Recover from expired signing credentials during large data-link uploads

Why

tw data-links upload fails partway through large uploads with HTTP 400 ExpiredToken (e.g. a 108 GiB upload died at ~75% after ~60 minutes). The presigned URLs the CLI receives at the start of a multipart upload are signed with temporary credentials that expire after ~1 hour, and the CLI had no way to recover — it uploaded parts strictly sequentially, with no retry and no ability to obtain fresh URLs, so any upload outliving the credential lifetime failed outright.

This is the client half of the fix; it pairs with the new Platform refresh endpoint introduced in 26.2.0 (API version 1.192.0).

How

The CLI now detects an expired-credentials error mid-upload, asks Platform to re-sign the URLs for the parts it still needs, and retries them — so a long-running upload transparently continues instead of failing.

Alongside that, the upload path gains general resilience it was missing:

Transient-error and network retry with exponential backoff for temporary failures (5xx, throttling, dropped connections), independent of the credential-refresh path.
Retry-safe progress reporting, so retried parts don't double-count against the progress bar.
Graceful degradation against older Platforms: if the refresh endpoint isn't available (404), the CLI stops with a clear, actionable message rather than a confusing low-level error.

Behavior differs per provider, matching what each backend can actually support:

Provider	Transient/network retry	Refresh on credential expiry
AWS S3 / Seqera Compute managed storage	✅	✅
Azure	✅	✗ (expiry is terminal)
Google Cloud Storage	✅	✗ (uses a long-lived resumable session)

Consuming the refresh endpoint requires a regenerated SDK, so this PR also bumps the Tower Java SDK version.

What it looks like

No new flags or usage changes — the same command now survives credential expiry:

$ tw data-links upload -w my-org/my-ws -n my-bucket -c my-creds ./sample.cram
  Uploading sample.cram [==============>          ] 61%
  # ~1h in, signing credentials expire; the CLI refreshes URLs and continues
  Uploading sample.cram [========================] 100%
  Uploaded 1 file to my-bucket

Against a Platform too old to support refresh, it fails fast with a clear reason instead of a raw ExpiredToken:

$ tw data-links upload -w my-org/my-ws -n my-bucket -c my-creds ./sample.cram
  ERROR: Token refresh is not supported for this Platform version.

Scope

CLI + SDK version bump only (paired with the Platform backend PR).
No changes to command syntax or output format.

…rge data-link uploads

…ndpoint Re-signing presigned upload URLs is now a mode of the regular data-link upload endpoint (POST /data-links/{id}/upload with uploadId + partNumbers) rather than a dedicated /upload/refresh endpoint. Adopt that on the client: - refreshUrls() calls generateDataLinkUploadUrl[WithPath] and zips the positional response back to the requested part numbers. - Detect an old Platform (that ignores the fields and initiates a new upload) by the echoed uploadId no longer matching, and fail with a clear message. - Drop the removed DataLinkRefreshMultiPartUpload*/PartUploadUrl models. - Bump tower-java-sdk 1.192.0 -> 1.194.0 (generated from the updated spec). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

sabulous · 2026-08-10T13:29:30Z

+  "methods":[{"name":"<init>","parameterTypes":[] }, {"name":"addPartNumbersItem","parameterTypes":["java.lang.Integer"] }, {"name":"contentLength","parameterTypes":["java.lang.Long"] }, {"name":"contentType","parameterTypes":["java.lang.String"] }, {"name":"dirPath","parameterTypes":["java.lang.String"] }, {"name":"equals","parameterTypes":["java.lang.Object"] }, {"name":"fileName","parameterTypes":["java.lang.String"] }, {"name":"getContentLength","parameterTypes":[] }, {"name":"getContentType","parameterTypes":[] }, {"name":"getDirPath","parameterTypes":[] }, {"name":"getFileName","parameterTypes":[] }, {"name":"getPartNumbers","parameterTypes":[] }, {"name":"getUploadId","parameterTypes":[] }, {"name":"hashCode","parameterTypes":[] }, {"name":"partNumbers","parameterTypes":["java.util.List"] }, {"name":"setContentLength","parameterTypes":["java.lang.Long"] }, {"name":"setContentType","parameterTypes":["java.lang.String"] }, {"name":"setDirPath","parameterTypes":["java.lang.String"] }, {"name":"setFileName","parameterTypes":["java.lang.String"] }, {"name":"setPartNumbers","parameterTypes":["java.util.List"] }, {"name":"setUploadId","parameterTypes":["java.lang.String"] }, {"name":"toIndentedString","parameterTypes":["java.lang.Object"] }, {"name":"toString","parameterTypes":[] }, {"name":"uploadId","parameterTypes":["java.lang.String"] }]
+},
+{
+  "name":"io.seqera.tower.model.DataLinkRefreshMultiPartUploadResponse",


I think this does not exist anymore, right?

sabulous · 2026-08-10T13:32:27Z

+        if (statusCode == 500 || statusCode == 502 || statusCode == 503 || statusCode == 504) {
+            return UploadErrorType.TRANSIENT;
+        }
+        return UploadErrorType.HARD_FAIL;


Should we maybe also remove 429 from the HARD_FAIL? It seems to me that it can be TRANSIENT as it could be returned for throttling and we might miss those cases. Wdyt?

sabulous

Looks good to me overall!

Here are my testing notes: built the branch against a locally-generated 1.196.0 SDK and ran it against a local Platform on #11943 — the full multipart happy path works end-to-end (initiate → parts → finish). I could not exercise the refresh path live, though: presigned URLs carry a 24h TTL (X-Amz-Expires=86400), so a normal upload finishes long before credentials expire and the re-sign never triggers. Flagging in case anyone has a way to force a real mid-upload expiry.

sabulous · 2026-08-11T08:13:37Z

+        // A Platform that predates re-signing ignores the uploadId/partNumbers fields and instead initiates a
+        // brand-new multi-part upload, returning a different uploadId. Detect that by the echoed uploadId and
+        // fail clearly rather than mixing URLs from a different upload into the in-progress one.
+        if (!uploadId.equals(response.getUploadId())) {


Not a blocker but worth noting I think: On a Platform that predates re-signing, generateDataLinkUploadUrl with partNumbers will initiate a new multipart upload server-side (new uploadId) before we detect the mismatch at line 231 and throw. The original upload gets aborted by the caller's catch, but that newly-created one is left in-progress on S3 (billable until a lifecycle rule reaps it). Only affects the old-Platform fallback path. Should we clean that orphan upload as well or is it not necessary?

Good point! Will update to clean up and finalize/cancel the uploads.

georgi-seqera · 2026-08-11T12:50:58Z

could not exercise the refresh path live, though: presigned URLs carry a 24h TTL (X-Amz-Expires=86400), so a normal upload finishes long before credentials expire and the re-sign never triggers. Flagging in case anyone has a way to force a real mid-upload expiry.

@sabulous Thanks for testing! I tested the real refresh scenario with steps described on the corresponding platform change https://github.com/seqeralabs/platform/pull/11943#issuecomment-5189733165 with a local Platform instance where with the AssumeRole request to STS can be configured for a shorter 15 minute duration.

sabulous · 2026-08-11T14:22:33Z

could not exercise the refresh path live, though: presigned URLs carry a 24h TTL (X-Amz-Expires=86400), so a normal upload finishes long before credentials expire and the re-sign never triggers. Flagging in case anyone has a way to force a real mid-upload expiry.

@sabulous Thanks for testing! I tested the real refresh scenario with steps described on the corresponding platform change seqeralabs/platform#11943 (comment) with a local Platform instance where with the AssumeRole request to STS can be configured for a shorter 15 minute duration.

Thanks @georgi-seqera , I could set it up and observe the second /upload call after ~17 minutes with uploadId and partNumbers!

Edit: One last thing, maybe it's worth exploring if we can communicate with the user through terminal that we are refreshing the creds. Totally optional tho.

feat: data-links - recover from expired signing credentials during la…

18f3d67

…rge data-link uploads

georgi-seqera requested review from sabulous, t0randr and weronikasosnowskaseqera July 30, 2026 09:10

georgi-seqera and others added 2 commits August 7, 2026 19:58

chore: version update and cleanup

bde4ef5

georgi-seqera marked this pull request as ready for review August 10, 2026 10:02

sabulous reviewed Aug 10, 2026

View reviewed changes

sabulous approved these changes Aug 11, 2026

View reviewed changes

Navigation Menu

Uh oh!

feat: data-links - recover from expired signing credentials during large data-link uploads - #652

feat: data-links - recover from expired signing credentials during large data-link uploads#652
georgi-seqera wants to merge 3 commits into
masterfrom
gh/feat/upload-refresh-token

georgi-seqera commented Jul 30, 2026

Uh oh!

sabulous Aug 10, 2026

Uh oh!

sabulous Aug 10, 2026

Uh oh!

sabulous left a comment

Uh oh!

sabulous Aug 11, 2026

Uh oh!

georgi-seqera Aug 11, 2026

Uh oh!

georgi-seqera commented Aug 11, 2026

Uh oh!

sabulous commented Aug 11, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

georgi-seqera commented Jul 30, 2026

Summary

Recover from expired signing credentials during large data-link uploads

Why

How

What it looks like

Scope

Uh oh!

sabulous Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

sabulous Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

sabulous left a comment

Choose a reason for hiding this comment

Uh oh!

sabulous Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

georgi-seqera Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

georgi-seqera commented Aug 11, 2026

Uh oh!

sabulous commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sabulous commented Aug 11, 2026 •

edited

Loading