feat(gsoc): fine grained API - #5497
Conversation
Add `Swarm-Soc-Fields` header to allow clients to request specific SOC fields (address, recoveredpubkey, identifier, signature, wrappedaddress, span, payload) in GSOC WebSocket messages. Add `Swarm-Cache-Wrapped-Chunk` header to enable caching of wrapped chunks on the node. Update GSOC handler to pass full SOC object instead of just payload, enabling access to all chunk properties. Adjust WebSocket buffer sizes to accommodate maximum SOC fields message size.
Add tests for `Swarm-Soc-Fields` header to verify requesting specific SOC fields (identifier, wrappedAddress, payload) and full wrapped chunk data (span + payload). Add test for `Swarm-Cache-Wrapped-Chunk` header to verify wrapped chunks are cached and retrievable. Update test helpers to support custom headers and return storer instance. Update gsoc handler signature to accept full SOC object instead of payload bytes.
There was a problem hiding this comment.
General comment: payload is the only variable-length field, but nothing stops clients from putting it before fixed fields (e.g. payload,identifier). That’s parseable only if you know all fixed sizes and derive payload_len = frame_len − fixed_total — which isn’t documented in OpenAPI. A typical left-to-right parser will misread or fail on those orderings. Worth either requiring payload to be last, or documenting field sizes and the parsing rule explicitly.
71f67e5 to
46186ef
Compare
|
great catch @sbackend123 ! I wouldn't put any check for the serialization order. The payload always can be recovered and define its length from the response bytes, because all other properties have fixed length -> if |
| return | ||
| default: | ||
| s.logger.Warning("gsoc ws: slow consumer, closing connection") | ||
| _ = conn.WriteControl(websocket.CloseMessage, |
There was a problem hiding this comment.
In gsocListeningWs, the subscriber callback registered via s.gsoc.Subscribe is invoked asynchronously in a goroutine spawned by gsoc.Handle.
When the subscriber buffer (dataC, capacity 2) fills up due to a slow client, the callback enters the default: branch:
default:
s.logger.Warning("gsoc ws: slow consumer, closing connection")
_ = conn.WriteControl(websocket.CloseMessage,
websocket.FormatCloseMessage(websocket.ClosePolicyViolation, "slow consumer"),
time.Now().Add(writeDeadline))
_ = conn.Close()
returnSimultaneously, the main gsocListeningWs goroutine is executing its event loop:
case b := <-dataC:
err = conn.SetWriteDeadline(time.Now().Add(writeDeadline))
...
err = conn.WriteMessage(websocket.BinaryMessage, b)
case <-ticker.C:
...
err = conn.WriteMessage(websocket.PingMessage, nil)gorilla/websocket.Conn is NOT thread-safe for concurrent writers. Calling conn.WriteControl and conn.Close from the callback goroutine while the main loop calls conn.WriteMessage or conn.SetWriteDeadline causes a data race on the connection's writer buffers and internal state.
Do not call conn.WriteControl or conn.Close directly inside the callback goroutine. Instead, signal the main loop to close the connection, or close a signal channel so that all WebSocket frame writes are strictly executed by the single writer goroutine in gsocListeningWs
| // parseSocFields parses the SwarmSocFieldsHeader value into a list of SOC field | ||
| // identifiers. When the header is empty it defaults to the payload field only, | ||
| // which preserves backward compatibility. | ||
| func parseSocFields(header string) ([]string, error) { |
There was a problem hiding this comment.
Vulnerability:
parseSocFields splits the Swarm-Soc-Fields header by commas and validates each string against validSocFields. However, it never de-duplicates fields or caps the number of fields.
If a client sends an HTTP header with repeated field names:
Swarm-Soc-Fields: payload,payload,payload,payload... (10,000 repetitions)
parseSocFieldsreturns a slice containing 10,000 "payload" items.- For every incoming SOC chunk,
socFieldsBytesiterates over all 10,000 items, appending the ~4KB chunk payload into a single bytes.Buffer. - Each chunk produces a ~40 MB buffer per subscriber. With 10 incoming chunks, this single connection allocates 400 MB of RAM.
An unauthenticated client can easily cause an Out-Of-Memory (OOM) panic on the Bee node.
As a fix maybe in parseSocFields, de-duplicate field names and enforce a maximum slice length equal to len(validSocFields)
| dataC = make(chan []byte, 2) // small buffer to decouple producer/consumer | ||
| gone = make(chan struct{}) | ||
| ticker = time.NewTicker(s.WsPingPeriod) | ||
| ctx, cancel = context.WithCancel(context.Background()) // for storing cached chunks |
There was a problem hiding this comment.
Would this be the case?
ctx, cancel = context.WithCancel(context.Background()) is created inside gsocListeningWs and canceled when the WebSocket connection terminates (defer cancel()).
When cacheWrappedChunk is enabled, s.storer.Cache().Put(ctx, c.WrappedChunk()) uses this connection-bound ctx. If the WebSocket connection closes while a chunk is being processed, cancel() is called and storer.Cache().Put fails with context.Canceled.
Caching a chunk in local storage is a side-effect for the node that should not be aborted simply because the subscriber's WebSocket closed.
| err error | ||
| ) | ||
| defer func() { | ||
| cancel() |
| } | ||
| } | ||
|
|
||
| // TestGsocWebsocketWrappedChunkData verifies that the Swarm-Soc-Fields header |
There was a problem hiding this comment.
The unit tests added in gsoc_test.go only test happy paths. For example the following test cases are missing:
- Invalid Headers: Passing invalid field names in Swarm-Soc-Fields and asserting HTTP 400 Bad Request.
- Duplicate Fields: Sending duplicate fields and verifying de-duplication/rejection.
- Slow Consumer: Verifying clean connection shutdown when a consumer falls behind.
- Ordering Verification: Verifying that sequential Handle calls preserve message ordering.
| responses: | ||
| "200": | ||
| description: Establishes a WebSocket subscription for incoming messages on the Single Owner Chunk address | ||
| description: > |
There was a problem hiding this comment.
Deficiencies
Undocumented Fixed Byte Sizes: The OpenAPI spec description lists available fields but fails to specify the exact binary byte size for each fixed field:
address: 32 bytes
recoveredPubKey: 33 bytes (compressed secp256k1 public key)
identifier: 32 bytes
signature: 65 bytes (secp256k1 signature + recovery id)
wrappedAddress: 32 bytes
span: 8 bytes (uint64)
payload: variable (0 to 4096 bytes)
Without explicit field lengths, API consumers cannot parse multi-field binary streams without inspecting Bee source code.
Variable-Length Field Order Warning: If payload is placed before fixed-length fields (e.g. Swarm-Soc-Fields: payload,span), parsing span requires scanning from the end of the frame. The spec should clearly state that payload MUST be specified as the last field when requesting multiple fields.
3 participants
Checklist
Description
Extends the GSOC (
/gsoc/subscribe/{address}) WebSocket endpoint so that clients can configure which SOC fields are serialized and sent on every incoming message, and optionally cache the wrapped chunk locally.New headers:
Swarm-Soc-Fields— comma-separated list of fields to include in the binary message. Allowed values:address,recoveredPubKey,identifier,signature,wrappedAddress,span,payload. Defaults topayloadfor backward compatibility.Swarm-Cache-Wrapped-Chunk— whentrue, the wrapped chunk of every incoming GSOC message is stored in the local cache so it can later be resolved through the bytes endpoint (useful when the SOC wraps a root chunk larger than 4 KB).API / handler changes:
gsoc.Handlersignature is changed fromfunc([]byte)tofunc(*soc.SOC)so subscribers have access to all SOC properties.ReadBufferSizeandWriteBufferSizeare sized toswarm.SocMaxChunkSizeandmaxSocFieldsSizerespectively to avoid split writes for the worst-case serialized message.OpenAPI:
SwarmCommon.yamland referenced them in the/gsoc/subscribe/{address}operation.8.1.0to8.2.0.Open API Spec Version Changes (if applicable)
Bee API:
8.1.0→8.2.0Motivation and Context (Optional)
Previously the GSOC subscription returned only the raw payload of the wrapped chunk. Consumers who also needed the span, the SOC address, the signature, or the recovered public key had no way to obtain them through the WebSocket. This change makes the subscription fully configurable while keeping the default behavior unchanged.
Caching the wrapped chunk is a separate convenience for users who publish large content through GSOC and want recipients to be able to fetch the full content via the standard bytes endpoint.
Related Issue (Optional)
Screenshots (if appropriate):
AI Disclosure