Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion deploy/lightspeed-stack/Containerfile
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ USER root
# Install gcc - required by polyleven python package on aarch64
# (dependency of autoevals, no pre-built binary wheels for linux on aarch64)
# cmake and cargo are required by fastuuid, maturin
RUN ${BUILDER_DNF_COMMAND} install -y --nodocs --setopt=keepcache=0 --setopt=tsflags=nodocs gcc gcc-c++ cmake cargo && \
RUN ${BUILDER_DNF_COMMAND} install -y --nodocs --setopt=keepcache=0 --setopt=tsflags=nodocs gcc gcc-c++ cmake cargo patch && \
${BUILDER_DNF_COMMAND} update -y --nodocs --setopt=keepcache=0 --setopt=tsflags=nodocs --allowerasing

# Install uv package manager
Expand Down Expand Up @@ -54,6 +54,21 @@ RUN if [ -f /cachi2/cachi2.env ]; then \
uv sync --locked --no-dev --group ogxlibdev; \
fi

# Apply patches to bundled Python packages (see patch/README.md).
# patch/<package>/NNN-*.patch files are applied in order to
# site-packages/<package>.
COPY ${LSC_SOURCE_DIR}/patch /tmp/patch
RUN set -eu; \
SITE_PACKAGES=/app-root/.venv/lib/python3.12/site-packages; \
for pkg_dir in /tmp/patch/*/; do \
pkg="$(basename "${pkg_dir}")"; \
for patch_file in "${pkg_dir}"[0-9][0-9][0-9]-*.patch; do \
echo "Applying ${patch_file}"; \
patch -p1 --batch --forward --fuzz=0 -d "${SITE_PACKAGES}/${pkg}" < "${patch_file}"; \
done; \
done; \
rm -rf /tmp/patch

# Explicitly remove some packages to mitigate some CVEs
# - GHSA-wj6h-64fc-37mp: python-ecdsa package won't fix it upstream.
# This package is required by python-jose. python-jose supports multiple
Expand Down
17 changes: 17 additions & 0 deletions patch/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Patches for bundled Python packages

Patches in this directory are applied to installed Python packages while
building the container image (see `deploy/lightspeed-stack/Containerfile`).
Use them only for urgent fixes that cannot wait for a new upstream release,
and remove them once the dependency is upgraded to a version with the fix.

Layout:

- One subdirectory per patched package, named after the package's import
directory in `site-packages` (for example `ogx`).
- Patch files are named `NNN-short-description.patch` and are applied in
numeric order.
- Paths in the patch are relative to the package directory and use `a/` and
`b/` prefixes; they are applied with
`patch -p1 -d site-packages/<package>`.
- Begin each patch with a description that links to the upstream change.
35 changes: 35 additions & 0 deletions patch/ogx/001-vertexai-number-streamed-tool-calls.patch
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
Number streamed Vertex AI tool calls sequentially

Backport of https://github.com/ogx-ai/ogx/pull/6775
(fixes https://github.com/ogx-ai/ogx/issues/6774).

Gemini sends each function call whole, and the tool call positions restart
at 0 in every streamed chunk. When parallel function calls arrive in
separate chunks they all get index 0, and consumers merge them into a
single call whose arguments are concatenated (invalid) JSON. Assign
indices sequentially across the stream.

Drop this patch once ogx includes the upstream fix.

--- a/providers/remote/inference/vertexai/vertexai.py
+++ b/providers/remote/inference/vertexai/vertexai.py
@@ -641,6 +641,7 @@ class VertexAIInferenceAdapter(NeedsRequestProviderData, BaseModel):
async def _iter() -> AsyncIterator[OpenAIChatCompletionChunk]:
is_first_chunk = True
last_chunk: Any = None
+ next_tool_call_index = 0
async for chunk in stream:
signatures: dict[str, str] = {}
openai_chunk = converters.convert_gemini_stream_chunk_to_openai(
@@ -650,6 +651,11 @@ class VertexAIInferenceAdapter(NeedsRequestProviderData, BaseModel):
is_first_chunk=is_first_chunk,
signatures_out=signatures,
)
+ # Gemini sends whole function calls, and positions restart in each chunk.
+ for choice in openai_chunk.choices:
+ for tool_call in choice.delta.tool_calls or []:
+ tool_call.index = next_tool_call_index
+ next_tool_call_index += 1
# Persist before yield: code after yield only runs when the
# consumer pulls the next item, and is skipped if the stream is
# closed after this chunk (client has the call id already).
Loading