fix(eval): use function calling for tool/input mocking so non-OpenAI models work (AE-1646)

[Chibionos]

Summary

Tool simulation and input generation in Studio Debug and Evaluation Set runs failed with AGENT_RUNTIME.UNEXPECTED_ERROR for non-OpenAI models (Anthropic Claude via Bedrock, Gemini), but worked with OpenAI/GPT.

Fixes AE-1646 (customer: Sarasota Memorial Health Care System).

Root cause (live-verified on alpha)

Both eval mockers requested structured output via OpenAI-only response_format json_schema and parsed response.choices[0].message.content. The normalized LLM Gateway only honors response_format for OpenAI models — and the failure shape differs per provider (verified by live calls, not assumed):

Provider	response_format behavior on the gateway
OpenAI (gpt-4.1-mini)	honored — valid JSON content, native $defs support
Claude (sonnet-4-5)	ignored — returns plain prose content (e.g. 'Tokyo')
Gemini (2.5-pro)	returns empty content

Both non-OpenAI shapes broke json.loads(...) → wrapped as UiPathMockResponseGenerationError → AGENT_RUNTIME.UNEXPECTED_ERROR. Note the prose case matters: an earlier revision of this PR fell back only on empty content, which fixed Gemini but left Claude — the customer's model — still failing. The live e2e below caught that.

Regression from #1555, which started routing the agent's model into simulations; before that, simulation always used a fixed OpenAI model.

Fix

Provider-aware strategy classes in eval/mocks/_structured_output.py (mirroring llm_as_judge_evaluator, whose docstring already states function calling is the cross-provider way to get structured output):

• OpenAIStructuredOutput — prefers response_format (reliable for OpenAI incl. $defs), falls back to a forced tool call on empty/non-JSON content or request error.
• AnthropicStructuredOutput / GeminiStructuredOutput — go straight to a forced tool call (their response_format is known-broken on the gateway; no wasted request per simulation).
• Unknown providers — response_format first with tool fallback (safe default).

The forced tool wraps the output/input schema under a response property (tool_choice=required), and reads tool_calls[0].arguments["response"] (already a parsed dict). $defs/$ref are inlined so tool parameters are self-contained (the gateway accepts $defs in response_format but not in tool parameters); sibling keys on $ref nodes (e.g. field descriptions) are merged over the inlined definition so LLM guidance survives. chat_completions now accepts raw-dict tools (pass-through) so arbitrary nested schemas reach the gateway verbatim.

E2E validation (live against alpha gateway)

Full customer path per model — @mockable → Mocker.response → generate_structured_output → real gateway → Pydantic coercion of the result. Tool simulation used a nested return model (enum + list of sub-models ⇒ real $defs/$ref); input generation ran generate_llm_input the same way.

Model	Tool simulation (nested $defs)	Input generation	Calls
gpt-4.1-mini-2025-04-14	✅ coerced to Calculation, total=42.0	✅	1 (response_format)
anthropic.claude-sonnet-4-5-20250929-v1:0	✅ coerced to Calculation, total=42.0	✅	1 (tool call)
gemini-2.5-pro	✅ coerced to Calculation, total=42.0	✅	1 (tool call)

Negative control: reverting the non-JSON-content fallback and re-running the Claude e2e reproduces the exact customer error (UiPathMockResponseGenerationError: Expecting value: line 1 column 1 (char 0)), confirming the e2e discriminates.

Tests

• Strategy routing: Claude/Gemini → single forced tool call, no response_format; OpenAI → response_format preferred; unknown → fallback chain (incl. prose-content, empty-content, and request-error triggers).
• $defs inlining: self-contained output, sibling-key merge, cyclic-ref handling, caller schema not mutated.
• test_raw_dict_tool_passthrough_mocked (platform): nested array schema forwarded byte-for-byte.
• Full tests/cli/eval suite (405) + platform mocked LLM tests green; ruff + mypy clean on both packages.

Review feedback addressed

• @mjnovice — separate per-provider classes: done (OpenAIStructuredOutput, AnthropicStructuredOutput, GeminiStructuredOutput, shared ToolCallStructuredOutput base).
• Copilot — description/implementation mismatch: resolved by the provider strategies (this description now matches); chat_completions docstring updated for raw-dict tools.

🤖 Generated with Claude Code

[github-actions]

🚨 Heads up: uipath-langchain cross-tests are FAILING 🚨

Your changes may break the uipath-langchain-python integration.

⚠️ These checks are NOT enforced by branch protection rules. Please review the failures before merging.

🔍 Inspect the failed run →

[Chibionos]

Re: the uipath-langchain cross-test heads-up above — that was from an earlier commit. After the adaptive fix (prefer response_format, fall back to function calling only when content is empty), the cross-tests pass on the latest commit: langchain-cross / {alpha,cloud,staging} and test-uipath-langchain (3.11/3.12/3.13 × ubuntu/windows) are all green. The heads-up comment is stale and can be disregarded.

Full suite is green: all eval test-cases (calculator-evals, simulation-testcase, tools-evals, etc.) pass across alpha/cloud/staging; lint, SonarCloud, build all pass.

[mjnovice]

Can we have separate classes for how we are doing Anthropic, OpenAI, Gemini etc. ?

[Chibionos]

Done in a8a3307 — split into per-provider strategy classes: OpenAIStructuredOutput (response_format preferred, tool fallback), AnthropicStructuredOutput and GeminiStructuredOutput (straight to a forced tool call, since their response_format is broken on the gateway — live-verified: Claude returns prose, Gemini empty content), with a shared ToolCallStructuredOutput base. Also avoids the wasted known-dead request per simulation for non-OpenAI models. E2E results on all 3 models are in the updated PR description.

[mjnovice]

Minor comment about making the generate_structured_output more modular.

[Copilot]

Pull request overview

This PR addresses eval tool/input simulation failures for non-OpenAI models by introducing a provider-agnostic structured-output helper and updating the eval mockers and LLM gateway integration to support function-calling style structured responses (including nested schemas via raw-dict tool passthrough).

Changes:

• Add generate_structured_output() helper to prefer response_format when available and fall back to a forced tool call when content is empty/unsupported.
• Update eval LLM mocker and input mocker to use the shared structured-output helper, and adjust/extend unit tests for the new behavior (including non-OpenAI fallback).
• Update UiPathLlmChatService.chat_completions() to accept raw-dict tools (pass-through) so nested JSON-schema tool parameters are preserved; bump package versions accordingly.

Reviewed changes

Copilot reviewed 10 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
packages/uipath/uv.lock	Bumps uipath and uipath-platform locked versions.
packages/uipath/pyproject.toml	Bumps uipath version and raises minimum uipath-platform dependency.
packages/uipath-platform/uv.lock	Bumps uipath-platform locked version.
packages/uipath-platform/pyproject.toml	Bumps uipath-platform version.
packages/uipath-platform/src/uipath/platform/chat/_llm_gateway_service.py	Allows passing raw-dict tools through to the normalized gateway request body.
packages/uipath-platform/tests/services/test_uipath_llm_integration.py	Adds coverage ensuring raw-dict tool schemas are forwarded unchanged and tool_choice serialization works.
packages/uipath/src/uipath/eval/mocks/_structured_output.py	New shared helper for structured output with response_format-first + tool-call fallback; includes schema $defs inlining logic.
packages/uipath/src/uipath/eval/mocks/_llm_mocker.py	Switches mock response generation to the shared structured-output helper; improves error propagation.
packages/uipath/src/uipath/eval/mocks/_input_mocker.py	Switches input generation to the shared structured-output helper.
packages/uipath/tests/cli/eval/mocks/test_structured_output.py	New unit tests validating schema wrapping/inlining and response extraction/fallback behavior.
packages/uipath/tests/cli/eval/mocks/test_mocks.py	Updates existing mocks and adds a non-OpenAI fallback regression test (AE-1646).
packages/uipath/tests/cli/eval/mocks/test_input_mocker.py	Adds assertion that OpenAI path uses response_format without tool fallback.

Comments suppressed due to low confidence (1)

packages/uipath/src/uipath/eval/mocks/_input_mocker.py:158

• generate_llm_input() no longer surfaces a clear JSON-parsing error when the LLM returns invalid JSON (the previous code raised UiPathInputMockingError("Failed to parse LLM response as JSON: ...")). Now a json.JSONDecodeError from generate_structured_output() is wrapped as a generic "Failed to generate input" error, which makes debugging structured-output failures harder.

    except UiPathInputMockingError:
        raise
    except Exception as e:
        raise UiPathInputMockingError(f"Failed to generate input: {str(e)}") from e

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[sonarqubecloud]

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
98.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud