From 5d5549704e1cd4a8f79c20e01eed2d1edee700ee Mon Sep 17 00:00:00 2001 From: JNK234 Date: Thu, 13 Aug 2026 22:06:03 -0500 Subject: [PATCH 1/2] feat: add structured output support - Add schema-constrained and JSON chat primitives with NetLogo conversion. - Send native response formats to OpenAI-compatible, Claude, Gemini, and Ollama providers. - Constrain llm:choose while preserving graceful fallback behavior. - Add provider, schema, conversion, history, and NetLogo integration coverage. - Document supported formats, validation boundaries, and provider differences. Implements NetLogo/Netlogo-LLM-Extension#22. Risk: provider support remains model-dependent; deterministic tests verify request payloads without live API calls. --- docs/API-REFERENCE.md | 193 +++++++++++++ src/main/LLMExtension.scala | 211 +++++++++++++- src/main/models/ChatRequest.scala | 7 +- src/main/models/ResponseFormat.scala | 268 ++++++++++++++++++ src/main/providers/BaseHttpProvider.scala | 24 +- src/main/providers/ClaudeProvider.scala | 47 ++- src/main/providers/GeminiProvider.scala | 18 +- src/main/providers/LLMProvider.scala | 16 +- src/main/providers/OllamaProvider.scala | 11 +- .../providers/OpenAICompatibleProvider.scala | 35 ++- src/main/utils/JsonToNetLogo.scala | 105 +++++++ src/test/DeterministicTestProvider.scala | 78 ++++- src/test/JsonToNetLogoSpec.scala | 142 ++++++++++ src/test/ResponseFormatSpec.scala | 164 +++++++++++ src/test/StructuredChatPlumbingSpec.scala | 64 +++++ src/test/StructuredOutputRequestSpec.scala | 233 +++++++++++++++ tests.txt | 223 +++++++++++++++ 17 files changed, 1823 insertions(+), 16 deletions(-) create mode 100644 src/main/models/ResponseFormat.scala create mode 100644 src/main/utils/JsonToNetLogo.scala create mode 100644 src/test/JsonToNetLogoSpec.scala create mode 100644 src/test/ResponseFormatSpec.scala create mode 100644 src/test/StructuredChatPlumbingSpec.scala create mode 100644 src/test/StructuredOutputRequestSpec.scala diff --git a/docs/API-REFERENCE.md b/docs/API-REFERENCE.md index fad341f..64b1399 100644 --- a/docs/API-REFERENCE.md +++ b/docs/API-REFERENCE.md @@ -13,6 +13,9 @@ The NetLogo Multi-LLM Extension provides a unified interface for multiple Large | `llm:chat-with-template file vars` | Chat | Send templated prompt with variable substitution | | `llm:chat-with-thinking text` | Chat | Returns `[answer thinking]` for reasoning-capable models | | `llm:choose prompt choices` | Chat | Force selection from provided options | +| `llm:chat-with-schema prompt schema` | Structured | Schema-constrained chat, returns parsed nested lists | +| `llm:chat-json prompt` | Structured | Force valid JSON output, returns the raw JSON string | +| `llm:get parsed key` | Structured | Look up a key in a `[[key value] ...]` list | | `llm:compile-error code` | Validation | `""` if valid, else compiler error; optional disallowed list | | `llm:set-thinking bool` | Reasoning | Enable/disable reasoning mode for current provider | | `llm:set-reasoning-effort level` | Reasoning | Set effort: `"low"`, `"medium"`, `"high"` | @@ -235,6 +238,117 @@ set color read-from-string color-choice - Forces LLM to return exactly one of the provided choices - Useful for agent decision-making in models - Maintains conversation context +- Where the provider supports constrained decoding, the option list is sent as a + schema constraint so the model cannot reply with anything else. Providers that + do not support it fall back to the prompt wording, and the reply is still + matched case-insensitively against the list — so behaviour is unchanged, just + more reliable where the constraint is available. +- Still raises an error if the reply matches no option. Wrap in `carefully` if a + model should tolerate that. + +## Structured Output Primitives + +These constrain the *shape* of a reply at the API level rather than by asking +nicely in the prompt. See [Structured Output](#structured-output-details) below +for the JSON-to-NetLogo mapping and provider support. + +### llm:chat-with-schema + +**Syntax**: `llm:chat-with-schema prompt schema` + +**Description**: Sends a prompt with the reply constrained to a JSON Schema, and +reports the parsed result as nested NetLogo lists. + +**Parameters**: + +- `prompt` (string): The question or context +- `schema` (string): A JSON Schema object, as a string + +**Returns**: List — the parsed JSON. Objects arrive as `[[key value] ...]` pairs; +use `llm:get` to read fields. + +**Example**: + +```netlogo +let schema "{\"type\":\"object\",\"properties\":{\"action\":{\"type\":\"string\"},\"confidence\":{\"type\":\"number\"}}}" +let result llm:chat-with-schema "What should the agent do?" schema +let action llm:get result "action" ; "explore" +let conf llm:get result "confidence" ; 0.85 +``` + +**Notes**: + +- The schema's structure is checked before any request is sent — that it is a + JSON object, that every level declares a supported `type`, that objects have + non-empty `properties`, that arrays declare `items`, and that any `enum` is a + non-empty array. A problem in any of those fails immediately with a message + naming the offending path, rather than as an opaque provider error after a + network round-trip. +- Other JSON Schema keywords are passed through to the provider unchecked. The + four providers accept different dialects, so validating every keyword here + would reject schemas a given provider would have honoured; a keyword one + provider rejects still surfaces as that provider's error. +- Object schemas are normalized to strict mode (`additionalProperties: false`, + every property listed in `required`) because OpenAI and Anthropic require it. + The same normalization is applied for every provider so one schema behaves + the same everywhere. +- If the reply is not valid JSON, this raises an error and conversation history + is left unchanged — a failed call never records a bogus exchange. + +### llm:chat-json + +**Syntax**: `llm:chat-json prompt` + +**Description**: Sends a prompt with the reply constrained to valid JSON, with no +schema, and reports the raw JSON string. + +**Returns**: String — the JSON text + +**Example**: + +```netlogo +let json-str llm:chat-json "List three actions as a JSON array" +``` + +**Notes**: + +- Use `llm:chat-with-schema` when the shape matters and you want NetLogo values + back; use this when you want the JSON text itself. +- The reply is checked to be valid JSON before it is reported. Not every + provider can enforce schemaless JSON natively — Anthropic has no such mode — + so a model that replies with prose raises an error rather than returning it, + and conversation history is left unchanged. +- Any JSON value is accepted, not just objects: an array or a scalar is still + valid JSON. The text is reported exactly as received, not reformatted. + +### llm:get + +**Syntax**: `llm:get parsed key` + +**Description**: Looks up a key in a list of `[key value]` pairs — the shape +`llm:chat-with-schema` reports. + +**Parameters**: + +- `parsed` (list): A list of `[key value]` pairs +- `key` (string): The key to find + +**Returns**: The value at that key — a string, number, boolean, or nested list + +**Example**: + +```netlogo +let result llm:chat-with-schema "Describe a turtle" schema +let name llm:get result "name" +let city llm:get (llm:get result "address") "city" ; nested lookup +``` + +**Notes**: + +- Matching is exact and case-sensitive, because JSON keys are. +- A missing key raises an error listing the available keys. It does not report a + blank — a silent `""` would be indistinguishable from a JSON null and would let + a typo travel through a run as data. ### llm:chat-with-template @@ -744,6 +858,85 @@ print llm:list-models ; Shows all providers, with Anthropic marked as ACTIVE - Custom models added via `models-override.yaml` are marked with `[custom]` - The currently active provider and model are marked with `[ACTIVE]` +## Structured Output Details + +### Why JSON becomes nested lists + +NetLogo's type system has strings, numbers, booleans, lists, and agents — there +is no map or dictionary type. A JSON object therefore arrives as a list of +`[key value]` pairs, which is the association-list shape NetLogo modelers +already use and which `llm:get` can search. + +| JSON | NetLogo | Example | +| ----------- | --------------------------- | ------------------------------ | +| Object `{}` | List of `[key value]` pairs | `[["name" "Alice"] ["age" 30]]` | +| Array `[]` | List | `["a" "b" "c"]` | +| String | String | `"hello"` | +| Number | Number | `42`, `3.14` | +| Boolean | Boolean | `true`, `false` | +| Null | Empty string | `""` | + +Null maps to `""` because NetLogo has no null, and `""` is a value a model can +compare against without a runtime error. + +### Provider support + +Each provider expresses the same constraint in its own request field: + +| Provider | Field | Schemaless JSON mode | +| ----------------------------------- | -------------------------------------- | --------------------------- | +| OpenAI, Groq, Together, OpenRouter | `response_format.json_schema.schema` | `response_format.json_object` | +| Anthropic (Claude) | `output_config.format.schema` | Prompt instruction only | +| Gemini | `generationConfig.responseJsonSchema` | `responseMimeType` only | +| Ollama | `format` | `format: "json"` | + +Notes on the two asymmetries: + +- **Anthropic has no schemaless JSON mode.** For `llm:chat-json`, no + `output_config.format` is sent — inventing one would be rejected by the API — + so the JSON instruction is carried in the prompt instead. `llm:chat-with-schema` + uses the native schema field. +- **Claude's `output_config` is shared.** Thinking depth (`effort`) and response + format (`format`) are siblings under one key, so both are merged into the same + object. Setting a schema does not disturb an existing `effort`, and vice versa. + +Whether a *particular model* honours the constraint is decided by the provider, +not this extension. Where a model ignores it, the reply is still returned and +`llm:choose` still matches it against the option list — nothing fails merely +because constrained decoding was unavailable. + +### Scope of this release + +`llm:set-response-format` and `llm:clear-response-format` — a persistent schema +applied to ordinary `llm:chat` calls — are **not** included. Two reasons: + +1. `llm:chat` declares a `StringType` return in its syntax. A persistent schema + would make it report JSON-as-a-string with no signal at the type level, so + model code could not tell which shape it was about to get. +2. The format would live in global config while conversation history is + per-agent, so a schema set for one agent would silently apply to every other + agent's calls mid-run. + +The one-shot primitives (`llm:chat-with-schema`, `llm:chat-json`) express the +same capability without either problem: the constraint and the return type are +visible at the call site. If a persistent form is added later, per-agent scoping +and a distinct return type are the open design questions. + +Generalizing `llm:choose` to non-string results (numbers, coordinates, ordered +plans) is likewise not a new set of primitives here — those are schema shapes for +`llm:chat-with-schema`. For example, a multi-step plan: + +```netlogo +let schema "{\"type\":\"object\",\"properties\":{\"steps\":{\"type\":\"array\",\"items\":{\"type\":\"string\"}}}}" +let plan llm:get (llm:chat-with-schema "Plan three moves" schema) "steps" +foreach plan [ s -> print s ] +``` + +If the steps are meant to be executed as NetLogo code rather than printed, check +each one with `llm:compile-error` first and keep `carefully` around the `run` — +a schema constrains the *shape* of a reply, not whether its contents are safe or +runnable. + ## Usage Patterns ### Basic Chat Bot diff --git a/src/main/LLMExtension.scala b/src/main/LLMExtension.scala index 450131c..54f2cd5 100644 --- a/src/main/LLMExtension.scala +++ b/src/main/LLMExtension.scala @@ -4,7 +4,8 @@ import org.nlogo.api._ import org.nlogo.core.{LogoList, Syntax} import org.nlogo.extensions.llm.config.{ConfigLoader, ConfigStore} import org.nlogo.extensions.llm.providers.{LLMProvider, ProviderDescriptor, ProviderFactory, ProviderRegistry, ProviderRegistrations, ModelRegistry, OllamaProvider, ReadinessCheck, RetryPolicy} -import org.nlogo.extensions.llm.models.{ChatMessage, ChatResponse} +import org.nlogo.extensions.llm.models.{ChatMessage, ChatResponse, EnumFormat, JsonObjectFormat, ResponseFormat} +import org.nlogo.extensions.llm.utils.JsonToNetLogo import scala.collection.mutable.{ArrayBuffer, WeakHashMap} import scala.concurrent.{Await, ExecutionContext, Future} import scala.concurrent.duration._ @@ -102,6 +103,11 @@ class LLMExtension extends DefaultClassManager { manager.addPrimitive("chat-with-thinking", ChatWithThinkingReporter) manager.addPrimitive("choose", ChooseReporter) + // Structured output primitives + manager.addPrimitive("chat-with-schema", ChatWithSchemaReporter) + manager.addPrimitive("chat-json", ChatJsonReporter) + manager.addPrimitive("get", GetReporter) + // Code validation primitive manager.addPrimitive("compile-error", CompileErrorReporter) @@ -621,8 +627,10 @@ class LLMExtension extends DefaultClassManager { tempHistory.prepend(ChatMessage.system(systemPrompt)) tempHistory += ChatMessage.user(userPrompt) - // Use chatWithFullResponse to access thinking field for thinking models - val responseFuture = provider.chatWithFullResponse(tempHistory.toSeq) + // Constrain the reply to the choice list where the provider supports it. + // The prompt still spells out the options, so a provider that ignores + // the constraint behaves exactly as it did before. + val responseFuture = provider.chatWithFormat(tempHistory.toSeq, EnumFormat(choices)) val response = Await.result(responseFuture, getAwaitTimeout) // Extract text: prefer content, fall back to thinking field @@ -631,11 +639,16 @@ class LLMExtension extends DefaultClassManager { .getOrElse("") .trim + // A constrained reply arrives as {"choice": "..."} while an unconstrained + // one is the bare option text. Accept both: which shape comes back + // depends on provider support, and the modeler asked for neither. + val candidate = extractEnumChoice(text).getOrElse(text) + // Exact match only (case-insensitive) - val chosenOption = choices.find(_.equalsIgnoreCase(text)) + val chosenOption = choices.find(_.equalsIgnoreCase(candidate)) .getOrElse { throw new ExtensionException( - s"llm:choose: response '$text' did not match any choice. " + + s"llm:choose: response '$candidate' did not match any choice. " + s"Choices: ${choices.mkString(", ")}" ) } @@ -652,6 +665,194 @@ class LLMExtension extends DefaultClassManager { } } + /** + * Read the selected option out of an enum-constrained reply. + * + * A provider enforcing the constraint returns `{"choice": "north"}`; one that + * ignores it returns `north`. Returns None for anything that is not the + * constrained shape, so the caller can fall back to the raw text. + */ + private def extractEnumChoice(text: String): Option[String] = + scala.util.Try { + ujson.read(text)(EnumFormat.ChoiceKey).str + }.toOption + + // Structured Output Primitives + + /** + * Chat with the reply constrained to a JSON Schema, reported as nested lists. + * + * llm:chat-with-schema prompt schema-string + * + * NetLogo has no map type, so the parsed JSON object arrives as a list of + * `[key value]` pairs — use `llm:get` to read fields out of it. + * + * The schema is validated before any request is sent: an unusable schema + * otherwise surfaces as an opaque provider 400 after a network round-trip, + * which a modeler cannot act on. + */ + object ChatWithSchemaReporter extends Reporter { + override def getSyntax: Syntax = Syntax.reporterSyntax( + right = List(Syntax.StringType, Syntax.StringType), + ret = Syntax.ListType + ) + + override def report(args: Array[Argument], context: Context): AnyRef = { + val inputText = args(0).getString + val schemaText = args(1).getString + val agent = context.getAgent + + // Validate before touching the provider so a bad schema costs nothing. + val format = + try ResponseFormat.parseSchema(schemaText) + catch { + case e: IllegalArgumentException => + throw new ExtensionException(s"llm:chat-with-schema: ${e.getMessage}") + } + + // This reporter's syntax promises a list, and only a JSON object converts + // to the [key value] pairs llm:get reads. A top-level scalar or array + // schema would report a bare string or a flat list instead, breaking that + // promise, so it is rejected here rather than at the provider. + val topLevelType = format.schema.value.get("type").collect { case ujson.Str(s) => s } + if (!topLevelType.contains("object")) { + throw new ExtensionException( + s"llm:chat-with-schema: schema must have type 'object' at the top level, but got " + + s"'${topLevelType.getOrElse("none")}'. llm:chat-with-schema reports a list of " + + "[key value] pairs, so the reply has to be a JSON object. Wrap it, e.g. " + + """{"type":"object","properties":{"value":{"type":"string"}}}""" + ) + } + + try { + val provider = ensureProvider() + + val userMessage = ChatMessage.user(inputText) + + val responseFuture = provider.chatWithFormat(snapshotHistory(agent) :+ userMessage, format) + val response = Await.result(responseFuture, getAwaitTimeout) + + val content = response.firstContent.getOrElse("") + + // Parse before committing: a reply that is not JSON means the call did + // not deliver what was asked for, so history must not record it as a + // successful exchange. + val parsed = + try JsonToNetLogo.parseObject(content) + catch { + case e: IllegalArgumentException => + throw new ExtensionException(s"llm:chat-with-schema: ${e.getMessage}") + } + + commitExchange(agent, userMessage, ChatMessage.assistant(content)) + + parsed + + } catch { + case e: Exception if !e.isInstanceOf[ExtensionException] => + throw new ExtensionException(s"llm:chat-with-schema failed: ${e.getMessage}") + } + } + } + + /** + * Chat with the reply constrained to valid JSON, reported as a raw string. + * + * llm:chat-json prompt + * + * Reports the JSON text rather than parsed lists, for modelers who want to + * hand it to another tool or inspect it directly. Use `llm:chat-with-schema` + * when the shape matters and you want NetLogo values back. + */ + object ChatJsonReporter extends Reporter { + override def getSyntax: Syntax = Syntax.reporterSyntax( + right = List(Syntax.StringType), + ret = Syntax.StringType + ) + + override def report(args: Array[Argument], context: Context): AnyRef = { + val inputText = args(0).getString + val agent = context.getAgent + + try { + val provider = ensureProvider() + + // Anthropic has no schemaless JSON mode and Gemini's mime type alone is + // only a hint, so the instruction is also stated in the prompt. Providers + // that do enforce JSON natively are unaffected by the extra sentence. + val jsonInstruction = ChatMessage.system( + "Respond with valid JSON only. No prose, no markdown code fences." + ) + val userMessage = ChatMessage.user(inputText) + + val tempHistory = ArrayBuffer.from(snapshotHistory(agent)) + tempHistory.prepend(jsonInstruction) + tempHistory += userMessage + + val responseFuture = provider.chatWithFormat(tempHistory.toSeq, JsonObjectFormat) + val response = Await.result(responseFuture, getAwaitTimeout) + + val content = response.firstContent.getOrElse("") + + // Not every provider can enforce schemaless JSON natively — Anthropic + // has no such mode at all — so the reply is verified here rather than + // trusted. Checking before the commit keeps a failed call out of + // history, and stops prose being reported from a primitive whose whole + // contract is that the result parses as JSON. Any JSON value is + // accepted: an array or scalar is still valid JSON. + try ujson.read(content) + catch { + case e: Exception => + throw new ExtensionException( + s"llm:chat-json: Response was not valid JSON: ${e.getMessage}. Response text: $content" + ) + } + + // Store the clean exchange only — the JSON instruction is prompt + // scaffolding, not part of the conversation. + commitExchange(agent, userMessage, ChatMessage.assistant(content)) + + content + + } catch { + case e: Exception if !e.isInstanceOf[ExtensionException] => + throw new ExtensionException(s"llm:chat-json failed: ${e.getMessage}") + } + } + } + + /** + * Look up a key in a list of `[key value]` pairs. + * + * llm:get parsed-result "key" + * + * Throws on a missing key rather than reporting a sentinel: a silent "" would + * be indistinguishable from a JSON null and would let a typo propagate as data + * through the rest of a run. + */ + object GetReporter extends Reporter { + override def getSyntax: Syntax = Syntax.reporterSyntax( + right = List(Syntax.ListType, Syntax.StringType), + ret = Syntax.WildcardType + ) + + override def report(args: Array[Argument], context: Context): AnyRef = { + val list = args(0).getList + val key = args(1).getString + + JsonToNetLogo.lookup(list, key).getOrElse { + val available = list.toVector.collect { + case pair: LogoList if pair.size == 2 => pair(0).toString + } + val detail = + if (list.size == 0) "The list is empty." + else if (available.isEmpty) "Available keys: (none - the list is not a list of [key value] pairs)" + else s"Available keys: ${available.mkString(", ")}" + throw new ExtensionException(s"""llm:get: key "$key" not found. $detail""") + } + } + } + // Thinking/Reasoning Primitives object ChatWithThinkingReporter extends Reporter { diff --git a/src/main/models/ChatRequest.scala b/src/main/models/ChatRequest.scala index dad09c9..bed533b 100644 --- a/src/main/models/ChatRequest.scala +++ b/src/main/models/ChatRequest.scala @@ -9,13 +9,18 @@ import upickle.default.{ReadWriter => RW, macroRW} * @param messages The conversation history as a sequence of messages * @param maxTokens Optional maximum number of tokens to generate * @param temperature Optional temperature for response randomness (0.0-2.0) + * @param thinkingConfig Optional reasoning/thinking configuration + * @param responseFormat Optional constraint on the shape of the reply. Each + * provider wraps this in its own request field; None + * leaves the request byte-for-byte as before. */ case class ChatRequest( model: String, messages: Seq[ChatMessage], maxTokens: Option[Int] = None, temperature: Option[Double] = None, - thinkingConfig: Option[ThinkingConfig] = None + thinkingConfig: Option[ThinkingConfig] = None, + responseFormat: Option[ResponseFormat] = None ) object ChatRequest { diff --git a/src/main/models/ResponseFormat.scala b/src/main/models/ResponseFormat.scala new file mode 100644 index 0000000..09e37f3 --- /dev/null +++ b/src/main/models/ResponseFormat.scala @@ -0,0 +1,268 @@ +// ABOUTME: Response-format model describing how an LLM's output should be constrained +// ABOUTME: Parses and normalizes modeler-supplied JSON Schema strings into a provider-neutral form +package org.nlogo.extensions.llm.models + +/** + * How a provider should constrain the shape of its reply. + * + * Provider-neutral by design: each provider decides how to wrap these in its own + * request body (see the `applyResponseFormat` hooks in the providers package). + */ +sealed trait ResponseFormat + +/** + * Constrain output to a JSON Schema. + * + * `schema` is always a JSON object, validated at construction by + * [[ResponseFormat.parseSchema]]. Providers that require strict-mode schemas + * pass it through [[ResponseFormat.strictSchema]] first. + */ +case class JsonSchemaFormat(schema: ujson.Obj, name: String = ResponseFormat.DefaultSchemaName) + extends ResponseFormat + +/** + * Constrain output to exactly one of a fixed list of strings. + * + * Rendered as a one-property object schema rather than a bare `{"enum": [...]}`, + * because the providers that support constrained decoding all accept an object + * schema, while a top-level enum is accepted only by some. One shape works + * everywhere, so `llm:choose` behaves the same regardless of provider. + */ +case class EnumFormat(choices: List[String]) extends ResponseFormat { + require(choices.nonEmpty, "Choice list cannot be empty") + + /** The enum rendered as an object schema with a single constrained property. */ + def schema: ujson.Obj = + ujson.Obj( + "type" -> "object", + "properties" -> ujson.Obj( + EnumFormat.ChoiceKey -> ujson.Obj( + "type" -> "string", + "enum" -> ujson.Arr(choices.map(c => ujson.Str(c))*) + ) + ), + "required" -> ujson.Arr(ujson.Str(EnumFormat.ChoiceKey)), + "additionalProperties" -> false + ) +} + +object EnumFormat { + /** Property name holding the selected option in an enum-constrained reply. */ + val ChoiceKey: String = "choice" +} + +/** Constrain output to syntactically valid JSON, with no schema. */ +case object JsonObjectFormat extends ResponseFormat + +object ResponseFormat { + + /** Schema name sent to providers that require one (OpenAI). */ + val DefaultSchemaName: String = "netlogo_schema" + + /** + * Serialization for the sealed hierarchy. + * + * Hand-written rather than derived: ujson.Obj has no upickle ReadWriter, and + * ChatRequest's derived instance needs one for this field to exist at all. + */ + implicit val rw: upickle.default.ReadWriter[ResponseFormat] = + upickle.default.readwriter[ujson.Value].bimap[ResponseFormat]( + { + case JsonSchemaFormat(schema, name) => + ujson.Obj("kind" -> "json_schema", "name" -> name, "schema" -> schema) + case EnumFormat(choices) => + ujson.Obj("kind" -> "enum", "choices" -> ujson.Arr(choices.map(ujson.Str(_))*)) + case JsonObjectFormat => + ujson.Obj("kind" -> "json_object") + }, + json => + json("kind").str match { + case "json_schema" => + JsonSchemaFormat(ujson.Obj.from(json("schema").obj), json("name").str) + case "enum" => + EnumFormat(json("choices").arr.map(_.str).toList) + case "json_object" => + JsonObjectFormat + case other => + throw new IllegalArgumentException(s"Unknown response format kind: $other") + } + ) + + /** + * Parse and validate a modeler-supplied JSON Schema string. + * + * Validation is deliberately front-loaded: an unusable schema otherwise fails + * as an opaque provider 400 after a network round-trip, which a modeler cannot + * act on. Errors here name what is wrong with the schema they wrote. + * + * @throws IllegalArgumentException if the string is not a usable object schema + */ + def parseSchema(schemaText: String): JsonSchemaFormat = { + if (schemaText == null || schemaText.trim.isEmpty) { + throw new IllegalArgumentException( + "Schema cannot be empty. Provide a JSON Schema object, e.g. " + + """{"type":"object","properties":{"action":{"type":"string"}}}""" + ) + } + + val parsed = + try ujson.read(schemaText) + catch { + case e: Exception => + throw new IllegalArgumentException( + s"Schema is not valid JSON: ${e.getMessage}" + ) + } + + val obj = parsed match { + case o: ujson.Obj => o + case other => + throw new IllegalArgumentException( + s"Schema must be a JSON object, but got ${typeName(other)}. " + + """Example: {"type":"object","properties":{"action":{"type":"string"}}}""" + ) + } + + validate(obj, path = "schema") + JsonSchemaFormat(obj) + } + + /** + * Check a schema is one a provider can actually enforce. + * + * Only the constraints every supported provider shares are checked. Anything + * provider-specific stays out, so a schema accepted here is not silently + * rejected by a different provider later. + */ + private def validate(schema: ujson.Obj, path: String): Unit = { + val typeValue = schema.value.get("type").map { + case ujson.Str(s) => s + case other => + throw new IllegalArgumentException( + s"$path: 'type' must be a string, but got ${typeName(other)}" + ) + }.getOrElse { + throw new IllegalArgumentException( + s"$path is missing a 'type' field. Every schema needs one, e.g. " + + """{"type":"object","properties":{...}}""" + ) + } + + // `enum` is checked because every provider requires a non-empty array here + // and `llm:choose` builds one itself. Other keywords are passed through + // unchecked on purpose: the four providers accept different JSON Schema + // dialects, so validating them all here would reject schemas a provider + // would have honoured. + schema.value.get("enum").foreach { + case arr: ujson.Arr => + if (arr.value.isEmpty) { + throw new IllegalArgumentException(s"$path: 'enum' must list at least one value") + } + case other => + throw new IllegalArgumentException( + s"$path: 'enum' must be an array of values, but got ${typeName(other)}. " + + """Example: {"type":"string","enum":["north","south"]}""" + ) + } + + typeValue match { + case "object" => + val props = schema.value.get("properties") match { + case Some(o: ujson.Obj) => o + case Some(other) => + throw new IllegalArgumentException( + s"$path: 'properties' must be a JSON object, but got ${typeName(other)}" + ) + case None => + throw new IllegalArgumentException( + s"$path of type 'object' is missing 'properties'. An object schema with no " + + "properties constrains nothing, so no provider can enforce it." + ) + } + if (props.value.isEmpty) { + throw new IllegalArgumentException( + s"$path: 'properties' is empty. An object schema must declare at least one property." + ) + } + props.value.foreach { case (key, value) => + value match { + case o: ujson.Obj => validate(o, s"$path.properties.$key") + case other => + throw new IllegalArgumentException( + s"$path.properties.$key must be a schema object, but got ${typeName(other)}" + ) + } + } + + case "array" => + schema.value.get("items") match { + case Some(o: ujson.Obj) => validate(o, s"$path.items") + case Some(other) => + throw new IllegalArgumentException( + s"$path.items must be a schema object, but got ${typeName(other)}" + ) + case None => + throw new IllegalArgumentException( + s"$path of type 'array' is missing 'items'. Declare the element schema, e.g. " + + """{"type":"array","items":{"type":"string"}}""" + ) + } + + case "string" | "number" | "integer" | "boolean" | "null" => () + + case other => + throw new IllegalArgumentException( + s"$path: unsupported 'type' value '$other'. Supported: " + + "object, array, string, number, integer, boolean, null" + ) + } + } + + /** + * Return a strict-mode copy of a schema. + * + * OpenAI and Anthropic both reject schemas that omit `additionalProperties: + * false` or leave properties out of `required`. Applying the same + * normalization for every provider means one schema behaves identically + * everywhere, rather than a modeler's schema working on Ollama and 400ing on + * OpenAI. + * + * The input is never mutated — ujson values are mutable, and the caller's + * schema is reused across requests. + */ + def strictSchema(schema: ujson.Value): ujson.Value = schema match { + case obj: ujson.Obj => + val out = ujson.Obj() + obj.value.foreach { + case ("properties", props: ujson.Obj) => + val strictProps = ujson.Obj() + props.value.foreach { case (k, v) => strictProps(k) = strictSchema(v) } + out("properties") = strictProps + case ("items", items) => + out("items") = strictSchema(items) + case (k, v) => + out(k) = v + } + + if (obj.value.get("type").exists(_ == ujson.Str("object"))) { + out("additionalProperties") = false + obj.value.get("properties") match { + case Some(props: ujson.Obj) => + out("required") = ujson.Arr(props.value.keys.map(k => ujson.Str(k)).toSeq*) + case _ => () + } + } + out + + case other => other + } + + private def typeName(value: ujson.Value): String = value match { + case _: ujson.Obj => "an object" + case _: ujson.Arr => "an array" + case _: ujson.Str => "a string" + case _: ujson.Num => "a number" + case _: ujson.Bool => "a boolean" + case ujson.Null => "null" + } +} diff --git a/src/main/providers/BaseHttpProvider.scala b/src/main/providers/BaseHttpProvider.scala index 1c80073..8997cdb 100644 --- a/src/main/providers/BaseHttpProvider.scala +++ b/src/main/providers/BaseHttpProvider.scala @@ -2,7 +2,7 @@ // ABOUTME: Reduces boilerplate by providing shared implementation of config, validation, and HTTP request handling package org.nlogo.extensions.llm.providers -import org.nlogo.extensions.llm.models.{ChatMessage, ChatRequest, ChatResponse} +import org.nlogo.extensions.llm.models.{ChatMessage, ChatRequest, ChatResponse, ResponseFormat} import org.nlogo.extensions.llm.config.ConfigStore import sttp.client4._ import sttp.client4.httpclient.HttpClientFutureBackend @@ -160,7 +160,10 @@ abstract class BaseHttpProvider(implicit ec: ExecutionContext) extends LLMProvid /** * Build a ChatRequest from current config and messages, resolving ThinkingConfig */ - protected def buildRequest(messages: Seq[ChatMessage]): ChatRequest = { + protected def buildRequest( + messages: Seq[ChatMessage], + responseFormat: Option[ResponseFormat] = None + ): ChatRequest = { val model = configStore.getOrElse(ConfigStore.MODEL, defaultModel) val temperature = configStore.get(ConfigStore.TEMPERATURE).map(_.toDouble) val maxTokens = configStore.get(ConfigStore.MAX_TOKENS).map(_.toInt) @@ -171,7 +174,8 @@ abstract class BaseHttpProvider(implicit ec: ExecutionContext) extends LLMProvid messages = messages, maxTokens = maxTokens, temperature = temperature, - thinkingConfig = thinkingConfig + thinkingConfig = thinkingConfig, + responseFormat = responseFormat ) } @@ -194,6 +198,20 @@ abstract class BaseHttpProvider(implicit ec: ExecutionContext) extends LLMProvid chat(request) } + /** + * Chat with the reply constrained to a response format. + * + * The format travels on the ChatRequest; each provider's createProviderRequest + * decides how to express it on the wire. + */ + override def chatWithFormat( + messages: Seq[ChatMessage], + format: ResponseFormat + ): Future[ChatResponse] = { + val request = buildRequest(messages, Some(format)) + chat(request) + } + /** * Send the actual HTTP request */ diff --git a/src/main/providers/ClaudeProvider.scala b/src/main/providers/ClaudeProvider.scala index 1db0180..1302d1b 100644 --- a/src/main/providers/ClaudeProvider.scala +++ b/src/main/providers/ClaudeProvider.scala @@ -3,7 +3,7 @@ package org.nlogo.extensions.llm.providers -import org.nlogo.extensions.llm.models.{ChatMessage, ChatRequest, ChatResponse} +import org.nlogo.extensions.llm.models.{ChatMessage, ChatRequest, ChatResponse, EnumFormat, JsonObjectFormat, JsonSchemaFormat, ResponseFormat} import org.nlogo.extensions.llm.config.ConfigStore import sttp.client4._ import sttp.model.Uri @@ -115,7 +115,7 @@ class ClaudeProvider(implicit ec: ExecutionContext) extends BaseHttpProvider { ClaudeModelCapabilities .effortValue(request.thinkingConfig.flatMap(_.reasoningEffort)) .foreach { effort => - baseRequest("output_config") = ujson.Obj("effort" -> effort) + outputConfig(baseRequest)("effort") = effort } } } else if (allowsSampling) { @@ -124,9 +124,52 @@ class ClaudeProvider(implicit ec: ExecutionContext) extends BaseHttpProvider { } } + applyResponseFormat(baseRequest, request) + baseRequest } + /** + * Get (creating if needed) the request's `output_config` object. + * + * `effort` and `format` are siblings under one key, so both writers must + * extend the same object. Assigning a fresh `ujson.Obj` from either would + * silently drop whatever the other had already set. + */ + private def outputConfig(baseRequest: ujson.Obj): ujson.Obj = + baseRequest.value.get("output_config") match { + case Some(existing: ujson.Obj) => existing + case _ => + val created = ujson.Obj() + baseRequest("output_config") = created + created + } + + /** + * Add Anthropic's native `output_config.format`. + * + * Only schema-bearing formats are sent. Anthropic has no schemaless JSON mode, + * so [[JsonObjectFormat]] adds nothing here — inventing a `json_object` type + * would be rejected by the API. The prompt-level instruction added by the + * extension carries that case instead. + */ + private def applyResponseFormat(baseRequest: ujson.Obj, request: ChatRequest): Unit = + request.responseFormat.foreach { + case JsonSchemaFormat(schema, _) => + outputConfig(baseRequest)("format") = ujson.Obj( + "type" -> "json_schema", + "schema" -> ResponseFormat.strictSchema(schema) + ) + + case enumFormat: EnumFormat => + outputConfig(baseRequest)("format") = ujson.Obj( + "type" -> "json_schema", + "schema" -> enumFormat.schema + ) + + case JsonObjectFormat => () + } + override protected def parseProviderResponse(responseBody: String, model: String): ChatResponse = { try { val parsed = ujson.read(responseBody) diff --git a/src/main/providers/GeminiProvider.scala b/src/main/providers/GeminiProvider.scala index d44df27..3ff489d 100644 --- a/src/main/providers/GeminiProvider.scala +++ b/src/main/providers/GeminiProvider.scala @@ -3,7 +3,7 @@ package org.nlogo.extensions.llm.providers -import org.nlogo.extensions.llm.models.{ChatMessage, ChatRequest, ChatResponse} +import org.nlogo.extensions.llm.models.{ChatMessage, ChatRequest, ChatResponse, EnumFormat, JsonObjectFormat, JsonSchemaFormat, ResponseFormat} import org.nlogo.extensions.llm.config.ConfigStore import sttp.client4._ import sttp.model.Uri @@ -131,6 +131,22 @@ class GeminiProvider(implicit ec: ExecutionContext) extends BaseHttpProvider { hasConfig = true } + // Structured output lives in generationConfig alongside thinking, so both + // can be set on one request. responseMimeType is mandatory whenever JSON is + // wanted; responseJsonSchema is the newer standard-JSON-Schema field, which + // takes lowercase types exactly as the modeler wrote them. + request.responseFormat.foreach { format => + generationConfig("responseMimeType") = "application/json" + hasConfig = true + format match { + case JsonSchemaFormat(schema, _) => + generationConfig("responseJsonSchema") = ResponseFormat.strictSchema(schema) + case enumFormat: EnumFormat => + generationConfig("responseJsonSchema") = enumFormat.schema + case JsonObjectFormat => () + } + } + if (hasConfig) { baseRequest("generationConfig") = generationConfig } diff --git a/src/main/providers/LLMProvider.scala b/src/main/providers/LLMProvider.scala index 44b1770..5c7e2c3 100644 --- a/src/main/providers/LLMProvider.scala +++ b/src/main/providers/LLMProvider.scala @@ -1,6 +1,6 @@ package org.nlogo.extensions.llm.providers -import org.nlogo.extensions.llm.models.{ChatMessage, ChatRequest, ChatResponse} +import org.nlogo.extensions.llm.models.{ChatMessage, ChatRequest, ChatResponse, ResponseFormat} import scala.concurrent.Future import scala.util.Try @@ -36,6 +36,20 @@ trait LLMProvider { */ def chatWithFullResponse(messages: Seq[ChatMessage]): Future[ChatResponse] + /** + * Chat with the reply constrained to a response format. + * + * Defaults to an unconstrained request so existing providers keep compiling + * and behave exactly as before; providers that can enforce a format override + * this. A provider that ignores the format still returns a usable answer, + * which is the graceful-degradation behaviour the extension relies on. + * + * @param messages The conversation history + * @param format How the reply should be shaped + */ + def chatWithFormat(messages: Seq[ChatMessage], format: ResponseFormat): Future[ChatResponse] = + chatWithFullResponse(messages) + /** * Set a configuration parameter for this provider * diff --git a/src/main/providers/OllamaProvider.scala b/src/main/providers/OllamaProvider.scala index 5e7ad54..06a995f 100644 --- a/src/main/providers/OllamaProvider.scala +++ b/src/main/providers/OllamaProvider.scala @@ -3,7 +3,7 @@ package org.nlogo.extensions.llm.providers -import org.nlogo.extensions.llm.models.{ChatMessage, ChatRequest, ChatResponse} +import org.nlogo.extensions.llm.models.{ChatMessage, ChatRequest, ChatResponse, EnumFormat, JsonObjectFormat, JsonSchemaFormat, ResponseFormat} import org.nlogo.extensions.llm.config.ConfigStore import sttp.client4._ import sttp.model.Uri @@ -55,6 +55,15 @@ class OllamaProvider(implicit ec: ExecutionContext) extends BaseHttpProvider { baseRequest("think") = true } + // Ollama takes the raw schema in `format`, or the bare string "json" for + // schemaless JSON mode. Enforcement is grammar-level at the sampling layer, + // so it is independent of `think`. + request.responseFormat.foreach { + case JsonSchemaFormat(schema, _) => baseRequest("format") = ResponseFormat.strictSchema(schema) + case enumFormat: EnumFormat => baseRequest("format") = enumFormat.schema + case JsonObjectFormat => baseRequest("format") = "json" + } + // Add options if parameters are specified val options = ujson.Obj() var hasOptions = false diff --git a/src/main/providers/OpenAICompatibleProvider.scala b/src/main/providers/OpenAICompatibleProvider.scala index dbf18f9..113fadd 100644 --- a/src/main/providers/OpenAICompatibleProvider.scala +++ b/src/main/providers/OpenAICompatibleProvider.scala @@ -2,7 +2,7 @@ // ABOUTME: Shared by OpenAI, OpenRouter, and Together AI — subclasses override hooks for headers, reasoning, and thinking package org.nlogo.extensions.llm.providers -import org.nlogo.extensions.llm.models.{ChatMessage, ChatRequest, ChatResponse, Choice} +import org.nlogo.extensions.llm.models.{ChatMessage, ChatRequest, ChatResponse, Choice, EnumFormat, JsonObjectFormat, JsonSchemaFormat, ResponseFormat} import org.nlogo.extensions.llm.config.ConfigStore import sttp.client4._ import sttp.model.Uri @@ -87,9 +87,42 @@ abstract class OpenAICompatibleProvider(implicit ec: ExecutionContext) extends B } } + applyResponseFormat(baseRequest, request) + baseRequest } + /** + * Add the OpenAI `response_format` field. + * + * A schema is normalized to strict mode first: OpenAI rejects a schema that + * omits `additionalProperties: false` or leaves any property out of + * `required`, and applying the same normalization for every provider keeps one + * modeler-written schema working across all of them. + */ + protected def applyResponseFormat(baseObj: ujson.Obj, request: ChatRequest): Unit = + request.responseFormat.foreach { + case JsonSchemaFormat(schema, name) => + baseObj("response_format") = jsonSchemaField(name, ResponseFormat.strictSchema(schema)) + + case enumFormat: EnumFormat => + baseObj("response_format") = + jsonSchemaField(ResponseFormat.DefaultSchemaName, enumFormat.schema) + + case JsonObjectFormat => + baseObj("response_format") = ujson.Obj("type" -> "json_object") + } + + private def jsonSchemaField(name: String, schema: ujson.Value): ujson.Obj = + ujson.Obj( + "type" -> "json_schema", + "json_schema" -> ujson.Obj( + "name" -> name, + "strict" -> true, + "schema" -> schema + ) + ) + override protected def parseProviderResponse(responseBody: String, model: String): ChatResponse = { try { val parsed = ujson.read(responseBody) diff --git a/src/main/utils/JsonToNetLogo.scala b/src/main/utils/JsonToNetLogo.scala new file mode 100644 index 0000000..a925f96 --- /dev/null +++ b/src/main/utils/JsonToNetLogo.scala @@ -0,0 +1,105 @@ +// ABOUTME: Converts parsed JSON into NetLogo values, mapping objects to [key value] pair lists +// ABOUTME: Also provides the key lookup that backs the llm:get primitive +package org.nlogo.extensions.llm.utils + +import org.nlogo.core.LogoList + +/** + * Bridges JSON structures into NetLogo's type system. + * + * NetLogo has no map/dictionary type — only strings, numbers, booleans, lists, + * and agents. A JSON object therefore becomes a list of `[key value]` pairs, + * which is the shape NetLogo modelers already use for association lists and + * which [[lookup]] can search. + */ +object JsonToNetLogo { + + /** + * Parse a JSON string and convert it to a NetLogo value. + * + * @throws IllegalArgumentException if the text is not valid JSON + */ + def parse(jsonText: String): AnyRef = { + val parsed = + try ujson.read(jsonText) + catch { + case e: Exception => + throw new IllegalArgumentException( + s"Response was not valid JSON: ${e.getMessage}. Response text: $jsonText" + ) + } + convert(parsed) + } + + /** + * Parse a JSON string that must be an object, and convert it. + * + * Callers that report a NetLogo list need the `[key value]` pair shape, which + * only a JSON object produces. A model that ignores its schema and replies + * with a bare scalar would otherwise yield a value of the wrong NetLogo type. + * + * @throws IllegalArgumentException if the text is not valid JSON or not an object + */ + def parseObject(jsonText: String): LogoList = { + val parsed = + try ujson.read(jsonText) + catch { + case e: Exception => + throw new IllegalArgumentException( + s"Response was not valid JSON: ${e.getMessage}. Response text: $jsonText" + ) + } + + parsed match { + case obj: ujson.Obj => convert(obj).asInstanceOf[LogoList] + case other => + throw new IllegalArgumentException( + s"expected a JSON object in the reply, but got ${typeName(other)}. Response text: $jsonText" + ) + } + } + + private def typeName(value: ujson.Value): String = value match { + case _: ujson.Obj => "an object" + case _: ujson.Arr => "an array" + case _: ujson.Str => "a string" + case _: ujson.Num => "a number" + case _: ujson.Bool => "a boolean" + case ujson.Null => "null" + } + + /** + * Recursively convert a JSON value to its NetLogo equivalent. + * + * Numbers box to Double because NetLogo has a single numeric type; null maps + * to the empty string because NetLogo has no null and an empty string is a + * value modelers can compare against without a runtime error. + */ + def convert(value: ujson.Value): AnyRef = value match { + case obj: ujson.Obj => + LogoList.fromIterator( + obj.value.iterator.map { case (key, v) => LogoList(key, convert(v)) } + ) + case arr: ujson.Arr => + LogoList.fromIterator(arr.value.iterator.map(convert)) + case ujson.Str(s) => s + case ujson.Num(n) => Double.box(n) + case ujson.Bool(b) => Boolean.box(b) + case ujson.Null => "" + } + + /** + * Find the value for `key` in a list of `[key value]` pairs. + * + * Matching is exact and case-sensitive, because JSON keys are. Entries that + * are not two-element lists are skipped rather than matched positionally, so + * asking a plain list such as `["a" "b"]` for key "a" misses instead of + * returning a neighbouring element. + * + * @return the value, or None if no pair has that key + */ + def lookup(list: LogoList, key: String): Option[AnyRef] = + list.toVector.collectFirst { + case pair: LogoList if pair.size == 2 && pair(0) == key => pair(1) + } +} diff --git a/src/test/DeterministicTestProvider.scala b/src/test/DeterministicTestProvider.scala index 263fcde..cca0ecb 100644 --- a/src/test/DeterministicTestProvider.scala +++ b/src/test/DeterministicTestProvider.scala @@ -1,7 +1,7 @@ package org.nlogo.extensions.llm.providers import org.nlogo.extensions.llm.config.ConfigStore -import org.nlogo.extensions.llm.models.{ChatMessage, ChatRequest, ChatResponse, Choice} +import org.nlogo.extensions.llm.models.{ChatMessage, ChatRequest, ChatResponse, Choice, EnumFormat, JsonObjectFormat, JsonSchemaFormat, ResponseFormat} import scala.concurrent.{ExecutionContext, Future} import scala.util.{Success, Try} @@ -18,6 +18,8 @@ class DeterministicTestProvider(implicit ec: ExecutionContext) extends LLMProvid private val testRespondRegex = """__TEST_RESPOND:(.+)""".r private val testThinkingRegex = """__TEST_THINKING:(.+)""".r private val testDelayRegex = """__TEST_DELAY:(\d+):(.*)""".r.unanchored + // Captures to end of line so a raw choice may contain spaces, braces, or quotes. + private val testRawChoiceRegex = """__TEST_RAW_CHOICE:(.*)""".r override def chat(request: ChatRequest): Future[ChatResponse] = { chat(request.messages).map { message => @@ -58,6 +60,80 @@ class DeterministicTestProvider(implicit ec: ExecutionContext) extends LLMProvid } } + /** + * Stands in for a provider that honours the requested format. + * + * An explicit `__TEST_RESPOND:` still wins, so a test can supply the exact + * reply text — including a deliberately malformed one. Otherwise an enum + * request answers in the constrained `{"choice": ...}` shape, which is what + * a real provider enforcing the constraint returns. + */ + override def chatWithFormat(messages: Seq[ChatMessage], format: ResponseFormat): Future[ChatResponse] = { + val lastUserMessage = messages.reverseIterator + .find(_.role == "user") + .map(_.content) + .getOrElse("") + + if (lastUserMessage.contains("__TEST_RESPOND:") || lastUserMessage.contains("__TEST_FAIL")) { + return chatWithFullResponse(messages) + } + + // __TEST_ECHO_FORMAT* markers report which format actually arrived, so a + // primitive that stopped constraining its request fails a test instead of + // silently degrading. The enum echo answers with the LAST choice because + // the unconstrained prompt-only path answers with the first. + if (lastUserMessage.contains("__TEST_ECHO_FORMAT_ENUM")) { + return Future.successful(structured(format match { + case EnumFormat(choices) => choices.last + case _ => "__NO_ENUM_FORMAT_RECEIVED__" + })) + } + + if (lastUserMessage.contains("__TEST_ECHO_FORMAT")) { + return Future.successful(structured(format match { + case JsonObjectFormat => ujson.Obj("format" -> "json_object").toString + case JsonSchemaFormat(schema, name) => + // Echo the schema exactly as handed over. Normalizing here would make + // the assertion test this stub rather than the primitive, and it would + // stay green even if every real provider stopped normalizing. + ujson.Obj( + "format" -> "json_schema", + "name" -> name, + "topLevelType" -> schema.value.get("type").getOrElse(ujson.Null) + ).toString + case EnumFormat(_) => ujson.Obj("format" -> "enum").toString + })) + } + + // __TEST_IGNORE_FORMAT simulates a provider that cannot enforce the + // requested format and answers in free text anyway — the graceful + // degradation path every caller has to survive. __TEST_RAW_CHOICE: + // pins that free-text reply so a specific shape can be asserted. + if (lastUserMessage.contains("__TEST_IGNORE_FORMAT")) { + val raw = testRawChoiceRegex.findFirstMatchIn(lastUserMessage) + .map(_.group(1).trim) + .getOrElse(respond(lastUserMessage).content) + return Future.successful(structured(raw)) + } + + format match { + case EnumFormat(choices) => + val body = ujson.Obj(EnumFormat.ChoiceKey -> choices.head).toString + Future.successful(structured(body)) + case JsonSchemaFormat(_, _) => + Future.successful(structured("{}")) + case JsonObjectFormat => + Future.successful(structured("{}")) + } + } + + private def structured(content: String): ChatResponse = + ChatResponse.simple( + id = "deterministic-test-response", + model = "deterministic-model", + message = ChatMessage.assistant(content) + ) + override def chat(messages: Seq[ChatMessage]): Future[ChatMessage] = { val lastUserMessage = messages.reverseIterator .find(_.role == "user") diff --git a/src/test/JsonToNetLogoSpec.scala b/src/test/JsonToNetLogoSpec.scala new file mode 100644 index 0000000..755b102 --- /dev/null +++ b/src/test/JsonToNetLogoSpec.scala @@ -0,0 +1,142 @@ +// ABOUTME: Deterministic tests for recursive JSON to NetLogo value conversion and key lookup +// ABOUTME: Covers every JSON type, nesting, and the lookup semantics behind llm:get +package org.nlogo.extensions.llm.utils + +import org.nlogo.core.LogoList +import org.scalatest.funsuite.AnyFunSuite + +class JsonToNetLogoSpec extends AnyFunSuite { + + private def convert(text: String): AnyRef = JsonToNetLogo.convert(ujson.read(text)) + + // --- Scalars --- + + test("a string converts to a NetLogo string") { + assert(convert(""""hello"""") == "hello") + } + + test("a number converts to a boxed Double") { + assert(convert("42") == Double.box(42.0)) + assert(convert("3.14") == Double.box(3.14)) + } + + test("a boolean converts to a boxed Boolean") { + assert(convert("true") == Boolean.box(true)) + assert(convert("false") == Boolean.box(false)) + } + + test("null converts to an empty string") { + // NetLogo has no null. Empty string is the value modelers can test with + // `= \"\"` without a runtime error, which nobody is available for otherwise. + assert(convert("null") == "") + } + + // --- Containers --- + + test("an array converts to a flat NetLogo list") { + val result = convert("""["a","b","c"]""").asInstanceOf[LogoList] + assert(result.toVector == Vector("a", "b", "c")) + } + + test("an object converts to a list of [key value] pairs preserving order") { + val result = convert("""{"name":"Alice","age":30}""").asInstanceOf[LogoList] + assert(result.size == 2) + val first = result(0).asInstanceOf[LogoList] + assert(first.toVector == Vector("name", "Alice")) + val second = result(1).asInstanceOf[LogoList] + assert(second.toVector == Vector("age", Double.box(30.0))) + } + + test("an empty object converts to an empty list") { + assert(convert("{}").asInstanceOf[LogoList].size == 0) + } + + test("nested objects convert recursively") { + val result = convert("""{"addr":{"city":"Chicago"}}""").asInstanceOf[LogoList] + val addrPair = result(0).asInstanceOf[LogoList] + assert(addrPair(0) == "addr") + val inner = addrPair(1).asInstanceOf[LogoList] + assert(inner(0).asInstanceOf[LogoList].toVector == Vector("city", "Chicago")) + } + + test("arrays of objects convert recursively") { + val result = convert("""{"steps":[{"n":1},{"n":2}]}""").asInstanceOf[LogoList] + val steps = result(0).asInstanceOf[LogoList](1).asInstanceOf[LogoList] + assert(steps.size == 2) + val firstStep = steps(0).asInstanceOf[LogoList](0).asInstanceOf[LogoList] + assert(firstStep.toVector == Vector("n", Double.box(1.0))) + } + + // --- parse entry point --- + + test("parse accepts a JSON string and converts it") { + val result = JsonToNetLogo.parse("""{"a":1}""").asInstanceOf[LogoList] + assert(result(0).asInstanceOf[LogoList].toVector == Vector("a", Double.box(1.0))) + } + + test("parse rejects malformed JSON with a message quoting the text") { + val ex = intercept[IllegalArgumentException] { JsonToNetLogo.parse("""{"a":""") } + assert(ex.getMessage.toLowerCase.contains("json")) + } + + // --- parseObject: the object-only entry point --- + + test("parseObject accepts an object and reports pairs") { + val result = JsonToNetLogo.parseObject("""{"a":1}""") + assert(result(0).asInstanceOf[LogoList].toVector == Vector("a", Double.box(1.0))) + } + + test("parseObject rejects a top-level scalar naming the type it got") { + val ex = intercept[IllegalArgumentException] { JsonToNetLogo.parseObject(""""hi"""") } + assert(ex.getMessage.contains("a string")) + assert(ex.getMessage.contains("expected a JSON object")) + } + + test("parseObject rejects a top-level array") { + // A flat list would not be readable by llm:get, so this must not slip through. + val ex = intercept[IllegalArgumentException] { JsonToNetLogo.parseObject("""["a"]""") } + assert(ex.getMessage.contains("an array")) + } + + test("parseObject rejects malformed JSON") { + val ex = intercept[IllegalArgumentException] { JsonToNetLogo.parseObject("""{"a":""") } + assert(ex.getMessage.contains("not valid JSON")) + } + + // --- Key lookup (llm:get) --- + + test("lookup finds a top-level key") { + val parsed = convert("""{"name":"Alice","age":30}""").asInstanceOf[LogoList] + assert(JsonToNetLogo.lookup(parsed, "name").contains("Alice")) + assert(JsonToNetLogo.lookup(parsed, "age").contains(Double.box(30.0))) + } + + test("lookup returns None for a missing key rather than guessing") { + val parsed = convert("""{"name":"Alice"}""").asInstanceOf[LogoList] + assert(JsonToNetLogo.lookup(parsed, "nope").isEmpty) + } + + test("lookup returns a nested structure so chained gets work") { + val parsed = convert("""{"addr":{"city":"Chicago"}}""").asInstanceOf[LogoList] + val addr = JsonToNetLogo.lookup(parsed, "addr").get.asInstanceOf[LogoList] + assert(JsonToNetLogo.lookup(addr, "city").contains("Chicago")) + } + + test("lookup is exact-match and case-sensitive, matching JSON key semantics") { + val parsed = convert("""{"Name":"Alice"}""").asInstanceOf[LogoList] + assert(JsonToNetLogo.lookup(parsed, "name").isEmpty) + assert(JsonToNetLogo.lookup(parsed, "Name").contains("Alice")) + } + + test("lookup ignores entries that are not [key value] pairs") { + // A plain list such as ["a" "b" "c"] is not a key-value structure; asking + // for a key must miss rather than match a bare element. + val plain = convert("""["a","b"]""").asInstanceOf[LogoList] + assert(JsonToNetLogo.lookup(plain, "a").isEmpty) + } + + test("lookup returns the first match when a key repeats") { + val list = LogoList(LogoList("k", "first"), LogoList("k", "second")) + assert(JsonToNetLogo.lookup(list, "k").contains("first")) + } +} diff --git a/src/test/ResponseFormatSpec.scala b/src/test/ResponseFormatSpec.scala new file mode 100644 index 0000000..0939fbc --- /dev/null +++ b/src/test/ResponseFormatSpec.scala @@ -0,0 +1,164 @@ +// ABOUTME: Deterministic tests for ResponseFormat parsing, validation, and schema normalization +// ABOUTME: Asserts modeler-supplied schema strings are rejected with clear messages or normalized consistently +package org.nlogo.extensions.llm.models + +import org.scalatest.funsuite.AnyFunSuite + +class ResponseFormatSpec extends AnyFunSuite { + + private def keys(v: ujson.Value): Set[String] = v.obj.keys.toSet + + // --- Parsing modeler-supplied schema strings --- + + test("a valid object schema parses into JsonSchemaFormat") { + val fmt = ResponseFormat.parseSchema( + """{"type":"object","properties":{"action":{"type":"string"}},"required":["action"]}""" + ) + assert(fmt.schema("type").str == "object") + } + + test("malformed JSON is rejected naming the schema, not a raw parser dump") { + val ex = intercept[IllegalArgumentException] { + ResponseFormat.parseSchema("""{"type":"object",""") + } + assert(ex.getMessage.toLowerCase.contains("schema")) + assert(ex.getMessage.toLowerCase.contains("valid json")) + } + + test("a JSON value that is not an object is rejected") { + val ex = intercept[IllegalArgumentException] { + ResponseFormat.parseSchema("""["a","b"]""") + } + assert(ex.getMessage.contains("JSON object")) + } + + test("an empty schema string is rejected") { + val ex = intercept[IllegalArgumentException] { + ResponseFormat.parseSchema(" ") + } + assert(ex.getMessage.toLowerCase.contains("empty")) + } + + test("a schema missing a type is rejected with a message naming 'type'") { + val ex = intercept[IllegalArgumentException] { + ResponseFormat.parseSchema("""{"properties":{"a":{"type":"string"}}}""") + } + assert(ex.getMessage.contains("type")) + } + + test("an object schema with no properties is rejected") { + val ex = intercept[IllegalArgumentException] { + ResponseFormat.parseSchema("""{"type":"object"}""") + } + assert(ex.getMessage.contains("properties")) + } + + test("an enum that is not an array is rejected locally rather than at the provider") { + // Every provider requires enum to be a list of values. Catching it here + // turns a provider-specific 400 into a message naming the offending field. + val ex = intercept[IllegalArgumentException] { + ResponseFormat.parseSchema("""{"type":"object","properties":{"d":{"type":"string","enum":"north"}}}""") + } + assert(ex.getMessage.contains("enum")) + assert(ex.getMessage.contains("d")) + } + + test("an empty enum array is rejected") { + val ex = intercept[IllegalArgumentException] { + ResponseFormat.parseSchema("""{"type":"object","properties":{"d":{"type":"string","enum":[]}}}""") + } + assert(ex.getMessage.contains("enum")) + } + + test("a valid enum is accepted and preserved through normalization") { + val fmt = ResponseFormat.parseSchema( + """{"type":"object","properties":{"d":{"type":"string","enum":["north","south"]}}}""" + ) + val strict = ResponseFormat.strictSchema(fmt.schema) + assert(strict("properties")("d")("enum").arr.map(_.str) == Seq("north", "south")) + } + + // --- Strict normalization (OpenAI/Anthropic requirements) --- + + test("normalization injects additionalProperties false on an object schema") { + val schema = ResponseFormat.parseSchema( + """{"type":"object","properties":{"a":{"type":"string"}},"required":["a"]}""" + ).schema + val strict = ResponseFormat.strictSchema(schema) + assert(strict("additionalProperties").bool == false) + } + + test("normalization lists every property in required, not just the declared ones") { + val schema = ResponseFormat.parseSchema( + """{"type":"object","properties":{"a":{"type":"string"},"b":{"type":"number"}},"required":["a"]}""" + ).schema + val strict = ResponseFormat.strictSchema(schema) + assert(strict("required").arr.map(_.str).toSet == Set("a", "b")) + } + + test("normalization recurses into nested object properties") { + val schema = ResponseFormat.parseSchema( + """{"type":"object","properties":{"addr":{"type":"object","properties":{"city":{"type":"string"}}}}}""" + ).schema + val strict = ResponseFormat.strictSchema(schema) + val nested = strict("properties")("addr") + assert(nested("additionalProperties").bool == false) + assert(nested("required").arr.map(_.str) == Seq("city")) + } + + test("normalization recurses into array item schemas") { + val schema = ResponseFormat.parseSchema( + """{"type":"object","properties":{"steps":{"type":"array","items":{"type":"object","properties":{"n":{"type":"number"}}}}}}""" + ).schema + val strict = ResponseFormat.strictSchema(schema) + val items = strict("properties")("steps")("items") + assert(items("additionalProperties").bool == false) + assert(items("required").arr.map(_.str) == Seq("n")) + } + + test("normalization does not mutate the caller's schema") { + val fmt = ResponseFormat.parseSchema( + """{"type":"object","properties":{"a":{"type":"string"}}}""" + ) + ResponseFormat.strictSchema(fmt.schema) + assert(!keys(fmt.schema).contains("additionalProperties"), "strictSchema must return a copy") + } + + // --- Enum format --- + + test("EnumFormat renders as a single-property object schema constraining the choice") { + val schema = EnumFormat(List("north", "south")).schema + assert(schema("type").str == "object") + val choiceProp = schema("properties")(EnumFormat.ChoiceKey) + assert(choiceProp("enum").arr.map(_.str) == Seq("north", "south")) + } + + test("EnumFormat rejects an empty choice list") { + intercept[IllegalArgumentException] { EnumFormat(Nil) } + } + + // --- Serialization --- + + test("a ChatRequest carrying a response format round-trips through upickle") { + // ChatRequest exposes a derived ReadWriter. Adding a sealed-trait field + // breaks that derivation unless ResponseFormat provides its own instance. + val request = ChatRequest( + model = "m", + messages = Seq(ChatMessage.user("hi")), + responseFormat = Some(EnumFormat(List("a", "b"))) + ) + val restored = upickle.default.read[ChatRequest](upickle.default.write(request)) + assert(restored.responseFormat.contains(EnumFormat(List("a", "b")))) + } + + test("a JsonSchemaFormat round-trips with its schema intact") { + val fmt = ResponseFormat.parseSchema("""{"type":"object","properties":{"a":{"type":"string"}}}""") + val restored = upickle.default.read[ResponseFormat](upickle.default.write[ResponseFormat](fmt)) + assert(restored.asInstanceOf[JsonSchemaFormat].schema("properties")("a")("type").str == "string") + } + + test("JsonObjectFormat round-trips") { + val restored = upickle.default.read[ResponseFormat](upickle.default.write[ResponseFormat](JsonObjectFormat)) + assert(restored == JsonObjectFormat) + } +} diff --git a/src/test/StructuredChatPlumbingSpec.scala b/src/test/StructuredChatPlumbingSpec.scala new file mode 100644 index 0000000..d47f1b8 --- /dev/null +++ b/src/test/StructuredChatPlumbingSpec.scala @@ -0,0 +1,64 @@ +// ABOUTME: Tests that a requested response format reaches the provider's request body unchanged +// ABOUTME: Covers the LLMProvider entry point used by llm:chat-with-schema and llm:chat-json +package org.nlogo.extensions.llm.providers + +import org.nlogo.extensions.llm.config.ConfigStore +import org.nlogo.extensions.llm.models._ +import org.scalatest.funsuite.AnyFunSuite +import scala.concurrent.ExecutionContext.Implicits.global + +/** + * Captures the ChatRequest that would have been sent, without any network I/O. + */ +private class CapturingProvider extends OllamaProvider { + var captured: Option[ChatRequest] = None + override def chat(request: ChatRequest): scala.concurrent.Future[ChatResponse] = { + captured = Some(request) + scala.concurrent.Future.successful( + ChatResponse.simple("id", request.model, ChatMessage.assistant("{}")) + ) + } +} + +class StructuredChatPlumbingSpec extends AnyFunSuite { + + ProviderRegistry.reset() + ProviderRegistrations.registerAll() + + private val messages = Seq(ChatMessage.user("hi")) + private val schema = ResponseFormat.parseSchema( + """{"type":"object","properties":{"a":{"type":"string"}}}""" + ) + + test("chatWithFormat forwards the requested format to the provider request") { + val provider = new CapturingProvider + provider.chatWithFormat(messages, schema) + assert(provider.captured.flatMap(_.responseFormat).contains(schema)) + } + + test("chatWithFormat still applies configured model and token settings") { + val provider = new CapturingProvider + provider.setConfig(ConfigStore.MODEL, "llama3.1") + provider.setConfig(ConfigStore.MAX_TOKENS, "1234") + provider.chatWithFormat(messages, JsonObjectFormat) + + val request = provider.captured.get + assert(request.model == "llama3.1") + assert(request.maxTokens.contains(1234)) + } + + test("the ordinary chat path still sends no response format") { + val provider = new CapturingProvider + provider.chat(messages) + assert(provider.captured.get.responseFormat.isEmpty) + } + + test("chatWithFormat returns the full response so callers can read content") { + val provider = new CapturingProvider + val response = scala.concurrent.Await.result( + provider.chatWithFormat(messages, JsonObjectFormat), + scala.concurrent.duration.Duration(5, "seconds") + ) + assert(response.firstContent.contains("{}")) + } +} diff --git a/src/test/StructuredOutputRequestSpec.scala b/src/test/StructuredOutputRequestSpec.scala new file mode 100644 index 0000000..d10ccf0 --- /dev/null +++ b/src/test/StructuredOutputRequestSpec.scala @@ -0,0 +1,233 @@ +// ABOUTME: Deterministic tests asserting each provider wraps a response format in its own request shape +// ABOUTME: No network I/O — only the JSON body each provider would have posted +package org.nlogo.extensions.llm.providers + +import org.nlogo.extensions.llm.models._ +import org.scalatest.funsuite.AnyFunSuite +import scala.concurrent.ExecutionContext.Implicits.global + +/** Exposes each provider's protected request builder for body assertions. */ +private class InspectableOpenAI extends OpenAIProvider { + def body(r: ChatRequest): ujson.Value = createProviderRequest(r) +} +private class InspectableGroq extends GroqProvider { + def body(r: ChatRequest): ujson.Value = createProviderRequest(r) +} +private class InspectableTogether extends TogetherProvider { + def body(r: ChatRequest): ujson.Value = createProviderRequest(r) +} +private class InspectableOpenRouter extends OpenRouterProvider { + def body(r: ChatRequest): ujson.Value = createProviderRequest(r) +} +private class InspectableClaude extends ClaudeProvider { + def body(r: ChatRequest): ujson.Value = createProviderRequest(r) +} +private class InspectableGemini extends GeminiProvider { + def body(r: ChatRequest): ujson.Value = createProviderRequest(r) +} +private class InspectableOllama extends OllamaProvider { + def body(r: ChatRequest): ujson.Value = createProviderRequest(r) +} + +class StructuredOutputRequestSpec extends AnyFunSuite { + + // Constructing a provider reads its default model from the registry, which is + // normally populated by LLMExtension.load(). Register here so the suite is + // self-contained and order-independent. + ProviderRegistry.reset() + ProviderRegistrations.registerAll() + + private val schemaText = + """{"type":"object","properties":{"action":{"type":"string"},"confidence":{"type":"number"}},"required":["action"]}""" + + private val schemaFormat = ResponseFormat.parseSchema(schemaText) + + private def request( + model: String, + format: Option[ResponseFormat], + thinking: Option[ThinkingConfig] = None, + temperature: Option[Double] = None, + maxTokens: Option[Int] = Some(4000) + ): ChatRequest = + ChatRequest( + model = model, + messages = Seq(ChatMessage.user("hi")), + maxTokens = maxTokens, + temperature = temperature, + thinkingConfig = thinking, + responseFormat = format + ) + + private def keys(v: ujson.Value): Set[String] = v.obj.keys.toSet + + // --- OpenAI-compatible family: openai, groq, together, openrouter --- + + private val openAiCompatible: Seq[(String, ChatRequest => ujson.Value)] = Seq( + "openai" -> (new InspectableOpenAI).body, + "groq" -> (new InspectableGroq).body, + "together" -> (new InspectableTogether).body, + "openrouter" -> (new InspectableOpenRouter).body + ) + + test("every OpenAI-compatible provider nests the schema under response_format.json_schema") { + openAiCompatible.foreach { case (name, build) => + val body = build(request("some-model", Some(schemaFormat))) + val rf = body("response_format") + assert(rf("type").str == "json_schema", s"$name response_format.type") + assert(rf("json_schema")("strict").bool, s"$name must request strict mode") + assert(rf("json_schema")("name").str.nonEmpty, s"$name must send a schema name") + assert( + rf("json_schema")("schema")("properties")("action")("type").str == "string", + s"$name must carry the caller's schema" + ) + } + } + + test("OpenAI-compatible providers normalize the schema to strict mode") { + openAiCompatible.foreach { case (name, build) => + val body = build(request("some-model", Some(schemaFormat))) + val schema = body("response_format")("json_schema")("schema") + assert(schema("additionalProperties").bool == false, s"$name additionalProperties") + assert( + schema("required").arr.map(_.str).toSet == Set("action", "confidence"), + s"$name must list every property as required" + ) + } + } + + test("OpenAI-compatible providers send response_format json_object for JSON mode") { + openAiCompatible.foreach { case (name, build) => + val body = build(request("some-model", Some(JsonObjectFormat))) + assert(body("response_format")("type").str == "json_object", s"$name json mode") + assert(!keys(body("response_format")).contains("json_schema"), s"$name json mode carries no schema") + } + } + + test("OpenAI-compatible providers send an enum as a constrained json_schema") { + openAiCompatible.foreach { case (name, build) => + val body = build(request("some-model", Some(EnumFormat(List("north", "south"))))) + val schema = body("response_format")("json_schema")("schema") + assert( + schema("properties")(EnumFormat.ChoiceKey)("enum").arr.map(_.str) == Seq("north", "south"), + s"$name enum values" + ) + } + } + + test("no response format leaves the OpenAI-compatible body unchanged") { + openAiCompatible.foreach { case (name, build) => + val body = build(request("some-model", None)) + assert(!keys(body).contains("response_format"), s"$name must omit response_format when unset") + } + } + + test("structured output survives alongside OpenAI reasoning fields") { + val body = (new InspectableOpenAI).body( + request("o3-mini", Some(schemaFormat), thinking = Some(ThinkingConfig(enabled = true, reasoningEffort = Some("high")))) + ) + assert(body("reasoning_effort").str == "high") + assert(body("response_format")("type").str == "json_schema") + } + + // --- Anthropic --- + + test("Claude nests the schema under output_config.format") { + val body = (new InspectableClaude).body(request("claude-sonnet-4-5-20250929", Some(schemaFormat))) + val format = body("output_config")("format") + assert(format("type").str == "json_schema") + assert(format("schema")("properties")("action")("type").str == "string") + assert(format("schema")("additionalProperties").bool == false) + } + + test("Claude keeps output_config.effort when a schema is also present") { + // effort and format share one output_config object; writing the schema must + // merge into it, not replace the thinking-depth setting. + val body = (new InspectableClaude).body( + request( + "claude-opus-5", + Some(schemaFormat), + thinking = Some(ThinkingConfig(enabled = true, reasoningEffort = Some("high"))) + ) + ) + assert(body("output_config")("effort").str == "high", "effort must survive a schema") + assert(body("output_config")("format")("type").str == "json_schema", "schema must survive effort") + assert(body("thinking")("type").str == "adaptive") + } + + test("Claude extended-thinking request still carries a schema") { + val body = (new InspectableClaude).body( + request("claude-haiku-4-5-20251001", Some(schemaFormat), thinking = Some(ThinkingConfig(enabled = true))) + ) + assert(body("thinking")("type").str == "enabled") + assert(body("output_config")("format")("type").str == "json_schema") + } + + test("Claude JSON mode with no schema sends no output_config.format") { + // Anthropic has no schemaless JSON mode, so there is nothing truthful to + // send; the prompt-level instruction carries it instead. + val body = (new InspectableClaude).body(request("claude-sonnet-4-5-20250929", Some(JsonObjectFormat))) + val hasFormat = keys(body).contains("output_config") && keys(body("output_config")).contains("format") + assert(!hasFormat, s"Claude must not invent a json_object format: $body") + } + + test("no response format leaves the Claude body without output_config.format") { + val body = (new InspectableClaude).body(request("claude-sonnet-4-5-20250929", None)) + assert(!keys(body).contains("output_config")) + } + + // --- Gemini --- + + test("Gemini sets responseJsonSchema and the JSON mime type") { + val body = (new InspectableGemini).body(request("gemini-2.5-flash", Some(schemaFormat))) + val gc = body("generationConfig") + assert(gc("responseMimeType").str == "application/json") + assert(gc("responseJsonSchema")("properties")("action")("type").str == "string") + } + + test("Gemini JSON mode sets the mime type with no schema") { + val gc = (new InspectableGemini).body(request("gemini-2.5-flash", Some(JsonObjectFormat)))("generationConfig") + assert(gc("responseMimeType").str == "application/json") + assert(!keys(gc).contains("responseJsonSchema")) + } + + test("Gemini keeps thinkingConfig alongside a response schema") { + val body = (new InspectableGemini).body( + request("gemini-2.5-flash", Some(schemaFormat), thinking = Some(ThinkingConfig(enabled = true, budgetTokens = Some(2048)))) + ) + val gc = body("generationConfig") + assert(gc("thinkingConfig")("thinkingBudget").num == 2048) + assert(gc("responseMimeType").str == "application/json") + } + + test("no response format leaves the Gemini body without responseMimeType") { + val body = (new InspectableGemini).body(request("gemini-2.5-flash", None, temperature = Some(0.7))) + assert(!keys(body("generationConfig")).contains("responseMimeType")) + assert(!keys(body("generationConfig")).contains("responseJsonSchema")) + } + + // --- Ollama --- + + test("Ollama puts the raw schema in format") { + val body = (new InspectableOllama).body(request("llama3.1", Some(schemaFormat))) + assert(body("format")("type").str == "object") + assert(body("format")("properties")("action")("type").str == "string") + } + + test("Ollama JSON mode sends the string format json") { + val body = (new InspectableOllama).body(request("llama3.1", Some(JsonObjectFormat))) + assert(body("format").str == "json") + } + + test("Ollama keeps think alongside a schema") { + val body = (new InspectableOllama).body( + request("llama3.1", Some(schemaFormat), thinking = Some(ThinkingConfig(enabled = true))) + ) + assert(body("think").bool) + assert(body("format")("type").str == "object") + } + + test("no response format leaves the Ollama body without format") { + val body = (new InspectableOllama).body(request("llama3.1", None)) + assert(!keys(body).contains("format")) + } +} diff --git a/tests.txt b/tests.txt index b3cedb8..b2ebac2 100644 --- a/tests.txt +++ b/tests.txt @@ -502,3 +502,226 @@ LLMCompileErrorBannedListValidation extensions [llm] (llm:compile-error "die" [1 2]) => ERROR Extension exception: llm:compile-error expects a list of primitive names as strings, but got: 1, 2 (llm:compile-error "fd 1" ["die" 5]) => ERROR Extension exception: llm:compile-error expects a list of primitive names as strings, but got: 5 + +LLMGetLookup + extensions [llm] + llm:get [["name" "Alice"] ["age" 30]] "name" => "Alice" + llm:get [["name" "Alice"] ["age" 30]] "age" => 30 + llm:get [["flag" true]] "flag" => true + llm:get [["skills" ["coding" "design"]]] "skills" => ["coding" "design"] + llm:get (llm:get [["addr" [["city" "Chicago"]]]] "addr") "city" => "Chicago" + llm:get [["name" "Alice"]] "missing" => ERROR Extension exception: llm:get: key "missing" not found. Available keys: name + llm:get [] "any" => ERROR Extension exception: llm:get: key "any" not found. The list is empty. + llm:get ["a" "b"] "a" => ERROR Extension exception: llm:get: key "a" not found. Available keys: (none - the list is not a list of [key value] pairs) + +LLMChatWithSchemaParsesJson + extensions [llm] + globals [result] + O> llm:set-api-key "test-key" + O> llm:set-provider "openai" + O> llm:clear-history + O> set result llm:chat-with-schema "__TEST_RESPOND:{\"action\":\"explore\",\"confidence\":0.8}" "{\"type\":\"object\",\"properties\":{\"action\":{\"type\":\"string\"},\"confidence\":{\"type\":\"number\"}}}" + llm:get result "action" => "explore" + llm:get result "confidence" => 0.8 + is-list? result => true + O> llm:clear-history + +LLMChatWithSchemaNested + extensions [llm] + globals [result] + O> llm:set-api-key "test-key" + O> llm:set-provider "openai" + O> llm:clear-history + O> set result llm:chat-with-schema "__TEST_RESPOND:{\"plan\":[\"a\",\"b\"],\"meta\":{\"n\":2}}" "{\"type\":\"object\",\"properties\":{\"plan\":{\"type\":\"array\",\"items\":{\"type\":\"string\"}},\"meta\":{\"type\":\"object\",\"properties\":{\"n\":{\"type\":\"number\"}}}}}" + llm:get result "plan" => ["a" "b"] + llm:get (llm:get result "meta") "n" => 2 + O> llm:clear-history + +LLMChatWithSchemaRejectsBadSchema + extensions [llm] + O> llm:set-api-key "test-key" + O> llm:set-provider "openai" + llm:chat-with-schema "hello" "not json at all" => ERROR Extension exception: llm:chat-with-schema: Schema is not valid JSON: expected null got "n" at index 0 + llm:chat-with-schema "hello" "" => ERROR Extension exception: llm:chat-with-schema: Schema cannot be empty. Provide a JSON Schema object, e.g. {"type":"object","properties":{"action":{"type":"string"}}} + +LLMChatWithSchemaRejectsNonObjectTopLevel + extensions [llm] + O> llm:set-api-key "test-key" + O> llm:set-provider "openai" + llm:chat-with-schema "hi" "{\"type\":\"string\"}" => ERROR Extension exception: llm:chat-with-schema: schema must have type 'object' at the top level, but got 'string'. llm:chat-with-schema reports a list of [key value] pairs, so the reply has to be a JSON object. Wrap it, e.g. {"type":"object","properties":{"value":{"type":"string"}}} + llm:chat-with-schema "hi" "{\"type\":\"array\",\"items\":{\"type\":\"string\"}}" => ERROR Extension exception: llm:chat-with-schema: schema must have type 'object' at the top level, but got 'array'. llm:chat-with-schema reports a list of [key value] pairs, so the reply has to be a JSON object. Wrap it, e.g. {"type":"object","properties":{"value":{"type":"string"}}} + +LLMChatWithSchemaRejectsNonObjectReply + extensions [llm] + O> llm:set-api-key "test-key" + O> llm:set-provider "openai" + O> llm:clear-history + llm:chat-with-schema "__TEST_RESPOND:\"just a string\"" "{\"type\":\"object\",\"properties\":{\"a\":{\"type\":\"string\"}}}" => ERROR Extension exception: llm:chat-with-schema: expected a JSON object in the reply, but got a string. Response text: "just a string" + llm:history => [] + O> llm:clear-history + +LLMChatWithSchemaHistory + extensions [llm] + globals [result] + O> llm:set-api-key "test-key" + O> llm:set-provider "openai" + O> llm:clear-history + O> set result llm:chat-with-schema "__TEST_RESPOND:{\"a\":1}" "{\"type\":\"object\",\"properties\":{\"a\":{\"type\":\"number\"}}}" + length llm:history => 2 + item 0 (item 0 llm:history) => "user" + item 0 (item 1 llm:history) => "assistant" + O> llm:clear-history + +LLMChatWithSchemaHistoryUnchangedOnFailure + extensions [llm] + O> llm:set-api-key "test-key" + O> llm:set-provider "openai" + O> llm:clear-history + O> carefully [ let r llm:chat-with-schema "__TEST_FAIL" "{\"type\":\"object\",\"properties\":{\"a\":{\"type\":\"number\"}}}" ] [ ] + llm:history => [] + O> llm:clear-history + +LLMChatWithSchemaRejectsNonJsonReply + extensions [llm] + O> llm:set-api-key "test-key" + O> llm:set-provider "openai" + O> llm:clear-history + llm:chat-with-schema "__TEST_RESPOND:sorry I cannot" "{\"type\":\"object\",\"properties\":{\"a\":{\"type\":\"number\"}}}" => ERROR Extension exception: llm:chat-with-schema: Response was not valid JSON: expected json value got "s" at index 0. Response text: sorry I cannot + llm:history => [] + O> llm:clear-history + +LLMChatJsonReturnsRawString + extensions [llm] + globals [result] + O> llm:set-api-key "test-key" + O> llm:set-provider "openai" + O> llm:clear-history + O> set result llm:chat-json "__TEST_RESPOND:{\"a\":1}" + is-string? result => true + result => "{\"a\":1}" + length llm:history => 2 + O> llm:clear-history + +LLMChatJsonRejectsNonJsonReply + extensions [llm] + O> llm:set-api-key "test-key" + O> llm:set-provider "openai" + O> llm:clear-history + llm:chat-json "__TEST_RESPOND:Sure! Here is the JSON you asked for." => ERROR Extension exception: llm:chat-json: Response was not valid JSON: expected json value got "S" at index 0. Response text: Sure! Here is the JSON you asked for. + llm:history => [] + O> llm:clear-history + +LLMChatJsonAcceptsNonObjectJson + extensions [llm] + globals [result] + O> llm:set-api-key "test-key" + O> llm:set-provider "openai" + O> llm:clear-history + O> set result llm:chat-json "__TEST_RESPOND:[1,2,3]" + result => "[1,2,3]" + length llm:history => 2 + O> llm:clear-history + +LLMChatJsonHistoryUnchangedOnFailure + extensions [llm] + O> llm:set-api-key "test-key" + O> llm:set-provider "openai" + O> llm:clear-history + O> carefully [ let r llm:chat-json "__TEST_FAIL" ] [ ] + llm:history => [] + O> llm:clear-history + +LLMChooseStructured + extensions [llm] + globals [choice] + O> llm:set-api-key "test-key" + O> llm:set-provider "openai" + O> llm:clear-history + O> set choice llm:choose "Pick a direction" ["north" "south" "east" "west"] + member? choice ["north" "south" "east" "west"] => true + length llm:history => 2 + O> llm:clear-history + +LLMChooseStructuredEmptyList + extensions [llm] + O> llm:set-api-key "test-key" + O> llm:set-provider "openai" + llm:choose "Pick" [] => ERROR Extension exception: Choice list cannot be empty + +LLMChooseSendsEnumConstraint + extensions [llm] + globals [choice] + O> llm:set-api-key "test-key" + O> llm:set-provider "openai" + O> llm:clear-history + O> set choice llm:choose "__TEST_ECHO_FORMAT_ENUM" ["north" "south"] + choice => "south" + length llm:history => 2 + item 1 (item 1 llm:history) => "south" + O> llm:clear-history + O> set choice llm:choose "__TEST_ECHO_FORMAT_ENUM" ["alpha" "beta" "gamma"] + choice => "gamma" + O> llm:clear-history + +LLMChatJsonSendsJsonObjectFormat + extensions [llm] + globals [result] + O> llm:set-api-key "test-key" + O> llm:set-provider "openai" + O> llm:clear-history + O> set result llm:chat-json "__TEST_ECHO_FORMAT" + result => "{\"format\":\"json_object\"}" + O> llm:clear-history + +LLMChatWithSchemaSendsSchemaFormat + extensions [llm] + globals [result] + O> llm:set-api-key "test-key" + O> llm:set-provider "openai" + O> llm:clear-history + O> set result llm:chat-with-schema "__TEST_ECHO_FORMAT" "{\"type\":\"object\",\"properties\":{\"a\":{\"type\":\"string\"}}}" + llm:get result "format" => "json_schema" + llm:get result "topLevelType" => "object" + llm:get result "name" => "netlogo_schema" + O> llm:clear-history + +LLMChooseDegradesWhenFormatIgnored + extensions [llm] + globals [choice] + O> llm:set-api-key "test-key" + O> llm:set-provider "openai" + O> llm:clear-history + O> set choice llm:choose "__TEST_IGNORE_FORMAT Pick a direction" ["north" "south" "east" "west"] + choice => "north" + length llm:history => 2 + item 1 (item 1 llm:history) => "north" + O> llm:clear-history + +LLMChooseDegradesCaseInsensitively + extensions [llm] + globals [choice] + O> llm:set-api-key "test-key" + O> llm:set-provider "openai" + O> llm:clear-history + O> set choice llm:choose "pick __TEST_IGNORE_FORMAT __TEST_RAW_CHOICE:NORTH" ["north" "south"] + choice => "north" + length llm:history => 2 + O> llm:clear-history + +LLMChooseDegradesOnMalformedConstrainedJson + extensions [llm] + O> llm:set-api-key "test-key" + O> llm:set-provider "openai" + O> llm:clear-history + llm:choose "__TEST_IGNORE_FORMAT __TEST_RAW_CHOICE:{\"choice\": pick" ["north" "south"] => ERROR Extension exception: llm:choose: response '{"choice": pick' did not match any choice. Choices: north, south + llm:history => [] + O> llm:clear-history + +LLMChooseUnmatchedAfterDegradationLeavesHistoryClean + extensions [llm] + O> llm:set-api-key "test-key" + O> llm:set-provider "openai" + O> llm:clear-history + llm:choose "__TEST_IGNORE_FORMAT __TEST_RAW_CHOICE:up" ["north" "south"] => ERROR Extension exception: llm:choose: response 'up' did not match any choice. Choices: north, south + llm:history => [] + O> llm:clear-history From 6ee23a93d576ed744d1fc924d6c98449daaff693 Mon Sep 17 00:00:00 2001 From: JNK234 Date: Thu, 13 Aug 2026 23:00:55 -0500 Subject: [PATCH 2/2] demo: add structured-outputs model exercising schema-constrained replies MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The structured output work shipped docs and tests but no runnable model, so there was nothing to open and press go. Six turtles forage, each asking the model what to do with the reply constrained to a JSON Schema. Two consequences are visible on screen rather than only described: - action is enum-constrained, so act-on compares it directly with no fuzzy matching or fallback branch - confidence arrives as a NUMBER, so recolor can write confidence >= 0.7 — lime above the threshold, orange below A second button shows llm:chat-json for contrast: valid JSON, but still a string the modeller has to handle. config.txt.example ships with placeholders; config.txt is gitignored. Verified live against Groq: 12 schema-constrained calls over 2 ticks, 0 failures. Turtles on food chose eat at 0.9 confidence while others chose explore at 0.8, so replies track state rather than repeating a default. Model compiles headless against NetLogo 7.0.3. Relates to #22. --- demos/structured-outputs/README.md | 97 +++++++ demos/structured-outputs/config.txt.example | 27 ++ .../structured-outputs.nlogox | 270 ++++++++++++++++++ 3 files changed, 394 insertions(+) create mode 100644 demos/structured-outputs/README.md create mode 100644 demos/structured-outputs/config.txt.example create mode 100644 demos/structured-outputs/structured-outputs.nlogox diff --git a/demos/structured-outputs/README.md b/demos/structured-outputs/README.md new file mode 100644 index 0000000..e02e100 --- /dev/null +++ b/demos/structured-outputs/README.md @@ -0,0 +1,97 @@ +# Structured Outputs + +A working demonstration of `llm:chat-with-schema`, `llm:chat-json`, and `llm:get` — +constraining a model's reply to a JSON Schema and reading the fields as real NetLogo +values instead of parsing a sentence. + +## The Problem + +`llm:chat` returns free text. A model that needs a number has to hope the phrasing +stays stable and pick it out of the string: + +```netlogo +let reply llm:chat "How confident are you, 0 to 1?" +;=> "I'd say about 0.8, though it depends on the situation." +; now what? substring? position? what if it says "eighty percent"? +``` + +That works until the model words things differently, and then it fails quietly. + +## The Fix + +```netlogo +let schema "{\"type\":\"object\",\"properties\":{\"action\":{\"type\":\"string\",\"enum\":[\"eat\",\"explore\",\"rest\"]},\"confidence\":{\"type\":\"number\"}},\"required\":[\"action\",\"confidence\"]}" + +let reply llm:chat-with-schema "A turtle sees food. What now?" schema +;=> [[action eat] [confidence 0.9]] + +let act llm:get reply "action" ;=> "eat" constrained to the enum +let conf llm:get reply "confidence" ;=> 0.9 a NUMBER, not text +``` + +## What the demo shows + +Six turtles forage. Each tick every turtle sends its energy and whether it is +standing on food, and gets back a reply matching the schema above. + +Two consequences are visible on screen: + +- **`action` is enum-constrained**, so `act-on` compares it directly. No fuzzy + matching, no fallback branch for unexpected wording. +- **`confidence` is a number**, so `recolor` writes `confidence >= 0.7`. Turtles + above the threshold turn lime, the rest orange. That comparison is only possible + because the value arrives typed. + +Press **show decisions** for each turtle's parsed fields, or **raw JSON (no schema)** +to see what `llm:chat-json` gives instead — valid JSON, but still a string. + +## How to Run + +1. `cp config.txt.example config.txt` and add your key. `config.txt` is gitignored. +2. Open `structured-outputs.nlogox` in NetLogo 7.0.3. +3. Press **setup**, then **go**. + +Structured output needs a provider that supports constrained decoding. Verified +against Groq; OpenAI, Anthropic, and Gemini also support it. Ollama depends on the +local model. + +## How JSON maps into NetLogo + +NetLogo has no dictionary type, so objects become `[key value]` pair lists: + +| JSON | NetLogo | +|---|---| +| `{"a": 1}` | `[[a 1]]` | +| `[1, 2]` | `[1 2]` | +| `"text"` | string | +| `10` | number | +| `true` | boolean | +| `null` | `""` (NetLogo has no null) | + +Nested objects are just more pair lists, so `llm:get` chains: + +```netlogo +let stats llm:get reply "stats" ;=> [[alive true] [speed 10]] +let speed llm:get stats "speed" ;=> 10 +``` + +## Gotchas + +- **The schema is a JSON string, not a NetLogo list.** The escaped quotes are + unavoidable. Passing a list raises an error naming the problem. +- **A missing key raises**, listing the keys that were available. Wrap `llm:get` in + `carefully` when a field is genuinely optional. +- **Key matching is exact and case-sensitive**, because JSON keys are. +- **The schema constrains shape, not truth.** A well-formed reply can still be a bad + decision — this removes parsing failures, not model error. + +## Verification + +Run headless against live Groq: 12 schema-constrained calls across 2 ticks, 0 +failures. Turtles standing on food chose `eat` at 0.9 confidence while others chose +`explore` at 0.8 — the replies track state rather than repeating a default. + +## Related + +- API reference: `docs/API-REFERENCE.md` → Structured Output +- Issue [#22](https://github.com/NetLogo/Netlogo-LLM-Extension/issues/22) diff --git a/demos/structured-outputs/config.txt.example b/demos/structured-outputs/config.txt.example new file mode 100644 index 0000000..a9675e7 --- /dev/null +++ b/demos/structured-outputs/config.txt.example @@ -0,0 +1,27 @@ +# Structured outputs demo — provider settings +# +# Copy this file to config.txt and fill in a key. config.txt is gitignored, +# so your key stays out of version control. +# +# Structured output needs a provider that supports constrained decoding. +# Verified working: Groq, OpenAI, Anthropic, Gemini. Ollama support depends +# on the local model. + +# --- Groq (free tier, fast; key from https://console.groq.com/keys) --- +provider=groq +groq_api_key=gsk_REPLACE_WITH_YOUR_KEY +model=openai/gpt-oss-20b + +# --- OpenAI --- +#provider=openai +#openai_api_key=sk-REPLACE_WITH_YOUR_KEY +#model=gpt-4o-mini + +# --- Local Ollama, no key needed --- +#provider=ollama +#model=llama3.2:3b +#base_url=http://localhost:11434 + +temperature=0.0 +max_tokens=300 +timeout_seconds=60 diff --git a/demos/structured-outputs/structured-outputs.nlogox b/demos/structured-outputs/structured-outputs.nlogox new file mode 100644 index 0000000..bb102bb --- /dev/null +++ b/demos/structured-outputs/structured-outputs.nlogox @@ -0,0 +1,270 @@ + + + = 0.7) [lime] [orange] +end + +;; Prints each turtle's most recent typed decision. +to show-decisions + clear-output + output-print "action confidence colour" + ask turtles [ + output-print (word + last-action + " " precision confidence 2 + " " (ifelse-value (confidence >= 0.7) ["lime"] ["orange"])) + ] + output-print "" + output-print (word "last parsed reply: " last-reply) +end + +;; The no-schema path: valid JSON, returned as raw text. +to show-raw-json + clear-output + carefully + [ let raw llm:chat-json + "Name two colours a turtle could be, as a JSON object with key colours." + output-print "llm:chat-json returns a STRING:" + output-print raw + output-print (word "is-string? " is-string? raw) ] + [ output-print (word "failed: " error-message) ] +end +]]> + + + + + + + + parsed-count + failed-count + + Lime turtles reported confidence of 0.7 or more, orange ones less. That comparison works because confidence arrives as a number, not as text to be parsed. + + [[action eat] [confidence 0.9] [reason found food here]] + + let act llm:get reply "action" ;=> "eat" a string + let conf llm:get reply "confidence" ;=> 0.9 a NUMBER + +Two things follow, and both are visible on screen: + +- `action` is constrained by the schema's `enum`, so `act-on` compares it directly + with no fuzzy matching or fallback branch. +- `confidence` arrives as a number, so `recolor` can write `confidence >= 0.7`. + Turtles above the threshold are lime, the rest orange. + +## HOW TO USE IT + +1. Put your provider settings in `config.txt` beside this model. +2. Press **setup**, then **go**. +3. Press **show decisions** to see each turtle's typed fields. +4. Press **raw JSON (no schema)** to see what `llm:chat-json` returns instead — + valid JSON, but as a string you would still have to handle yourself. + +## THINGS TO NOTICE + +The schema is a JSON **string**, not a NetLogo list. The escaped quotes in `setup` +are unavoidable, and passing a list instead raises an error. + +Nesting has no dotted-path syntax. A nested object is another `[[key value] ...]` +list, so you call `llm:get` again on the result. + +A missing key raises rather than reporting a default, and names the keys that were +available. Wrap `llm:get` in `carefully` when a field is genuinely optional. + +`null` in JSON becomes an empty string, because NetLogo has no null. + +## EXTENDING THE MODEL + +Add a field to `decision-schema` — say a `target` object with `xcor` and `ycor` — +and read it with a second `llm:get`. Nothing else has to change. + +The **failed** monitor counts replies that did not parse. Schema constraint is +enforced by the provider, so this counts network and model failures rather than +malformed JSON; a run that only ever increments **parsed** is the expected case. + +## RELATED PRIMITIVES + +`llm:chat-with-schema` — schema-constrained reply, parsed into nested lists +`llm:chat-json` — valid JSON with no schema, returned as a string +`llm:get` — read a key from a `[[key value] ...]` list +]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +