diff --git a/CHANGELOG.md b/CHANGELOG.md index b6930b4e76..6a19a305bd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ Full release notes with details on each version: [GitHub Releases](https://githu ## 0.9.48 (2026-08-20) +- Feature: Microsoft Dynamics 365 Business Central AL (`.al`) extraction now models application objects, permission sets and extensions, ControlAddIns and hosted UserControls, procedures and overloads, scoped field/action/UserControl triggers, fields, enum values, extensions, interfaces, typed calls, ControlAddIn event bindings, event subscriptions, enum implementations, application dependencies, TestPage targets, and test handler bindings with case-insensitive cross-file resolution. Install the optional `[al]` extra for full Tree-sitter extraction on Python 3.12+; other supported Python versions retain structural object/procedure/trigger extraction through the built-in fallback. - Fix: a control character in a node label or id no longer aborts the whole export; the GraphML and Obsidian exporters scrub only the characters those formats forbid (tab, newline, and non-ASCII letters are preserved), and `graph.json` and its byte-identity round-trip are untouched (#2897, thanks @abhay-codes07). - Fix: `graphify update` / `label` / `cluster-only` no longer leave a large graph without a `graph.html`; the aggregated community view renders instead of raising, a failed render preserves the previous file, and a missing `graph.html` is regenerated on the no-change fast path without reclustering (#2853, thanks @oleksii-tumanov). - Feature: `graphify extract --no-dedup` skips the fuzzy near-duplicate merge on build and incremental merge, for operators who would rather keep distinct symbols that fuzzy-matched; exact-id uniqueness is unaffected and the flag arms the shrink guard so a surprising node drop is refused loudly (#2881, thanks @rajarshidattapy). diff --git a/README.md b/README.md index 0c14d207c9..e61774e911 100644 --- a/README.md +++ b/README.md @@ -267,6 +267,7 @@ Codex users also need `multi_agent = true` under `[features]` in `~/.codex/confi | `dm` | BYOND DreamMaker `.dm`/`.dme` AST extraction (may need a C compiler + `python3-dev` if no wheel matches your platform) | `uv tool install "graphifyy[dm]"` | | `terraform` | Terraform / HCL `.tf`/`.tfvars`/`.hcl` AST extraction | `uv tool install "graphifyy[terraform]"` | | `pascal` | Pascal / Delphi `.pas`/`.dpr`/`.dpk`/`.inc` AST extraction (more accurate `calls`/`inherits` edges; falls back to a regex extractor when absent) | `uv tool install "graphifyy[pascal]"` | +| `al` | Microsoft Dynamics 365 Business Central `.al` extraction (full Tree-sitter objects, permission sets and extensions, ControlAddIns and UserControls, app dependencies, calls, events, and test-app relationships on Python 3.12+; structural object/procedure/trigger fallback otherwise) | `uv tool install "graphifyy[al]"` | | `ocaml` | OCaml `.ml`/`.mli` AST extraction | `uv tool install "graphifyy[ocaml]"` | | `commonlisp` | Common Lisp `.lisp`/`.cl`/`.lsp`/`.asd` AST extraction | `uv tool install "graphifyy[commonlisp]"` | | `chinese` | Chinese query segmentation (jieba) | `uv tool install "graphifyy[chinese]"` | @@ -338,7 +339,7 @@ To remove graphify from all platforms at once: `graphify uninstall` (add `--purg | Type | Extensions | |------|-----------| -| Code (37 tree-sitter grammars) | `.py .ts .mts .cts .js .jsx .tsx .mjs .go .rs .java .c .cpp .cc .cxx .h .hpp .cu .cuh .metal .rb .cs .kt .kts .scala .php .swift .lua .luau .toc .zig .ps1 .psm1 .psd1 .ex .exs .m .mm .ml .mli .jl .vue .svelte .astro .groovy .gradle .dart .v .sv .svh .sql .f .f90 .f95 .f03 .f08 .pas .pp .dpr .dpk .lpr .inc .dfm .lfm .lpk .sh .bash .json .dm .dme .dmi .dmm .dmf .sln .slnx .csproj .fsproj .vbproj .xaml .razor .cshtml` (`.dm`/`.dme` requires `uv tool install graphifyy[dm]`, `.ml`/`.mli` requires `uv tool install graphifyy[ocaml]`; `.mts`/`.cts` reuse the TypeScript grammar, `.cc`/`.cxx` and CUDA `.cu`/`.cuh` and Metal `.metal` reuse the C++ grammar) | +| Code (38 tree-sitter grammars) | `.py .ts .mts .cts .js .jsx .tsx .mjs .go .rs .java .c .cpp .cc .cxx .h .hpp .cu .cuh .metal .rb .cs .kt .kts .scala .php .swift .lua .luau .toc .zig .ps1 .psm1 .psd1 .ex .exs .m .mm .ml .mli .jl .vue .svelte .astro .groovy .gradle .dart .v .sv .svh .sql .f .f90 .f95 .f03 .f08 .pas .pp .dpr .dpk .lpr .inc .dfm .lfm .lpk .al .sh .bash .json .dm .dme .dmi .dmm .dmf .sln .slnx .csproj .fsproj .vbproj .xaml .razor .cshtml` (`.al` uses `uv tool install graphifyy[al]` for full extraction on Python 3.12+ and otherwise falls back to structural extraction; `.dm`/`.dme` requires `uv tool install graphifyy[dm]`, `.ml`/`.mli` requires `uv tool install graphifyy[ocaml]`; `.mts`/`.cts` reuse the TypeScript grammar, `.cc`/`.cxx` and CUDA `.cu`/`.cuh` and Metal `.metal` reuse the C++ grammar) | | Salesforce Apex | `.cls .trigger` (regex-based; classes, interfaces, enums, methods, triggers, SOQL/DML edges) | | Terraform / HCL | `.tf .tfvars .hcl` (requires `uv tool install graphifyy[terraform]`) | | OCaml | `.ml .mli` (requires `uv tool install graphifyy[ocaml]`) | diff --git a/graphify/al_resolution.py b/graphify/al_resolution.py new file mode 100644 index 0000000000..ab629bf9f5 --- /dev/null +++ b/graphify/al_resolution.py @@ -0,0 +1,434 @@ +"""Application-level symbol resolution for Business Central AL.""" +from __future__ import annotations + +import json +import re +from pathlib import Path + + +_EXTENSION_BASE_KINDS = { + "tableextension": "table", + "pageextension": "page", + "enumextension": "enum", + "reportextension": "report", + "permissionsetextension": "permissionset", +} +_APP_MANIFEST = "app.json" + + +def _key(value: object) -> str: + return str(value or "").strip().strip('"').replace('""', '"').casefold() + + +def _same_source(node_source: object, fact_source: object) -> bool: + node_parts = tuple( + part for part in str(node_source or "").replace("\\", "/").casefold().split("/") + if part and part != "." + ) + fact_parts = tuple( + part for part in str(fact_source or "").replace("\\", "/").casefold().split("/") + if part and part != "." + ) + return bool( + node_parts + and len(node_parts) <= len(fact_parts) + and fact_parts[-len(node_parts):] == node_parts + ) + + +def _unique_member_id(candidates: list[str]) -> str | None: + unique = set(candidates) + return next(iter(unique)) if len(unique) == 1 else None + + +def _manifest_context(source_file: str, cache: dict[Path, dict]) -> dict: + current = Path(source_file).resolve().parent + for directory in (current, *current.parents): + manifest = directory / _APP_MANIFEST + if not manifest.is_file(): + continue + if manifest not in cache: + try: + data = json.loads(manifest.read_text(encoding="utf-8-sig")) + if not isinstance(data, dict): + raise TypeError("app.json root must be an object") + dependencies = { + str(item.get("id", "")).casefold() + for item in data.get("dependencies", []) + if isinstance(item, dict) and item.get("id") + } + cache[manifest] = { + "id": str(data.get("id", "")), + "name": str(data.get("name", "")), + "dependencies": dependencies, + "manifest": str(manifest), + } + except (OSError, ValueError, TypeError): + cache[manifest] = {} + return cache[manifest] + return {} + + +def _reference_name(value: object) -> str: + text = str(value or "").strip() + if "::" in text: + text = text.rsplit("::", 1)[1] + if len(text) >= 2 and text[0] == text[-1] and text[0] in {'"', "'"}: + text = text[1:-1] + return text.replace('""', '"').replace("''", "'") + + +class _ALSymbolResolver: + def __init__(self, per_file: list[dict], all_nodes: list[dict], all_edges: list[dict]) -> None: + self.results = [ + result for result in per_file + if isinstance(result, dict) and result.get("al_facts") + ] + self.all_nodes = all_nodes + self.all_edges = all_edges + self.node_by_id = {node.get("id"): node for node in all_nodes} + self.object_nodes = [ + node for node in all_nodes + if node.get("language") == "al" and node.get("object_kind") + ] + self.member_nodes = [ + node for node in all_nodes + if node.get("language") == "al" and node.get("member_kind") + ] + self.parent_of = { + edge.get("target"): edge.get("source") + for edge in all_edges + if edge.get("relation") == "contains" + } + self.manifest_cache: dict[Path, dict] = {} + self.object_fact_to_nid: dict[str, str] = {} + self.member_fact_to_nid: dict[str, str] = {} + self.object_facts: list[dict] = [] + self.member_facts: list[dict] = [] + self.result_context: dict[int, dict] = {} + self.object_by_name: dict[str, list[dict]] = {} + self.object_by_kind_name: dict[tuple[str, str], list[dict]] = {} + self.members_by_parent_name: dict[tuple[str, str], list[str]] = {} + self.member_parameter_counts: dict[str, int] = {} + self.existing = { + (edge.get("source"), edge.get("target"), edge.get("relation"), edge.get("context")) + for edge in all_edges + } + + def resolve(self) -> None: + if not self.results: + return + self._map_facts() + self._build_indexes() + self._emit_results() + self._emit_manifest_dependencies() + + def _context_for_result(self, result: dict) -> dict: + facts = result["al_facts"] + first_object = next(iter(facts.get("objects", [])), {}) + source_file = str(first_object.get("source_file", "")) + app = _manifest_context(source_file, self.manifest_cache) if source_file else {} + context = { + "namespace": str(facts.get("namespace", "")), + "usings": {_key(value) for value in facts.get("usings", [])}, + "app": app, + } + self.result_context[id(result)] = context + return context + + def _map_object_fact(self, fact: dict, app: dict) -> None: + candidates = [ + node for node in self.object_nodes + if node.get("object_kind") == fact.get("kind") + and _key(node.get("qualified_name")) == _key(fact.get("qualified_name")) + and _same_source(node.get("source_file"), fact.get("source_file")) + ] + if len(candidates) == 1: + target = candidates[0] + self.object_fact_to_nid[str(fact.get("nid"))] = target["id"] + fact["final_nid"] = target["id"] + fact["app"] = app + if app: + target["application_id"] = app.get("id") or None + target["application_name"] = app.get("name") or None + self.object_facts.append(fact) + + def _map_member_fact(self, fact: dict) -> None: + final_parent = self.object_fact_to_nid.get(str(fact.get("parent"))) + candidates = [ + node for node in self.member_nodes + if node.get("member_kind") == fact.get("kind") + and _key(str(node.get("label", "")).removesuffix("()")) == _key(fact.get("name")) + and self.parent_of.get(node.get("id")) == final_parent + and str(node.get("signature", "")) == str(fact.get("signature", "")) + and node.get("source_location") == f"L{fact.get('line')}" + ] + if len(candidates) == 1: + self.member_fact_to_nid[str(fact.get("nid"))] = candidates[0]["id"] + fact["final_nid"] = candidates[0]["id"] + self.member_facts.append(fact) + + def _map_facts(self) -> None: + for result in self.results: + context = self._context_for_result(result) + facts = result["al_facts"] + for fact in facts.get("objects", []): + self._map_object_fact(fact, context["app"]) + for fact in facts.get("members", []): + self._map_member_fact(fact) + + def _index_objects(self) -> None: + for fact in self.object_facts: + if not fact.get("final_nid"): + continue + names = {_key(fact.get("name")), _key(fact.get("qualified_name"))} + for name in names: + self.object_by_name.setdefault(name, []).append(fact) + key = (str(fact.get("kind", "")), name) + self.object_by_kind_name.setdefault(key, []).append(fact) + + def _index_members(self) -> None: + for fact in self.member_facts: + parent = self.object_fact_to_nid.get(str(fact.get("parent"))) + target = fact.get("final_nid") + if parent and target: + key = (parent, _key(fact.get("name"))) + self.members_by_parent_name.setdefault(key, []).append(target) + self.member_parameter_counts[target] = int(fact.get("parameter_count", 0)) + + def _build_indexes(self) -> None: + self._index_objects() + self._index_members() + + def _visible(self, candidate: dict, context: dict) -> bool: + candidate_namespace = _key(candidate.get("namespace")) + source_namespace = _key(context.get("namespace")) + if candidate_namespace and candidate_namespace != source_namespace: + if candidate_namespace not in context.get("usings", set()): + return False + source_app = context.get("app") or {} + candidate_app = candidate.get("app") or {} + if source_app.get("id") and candidate_app.get("id"): + if _key(source_app["id"]) != _key(candidate_app["id"]): + if _key(candidate_app["id"]) not in source_app.get("dependencies", set()): + return False + return True + + def _resolve_object(self, name: object, kind: str | None, context: dict) -> dict | None: + lookup = _key(_reference_name(name)) + normalized_kind = str(kind or "").casefold().removesuffix("_keyword") + if normalized_kind == "record": + normalized_kind = "table" + elif normalized_kind == "testpage": + normalized_kind = "page" + candidates = ( + self.object_by_kind_name.get((normalized_kind, lookup), []) + if normalized_kind else self.object_by_name.get(lookup, []) + ) + visible_candidates = [candidate for candidate in candidates if self._visible(candidate, context)] + unique = {candidate["final_nid"]: candidate for candidate in visible_candidates} + return next(iter(unique.values())) if len(unique) == 1 else None + + def _add_edge( + self, source: str | None, target: str | None, + relation: str, context: str, line: object, + ) -> None: + if not source or not target or source == target: + return + key = (source, target, relation, context) + if key in self.existing: + return + self.existing.add(key) + source_node = self.node_by_id.get(source, {}) + self.all_edges.append({ + "source": source, + "target": target, + "relation": relation, + "context": context, + "confidence": "EXTRACTED", + "confidence_score": 1.0, + "source_file": source_node.get("source_file", ""), + "source_location": f"L{line}" if line else source_node.get("source_location"), + "weight": 1.0, + }) + + def _emit_objects(self, facts: dict, context: dict) -> None: + for fact in facts.get("objects", []): + source = self.object_fact_to_nid.get(str(fact.get("nid"))) + if fact.get("base"): + target = self._resolve_object( + fact["base"], _EXTENSION_BASE_KINDS.get(str(fact.get("kind"))), context + ) + self._add_edge( + source, target and target["final_nid"], + "extends", "extension", fact.get("line"), + ) + for interface in fact.get("interfaces", []): + target = self._resolve_object(interface, "interface", context) + self._add_edge( + source, target and target["final_nid"], + "implements", "interface", fact.get("line"), + ) + + def _emit_references(self, facts: dict, context: dict) -> None: + for reference in facts.get("references", []): + raw_source = str(reference.get("source")) + source = self.member_fact_to_nid.get(raw_source) or self.object_fact_to_nid.get(raw_source) + target = self._resolve_object( + reference.get("name"), str(reference.get("kind", "")), context + ) + reference_context = ( + "test_target" if str(reference.get("kind", "")).casefold() == "testpage" else "type" + ) + self._add_edge( + source, target and target["final_nid"], "references", + reference_context, reference.get("line"), + ) + + def _emit_calls(self, facts: dict, context: dict) -> None: + for call in facts.get("calls", []): + source = self.member_fact_to_nid.get(str(call.get("source"))) + owner = self.parent_of.get(source) + target_owner = owner + if call.get("receiver_type"): + target_object = self._resolve_object( + call["receiver_type"], call.get("receiver_kind"), context + ) + target_owner = target_object and target_object["final_nid"] + candidates = self.members_by_parent_name.get( + (str(target_owner), _key(call.get("name"))), [] + ) + if call.get("argument_count") is not None: + candidates = [ + candidate for candidate in candidates + if self.member_parameter_counts.get(candidate) == call["argument_count"] + ] + target = _unique_member_id(candidates) + self._add_edge(source, target, "calls", "call", call.get("line")) + + def _emit_subscribers(self, facts: dict, context: dict) -> None: + for subscriber in facts.get("event_subscribers", []): + arguments = subscriber.get("arguments", []) + if len(arguments) < 3: + continue + object_kind = _reference_name(arguments[0]).casefold() + publisher_object = self._resolve_object(arguments[1], object_kind, context) + event_name = _reference_name(arguments[2]) + candidates = self.members_by_parent_name.get( + (str(publisher_object and publisher_object["final_nid"]), _key(event_name)), [] + ) + target = _unique_member_id(candidates) + self._add_edge( + self.member_fact_to_nid.get(str(subscriber.get("source"))), + target, + "references", + "event_subscription", + subscriber.get("line"), + ) + + def _emit_enum_mappings(self, facts: dict, context: dict) -> None: + for mapping in facts.get("enum_mappings", []): + source = self.member_fact_to_nid.get(str(mapping.get("source"))) + interface = self._resolve_object(mapping.get("interface"), "interface", context) + implementation = self._resolve_object(mapping.get("implementation"), "codeunit", context) + self._add_edge( + source, interface and interface["final_nid"], "implements", + "enum_implementation", mapping.get("line"), + ) + self._add_edge( + source, implementation and implementation["final_nid"], "references", + "enum_implementation", mapping.get("line"), + ) + + def _emit_test_handlers(self, facts: dict) -> None: + for binding in facts.get("test_handlers", []): + source = self.member_fact_to_nid.get(str(binding.get("source"))) + owner = self.parent_of.get(source) + for argument in binding.get("arguments", []): + handler_names = ( + name.strip() for name in _reference_name(argument).split(",") + ) + for handler_name in filter(None, handler_names): + candidates = self.members_by_parent_name.get( + (str(owner), _key(handler_name)), [] + ) + target = _unique_member_id(candidates) + self._add_edge( + source, target, "references", "test_handler", binding.get("line") + ) + + def _emit_control_addin_events(self, facts: dict, context: dict) -> None: + for binding in facts.get("control_addin_events", []): + controladdin = self._resolve_object( + binding.get("controladdin"), "controladdin", context + ) + candidates = self.members_by_parent_name.get( + ( + str(controladdin and controladdin["final_nid"]), + _key(binding.get("event")), + ), + [], + ) + target = _unique_member_id(candidates) + self._add_edge( + self.member_fact_to_nid.get(str(binding.get("source"))), + target, + "references", + "control_addin_event", + binding.get("line"), + ) + + def _emit_core_facts(self, facts: dict, context: dict) -> None: + self._emit_objects(facts, context) + self._emit_references(facts, context) + self._emit_calls(facts, context) + + def _emit_attribute_facts(self, facts: dict, context: dict) -> None: + self._emit_subscribers(facts, context) + self._emit_enum_mappings(facts, context) + self._emit_test_handlers(facts) + self._emit_control_addin_events(facts, context) + + def _emit_results(self) -> None: + for result in self.results: + facts = result["al_facts"] + context = self.result_context[id(result)] + self._emit_core_facts(facts, context) + self._emit_attribute_facts(facts, context) + + def _manifest_node(self, manifest_nodes: list[dict], manifest: object) -> dict | None: + candidates = [ + node for node in manifest_nodes + if _same_source(node.get("source_file"), manifest) + ] + return candidates[0] if len(candidates) == 1 else None + + def _emit_manifest_dependencies(self) -> None: + manifest_nodes = [ + node for node in self.all_nodes + if str(node.get("source_file", "")).casefold().endswith(_APP_MANIFEST) + and str(node.get("label", "")).casefold().endswith(_APP_MANIFEST) + ] + apps_by_id: dict[str, list[dict]] = {} + for app in self.manifest_cache.values(): + if app.get("id"): + apps_by_id.setdefault(_key(app["id"]), []).append(app) + for app in self.manifest_cache.values(): + source_node = self._manifest_node(manifest_nodes, app.get("manifest")) + if source_node is None: + continue + for dependency_id in app.get("dependencies", set()): + targets = apps_by_id.get(_key(dependency_id), []) + if len(targets) != 1: + continue + target_node = self._manifest_node(manifest_nodes, targets[0].get("manifest")) + if target_node is not None: + self._add_edge( + source_node["id"], target_node["id"], + "depends_on", "application", 1, + ) + + +def resolve_al_symbols(per_file: list[dict], all_nodes: list[dict], all_edges: list[dict]) -> None: + """Resolve AL facts without guessing when multiple candidates remain.""" + _ALSymbolResolver(per_file, all_nodes, all_edges).resolve() \ No newline at end of file diff --git a/graphify/detect.py b/graphify/detect.py index d16b5800ce..5266bc0700 100644 --- a/graphify/detect.py +++ b/graphify/detect.py @@ -42,7 +42,7 @@ class FileType(str, Enum): _MTIME_COARSE_S = 2.0 _MTIME_SUBSECOND_S = 0.05 -CODE_EXTENSIONS = {'.py', '.ts', '.tsx', '.mts', '.cts', '.js', '.jsx', '.mjs', '.cjs', '.ejs', '.ets', '.go', '.rs', '.java', '.groovy', '.gradle', '.cpp', '.cc', '.cxx', '.c', '.h', '.hpp', '.cu', '.cuh', '.metal', '.rb', '.rake', '.swift', '.kt', '.kts', '.cs', '.scala', '.php', '.lua', '.luau', '.toc', '.zig', '.ps1', '.psm1', '.psd1', '.ex', '.exs', '.m', '.mm', '.ml', '.mli', '.jl', '.vue', '.svelte', '.astro', '.dart', '.v', '.sv', '.svh', '.sql', '.r', '.f', '.F', '.f90', '.F90', '.f95', '.F95', '.f03', '.F03', '.f08', '.F08', '.pas', '.pp', '.dpr', '.dpk', '.lpr', '.inc', '.dfm', '.lfm', '.lpk', '.sh', '.bash', '.json', '.tf', '.tfvars', '.hcl', '.dm', '.dme', '.dmi', '.dmm', '.dmf', '.sln', '.slnx', '.csproj', '.fsproj', '.vbproj', '.xaml', '.razor', '.cshtml', '.cls', '.trigger', '.lisp', '.cl', '.lsp', '.asd'} +CODE_EXTENSIONS = {'.py', '.ts', '.tsx', '.mts', '.cts', '.js', '.jsx', '.mjs', '.cjs', '.ejs', '.ets', '.go', '.rs', '.java', '.groovy', '.gradle', '.cpp', '.cc', '.cxx', '.c', '.h', '.hpp', '.cu', '.cuh', '.metal', '.rb', '.rake', '.swift', '.kt', '.kts', '.cs', '.scala', '.php', '.lua', '.luau', '.toc', '.zig', '.ps1', '.psm1', '.psd1', '.ex', '.exs', '.m', '.mm', '.ml', '.mli', '.jl', '.vue', '.svelte', '.astro', '.dart', '.v', '.sv', '.svh', '.sql', '.r', '.f', '.F', '.f90', '.F90', '.f95', '.F95', '.f03', '.F03', '.f08', '.F08', '.pas', '.pp', '.dpr', '.dpk', '.lpr', '.inc', '.dfm', '.lfm', '.lpk', '.al', '.sh', '.bash', '.json', '.tf', '.tfvars', '.hcl', '.dm', '.dme', '.dmi', '.dmm', '.dmf', '.sln', '.slnx', '.csproj', '.fsproj', '.vbproj', '.xaml', '.razor', '.cshtml', '.cls', '.trigger', '.lisp', '.cl', '.lsp', '.asd'} DOC_EXTENSIONS = {'.md', '.mdx', '.qmd', '.skill', '.txt', '.rst', '.html', '.yaml', '.yml'} PAPER_EXTENSIONS = {'.pdf'} IMAGE_EXTENSIONS = {'.png', '.jpg', '.jpeg', '.gif', '.webp', '.svg'} diff --git a/graphify/extract.py b/graphify/extract.py index ffc6153f82..9bae1c4d23 100644 --- a/graphify/extract.py +++ b/graphify/extract.py @@ -23,6 +23,7 @@ ) from .ruby_resolution import resolve_ruby_member_calls from .pascal_resolution import resolve_pascal_inherited_calls +from .al_resolution import resolve_al_symbols # --- migrated to graphify/extractors/ (see graphify/extractors/MIGRATION.md) --- from graphify.extractors.base import ( # noqa: F401 @@ -61,6 +62,7 @@ from graphify.paths import disambiguate_ambiguous_candidates from graphify.extractors.models import LanguageConfig, _JS_CACHE_BYPASS_SUFFIXES, _NamespaceExportFact, _StarExportFact, _SymbolAliasFact, _SymbolDeclarationFact, _SymbolExportFact, _SymbolImportFact, _SymbolResolutionFacts, _SymbolUseFact, _WORKSPACE_PACKAGE_CACHE # noqa: E402,F401 +from graphify.extractors.al import extract_al # noqa: E402,F401 from graphify.extractors.resolution import ( # noqa: E402,F401 _DECLDEF_HEADER_SUFFIXES, @@ -3945,6 +3947,9 @@ def _resolve_kotlin_qualified_calls( resolve_pascal_inherited_calls, ) ) +register_language_resolver( + LanguageResolver("al_symbols", frozenset({".al", ".AL"}), resolve_al_symbols) +) # Kotlin fully-qualified call resolution (#2550): `com.pkg.Fn()` / # `com.pkg.Object.method()` raw_calls the shared pass skips (member calls with # no receiver). Runs in the tail registry like the other member-call resolvers; @@ -4965,6 +4970,7 @@ def add_existing_edge(edge: dict) -> None: _DISPATCH: dict[str, Any] = { + ".al": extract_al, ".py": extract_python, ".js": extract_js, ".jsx": extract_js, @@ -5073,6 +5079,7 @@ def add_existing_edge(edge: dict) -> None: # rather than falling back like Pascal does. Used by the #1745 warning in # extract() to tell the user which extra restores the language. _EXTRA_FOR_EXTENSION = { + ".al": "al", ".sql": "sql", ".tf": "terraform", ".tfvars": "terraform", @@ -5686,12 +5693,29 @@ def extract( # dependency when there is one. _missing_dep_count: dict[str, int] = {} _missing_dep_error: dict[str, str] = {} + _missing_dep_fallback: dict[str, bool] = {} for i, _p in enumerate(paths): - _err = (per_file[i] or {}).get("error") or "" + _result = per_file[i] or {} + _diagnostics = ( + str(_result.get("dependency_warning") or ""), + str(_result.get("error") or ""), + ) + _err = next( + ( + message for message in _diagnostics + if _DEP_MISSING_MARKER in message + or _DEP_LOAD_FAILED_MARKER in message + ), + "", + ) if _DEP_MISSING_MARKER in _err or _DEP_LOAD_FAILED_MARKER in _err: _ext = _p.suffix.lower() _missing_dep_count[_ext] = _missing_dep_count.get(_ext, 0) + 1 _missing_dep_error.setdefault(_ext, _err) + _missing_dep_fallback[_ext] = ( + _missing_dep_fallback.get(_ext, False) + or bool(_result.get("dependency_warning")) + ) for _ext, _n in sorted(_missing_dep_count.items(), key=lambda kv: (-kv[1], kv[0])): _extra = _EXTRA_FOR_EXTENSION.get(_ext) _err_text = _missing_dep_error[_ext] @@ -5707,11 +5731,17 @@ def extract( _hint = "" _cause = ("a dependency is missing" if _DEP_MISSING_MARKER in _err_text else "a dependency failed to load") - print( - f" warning: {_n} {_ext} file(s) contributed nothing to the graph " - f"because {_cause}: {_reason}.{_hint} (#1745)", - file=sys.stderr, flush=True, - ) + if _missing_dep_fallback.get(_ext): + _message = ( + f" warning: {_n} {_ext} file(s) used fallback extraction because " + f"{_cause}: {_reason}.{_hint} (#1745)" + ) + else: + _message = ( + f" warning: {_n} {_ext} file(s) contributed nothing to the graph " + f"because {_cause}: {_reason}.{_hint} (#1745)" + ) + print(_message, file=sys.stderr, flush=True) # #2551: a file the parser ACCEPTED but only with ERROR recovery (e.g. the # Kotlin grammar rejecting one-line `class C { val x }` bodies, or Luau diff --git a/graphify/extractors/__init__.py b/graphify/extractors/__init__.py index 68ff3340c3..ad5fe64cb8 100644 --- a/graphify/extractors/__init__.py +++ b/graphify/extractors/__init__.py @@ -10,6 +10,7 @@ from pathlib import Path from typing import Callable +from graphify.extractors.al import extract_al from graphify.extractors.apex import extract_apex from graphify.extractors.bash import extract_bash from graphify.extractors.blade import extract_blade @@ -35,6 +36,7 @@ from graphify.extractors.zig import extract_zig LANGUAGE_EXTRACTORS: dict[str, Callable[[Path], dict]] = { + "al": extract_al, "apex": extract_apex, "bash": extract_bash, "blade": extract_blade, diff --git a/graphify/extractors/al.py b/graphify/extractors/al.py new file mode 100644 index 0000000000..4d36fb9bff --- /dev/null +++ b/graphify/extractors/al.py @@ -0,0 +1,902 @@ +"""Microsoft Dynamics 365 Business Central AL extraction.""" +from __future__ import annotations + +import re +from pathlib import Path + +from graphify.extractors.base import _file_stem, _make_id + + +_AL_IDENTIFIER = r'(?P"(?:[^"]|"")+"|[A-Za-z_][\w.]*)' +_AL_OBJECT_RE = re.compile( + rf"(?im)^\s*(?Pcodeunit|tableextension|table|pageextension|page|" + rf"enumextension|enum|interface|reportextension|report|query|xmlport|" + rf"permissionsetextension|permissionset|controladdin)\s+" + rf"(?:(?P\d+)\s+)?{_AL_IDENTIFIER}\s*" + rf"(?:extends\s+(?P\"(?:[^\"]|\"\")+\"|[A-Za-z_][\w.]*))?" + rf"(?:implements\s+(?P[^{{]+))?\s*{{" +) +_AL_CALLABLE_RE = re.compile( + rf"(?im)^\s*(?:(?Plocal|internal|protected|public)\s+)?" + rf"(?Pprocedure|trigger)\s+{_AL_IDENTIFIER}\s*" + rf"\((?P[^)]*)\)\s*(?::\s*(?P[^;\r\n]+))?" +) +_AL_NAMESPACE_RE = re.compile(r"(?im)^\s*namespace\s+([A-Za-z_][\w.]*)\s*;") +_AL_OBJECT_TYPES = { + "codeunit_declaration": "codeunit", + "table_declaration": "table", + "tableextension_declaration": "tableextension", + "page_declaration": "page", + "pageextension_declaration": "pageextension", + "enum_declaration": "enum", + "enumextension_declaration": "enumextension", + "interface_declaration": "interface", + "report_declaration": "report", + "reportextension_declaration": "reportextension", + "query_declaration": "query", + "xmlport_declaration": "xmlport", + "permissionset_declaration": "permissionset", + "permissionsetextension_declaration": "permissionsetextension", + "controladdin_declaration": "controladdin", +} +_AL_CALLABLE_TYPES = { + "procedure": "procedure", + "interface_procedure": "procedure", + "preproc_split_procedure": "procedure", + "trigger_declaration": "trigger", + "event_declaration": "event", +} +_AL_MEMBER_SCOPE_TYPES = { + "field_declaration", + "page_field", + "action_declaration", + "action_group_section", + "report_dataitem", + "query_dataitem", + "request_page", + "request_page_section", + "usercontrol_section", +} + + +def _decode_al_identifier(value: str | None) -> str: + """Return an AL identifier without delimiters while preserving its spelling.""" + if not value: + return "" + value = value.strip() + if len(value) >= 2 and value[0] == value[-1] == '"': + return value[1:-1].replace('""', '"') + return value + + +def _al_lookup_key(value: str) -> str: + return _decode_al_identifier(value).casefold() + + +def _mask_al_code(chars: list[str], index: int, char: str, following: str) -> tuple[int, str]: + if char == "/" and following in {"/", "*"}: + chars[index] = chars[index + 1] = " " + state = "line_comment" if following == "/" else "block_comment" + return index + 2, state + if char == '"': + return index + 1, "quoted_identifier" + if char == "'": + chars[index] = " " + return index + 1, "string" + return index + 1, "code" + + +def _mask_al_line_comment(chars: list[str], index: int, char: str, _following: str) -> tuple[int, str]: + if char == "\n": + return index + 1, "code" + chars[index] = " " + return index + 1, "line_comment" + + +def _mask_al_block_comment( + chars: list[str], index: int, char: str, following: str +) -> tuple[int, str]: + if char == "*" and following == "/": + chars[index] = chars[index + 1] = " " + return index + 2, "code" + if char != "\n": + chars[index] = " " + return index + 1, "block_comment" + + +def _mask_al_string(chars: list[str], index: int, char: str, following: str) -> tuple[int, str]: + chars[index] = " " if char != "\n" else "\n" + if char == "'" and following == "'": + chars[index + 1] = " " + return index + 2, "string" + return index + 1, "code" if char == "'" else "string" + + +def _mask_al_quoted_identifier( + _chars: list[str], index: int, char: str, following: str +) -> tuple[int, str]: + if char == '"' and following == '"': + return index + 2, "quoted_identifier" + return index + 1, "code" if char == '"' else "quoted_identifier" + + +def _mask_al_comments_and_strings(source: str) -> str: + """Mask comments and string literals without changing offsets or newlines.""" + chars = list(source) + if chars and chars[0] == "\ufeff": + chars[0] = " " + index = 0 + state = "code" + handlers = { + "code": _mask_al_code, + "line_comment": _mask_al_line_comment, + "block_comment": _mask_al_block_comment, + "string": _mask_al_string, + "quoted_identifier": _mask_al_quoted_identifier, + } + while index < len(source): + char = source[index] + following = source[index + 1] if index + 1 < len(source) else "" + index, state = handlers[state](chars, index, char, following) + return "".join(chars) + + +def _matching_brace(masked_source: str, opening: int) -> int: + """Find the closing brace in source with comments and strings already masked.""" + depth = 0 + for index in range(opening, len(masked_source)): + if masked_source[index] == "{": + depth += 1 + elif masked_source[index] == "}": + depth -= 1 + if depth == 0: + return index + return len(masked_source) + + +def _line_number(source: str, offset: int) -> int: + return source.count("\n", 0, offset) + 1 + + +def _walk_al(node): + yield node + for child in node.named_children: + yield from _walk_al(child) + + +def _node_text(node, source: bytes) -> str: + return source[node.start_byte:node.end_byte].decode("utf-8", errors="replace") + + +def _field_text(node, field: str, source: bytes) -> str: + child = node.child_by_field_name(field) + return _node_text(child, source) if child else "" + + +def _first_descendant(node, types: set[str]): + return next((child for child in _walk_al(node) if child.type in types), None) + + +def _attribute_metadata(node, source: bytes) -> list[dict]: + attributes: list[dict] = [] + sibling = node.prev_named_sibling + while sibling is not None and sibling.type == "attribute_item": + content = sibling.child_by_field_name("attribute") + if content is not None: + name = _field_text(content, "name", source) + arguments = content.child_by_field_name("arguments") + argument_list = _first_descendant(arguments, {"attribute_argument_list"}) if arguments else None + attributes.append({ + "name": _decode_al_identifier(name), + "arguments": [ + _node_text(child, source).strip() + for child in (argument_list.named_children if argument_list else []) + ], + }) + sibling = sibling.prev_named_sibling + attributes.reverse() + return attributes + + +def _parameter_metadata(node, source: bytes) -> list[dict]: + parameters = node.child_by_field_name("parameters") + if parameters is None: + return [] + result: list[dict] = [] + for parameter in parameters.named_children: + if parameter.type != "parameter": + continue + type_node = parameter.child_by_field_name("type") + result.append({ + "name": _decode_al_identifier(_field_text(parameter, "name", source)), + "type": _node_text(type_node, source).strip() if type_node else "", + "modifier": _field_text(parameter, "modifier", source).strip() or None, + }) + return result + + +def _type_reference(type_node, source: bytes) -> tuple[str, str] | None: + reference_node = _first_descendant(type_node, {"object_reference_type", "record_type"}) + if reference_node is None: + return None + reference = _decode_al_identifier(_field_text(reference_node, "reference", source)) + if not reference: + return None + object_type = _field_text(reference_node, "object_type", source).casefold() + if not object_type: + object_type = "record" if reference_node.type == "record_type" else "object" + return object_type, reference + + +def _member_scope_seed(member_node, source: bytes) -> str | None: + seeds: list[str] = [] + current = member_node.parent + while current is not None and current.type not in _AL_OBJECT_TYPES: + if current.type in _AL_MEMBER_SCOPE_TYPES: + name = _decode_al_identifier(_field_text(current, "name", source)) + identifier = _field_text(current, "id", source) + reference = _decode_al_identifier(_field_text(current, "source", source)) + if name or identifier or reference: + seeds.append(f"{current.type}:{identifier}:{name}:{reference}") + current = current.parent + return "/".join(reversed(seeds)) or None + + +class _ALTreeContext: + def __init__(self, path: Path, source: str, tree) -> None: + self.path = path + self.source = source.encode("utf-8") + self.str_path = str(path) + self.stem = _file_stem(path) + self.root = tree.root_node + self.namespace, usings = _al_namespace_and_usings(self.root, self.source) + self.file_nid = _make_id(self.str_path) + self.nodes: list[dict] = [{ + "id": self.file_nid, + "label": path.name, + "file_type": "code", + "source_file": self.str_path, + "source_location": "L1", + "language": "al", + "namespace": self.namespace or None, + "extraction_tier": "tree_sitter", + }] + self.edges: list[dict] = [] + self.seen_ids = {self.file_nid} + self.facts: dict[str, object] = { + "namespace": self.namespace, + "usings": usings, + "objects": [], + "members": [], + "references": [], + "calls": [], + "event_subscribers": [], + "event_publishers": [], + "enum_mappings": [], + "test_handlers": [], + "control_addin_events": [], + } + + def add_node(self, node: dict) -> None: + if node["id"] not in self.seen_ids: + self.seen_ids.add(node["id"]) + self.nodes.append(node) + + def add_edge(self, parent: str, child: str, relation: str, line: int) -> None: + self.edges.append({ + "source": parent, + "target": child, + "relation": relation, + "confidence": "EXTRACTED", + "source_file": self.str_path, + "source_location": f"L{line}", + "weight": 1.0, + }) + + def result(self) -> dict: + syntax_errors = [ + {"line": node.start_point.row + 1, "type": node.type} + for node in _walk_al(self.root) + if node.type == "ERROR" or node.is_missing + ] + result = {"nodes": self.nodes, "edges": self.edges, "al_facts": self.facts} + if syntax_errors: + result["syntax_errors"] = syntax_errors + return result + + +def _al_namespace_and_usings(root, source: bytes) -> tuple[str, list[str]]: + namespace_node = next( + (node for node in root.named_children if node.type == "namespace_declaration"), None + ) + namespace = _field_text(namespace_node, "name", source) if namespace_node else "" + usings = [ + _field_text(node, "namespace", source) + for node in root.named_children + if node.type == "using_statement" + ] + return namespace, usings + + +def _al_object_info(context: _ALTreeContext, object_node) -> dict: + kind = _AL_OBJECT_TYPES[object_node.type] + name = _decode_al_identifier(_field_text(object_node, "object_name", context.source)) + qualified_name = f"{context.namespace}.{name}" if context.namespace else name + interfaces = [ + _decode_al_identifier(_node_text(child, context.source)) + for clause in object_node.named_children + if clause.type == "implements_clause" + for index, child in enumerate(clause.children) + if clause.field_name_for_child(index) == "interface" + ] + return { + "nid": _make_id(context.stem, kind, qualified_name), + "kind": kind, + "name": name, + "qualified_name": qualified_name, + "lookup_key": _al_lookup_key(qualified_name), + "object_id": _field_text(object_node, "object_id", context.source) or None, + "base": _decode_al_identifier(_field_text(object_node, "base_object", context.source)) or None, + "interfaces": [interface for interface in interfaces if interface], + "line": object_node.start_point.row + 1, + } + + +def _al_emit_object(context: _ALTreeContext, info: dict) -> None: + context.add_node({ + "id": info["nid"], + "label": info["name"], + "file_type": "code", + "source_file": context.str_path, + "source_location": f"L{info['line']}", + "language": "al", + "object_kind": info["kind"], + "object_id": info["object_id"], + "qualified_name": info["qualified_name"], + "namespace": context.namespace or None, + "lookup_key": info["lookup_key"], + "extraction_tier": "tree_sitter", + }) + context.add_edge(context.file_nid, info["nid"], "contains", info["line"]) + context.facts["objects"].append({ + **info, + "namespace": context.namespace, + "source_file": context.str_path, + }) + + +def _al_object_variables(context: _ALTreeContext, object_node, object_nid: str) -> dict: + variable_types: dict[str, tuple[str, str]] = {} + for declaration in (node for node in _walk_al(object_node) if node.type == "variable_declaration"): + ancestor = declaration.parent + while ancestor is not None and ancestor is not object_node: + if ancestor.type in _AL_CALLABLE_TYPES: + break + ancestor = ancestor.parent + if ancestor is not object_node: + continue + variable_name = _decode_al_identifier(_field_text(declaration, "name", context.source)) + type_node = declaration.child_by_field_name("type") + reference = _type_reference(type_node, context.source) if type_node else None + if variable_name and reference: + variable_types[_al_lookup_key(variable_name)] = reference + context.facts["references"].append({ + "source": object_nid, + "name": reference[1], + "kind": reference[0], + "line": declaration.start_point.row + 1, + }) + return variable_types + + +def _al_member_nodes(object_node) -> list: + return [ + node for node in _walk_al(object_node) + if node.type in _AL_CALLABLE_TYPES + or node.type in { + "field_declaration", + "enum_value_declaration", + "usercontrol_section", + } + ] + + +def _al_member_name_counts(context: _ALTreeContext, member_nodes: list) -> dict: + counts: dict[tuple[str, str], int] = {} + for candidate in member_nodes: + if candidate.type not in _AL_CALLABLE_TYPES: + continue + name = _decode_al_identifier(_field_text(candidate, "name", context.source)) + key = (_AL_CALLABLE_TYPES[candidate.type], _al_lookup_key(name)) + counts[key] = counts.get(key, 0) + 1 + return counts + + +def _al_member_declaration(context: _ALTreeContext, member_node) -> dict | None: + if member_node.type in _AL_CALLABLE_TYPES: + kind, name_field, suffix = _AL_CALLABLE_TYPES[member_node.type], "name", "()" + elif member_node.type == "field_declaration": + kind, name_field, suffix = "field", "name", "" + elif member_node.type == "enum_value_declaration": + kind, name_field, suffix = "enum_value", "value_name", "" + else: + kind, name_field, suffix = "usercontrol", "name", "" + name = _decode_al_identifier(_field_text(member_node, name_field, context.source)) + if not name: + return None + parameters = _parameter_metadata(member_node, context.source) + return { + "kind": kind, + "name": name, + "suffix": suffix, + "lookup_key": _al_lookup_key(name), + "attributes": _attribute_metadata(member_node, context.source), + "parameters": parameters, + "signature": ",".join( + f"{parameter.get('modifier') or ''}:{parameter.get('type') or ''}" + for parameter in parameters + ), + "line": member_node.start_point.row + 1, + } + + +def _al_member_nid( + context: _ALTreeContext, member_node, object_nid: str, info: dict, counts: dict +) -> str: + scope_seed = _member_scope_seed(member_node, context.source) + identity_seeds = [scope_seed] if scope_seed else [] + if counts.get((info["kind"], info["lookup_key"]), 0) > 1: + identity_seeds.append(f"signature:{info['signature']}") + return _make_id(object_nid, info["kind"], *identity_seeds, info["name"]) + + +def _al_callable_metadata(context: _ALTreeContext, member_node, info: dict) -> dict: + return_type = member_node.child_by_field_name("return_type") + if return_type is None and member_node.type == "interface_procedure": + suffix = _first_descendant(member_node, {"interface_procedure_suffix"}) + return_type = suffix.child_by_field_name("return_type") if suffix else None + modifier = member_node.child_by_field_name("modifier") + return { + "visibility": _node_text(modifier, context.source).strip() if modifier else None, + "parameters": info["parameters"], + "return_type": _node_text(return_type, context.source).strip() + if return_type else None, + "attributes": info["attributes"], + "signature": info["signature"], + "_callable": True, + } + + +def _al_data_member_metadata( + context: _ALTreeContext, member_node, info: dict +) -> dict: + data_type = member_node.child_by_field_name( + "source" if info["kind"] == "usercontrol" else "type" + ) + return { + "member_id": _field_text(member_node, "id", context.source) + or _field_text(member_node, "value_id", context.source) or None, + "data_type": _node_text(data_type, context.source).strip() if data_type else None, + } + + +def _al_member_metadata( + context: _ALTreeContext, member_node, object_nid: str, member_nid: str, info: dict +) -> dict: + metadata = { + "id": member_nid, + "label": f"{info['name']}{info['suffix']}", + "file_type": "code", + "source_file": context.str_path, + "source_location": f"L{info['line']}", + "language": "al", + "member_kind": info["kind"], + "parent_object": object_nid, + "lookup_key": info["lookup_key"], + "extraction_tier": "tree_sitter", + } + if info["kind"] in _AL_CALLABLE_TYPES.values(): + metadata.update(_al_callable_metadata(context, member_node, info)) + else: + metadata.update(_al_data_member_metadata(context, member_node, info)) + return metadata + + +def _al_callable_types( + context: _ALTreeContext, member_node, member_nid: str, info: dict, object_types: dict +) -> dict: + callable_types = dict(object_types) + for declaration in ( + node for node in _walk_al(member_node) if node.type == "variable_declaration" + ): + name = _decode_al_identifier(_field_text(declaration, "name", context.source)) + type_node = declaration.child_by_field_name("type") + reference = _type_reference(type_node, context.source) if type_node else None + if name and reference: + callable_types[_al_lookup_key(name)] = reference + context.facts["references"].append({ + "source": member_nid, + "name": reference[1], + "kind": reference[0], + "line": declaration.start_point.row + 1, + }) + for parameter, parameter_data in zip( + (node for node in _walk_al(member_node) if node.type == "parameter"), + info["parameters"], + ): + type_node = parameter.child_by_field_name("type") + reference = _type_reference(type_node, context.source) if type_node else None + if parameter_data["name"] and reference: + callable_types[_al_lookup_key(parameter_data["name"])] = reference + context.facts["references"].append({ + "source": member_nid, + "name": reference[1], + "kind": reference[0], + "line": parameter.start_point.row + 1, + }) + return callable_types + + +def _al_extract_calls( + context: _ALTreeContext, member_node, member_nid: str, callable_types: dict +) -> None: + for call in (node for node in _walk_al(member_node) if node.type == "call_expression"): + function = call.child_by_field_name("function") + if function is None: + continue + if function.type == "member_expression": + receiver = _decode_al_identifier(_field_text(function, "object", context.source)) + call_name = _decode_al_identifier(_field_text(function, "member", context.source)) + else: + receiver = "" + call_name = _decode_al_identifier(_node_text(function, context.source)) + receiver_type = callable_types.get(_al_lookup_key(receiver)) if receiver else None + arguments = call.child_by_field_name("arguments") + context.facts["calls"].append({ + "source": member_nid, + "name": call_name, + "receiver": receiver or None, + "receiver_kind": receiver_type[0] if receiver_type else None, + "receiver_type": receiver_type[1] if receiver_type else None, + "argument_count": len(arguments.named_children) if arguments else 0, + "line": call.start_point.row + 1, + }) + + +def _al_extract_callable_attributes( + context: _ALTreeContext, member_nid: str, object_nid: str, info: dict +) -> None: + attributes = {_al_lookup_key(item["name"]): item for item in info["attributes"]} + if event_attribute := attributes.get("eventsubscriber"): + context.facts["event_subscribers"].append({ + "source": member_nid, + "arguments": event_attribute["arguments"], + "line": info["line"], + }) + if attributes.keys() & {"integrationevent", "businessevent"}: + context.facts["event_publishers"].append({ + "nid": member_nid, + "object": object_nid, + "name": info["name"], + "lookup_key": info["lookup_key"], + }) + if handler_attribute := attributes.get("handlerfunctions"): + context.facts["test_handlers"].append({ + "source": member_nid, + "arguments": handler_attribute["arguments"], + "line": info["line"], + }) + + +def _al_enum_mapping(context: _ALTreeContext, prop, member_nid: str) -> None: + comparison = _first_descendant(prop, {"comparison_expression"}) + if comparison is None: + return + context.facts["enum_mappings"].append({ + "source": member_nid, + "interface": _decode_al_identifier(_field_text(comparison, "left", context.source)), + "implementation": _decode_al_identifier(_field_text(comparison, "right", context.source)), + "line": prop.start_point.row + 1, + }) + + +def _al_extract_enum_mappings(context: _ALTreeContext, member_node, member_nid: str) -> None: + for prop in (node for node in _walk_al(member_node) if node.type == "property"): + property_name = _node_text(prop.child_by_field_name("name"), context.source) + if _al_lookup_key(property_name) == "implementation": + _al_enum_mapping(context, prop, member_nid) + + +def _al_postprocess_member( + context: _ALTreeContext, member_node, object_nid: str, member_nid: str, + info: dict, object_types: dict, +) -> None: + if info["kind"] in _AL_CALLABLE_TYPES.values(): + callable_types = _al_callable_types(context, member_node, member_nid, info, object_types) + _al_extract_calls(context, member_node, member_nid, callable_types) + _al_extract_callable_attributes(context, member_nid, object_nid, info) + elif info["kind"] == "enum_value": + _al_extract_enum_mappings(context, member_node, member_nid) + + +def _al_emit_member( + context: _ALTreeContext, member_node, object_nid: str, member_nid: str, info: dict +) -> None: + context.add_node( + _al_member_metadata(context, member_node, object_nid, member_nid, info) + ) + context.add_edge(object_nid, member_nid, "contains", info["line"]) + context.facts["members"].append({ + "nid": member_nid, + "parent": object_nid, + "name": info["name"], + "lookup_key": info["lookup_key"], + "kind": info["kind"], + "signature": info["signature"], + "parameter_count": len(info["parameters"]), + "line": info["line"], + }) + + +def _al_controladdin_name(context: _ALTreeContext, node) -> str: + return _decode_al_identifier(_field_text(node, "source", context.source)) + + +def _al_collect_member_control_facts( + context: _ALTreeContext, member_node, member_nid: str, info: dict +) -> None: + if info["kind"] == "usercontrol": + controladdin = _al_controladdin_name(context, member_node) + if controladdin: + context.facts["references"].append({ + "source": member_nid, + "name": controladdin, + "kind": "controladdin", + "line": info["line"], + }) + return + if info["kind"] != "trigger": + return + parent = member_node.parent + while parent is not None and parent.type not in _AL_OBJECT_TYPES: + if parent.type == "usercontrol_section": + controladdin = _al_controladdin_name(context, parent) + if controladdin: + context.facts["control_addin_events"].append({ + "source": member_nid, + "controladdin": controladdin, + "event": info["name"], + "line": info["line"], + }) + return + parent = parent.parent + + +def _al_extract_member( + context: _ALTreeContext, member_node, object_nid: str, counts: dict, object_types: dict +) -> None: + info = _al_member_declaration(context, member_node) + if info is None: + return + member_nid = _al_member_nid(context, member_node, object_nid, info, counts) + _al_emit_member(context, member_node, object_nid, member_nid, info) + _al_collect_member_control_facts(context, member_node, member_nid, info) + _al_postprocess_member(context, member_node, object_nid, member_nid, info, object_types) + + +def _al_register_usercontrol_types( + context: _ALTreeContext, members: list, object_types: dict +) -> None: + for usercontrol in ( + node for node in members if node.type == "usercontrol_section" + ): + name = _decode_al_identifier(_field_text(usercontrol, "name", context.source)) + controladdin = _al_controladdin_name(context, usercontrol) + if name and controladdin: + reference = ("controladdin", controladdin) + object_types[_al_lookup_key(name)] = reference + object_types[_al_lookup_key(f"CurrPage.{name}")] = reference + + +def _al_extract_members( + context: _ALTreeContext, object_node, object_nid: str, object_types: dict +) -> None: + members = _al_member_nodes(object_node) + counts = _al_member_name_counts(context, members) + _al_register_usercontrol_types(context, members, object_types) + for member_node in members: + _al_extract_member(context, member_node, object_nid, counts, object_types) + + +def _al_extract_object(context: _ALTreeContext, object_node) -> None: + info = _al_object_info(context, object_node) + if not info["name"]: + return + _al_emit_object(context, info) + object_types = _al_object_variables(context, object_node, info["nid"]) + _al_extract_members(context, object_node, info["nid"], object_types) + + +def _extract_al_tree_sitter(path: Path, source: str, tree) -> dict: + context = _ALTreeContext(path, source, tree) + for object_node in (node for node in _walk_al(context.root) if node.type in _AL_OBJECT_TYPES): + _al_extract_object(context, object_node) + return context.result() + + +class _ALFallbackExtractor: + def __init__(self, path: Path, source: str) -> None: + self.path = path + self.str_path = str(path) + self.stem = _file_stem(path) + self.masked = _mask_al_comments_and_strings(source) + namespace_match = _AL_NAMESPACE_RE.search(self.masked) + self.namespace = namespace_match.group(1) if namespace_match else "" + self.file_nid = _make_id(self.str_path) + self.nodes: list[dict] = [{ + "id": self.file_nid, + "label": path.name, + "file_type": "code", + "source_file": self.str_path, + "source_location": "L1", + "language": "al", + "extraction_tier": "fallback", + }] + self.edges: list[dict] = [] + self.seen_ids = {self.file_nid} + + def extract(self) -> dict: + self._extract_objects() + return {"nodes": self.nodes, "edges": self.edges} + + def _add_node(self, node: dict) -> None: + if node["id"] not in self.seen_ids: + self.seen_ids.add(node["id"]) + self.nodes.append(node) + + def _add_contains(self, parent: str, child: str, line: int) -> None: + self.edges.append({ + "source": parent, + "target": child, + "relation": "contains", + "confidence": "EXTRACTED", + "source_file": self.str_path, + "source_location": f"L{line}", + "weight": 1.0, + }) + + def _extract_objects(self) -> None: + for match in _AL_OBJECT_RE.finditer(self.masked): + info = self._object_info(match) + self._emit_object(info) + self._extract_callables(match, info["nid"]) + + def _object_info(self, match) -> dict: + kind = match.group("kind").casefold() + name = _decode_al_identifier(match.group("name")) + qualified_name = f"{self.namespace}.{name}" if self.namespace else name + return { + "nid": _make_id(self.stem, kind, qualified_name), + "kind": kind, + "name": name, + "qualified_name": qualified_name, + "object_id": match.group("object_id"), + "line": _line_number(self.masked, match.start()), + "lookup_key": _al_lookup_key(qualified_name), + } + + def _emit_object(self, info: dict) -> None: + self._add_node({ + "id": info["nid"], + "label": info["name"], + "file_type": "code", + "source_file": self.str_path, + "source_location": f"L{info['line']}", + "language": "al", + "object_kind": info["kind"], + "object_id": info["object_id"], + "qualified_name": info["qualified_name"], + "namespace": self.namespace or None, + "lookup_key": info["lookup_key"], + "extraction_tier": "fallback", + }) + self._add_contains(self.file_nid, info["nid"], info["line"]) + + def _extract_callables(self, match, object_nid: str) -> None: + opening = match.end() - 1 + closing = _matching_brace(self.masked, opening) + body = self.masked[opening + 1:closing] + body_offset = opening + 1 + occurrences: dict[tuple[str, str], int] = {} + for callable_match in _AL_CALLABLE_RE.finditer(body): + kind = callable_match.group("callable_kind").casefold() + name = _decode_al_identifier(callable_match.group("name")) + key = (kind, _al_lookup_key(name)) + occurrences[key] = occurrences.get(key, 0) + 1 + info = self._callable_info( + callable_match, object_nid, body_offset, occurrences[key] + ) + self._emit_callable(info) + + def _callable_info( + self, match, object_nid: str, body_offset: int, occurrence: int + ) -> dict: + name = _decode_al_identifier(match.group("name")) + kind = match.group("callable_kind").casefold() + identity = (name,) if occurrence == 1 else (name, str(occurrence)) + return { + "nid": _make_id(object_nid, kind, *identity), + "parent": object_nid, + "name": name, + "kind": kind, + "line": _line_number(self.masked, body_offset + match.start()), + "visibility": match.group("visibility"), + "parameters": match.group("parameters").strip(), + "return_type": (match.group("return_type") or "").strip() or None, + "lookup_key": _al_lookup_key(name), + } + + def _emit_callable(self, info: dict) -> None: + self._add_node({ + "id": info["nid"], + "label": f"{info['name']}()", + "file_type": "code", + "source_file": self.str_path, + "source_location": f"L{info['line']}", + "language": "al", + "member_kind": info["kind"], + "visibility": info["visibility"], + "parameters": info["parameters"], + "return_type": info["return_type"], + "lookup_key": info["lookup_key"], + "extraction_tier": "fallback", + "_callable": True, + }) + self._add_contains(info["parent"], info["nid"], info["line"]) + + +def _extract_al_fallback(path: Path, source: str) -> dict: + return _ALFallbackExtractor(path, source).extract() + + +def _al_fallback_result(path: Path, source: str, warning: str) -> dict: + result = _extract_al_fallback(path, source) + result["dependency_warning"] = warning + return result + + +def extract_al(path: Path, source_override: str | None = None) -> dict: + """Extract Business Central AL, falling back to structural regex parsing.""" + try: + source = source_override if source_override is not None else path.read_text( + encoding="utf-8", errors="replace" + ) + except OSError as exc: + return {"nodes": [], "edges": [], "error": str(exc)} + + try: + from tree_sitter import Language, Parser + except ImportError as exc: + return _al_fallback_result(path, source, f"tree_sitter failed to load: {exc}") + try: + import tree_sitter_al + except ImportError as exc: + import importlib.util + + if importlib.util.find_spec("tree_sitter_al") is None: + return _al_fallback_result( + path, source, + "tree_sitter_al not installed. Run: pip install tree-sitter-al", + ) + return _al_fallback_result( + path, source, f"tree_sitter_al failed to load: {exc}" + ) + try: + language = Language(tree_sitter_al.language()) + parser = Parser(language) + tree = parser.parse(source.encode("utf-8")) + except Exception as exc: + return _al_fallback_result( + path, source, f"tree_sitter_al failed to initialize: {exc}" + ) + return _extract_al_tree_sitter(path, source, tree) \ No newline at end of file diff --git a/pyproject.toml b/pyproject.toml index 15ea9dd57c..9dfbe68187 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -80,6 +80,9 @@ sql = ["tree-sitter-sql"] # absent (#781), so this stays optional. Unlike tree-sitter-dm below, it ships # prebuilt wheels for every platform (win/macOS/Linux), so no C toolchain needed. pascal = ["tree-sitter-pascal"] +# Current tree-sitter-al wheels require Python 3.12+, while graphify supports +# Python 3.10+. The extractor provides a structural fallback on older runtimes. +al = ["tree-sitter-al>=4,<5; python_version >= '3.12'"] # tree-sitter-dm (BYOND DreamMaker) ships only a Windows wheel, so on Linux/Mac it # must compile from source (needs a C toolchain + python3-dev). Keeping it optional # avoids breaking the default `uv tool install graphifyy` for everyone (#1104). @@ -91,7 +94,7 @@ ocaml = ["tree-sitter-ocaml"] # tree-sitter-commonlisp ships prebuilt abi3 wheels for every platform; optional # because Common Lisp is a niche corpus language. commonlisp = ["tree-sitter-commonlisp"] -all = ["mcp>=1,<3", "starlette>=1.3.1,<2", "neo4j", "falkordb", "pypdf>=6.12.0", "markdownify", "watchdog", "graspologic; python_version < '3.13'", "python-docx", "openpyxl", "faster-whisper; python_version >= '3.11'", "yt-dlp>=2026.6.9", "matplotlib", "numpy>=2.0; python_version >= '3.13'", "openai", "tiktoken", "boto3", "anthropic", "tree-sitter-sql", "jieba", "tree-sitter-dm", "tree-sitter-hcl", "tree-sitter-pascal", "tree-sitter-ocaml", "tree-sitter-commonlisp"] +all = ["mcp>=1,<3", "starlette>=1.3.1,<2", "neo4j", "falkordb", "pypdf>=6.12.0", "markdownify", "watchdog", "graspologic; python_version < '3.13'", "python-docx", "openpyxl", "faster-whisper; python_version >= '3.11'", "yt-dlp>=2026.6.9", "matplotlib", "numpy>=2.0; python_version >= '3.13'", "openai", "tiktoken", "boto3", "anthropic", "tree-sitter-sql", "jieba", "tree-sitter-dm", "tree-sitter-hcl", "tree-sitter-pascal", "tree-sitter-al>=4,<5; python_version >= '3.12'", "tree-sitter-ocaml", "tree-sitter-commonlisp"] [project.scripts] graphify = "graphify.__main__:main" diff --git a/tests/fixtures/sample.al b/tests/fixtures/sample.al new file mode 100644 index 0000000000..ea2353eeb4 --- /dev/null +++ b/tests/fixtures/sample.al @@ -0,0 +1,27 @@ +namespace Acme.Comments; + +using Acme.Shared; + +table 75000 "Comment Entry" +{ + trigger OnInsert() + begin + Initialize(); + end; + + [IntegrationEvent(false, false)] + local procedure Initialize(): Boolean + begin + // Braces in comments must not terminate the object: } + exit('{ready}'); + end; +} + +#if TEST +tableextension 75001 "Customer Comments" extends Customer +{ + procedure AddComment(CommentText: Text) + begin + end; +} +#endif \ No newline at end of file diff --git a/tests/fixtures/semantic.al b/tests/fixtures/semantic.al new file mode 100644 index 0000000000..c236f6d0c9 --- /dev/null +++ b/tests/fixtures/semantic.al @@ -0,0 +1,75 @@ +namespace Example.App; + +using Example.Shared; + +interface "IWorker" +{ + procedure Run(Target: Record "Work Item"): Boolean; +} + +enum 75100 "Work Kind" implements "IWorker" +{ + value(0; Standard) + { + Implementation = "IWorker" = "Worker Impl"; + } +} + +enumextension 75101 "More Work Kinds" extends "Work Kind" { } + +table 75102 "Work Item" +{ + fields + { + field(1; "Entry No."; Integer) + { + trigger OnValidate() + begin + end; + } + field(2; Description; Text[100]) + { + trigger OnValidate() + begin + end; + } + } +} + +tableextension 75103 "Work Item Ext" extends "Work Item" { } +page 75104 "Work Items" { } +pageextension 75105 "Work Items Ext" extends "Work Items" { } +report 75106 "Work Report" { } +reportextension 75107 "Work Report Ext" extends "Work Report" { } +query 75108 "Work Query" { } +xmlport 75109 "Work Export" { } +permissionset 75112 "Work Permissions" +{ + Assignable = true; + Permissions = tabledata "Work Item" = R; +} +permissionsetextension 75113 "Extra Work Permissions" extends "Work Permissions" { } + +codeunit 75110 "Worker Impl" implements "IWorker" +{ + [IntegrationEvent(false, false)] + local procedure OnWorked() + begin + end; + + procedure Run(Target: Record "Work Item"): Boolean + var + OtherWorker: Codeunit "Worker Impl"; + begin + OtherWorker.OnWorked(); + exit(true); + end; +} + +codeunit 75111 "Worker Subscriber" +{ + [EventSubscriber(ObjectType::Codeunit, Codeunit::"Worker Impl", 'OnWorked', '', false, false)] + local procedure HandleWorked() + begin + end; +} \ No newline at end of file diff --git a/tests/fixtures/special_identifiers.al b/tests/fixtures/special_identifiers.al new file mode 100644 index 0000000000..d272fe0aba --- /dev/null +++ b/tests/fixtures/special_identifiers.al @@ -0,0 +1,80 @@ +table 70210 "Übernahme-Plan (Nord & Süd)" +{ + fields + { + field(1; "Externe Nr. (Alt)"; Code[20]) + { + trigger OnValidate() + begin + end; + } + field(2; "Prüfstatus & Hinweis"; Text[50]) + { + } + } + + procedure "Setze Prüfstatus"(Value: Text) + begin + end; +} + +page 70211 "Planübersicht (Täglich)" +{ + actions + { + area(processing) + { + group("Tägliche Auswahl") + { + action("Auswahl & starten") + { + trigger OnAction() + begin + "Prüfe & Starte (Auswahl)"(); + end; + } + } + group("Spätere Auswahl") + { + action("Auswahl & starten") + { + trigger OnAction() + begin + end; + } + } + } + } + + procedure "Prüfe & Starte (Auswahl)"() + var + "Gewählter Plan": Record "Übernahme-Plan (Nord & Süd)"; + begin + "Gewählter Plan"."Setze Prüfstatus"('Bereit'); + end; +} + +report 70212 "Prüfliste (Regionen)" +{ + dataset + { + dataitem("Nördliche Auswahl"; Customer) + { + trigger OnAfterGetRecord() + begin + "Sammle Ergebnis"(); + end; + } + dataitem("Südliche Auswahl"; Vendor) + { + trigger OnAfterGetRecord() + begin + "Sammle Ergebnis"(); + end; + } + } + + procedure "Sammle Ergebnis"() + begin + end; +} diff --git a/tests/test_al.py b/tests/test_al.py new file mode 100644 index 0000000000..eb664fac74 --- /dev/null +++ b/tests/test_al.py @@ -0,0 +1,793 @@ +from pathlib import Path +import builtins +import importlib.util +import sys +from types import SimpleNamespace + +import pytest + +from graphify.detect import CODE_EXTENSIONS, FileType, classify_file +from graphify.extract import _get_extractor, extract +from graphify.al_resolution import _same_source, _unique_member_id +from graphify.extractors.al import ( + _extract_al_fallback, + _mask_al_comments_and_strings, + _matching_brace, + extract_al, +) + + +def test_al_extension_is_detected_case_insensitively(): + assert ".al" in CODE_EXTENSIONS + assert classify_file(Path("Comment.Codeunit.al")) == FileType.CODE + assert classify_file(Path("Comment.Codeunit.AL")) == FileType.CODE + + +def test_al_extension_dispatches_to_al_extractor(): + assert _get_extractor(Path("Comment.Codeunit.al")) is extract_al + assert _get_extractor(Path("Comment.Codeunit.AL")) is extract_al + + +def test_al_source_matching_uses_complete_path_components(): + assert _same_source("app/foo.al", "C:/repo/app/foo.al") + assert _same_source("APP\\FOO.AL", "c:/repo/app/foo.al") + assert not _same_source("app/foo.al", "C:/repo/myapp/foo.al") + assert not _same_source("foo.al", "C:/repo/notfoo.al") + + +def test_al_unique_member_id_deduplicates_string_ids_without_selecting_ambiguity(): + assert _unique_member_id(["member", "member"]) == "member" + assert _unique_member_id(["first", "second"]) is None + assert _unique_member_id([]) is None + + +def test_al_mask_preserves_offsets_and_newlines_across_lexical_states(): + source = "code // comment\nnext /* block\ncomment */ more 'it''s' end" + + masked = _mask_al_comments_and_strings(source) + + assert len(masked) == len(source) + assert [index for index, char in enumerate(masked) if char == "\n"] == [ + index for index, char in enumerate(source) if char == "\n" + ] + assert masked.replace(" ", "") == "code\nnext\nmoreend" + + +def test_al_mask_preserves_comment_markers_inside_quoted_identifiers(): + source = ( + 'codeunit 1 "Name // Part" { } // comment\n' + 'codeunit 2 "Name /* Part" { }\n' + 'codeunit 3 "A""//""B" { }\n' + ) + + masked = _mask_al_comments_and_strings(source) + + assert '"Name // Part"' in masked + assert '"Name /* Part"' in masked + assert '"A""//""B"' in masked + assert "// comment" not in masked + + +def test_al_mask_preserves_every_quoted_identifier_character(): + identifiers = ['"Name // Part"', '"Name /* Part"', '"A""//""B"'] + source = " ".join(identifiers) + " // trailing comment" + + masked = _mask_al_comments_and_strings(source) + + for identifier in identifiers: + start = source.index(identifier) + assert masked[start:start + len(identifier)] == identifier + + +def test_al_fallback_preserves_comment_markers_inside_quoted_identifiers(): + result = _extract_al_fallback( + Path("quoted.al"), + 'codeunit 1 "Name // Part" ' + '{ procedure "Run /* Now"() begin end; }', + ) + + labels = {node["label"] for node in result["nodes"]} + + assert "Name // Part" in labels + assert "Run /* Now()" in labels + + +def test_al_matching_brace_uses_masked_comments_and_strings(): + source = "{ value := '{'; /* } */ nested { } } trailing" + masked = _mask_al_comments_and_strings(source) + + closing = _matching_brace(masked, 0) + + assert closing == source.index("} trailing") + + +def test_al_missing_parser_reports_optional_extra(tmp_path, capsys, monkeypatch): + monkeypatch.setitem(sys.modules, "tree_sitter_al", None) + source = tmp_path / "Comment.Codeunit.al" + source.write_text('codeunit 75000 "Comment Mgt." { }', encoding="utf-8") + + result = extract([source], cache_root=tmp_path) + err = capsys.readouterr().err + + assert "used fallback extraction" in err + assert "tree_sitter_al not installed" in err + # The published distribution is intentionally named graphifyy. + assert 'graphifyy[al]' in err + assert result["failed_sources"] == [] + assert any(node.get("object_kind") == "codeunit" for node in result["nodes"]) + + +def test_al_missing_tree_sitter_core_uses_fallback(tmp_path, monkeypatch): + source = tmp_path / "Comment.Codeunit.al" + source.write_text('codeunit 75000 "Comment Mgt." { }', encoding="utf-8") + original_import = builtins.__import__ + + def broken_import(name, *args, **kwargs): + if name == "tree_sitter": + raise ImportError("core unavailable") + return original_import(name, *args, **kwargs) + + monkeypatch.setattr(builtins, "__import__", broken_import) + + result = extract_al(source) + + assert "tree_sitter failed to load" in result.get("dependency_warning", "") + assert any(node.get("object_kind") == "codeunit" for node in result["nodes"]) + + +def test_al_parser_load_failure_uses_fallback(tmp_path, monkeypatch): + source = tmp_path / "Comment.Codeunit.al" + source.write_text('codeunit 75000 "Comment Mgt." { }', encoding="utf-8") + original_import = builtins.__import__ + original_find_spec = importlib.util.find_spec + + def broken_import(name, *args, **kwargs): + if name == "tree_sitter_al": + raise ImportError("incompatible AL parser binary") + return original_import(name, *args, **kwargs) + + def installed_spec(name, *args, **kwargs): + if name == "tree_sitter_al": + return object() + return original_find_spec(name, *args, **kwargs) + + monkeypatch.setattr(builtins, "__import__", broken_import) + monkeypatch.setattr(importlib.util, "find_spec", installed_spec) + + result = extract_al(source) + warning = result.get("dependency_warning") or "" + assert "tree_sitter_al failed to load" in warning + assert "incompatible AL parser binary" in warning + assert not result.get("error") + assert any(node.get("object_kind") == "codeunit" for node in result["nodes"]) + + +def test_al_parser_initialization_failure_uses_fallback(tmp_path, monkeypatch): + source = tmp_path / "Comment.Codeunit.al" + source.write_text('codeunit 75000 "Comment Mgt." { }', encoding="utf-8") + + class BrokenLanguage: + def __init__(self, *_args, **_kwargs): + raise TypeError("incompatible language capsule") + + monkeypatch.setitem( + sys.modules, + "tree_sitter", + SimpleNamespace(Language=BrokenLanguage, Parser=object), + ) + monkeypatch.setitem( + sys.modules, + "tree_sitter_al", + SimpleNamespace(language=lambda: object()), + ) + + result = extract_al(source) + + assert "failed to initialize" in result.get("dependency_warning", "") + assert not result.get("error") + assert any(node.get("object_kind") == "codeunit" for node in result["nodes"]) + + +def test_al_fallback_extracts_objects_procedures_and_triggers(monkeypatch): + monkeypatch.setitem(sys.modules, "tree_sitter_al", None) + fixture = Path(__file__).parent / "fixtures" / "sample.al" + + result = extract_al(fixture) + labels = {node["label"] for node in result["nodes"]} + objects = {node["label"]: node for node in result["nodes"] if node.get("object_kind")} + + assert {"Comment Entry", "Customer Comments"} <= labels + assert {"OnInsert()", "Initialize()", "AddComment()"} <= labels + assert objects["Comment Entry"]["object_id"] == "75000" + assert objects["Comment Entry"]["qualified_name"] == "Acme.Comments.Comment Entry" + assert objects["Customer Comments"]["object_kind"] == "tableextension" + assert all(node.get("extraction_tier") == "fallback" for node in result["nodes"]) + assert {edge["relation"] for edge in result["edges"]} == {"contains"} + + +def test_al_fallback_extracts_permission_sets(): + result = _extract_al_fallback( + Path("Sample.permissionset.al"), + 'permissionset 70000 "Sample Admin"\n{\n}\n', + ) + + permission_set = next( + node for node in result["nodes"] if node.get("object_kind") == "permissionset" + ) + assert permission_set["label"] == "Sample Admin" + assert permission_set["object_id"] == "70000" + + +def test_al_fallback_extracts_permission_set_extensions(): + result = _extract_al_fallback( + Path("Sample.permissionsetextension.al"), + 'permissionsetextension 70001 "Extra Sample Rights" ' + 'extends "Sample Rights"\n{\n}\n', + ) + + extension = next( + node + for node in result["nodes"] + if node.get("object_kind") == "permissionsetextension" + ) + assert extension["label"] == "Extra Sample Rights" + assert extension["object_id"] == "70001" + + +def test_al_fallback_extracts_controladdins(): + result = _extract_al_fallback( + Path("Sample.ControlAddin.al"), + "controladdin SampleControl\n" + "{\n" + " procedure Run(Value: Text);\n" + "}\n", + ) + + controladdin = next( + node for node in result["nodes"] + if node.get("object_kind") == "controladdin" + ) + + assert controladdin["label"] == "SampleControl" + assert "Run()" in {node["label"] for node in result["nodes"]} + + +def test_al_fallback_preserves_spelling_and_casefolds_lookup(monkeypatch, tmp_path): + monkeypatch.setitem(sys.modules, "tree_sitter_al", None) + source = tmp_path / "Mixed.AL" + source.write_text( + 'codeunit 75002 "Mixed ""Case""" { local procedure DoWork() begin end; }', + encoding="utf-8", + ) + + result = extract_al(source) + object_node = next(node for node in result["nodes"] if node.get("object_kind")) + callable_node = next(node for node in result["nodes"] if node.get("member_kind")) + + assert object_node["label"] == 'Mixed "Case"' + assert object_node["lookup_key"] == 'mixed "case"' + assert callable_node["label"] == "DoWork()" + assert callable_node["lookup_key"] == "dowork" + + +def test_al_fallback_accepts_utf8_bom(monkeypatch, tmp_path): + monkeypatch.setitem(sys.modules, "tree_sitter_al", None) + source = tmp_path / "Bom.Codeunit.al" + source.write_text( + '\ufeffcodeunit 70220 "Posting Sample"\n' + "{\n" + " procedure Execute()\n" + " begin\n" + " end;\n" + "}\n", + encoding="utf-8", + ) + + result = extract_al(source) + labels = {node["label"] for node in result["nodes"]} + + assert "Posting Sample" in labels + assert "Execute()" in labels + + +def test_al_fallback_preserves_quoted_special_identifiers(monkeypatch): + monkeypatch.setitem(sys.modules, "tree_sitter_al", None) + fixture = Path(__file__).parent / "fixtures" / "special_identifiers.al" + + result = extract_al(fixture) + labels = {node["label"] for node in result["nodes"]} + + assert "Übernahme-Plan (Nord & Süd)" in labels + assert "Planübersicht (Täglich)" in labels + assert "Setze Prüfstatus()" in labels + assert "Prüfe & Starte (Auswahl)()" in labels + assert "Prüfliste (Regionen)" in labels + assert "Sammle Ergebnis()" in labels + assert len([node for node in result["nodes"] if node["label"] == "OnAction()"]) == 2 + assert len( + [node for node in result["nodes"] if node["label"] == "OnAfterGetRecord()"] + ) == 2 + + +def test_al_fallback_preserves_duplicate_triggers(monkeypatch): + monkeypatch.setitem(sys.modules, "tree_sitter_al", None) + fixture = Path(__file__).parent / "fixtures" / "semantic.al" + + result = extract_al(fixture) + triggers = [node for node in result["nodes"] if node["label"] == "OnValidate()"] + trigger_ids = {node["id"] for node in triggers} + trigger_edges = [ + edge for edge in result["edges"] + if edge["relation"] == "contains" and edge["target"] in trigger_ids + ] + + assert len(triggers) == 2 + assert len(trigger_ids) == 2 + assert len(trigger_edges) == 2 + + +def test_al_fallback_reports_read_errors(monkeypatch, tmp_path): + monkeypatch.setitem(sys.modules, "tree_sitter_al", None) + result = extract_al(tmp_path / "missing.al") + assert result["nodes"] == [] + assert result["edges"] == [] + assert result.get("error") + + +def test_al_tree_sitter_extracts_supported_objects_and_members(): + pytest.importorskip("tree_sitter_al") + fixture = Path(__file__).parent / "fixtures" / "semantic.al" + + result = extract_al(fixture) + object_nodes = [node for node in result["nodes"] if node.get("object_kind")] + kinds = {node["object_kind"] for node in object_nodes} + member_kinds = {node.get("member_kind") for node in result["nodes"]} + + assert kinds == { + "codeunit", "table", "tableextension", "page", "pageextension", + "enum", "enumextension", "interface", "report", "reportextension", + "query", "xmlport", "permissionset", "permissionsetextension", + } + assert {"procedure", "field", "enum_value"} <= member_kinds + on_validate = [node for node in result["nodes"] if node["label"] == "OnValidate()"] + assert len(on_validate) == 2 + assert len({node["id"] for node in on_validate}) == 2 + assert all(node["extraction_tier"] == "tree_sitter" for node in result["nodes"]) + assert not result.get("syntax_errors") + + +def test_al_tree_sitter_resolves_quoted_special_identifiers(): + pytest.importorskip("tree_sitter_al") + fixture = Path(__file__).parent / "fixtures" / "special_identifiers.al" + + result = extract([fixture], cache_root=fixture.parent) + nodes = {node["label"]: node for node in result["nodes"]} + action_triggers = [node for node in result["nodes"] if node["label"] == "OnAction()"] + dataitem_triggers = [ + node for node in result["nodes"] if node["label"] == "OnAfterGetRecord()" + ] + relations = { + (edge["source"], edge["target"], edge["relation"]) + for edge in result["edges"] + } + + assert nodes["Übernahme-Plan (Nord & Süd)"]["lookup_key"] == ( + "übernahme-plan (nord & süd)" + ) + assert nodes["Externe Nr. (Alt)"]["member_kind"] == "field" + assert nodes["Prüfe & Starte (Auswahl)()"]["lookup_key"] == ( + "prüfe & starte (auswahl)" + ) + assert len(action_triggers) == 2 + assert len({node["id"] for node in action_triggers}) == 2 + assert len(dataitem_triggers) == 2 + assert len({node["id"] for node in dataitem_triggers}) == 2 + assert any( + (trigger["id"], nodes["Prüfe & Starte (Auswahl)()"]["id"], "calls") + in relations + for trigger in action_triggers + ) + assert ( + nodes["Prüfe & Starte (Auswahl)()"]["id"], + nodes["Setze Prüfstatus()"]["id"], + "calls", + ) in relations + assert all( + (trigger["id"], nodes["Sammle Ergebnis()"]["id"], "calls") in relations + for trigger in dataitem_triggers + ) + + +def test_al_tree_sitter_preserves_callable_and_field_metadata(): + pytest.importorskip("tree_sitter_al") + fixture = Path(__file__).parent / "fixtures" / "semantic.al" + + result = extract_al(fixture) + run = next(node for node in result["nodes"] if node["label"] == "Run()" and node.get("_callable")) + field = next(node for node in result["nodes"] if node.get("member_kind") == "field") + publisher = next(node for node in result["nodes"] if node["label"] == "OnWorked()") + + assert run["parameters"][0]["name"] == "Target" + assert run["return_type"] == "Boolean" + assert field["member_id"] == "1" + assert field["data_type"] == "Integer" + assert publisher["visibility"] == "local" + assert publisher["attributes"][0]["name"] == "IntegrationEvent" + + +def test_al_tree_sitter_collects_resolution_facts_without_error_nodes(): + pytest.importorskip("tree_sitter_al") + fixture = Path(__file__).parent / "fixtures" / "semantic.al" + + result = extract_al(fixture) + facts = result["al_facts"] + + assert facts["namespace"] == "Example.App" + assert facts["usings"] == ["Example.Shared"] + assert any(item["base"] == "Work Item" for item in facts["objects"]) + assert any(item["interfaces"] == ["IWorker"] for item in facts["objects"]) + assert any(call["receiver_type"] == "Worker Impl" for call in facts["calls"]) + assert facts["event_publishers"][0]["name"] == "OnWorked" + assert facts["enum_mappings"] == [{ + "source": next(node["id"] for node in result["nodes"] if node["label"] == "Standard"), + "interface": "IWorker", + "implementation": "Worker Impl", + "line": 14, + }] + assert not any(node.get("label") == "ERROR" for node in result["nodes"]) + + +def test_al_resolver_emits_language_relationships(tmp_path): + pytest.importorskip("tree_sitter_al") + source = tmp_path / "semantic.al" + source.write_text( + (Path(__file__).parent / "fixtures" / "semantic.al").read_text(encoding="utf-8"), + encoding="utf-8", + ) + + result = extract([source], cache_root=tmp_path) + nodes = {node["label"]: node["id"] for node in result["nodes"]} + relations = { + (edge["source"], edge["target"], edge["relation"], edge.get("context")) + for edge in result["edges"] + } + + assert (nodes["Work Item Ext"], nodes["Work Item"], "extends", "extension") in relations + assert ( + nodes["Extra Work Permissions"], + nodes["Work Permissions"], + "extends", + "extension", + ) in relations + assert (nodes["Worker Impl"], nodes["IWorker"], "implements", "interface") in relations + assert (nodes["Standard"], nodes["IWorker"], "implements", "enum_implementation") in relations + assert (nodes["Standard"], nodes["Worker Impl"], "references", "enum_implementation") in relations + assert (nodes["HandleWorked()"], nodes["OnWorked()"], "references", "event_subscription") in relations + assert any( + target == nodes["OnWorked()"] and relation == "calls" + for _, target, relation, _ in relations + ) + + +def test_al_resolver_preserves_all_implemented_interfaces(tmp_path): + pytest.importorskip("tree_sitter_al") + source = tmp_path / "interfaces.al" + source.write_text( + "interface FirstContract { }\n" + "interface SecondContract { }\n" + "codeunit 1 Worker implements FirstContract, SecondContract { }\n", + encoding="utf-8", + ) + + result = extract([source], cache_root=tmp_path) + nodes = {node["label"]: node["id"] for node in result["nodes"]} + implemented = { + edge["target"] + for edge in result["edges"] + if edge["source"] == nodes["Worker"] + and edge["relation"] == "implements" + } + + assert implemented == {nodes["FirstContract"], nodes["SecondContract"]} + + +def test_al_resolves_usercontrols_and_controladdin_calls(tmp_path): + pytest.importorskip("tree_sitter_al") + source = tmp_path / "controls.al" + source.write_text( + "controladdin DemoAddIn\n" + "{\n" + " procedure Run(Value: Text);\n" + " event OnRaised(Value: Text);\n" + "}\n" + "page 1 DemoPage\n" + "{\n" + " layout\n" + " {\n" + " area(Content)\n" + " {\n" + " usercontrol(FirstHost; DemoAddIn)\n" + " {\n" + " trigger OnRaised(Value: Text)\n" + " begin\n" + " CurrPage.FirstHost.Run(Value);\n" + " end;\n" + " }\n" + " usercontrol(SecondHost; DemoAddIn)\n" + " {\n" + " trigger OnRaised(Value: Text)\n" + " begin\n" + " end;\n" + " }\n" + " }\n" + " }\n" + "}\n", + encoding="utf-8", + ) + + result = extract([source], cache_root=tmp_path) + controladdin = next( + node for node in result["nodes"] + if node.get("object_kind") == "controladdin" + ) + usercontrols = [ + node for node in result["nodes"] + if node.get("member_kind") == "usercontrol" + ] + triggers = [ + node for node in result["nodes"] + if node["label"] == "OnRaised()" and node.get("member_kind") == "trigger" + ] + run = next(node for node in result["nodes"] if node["label"] == "Run()") + event = next( + node for node in result["nodes"] + if node["label"] == "OnRaised()" and node.get("member_kind") == "event" + ) + relations = { + (edge["source"], edge["target"], edge["relation"], edge.get("context")) + for edge in result["edges"] + } + + assert {node["label"] for node in usercontrols} == {"FirstHost", "SecondHost"} + assert all(node["data_type"] == "DemoAddIn" for node in usercontrols) + assert len({node["id"] for node in triggers}) == 2 + assert all( + (node["id"], controladdin["id"], "references", "type") in relations + for node in usercontrols + ) + assert any( + (trigger["id"], run["id"], "calls", "call") in relations + for trigger in triggers + ) + assert all( + ( + trigger["id"], + event["id"], + "references", + "control_addin_event", + ) in relations + for trigger in triggers + ) + + +def test_al_resolver_is_case_insensitive_and_avoids_ambiguous_targets(tmp_path): + pytest.importorskip("tree_sitter_al") + first = tmp_path / "first.al" + second = tmp_path / "second.al" + caller = tmp_path / "caller.al" + first.write_text('namespace One; codeunit 1 Worker { procedure Run() begin end; }', encoding="utf-8") + second.write_text('namespace Two; codeunit 2 WORKER { procedure Run() begin end; }', encoding="utf-8") + caller.write_text( + 'codeunit 3 Caller { procedure Start() var W: Codeunit worker; begin W.Run(); end; }', + encoding="utf-8", + ) + + result = extract([first, second, caller], cache_root=tmp_path) + start = next(node["id"] for node in result["nodes"] if node["label"] == "Start()") + assert not any(edge["source"] == start and edge["relation"] == "calls" for edge in result["edges"]) + + +def test_al_resolver_preserves_and_resolves_procedure_overloads(tmp_path): + pytest.importorskip("tree_sitter_al") + source = tmp_path / "overloads.al" + source.write_text( + '''codeunit 1 Worker +{ + procedure Start() + begin + Run(1); + end; + + local procedure Run() + begin + end; + + local procedure Run(Value: Integer) + begin + end; +}''', + encoding="utf-8", + ) + + result = extract([source], cache_root=tmp_path) + run_nodes = [node for node in result["nodes"] if node["label"] == "Run()"] + start = next(node for node in result["nodes"] if node["label"] == "Start()") + one_parameter = next(node for node in run_nodes if len(node["parameters"]) == 1) + + assert len(run_nodes) == 2 + assert len({node["id"] for node in run_nodes}) == 2 + assert any( + edge["source"] == start["id"] + and edge["target"] == one_parameter["id"] + and edge["relation"] == "calls" + for edge in result["edges"] + ) + + +def test_al_resolver_uses_namespace_imports_and_manifest_context(tmp_path): + pytest.importorskip("tree_sitter_al") + (tmp_path / "app.json").write_text( + '{"id":"app-id","name":"Example App","dependencies":[]}', encoding="utf-8" + ) + worker = tmp_path / "worker.al" + caller = tmp_path / "caller.al" + worker.write_text( + 'namespace Shared; codeunit 1 Worker { procedure Run() begin end; }', encoding="utf-8" + ) + caller.write_text( + 'namespace Main; using Shared; codeunit 2 Caller { procedure Start() var W: Codeunit Worker; begin W.run(); end; }', + encoding="utf-8", + ) + + result = extract([worker, caller], cache_root=tmp_path) + nodes = {node["label"]: node for node in result["nodes"]} + assert nodes["Worker"]["application_id"] == "app-id" + assert nodes["Caller"]["application_name"] == "Example App" + assert any( + edge["source"] == nodes["Start()"]["id"] + and edge["target"] == nodes["Run()"]["id"] + and edge["relation"] == "calls" + for edge in result["edges"] + ) + + +def test_al_resolution_is_independent_of_file_order(tmp_path): + pytest.importorskip("tree_sitter_al") + worker = tmp_path / "worker.al" + caller = tmp_path / "caller.al" + worker.write_text( + "namespace Shared; codeunit 1 Worker { procedure Run() begin end; }", + encoding="utf-8", + ) + caller.write_text( + "namespace Main; using Shared; codeunit 2 Caller " + "{ procedure Start() var W: Codeunit Worker; begin W.Run(); end; }", + encoding="utf-8", + ) + + relationships = [] + for index, files in enumerate(([worker, caller], [caller, worker])): + result = extract(files, cache_root=tmp_path / f"cache-{index}") + labels = {node["id"]: node["label"] for node in result["nodes"]} + relationships.append({ + ( + labels.get(edge["source"]), + labels.get(edge["target"]), + edge["relation"], + edge.get("context"), + ) + for edge in result["edges"] + if edge["relation"] in {"calls", "references"} + }) + + assert relationships[0] == relationships[1] + assert ("Start()", "Run()", "calls", "call") in relationships[0] + + +def test_al_resolver_connects_test_app_targets_handlers_and_dependency(tmp_path): + pytest.importorskip("tree_sitter_al") + main = tmp_path / "MainApp" + tests = tmp_path / "TestApp" + main.mkdir() + tests.mkdir() + (main / "app.json").write_text( + '{"id":"main-id","name":"Main App","dependencies":[]}', encoding="utf-8" + ) + (tests / "app.json").write_text( + '{"id":"test-id","name":"Test App","dependencies":' + '[{"id":"main-id","name":"Main App"}]}', + encoding="utf-8", + ) + (main / "Card.Page.al").write_text( + 'page 1 "Example Card" { }', encoding="utf-8" + ) + (tests / "CardTest.Codeunit.al").write_text( + '''codeunit 2 "Example Tests" +{ + Subtype = Test; + + [Test] + [HandlerFunctions('ConfirmHandler, SecondHandler')] + procedure OpensCard() + var + Card: TestPage "Example Card"; + begin + Card.OpenView(); + end; + + [ConfirmHandler] + procedure ConfirmHandler(Question: Text; var Reply: Boolean) + begin + Reply := false; + end; + + [ConfirmHandler] + procedure SecondHandler(Question: Text; var Reply: Boolean) + begin + Reply := true; + end; +}''', + encoding="utf-8", + ) + + files = sorted(path for path in tmp_path.rglob("*") if path.is_file()) + result = extract(files, cache_root=tmp_path) + nodes = {node["label"]: node["id"] for node in result["nodes"]} + relations = { + (edge["source"], edge["target"], edge["relation"], edge.get("context")) + for edge in result["edges"] + } + manifests = { + Path(node["source_file"]).parent.name: node["id"] + for node in result["nodes"] + if str(node.get("source_file", "")).endswith("app.json") + and str(node.get("label", "")).endswith("app.json") + } + + assert (nodes["OpensCard()"], nodes["Example Card"], "references", "test_target") in relations + assert ( + nodes["OpensCard()"], nodes["ConfirmHandler()"], "references", "test_handler" + ) in relations + assert ( + nodes["OpensCard()"], nodes["SecondHandler()"], "references", "test_handler" + ) in relations + assert ( + manifests["TestApp"], manifests["MainApp"], "depends_on", "application" + ) in relations + assert not any( + edge["source"] == nodes["OpensCard()"] and edge["relation"] == "calls" + for edge in result["edges"] + ) + + +def test_al_resolver_tolerates_invalid_manifest(tmp_path): + pytest.importorskip("tree_sitter_al") + (tmp_path / "app.json").write_text("not-json", encoding="utf-8") + source = tmp_path / "simple.al" + source.write_text('codeunit 1 Simple { procedure Run() begin end; }', encoding="utf-8") + result = extract([source], cache_root=tmp_path) + assert any(node["label"] == "Simple" for node in result["nodes"]) + + +@pytest.mark.parametrize("manifest", ["[]", '"not-an-object"', "null"]) +def test_al_resolver_tolerates_non_object_manifest(tmp_path, manifest): + pytest.importorskip("tree_sitter_al") + (tmp_path / "app.json").write_text(manifest, encoding="utf-8") + source = tmp_path / "simple.al" + source.write_text('codeunit 1 Simple { procedure Run() begin end; }', encoding="utf-8") + + result = extract([source], cache_root=tmp_path) + + assert any(node["label"] == "Simple" for node in result["nodes"]) + + +def test_al_corpus_continues_after_one_file_fails(tmp_path): + pytest.importorskip("tree_sitter_al") + valid = tmp_path / "valid.al" + missing = tmp_path / "missing.al" + valid.write_text('codeunit 1 Valid { procedure Run() begin end; }', encoding="utf-8") + + result = extract([missing, valid], cache_root=tmp_path) + + assert any(node["label"] == "Valid" for node in result["nodes"]) + assert result["failed_sources"] == [str(missing)] \ No newline at end of file diff --git a/tests/test_extract.py b/tests/test_extract.py index c9790e4ab5..7d230512e9 100644 --- a/tests/test_extract.py +++ b/tests/test_extract.py @@ -3709,6 +3709,61 @@ def _broken_import(name, *args, **kwargs): assert failed == {"schema.sql", "views.sql"} +def test_extract_preserves_fallback_warning_across_same_extension( + tmp_path, capsys, monkeypatch +): + first = tmp_path / "first.al" + second = tmp_path / "second.al" + first.write_text("codeunit 1 First { }", encoding="utf-8") + second.write_text("codeunit 2 Second { }", encoding="utf-8") + + def mixed_extractor(path): + if path == first: + return { + "nodes": [{"id": "first", "label": "First"}], + "edges": [], + "dependency_warning": "tree_sitter_al failed to load", + } + return { + "nodes": [], + "edges": [], + "error": "tree_sitter_al failed to load", + } + + monkeypatch.setitem(_DISPATCH, ".al", mixed_extractor) + + extract([first, second], cache_root=tmp_path) + err = capsys.readouterr().err + + assert "2 .al file(s) used fallback extraction" in err + assert "contributed nothing" not in err + + +def test_extract_preserves_dependency_warning_when_result_also_has_error( + tmp_path, capsys, monkeypatch +): + source = tmp_path / "broken.al" + source.write_text("codeunit 1 Broken { }", encoding="utf-8") + + monkeypatch.setitem( + _DISPATCH, + ".al", + lambda _path: { + "nodes": [{"id": "broken", "label": "Broken"}], + "edges": [], + "error": "post-processing failed", + "dependency_warning": "tree_sitter_al failed to load", + }, + ) + + extract([source], cache_root=tmp_path) + err = capsys.readouterr().err + + assert "1 .al file(s) used fallback extraction" in err + assert "tree_sitter_al failed to load" in err + assert "post-processing failed" not in err + + def test_extract_progress_final_line_uses_consistent_denominator(tmp_path, capsys): # #1693: intermediate progress lines count against uncached_work; the final # "100%" line must NOT switch to total_files (which includes cached hits and diff --git a/uv.lock b/uv.lock index 881314a5b5..327e606db2 100644 --- a/uv.lock +++ b/uv.lock @@ -6,11 +6,14 @@ resolution-markers = [ "python_full_version >= '3.14' and sys_platform == 'emscripten'", "python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32'", "python_full_version == '3.13.*' and sys_platform == 'win32'", - "python_full_version >= '3.11' and python_full_version < '3.13' and sys_platform == 'win32'", + "python_full_version == '3.12.*' and sys_platform == 'win32'", + "python_full_version == '3.11.*' and sys_platform == 'win32'", "python_full_version == '3.13.*' and sys_platform == 'emscripten'", - "python_full_version >= '3.11' and python_full_version < '3.13' and sys_platform == 'emscripten'", + "python_full_version == '3.12.*' and sys_platform == 'emscripten'", + "python_full_version == '3.11.*' and sys_platform == 'emscripten'", "python_full_version == '3.13.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", - "python_full_version >= '3.11' and python_full_version < '3.13' and sys_platform != 'emscripten' and sys_platform != 'win32'", + "python_full_version == '3.12.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", + "python_full_version == '3.11.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", "python_full_version < '3.11'", ] @@ -548,11 +551,14 @@ resolution-markers = [ "python_full_version >= '3.14' and sys_platform == 'emscripten'", "python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32'", "python_full_version == '3.13.*' and sys_platform == 'win32'", - "python_full_version >= '3.11' and python_full_version < '3.13' and sys_platform == 'win32'", + "python_full_version == '3.12.*' and sys_platform == 'win32'", + "python_full_version == '3.11.*' and sys_platform == 'win32'", "python_full_version == '3.13.*' and sys_platform == 'emscripten'", - "python_full_version >= '3.11' and python_full_version < '3.13' and sys_platform == 'emscripten'", + "python_full_version == '3.12.*' and sys_platform == 'emscripten'", + "python_full_version == '3.11.*' and sys_platform == 'emscripten'", "python_full_version == '3.13.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", - "python_full_version >= '3.11' and python_full_version < '3.13' and sys_platform != 'emscripten' and sys_platform != 'win32'", + "python_full_version == '3.12.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", + "python_full_version == '3.11.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", ] dependencies = [ { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' and python_full_version < '3.13'" }, @@ -1127,6 +1133,9 @@ dependencies = [ ] [package.optional-dependencies] +al = [ + { name = "tree-sitter-al", marker = "python_full_version >= '3.12'" }, +] all = [ { name = "anthropic" }, { name = "boto3" }, @@ -1145,6 +1154,7 @@ all = [ { name = "python-docx" }, { name = "starlette" }, { name = "tiktoken" }, + { name = "tree-sitter-al", marker = "python_full_version >= '3.12'" }, { name = "tree-sitter-commonlisp" }, { name = "tree-sitter-dm" }, { name = "tree-sitter-hcl" }, @@ -1303,6 +1313,8 @@ requires-dist = [ { name = "tiktoken", marker = "extra == 'kimi'" }, { name = "tiktoken", marker = "extra == 'openai'" }, { name = "tree-sitter", specifier = ">=0.23.0,<0.26" }, + { name = "tree-sitter-al", marker = "python_full_version >= '3.12' and extra == 'al'", specifier = ">=4,<5" }, + { name = "tree-sitter-al", marker = "python_full_version >= '3.12' and extra == 'all'", specifier = ">=4,<5" }, { name = "tree-sitter-bash", specifier = ">=0.23,<0.27" }, { name = "tree-sitter-c", specifier = ">=0.23,<0.25" }, { name = "tree-sitter-c-sharp", specifier = ">=0.23,<0.25" }, @@ -1345,7 +1357,7 @@ requires-dist = [ { name = "yt-dlp", marker = "extra == 'all'", specifier = ">=2026.6.9" }, { name = "yt-dlp", marker = "extra == 'video'", specifier = ">=2026.6.9" }, ] -provides-extras = ["mcp", "neo4j", "falkordb", "pdf", "watch", "svg", "leiden", "office", "google", "postgres", "video", "kimi", "ollama", "bedrock", "anthropic", "gemini", "openai", "chinese", "sql", "pascal", "dm", "terraform", "ocaml", "commonlisp", "all"] +provides-extras = ["mcp", "neo4j", "falkordb", "pdf", "watch", "svg", "leiden", "office", "google", "postgres", "video", "kimi", "ollama", "bedrock", "anthropic", "gemini", "openai", "chinese", "sql", "pascal", "al", "dm", "terraform", "ocaml", "commonlisp", "all"] [package.metadata.requires-dev] dev = [ @@ -2264,11 +2276,14 @@ resolution-markers = [ "python_full_version >= '3.14' and sys_platform == 'emscripten'", "python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32'", "python_full_version == '3.13.*' and sys_platform == 'win32'", - "python_full_version >= '3.11' and python_full_version < '3.13' and sys_platform == 'win32'", + "python_full_version == '3.12.*' and sys_platform == 'win32'", + "python_full_version == '3.11.*' and sys_platform == 'win32'", "python_full_version == '3.13.*' and sys_platform == 'emscripten'", - "python_full_version >= '3.11' and python_full_version < '3.13' and sys_platform == 'emscripten'", + "python_full_version == '3.12.*' and sys_platform == 'emscripten'", + "python_full_version == '3.11.*' and sys_platform == 'emscripten'", "python_full_version == '3.13.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", - "python_full_version >= '3.11' and python_full_version < '3.13' and sys_platform != 'emscripten' and sys_platform != 'win32'", + "python_full_version == '3.12.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", + "python_full_version == '3.11.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", ] sdist = { url = "https://files.pythonhosted.org/packages/6a/51/63fe664f3908c97be9d2e4f1158eb633317598cfa6e1fc14af5383f17512/networkx-3.6.1.tar.gz", hash = "sha256:26b7c357accc0c8cde558ad486283728b65b6a95d85ee1cd66bafab4c8168509", size = 2517025, upload-time = "2025-12-08T17:02:39.908Z" } wheels = [ @@ -2331,9 +2346,12 @@ name = "numpy" version = "1.26.4" source = { registry = "https://pypi.org/simple" } resolution-markers = [ - "python_full_version >= '3.11' and python_full_version < '3.13' and sys_platform == 'win32'", - "python_full_version >= '3.11' and python_full_version < '3.13' and sys_platform == 'emscripten'", - "python_full_version >= '3.11' and python_full_version < '3.13' and sys_platform != 'emscripten' and sys_platform != 'win32'", + "python_full_version == '3.12.*' and sys_platform == 'win32'", + "python_full_version == '3.11.*' and sys_platform == 'win32'", + "python_full_version == '3.12.*' and sys_platform == 'emscripten'", + "python_full_version == '3.11.*' and sys_platform == 'emscripten'", + "python_full_version == '3.12.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", + "python_full_version == '3.11.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", "python_full_version < '3.11'", ] sdist = { url = "https://files.pythonhosted.org/packages/65/6e/09db70a523a96d25e115e71cc56a6f9031e7b8cd166c1ac8438307c14058/numpy-1.26.4.tar.gz", hash = "sha256:2a02aba9ed12e4ac4eb3ea9421c420301a0c6460d9830d74a9df87efa4912010", size = 15786129, upload-time = "2024-02-06T00:26:44.495Z" } @@ -2607,9 +2625,12 @@ name = "pandas" version = "3.0.3" source = { registry = "https://pypi.org/simple" } resolution-markers = [ - "python_full_version >= '3.11' and python_full_version < '3.13' and sys_platform == 'win32'", - "python_full_version >= '3.11' and python_full_version < '3.13' and sys_platform == 'emscripten'", - "python_full_version >= '3.11' and python_full_version < '3.13' and sys_platform != 'emscripten' and sys_platform != 'win32'", + "python_full_version == '3.12.*' and sys_platform == 'win32'", + "python_full_version == '3.11.*' and sys_platform == 'win32'", + "python_full_version == '3.12.*' and sys_platform == 'emscripten'", + "python_full_version == '3.11.*' and sys_platform == 'emscripten'", + "python_full_version == '3.12.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", + "python_full_version == '3.11.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", ] dependencies = [ { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' and python_full_version < '3.13'" }, @@ -3930,9 +3951,12 @@ name = "scikit-learn" version = "1.8.0" source = { registry = "https://pypi.org/simple" } resolution-markers = [ - "python_full_version >= '3.11' and python_full_version < '3.13' and sys_platform == 'win32'", - "python_full_version >= '3.11' and python_full_version < '3.13' and sys_platform == 'emscripten'", - "python_full_version >= '3.11' and python_full_version < '3.13' and sys_platform != 'emscripten' and sys_platform != 'win32'", + "python_full_version == '3.12.*' and sys_platform == 'win32'", + "python_full_version == '3.11.*' and sys_platform == 'win32'", + "python_full_version == '3.12.*' and sys_platform == 'emscripten'", + "python_full_version == '3.11.*' and sys_platform == 'emscripten'", + "python_full_version == '3.12.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", + "python_full_version == '3.11.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", ] dependencies = [ { name = "joblib", marker = "python_full_version >= '3.11' and python_full_version < '3.13'" }, @@ -4044,9 +4068,12 @@ name = "scipy" version = "1.17.1" source = { registry = "https://pypi.org/simple" } resolution-markers = [ - "python_full_version >= '3.11' and python_full_version < '3.13' and sys_platform == 'win32'", - "python_full_version >= '3.11' and python_full_version < '3.13' and sys_platform == 'emscripten'", - "python_full_version >= '3.11' and python_full_version < '3.13' and sys_platform != 'emscripten' and sys_platform != 'win32'", + "python_full_version == '3.12.*' and sys_platform == 'win32'", + "python_full_version == '3.11.*' and sys_platform == 'win32'", + "python_full_version == '3.12.*' and sys_platform == 'emscripten'", + "python_full_version == '3.11.*' and sys_platform == 'emscripten'", + "python_full_version == '3.12.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", + "python_full_version == '3.11.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", ] dependencies = [ { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' and python_full_version < '3.13'" }, @@ -4493,6 +4520,19 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/a6/6e/e64621037357acb83d912276ffd30a859ef117f9c680f2e3cb955f47c680/tree_sitter-0.25.2-cp314-cp314-win_arm64.whl", hash = "sha256:b8d4429954a3beb3e844e2872610d2a4800ba4eb42bb1990c6a4b1949b18459f", size = 117470, upload-time = "2025-09-25T17:37:58.431Z" }, ] +[[package]] +name = "tree-sitter-al" +version = "4.0.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/19/85/c76fd3cfd578ab2df28ef00c64a48b8b1fd12d7f92a8413900ffa89b7846/tree_sitter_al-4.0.1.tar.gz", hash = "sha256:cb257caf4741a041818b9e1a0b355185c8dc9ddc117410bad0b76d0b504df0ec", size = 1498600, upload-time = "2026-08-12T09:57:53.118Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c0/6c/133ced7b42e29baefaabf1344634da22e026a4521324623cbde308ce1af5/tree_sitter_al-4.0.1-cp312-abi3-macosx_11_0_arm64.whl", hash = "sha256:293f6a9336f4d382c21346b0a47d3daec070a0881bc2ef5ec90bdeb993b509d9", size = 587404, upload-time = "2026-08-12T09:57:46.826Z" }, + { url = "https://files.pythonhosted.org/packages/51/a0/b8e8eb07a496f6d0fbb92087547fd4ae3e3e108d82665b6157c911cee1cb/tree_sitter_al-4.0.1-cp312-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:dd8e3639a7734a3782b9fb3a115ab03f0dff0a1517ea5ab2c9eb2aa71ae4e514", size = 587554, upload-time = "2026-08-12T09:57:48.33Z" }, + { url = "https://files.pythonhosted.org/packages/9b/24/a37dbb76984e8236f7ef82ed335956b071ab2655d3188fd87137acf515ec/tree_sitter_al-4.0.1-cp312-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:3b731a33140f68b13f8cbc34357475e29cd830fc58865aeb10d001b08e16a267", size = 585229, upload-time = "2026-08-12T09:57:49.532Z" }, + { url = "https://files.pythonhosted.org/packages/af/6a/2fd01091bfcdded3ffaab0eecd622dacf2e1003c1d76324e5c233f10b0fc/tree_sitter_al-4.0.1-cp312-abi3-win32.whl", hash = "sha256:b993e473d9a975a44088b96d23b0eb03d79663b995c304b1f7fc4a5a4a456552", size = 508730, upload-time = "2026-08-12T09:57:50.739Z" }, + { url = "https://files.pythonhosted.org/packages/ce/04/2ca63166fc3f0a2dd8577ed7c13f721bf0dea1be02600b7cf7694111eb7b/tree_sitter_al-4.0.1-cp312-abi3-win_amd64.whl", hash = "sha256:7ee072098af777b7981e5f68cf65f8b1148963232b90f1295a0ec8875b7689cc", size = 509693, upload-time = "2026-08-12T09:57:51.935Z" }, +] + [[package]] name = "tree-sitter-bash" version = "0.25.1"