Skip to content

Fix custom layers: MAL v2 Layer.NAME compilation + layer-extensions.yml jar shadowing#13918

Merged
wu-sheng merged 4 commits into
apache:masterfrom
toffentoffen:fix-custom-layers
Jun 19, 2026
Merged

Fix custom layers: MAL v2 Layer.NAME compilation + layer-extensions.yml jar shadowing#13918
wu-sheng merged 4 commits into
apache:masterfrom
toffentoffen:fix-custom-layers

Conversation

@toffentoffen

@toffentoffen toffentoffen commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Fix two independent bugs that break custom layers end-to-end. Closes #13917.

  • Add a unit test to verify that the fix works.
  • Explain briefly why the bug exists and how to fix it.

Custom layers (declared without modifying OAP source — #13856 / #13883) were broken in two independent ways. Either one alone prevents a custom layer from working.

Bug 1 — MAL v2 compiler cannot compile a custom-layer reference

A custom layer has no generated Layer.* static field — it lives only in the name registry — so referencing it in a MAL expression failed v2 code generation. The natural form service(['svc'], Layer.IOT_FLEET) parses as the static-field enumRef (IDENTIFIER DOT IDENTIFIER, exactly like Layer.GENERAL), and codegen emitted it verbatim as a Layer.IOT_FLEET field access — a field that does not exist on Layer — so Javassist failed to compile the generated source.

Fix: MALMethodChainCodegen now lowers every Layer.NAME static-field reference to a runtime Layer.nameOf("NAME") registry lookup, scoped to Layer only (via MALCodegenHelper.LAYER_ENUM_TYPE). Layer is a registry-backed type, not a Java enum, and Layer.nameOf resolves both built-in and custom layers by name; for a built-in this is behavior-preserving, because Layer.nameOf("GENERAL") returns the same instance as the Layer.GENERAL field. The other ENUM_FQCN types (DetectPoint, DownsamplingType, TimeUnit, K8sRetagType) are real Java enums with no nameOf(String) and keep their direct static-field reference. A custom layer is therefore referenced exactly like a built-in one, with no grammar / AST / parser change. docs/.../mal.md is updated to the Layer.NAME form.

Tests (MALClassGeneratorTest): a custom-layer field form lowers to Layer.nameOf("...") and compiles; a built-in field form lowers identically; a non-Layer enum (DetectPoint) keeps its direct field access. All 1350 bundled-rule execution tests pass — every built-in Layer.* reference now routes through nameOf and resolves to the same layer (no rule used a custom layer before, which is why CI never caught it).

Note: an earlier revision of this PR fixed only the explicit Layer.nameOf('NAME') method form via a parser reinterpretation (EnumStaticCallArgument). That was superseded by the field-form lowering above — the field form is the natural, documented-as-built-in syntax and needs no grammar/AST special-casing. Neither form had shipped, so there is no compatibility concern.

Bug 2 — layer-extensions.yml bundled in the jar shadows the operator's config/ copy

layer-extensions.yml (operator-managed, documented as living in config/) was packaged inside skywalking-oap.jar and not shipped to the distribution config/. The launch script (dist-material/bin/oapService.sh:38-41) prepends every oap-libs/*.jar to CLASSPATH, leaving config/ last, so ResourceUtils.read("layer-extensions.yml")getClassLoader().getResource(...) deterministically resolved the empty jar-bundled template (layers: []) and silently ignored the operator's file — custom layers declared there never registered.

Fix: add layer-extensions.yml to the maven-jar-plugin <excludes> (oap-server/server-starter/pom.xml) and to the assembly <includes> (apm-dist/src/main/assembly/binary.xml), so it follows the same exclude-from-jar + copy-to-config/ packaging as every other operator-editable config. Verified by rebuilding skywalking-oap.jar and confirming the file is no longer present.


…ns.yml jar shadowing

Two independent bugs prevented custom layers (apache#13856 / apache#13883) from working end to end.

1. The v2 MAL compiler could not compile an expression that references a
   custom layer via Layer.nameOf('NAME') — the documented form for layers
   that have no generated Layer.* static field. The grammar's enumRef models
   only the static-field form (Layer.GENERAL); Layer.nameOf('X') was
   mis-parsed as a metric named "Layer" with a .nameOf("X") method chain and
   failed code generation with "nameOf(java.lang.String) not found in
   SampleFamily". The parser now reinterprets the EnumType.method('arg')
   shape, keyed on the known enum-type set so real metric.method('x') calls
   are unaffected, and emits a static call on the enum FQCN. The v1 Groovy
   compiler already evaluated this natively.

2. layer-extensions.yml was bundled in skywalking-oap.jar and not shipped to
   the distribution config/ directory. Because the launch script puts
   oap-libs/*.jar ahead of config/ on the classpath, ResourceUtils.read()
   always resolved the empty jar-bundled template (layers: []) and an
   operator's config/layer-extensions.yml was silently shadowed, so custom
   layers declared there never registered. The file now follows the same
   exclude-from-jar + copy-to-config/ packaging as every other
   operator-editable config (application.yml, alarm-settings.yml, ...).

Closes apache#13917

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@wu-sheng wu-sheng added bug Something isn't working and you are sure it's a bug! backend OAP backend related. labels Jun 19, 2026
@wu-sheng wu-sheng added this to the 11.0.0 milestone Jun 19, 2026
wu-sheng and others added 2 commits June 19, 2026 11:38
A custom layer (layerDefinitions: / layer-extensions.yml / LayerExtension SPI)
has no generated Layer.* static field, so referencing it as Layer.IOT_FLEET
failed v2 MAL code generation. The compiler now lowers every Layer.NAME
static-field reference to a runtime Layer.nameOf("NAME") registry lookup,
scoped to Layer only (the other ENUM_FQCN types are real enums and keep their
direct field access). For a built-in layer this is equivalent, since
Layer.nameOf("GENERAL") returns the same instance as Layer.GENERAL, so a custom
layer is referenced exactly like a built-in one.

This replaces the earlier Layer.nameOf('NAME') parser-reinterpretation approach
(EnumStaticCallArgument): neither form had shipped, and the field form needs no
grammar/AST/parser special-casing. Reverts that path in MALExpressionModel /
MALScriptParser and updates docs to the field form.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@wu-sheng wu-sheng changed the title Fix custom layers: MAL v2 Layer.nameOf() compilation + layer-extensions.yml jar shadowing Fix custom layers: MAL v2 Layer.NAME compilation + layer-extensions.yml jar shadowing Jun 19, 2026
@wu-sheng wu-sheng merged commit dcfb070 into apache:master Jun 19, 2026
437 of 439 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend OAP backend related. bug Something isn't working and you are sure it's a bug!

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Custom layers broken in 11.0.0: MAL v2 Layer.nameOf() fails to compile + layer-extensions.yml shadowed by jar copy

2 participants