[GSD-12905] ovms crash on arc iGPU when upgrading from 26.01 to 26.18

[jpm-canonical]

Pre-submission Checklist

• I am using the latest GPU driver version (releases)
• I have searched for similar issues and found none

GPU Hardware

Intel Corporation Lunar Lake [Intel Arc Graphics 130V / 140V] (rev 04)

DRI Devices Information

$ ls -la /dev/dri/by-path/
total 0
drwxr-xr-x 2 root root 100 Jun  2 14:11 .
drwxr-xr-x 3 root root 100 Jun  2 14:11 ..
lrwxrwxrwx+1 root root   8 Jun  2 14:11 pci-0000:00:02.0-card -> ../card1
lrwxrwxrwx 1 root root   8 Jun  2 14:11 pci-0000:00:02.0-platform-simple-framebuffer.0-card -> ../card0
lrwxrwxrwx+1 root root  13 Jun  2 14:11 pci-0000:00:02.0-render -> ../renderD128

GPU Detailed Information (lspci output)

$ sudo lspci -vvv -k -s 0000:00:02.0
00:02.0 VGA compatible controller: Intel Corporation Lunar Lake [Intel Arc Graphics 130V / 140V] (rev 04) (prog-if 00 [VGA controller])
	Subsystem: Dell Device 0cc9
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin ? routed to IRQ 212
	IOMMU group: 4
	Region 0: Memory at 3014000000 (64-bit, prefetchable) [size=16M]
	Region 2: Memory at 2000000000 (64-bit, prefetchable) [size=256M]
	Expansion ROM at 000c0000 [disabled] [size=128K]
	Capabilities: [40] Vendor Specific Information: Intel Capabilities v1
		CapA: Peg60Dis- Peg12Dis- Peg11Dis- Peg10Dis- PeLWUDis- DmiWidth=x4
		      EccDis- ForceEccEn- VTdDis- DmiG2Dis- PegG2Dis- DDRMaxSize=Unlimited
		      1NDis- CDDis- DDPCDis- X2APICEn- PDCDis- IGDis- CDID=0 CRID=0
		      DDROCCAP- OCEn- DDRWrtVrefEn+ DDR3LEn+
		CapB: ImguDis- OCbySSKUCap- OCbySSKUEn- SMTCap- CacheSzCap 0x0
		      SoftBinCap- DDR3MaxFreqWithRef100=Disabled PegG3Dis-
		      PkgTyp- AddGfxEn- AddGfxCap- PegX16Dis- DmiG3Dis- GmmDis-
		      DDR3MaxFreq=2932MHz LPDDR3En-
	Capabilities: [70] Express (v2) Root Complex Integrated Endpoint, IntMsgNum 0
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag+ RBE+ FLReset+ TEE-IO-
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
			 10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
			 AtomicOpsCtl: ReqEn-
			 IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
			 10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
	Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
		Address: 00000000fee00018  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [d0] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Null
	Capabilities: [110 v1] Process Address Space ID (PASID)
		PASIDCap: Exec- Priv-, Max PASID Width: 14
		PASIDCtl: Enable- Exec- Priv-
	Capabilities: [200 v1] Address Translation Service (ATS)
		ATSCap:	Invalidate Queue Depth: 00
		ATSCtl:	Enable+, Smallest Translation Unit: 00
	Capabilities: [420 v1] Physical Resizable BAR
		BAR 2: current size: 256MB, supported: 256MB
	Capabilities: [400 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Kernel driver in use: xe
	Kernel modules: xe

Driver Version

26.18.38308.1

Installed GPU Driver Packages

$ sudo dpkg --list | grep -iE "igc|gmm|opencl|level-zero|fc|level_zero|ocloc|libze"
ii  bpfcc-tools                                0.35.0+ds-1ubuntu2                          all          tools for BPF Compiler Collection (BCC)
ii  clinfo                                     3.0.25.02.14-1build1                        amd64        Query OpenCL system information
ii  intel-opencl-icd                           26.05.37020.3-1                             amd64        Intel graphics compute runtime for OpenCL
ii  intel-opencl-icd-legacy                    24.35.30872.45-1                            amd64        Intel graphics compute runtime for OpenCL -- legacy platforms
ii  libbpfcc:amd64                             0.35.0+ds-1ubuntu2                          amd64        shared library for BPF Compiler Collection (BCC)
ii  libcbor0.10:amd64                          0.10.2-2ubuntu3                             amd64        library for parsing and generating CBOR (RFC 7049)
ii  libdebconfclient0:amd64                    0.280ubuntu1                                amd64        Debian Configuration Management System (C-implementation library)
ii  libfile-fcntllock-perl                     0.22-4ubuntu6                               amd64        Perl module for file locking with fcntl(2)
ii  libigc1:amd64                              1.0.17791.18+1-3                            amd64        Intel graphics compiler for OpenCL -- core libs
ii  libigc2:amd64                              2.28.4-4                                    amd64        Intel graphics compiler for OpenCL -- core libs
ii  libigdfcl1:amd64                           1.0.17791.18+1-3                            amd64        Intel graphics compiler for OpenCL -- OpenCL library
ii  libigdfcl2:amd64                           2.28.4-4                                    amd64        Intel graphics compiler for OpenCL -- OpenCL library
ii  libigdgmm12:amd64                          22.9.0+ds1-1                                amd64        Intel Graphics Memory Management Library -- shared library
ii  libref-array1t64:amd64                     0.6.2-3build1                               amd64        refcounted array for C
ii  libze1:amd64                               1.28.2-2                                    amd64        oneAPI Level Zero -- share libraries
ii  linux-firmware-qlogic                      20260319.git217ca6e4-0ubuntu1               all          Firmware for QLogic SCSI, FC, and IB host bus and Ethernet adapters
ii  ocl-icd-libopencl1:amd64                   2.3.4-1                                     amd64        Generic OpenCL ICD Loader
ii  python3-bpfcc                              0.35.0+ds-1ubuntu2                          all          Python 3 wrappers for BPF Compiler Collection (BCC)
ii  python3-idna                               3.11-1                                      all          Python IDNA2008 (RFC 5891) handling (Python 3)

Driver Installation Details

Host machine is Ubuntu 26.04.

Software and compute runtime are confined inside a snap using Ubuntu 24.04 as base.

PR for reference: canonical/gemma4-snap#48

Linux Distribution

Ubuntu 24.04 LTS

Other Linux Distribution

No response

Kernel Version & Boot Parameters

$ uname -r
7.0.0-22-generic
$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-7.0.0-22-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro quiet splash crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M
$ lsmod | grep -E 'i915|xe'
xe                   4415488  75
drm_gpusvm_helper      57344  1 xe
gpu_sched              69632  1 xe
drm_gpuvm              57344  1 xe
drm_buddy              28672  1 xe
drm_ttm_helper         20480  1 xe
ttm                   135168  2 drm_ttm_helper,xe
drm_exec               12288  2 drm_gpuvm,xe
drm_suballoc_helper    24576  1 xe
drm_display_helper    303104  1 xe
cec                   106496  2 drm_display_helper,xe
i2c_algo_bit           16384  1 xe
intel_vsec             24576  3 intel_pmc_ssram_telemetry,pmt_telemetry,xe
video                  77824  3 dell_wmi,dell_laptop,xe

Actual Behavior

When running OpenVINO Model Server, targeting the iGPU of my laptop, it crashes. This is after updating the compute runtime to 26.18.38308.1. Before I used 26.01.36711.4 on which it did not crash, and ran correctly.

Expected Behavior

Newer compute runtime should not crash.

Reproduction Rate

Always reproduces - 100%

Steps to Reproduce

1. sudo snap install gemma4 --channel latest/edge
2. gemma4 chat runs as expected, using the GPU
3. sudo snap refresh gemma4 --channel latest/edge/pr-48
4. gemma4 chat does not work
5. sudo snap logs -f gemma4 will show openvino model server crashing and restarting

Is this a regression?

• Yes, this is a regression - functionality that previously worked is now broken

Last Known Working Driver Version

26.01.36711.4

First Known Failing Driver Version

26.18.38308.1

API Call Logs

+ ovms --rest_port 8336 --rest_bind_address 127.0.0.1 --model_name gemma4-e4b-it-int4-ov --model_path /tmp/gemma4-e4b-it-int4-ov --pipeline_type VLM --task text_generation --target_device GPU
[2026-06-11 16:40:48.283][1702589][modelmanager][info][modelmanager.cpp:180] Available devices for Open VINO: CPU, GPU
[2026-06-11 16:40:48.283][1702589][serving][info][capimodule.cpp:40] C-APIModule starting
[2026-06-11 16:40:48.283][1702589][serving][info][capimodule.cpp:42] C-APIModule started
[2026-06-11 16:40:48.283][1702589][serving][info][grpcservermodule.cpp:110] GRPCServerModule starting
[2026-06-11 16:40:48.283][1702589][serving][info][grpcservermodule.cpp:114] GRPCServerModule started
[2026-06-11 16:40:48.283][1702589][serving][info][grpcservermodule.cpp:115] Port was not set. GRPC server will not be started.
[2026-06-11 16:40:48.283][1702589][serving][info][httpservermodule.cpp:35] HTTPServerModule starting
[2026-06-11 16:40:48.283][1702589][serving][info][httpservermodule.cpp:39] Will start 8 REST workers
[2026-06-11 16:40:48.284][1702638][serving][info][drogon_http_server.cpp:157] Binding REST server to address: 127.0.0.1:8336
[2026-06-11 16:40:48.334][1702589][serving][info][drogon_http_server.cpp:184] REST server listening on port 8336 with 8 unary threads and 8 streaming threads
[2026-06-11 16:40:48.334][1702589][serving][info][http_server.cpp:249] API key not provided via --api_key_file or API_KEY environment variable. Authentication will be disabled.
[2026-06-11 16:40:48.334][1702589][serving][info][httpservermodule.cpp:52] HTTPServerModule started
[2026-06-11 16:40:48.334][1702589][serving][info][httpservermodule.cpp:53] Started REST server at 127.0.0.1:8336
[2026-06-11 16:40:48.336][1702589][serving][info][server.cpp:461] Graph config created in memory from model_path: /tmp/gemma4-e4b-it-int4-ov
[2026-06-11 16:40:48.336][1702589][serving][info][servablemanagermodule.cpp:51] ServableManagerModule starting
[2026-06-11 16:40:48.337][1702589][serving][info][mediapipegraphdefinition.cpp:101] Graph queue globally disabled via OVMS_GRAPH_QUEUE_OFF=1 for mediapipe: gemma4-e4b-it-int4-ov
[2026-06-11 16:40:48.337][1702589][serving][info][mediapipegraphdefinition.cpp:479] MediapipeGraphDefinition initializing graph nodes
[2026-06-11 16:40:48.337][1702589][modelmanager][info][servable_initializer.cpp:467] Initializing Visual Language Model Legacy servable
[2026-06-11 16:40:53.554][1702589][serving][error][servable_initializer.cpp:88] Error during llm node initialization for models_path: /tmp/gemma4-e4b-it-int4-ov exception: Exception from src/inference/src/cpp/core.cpp:117:
Exception from src/inference/src/dev/plugin.cpp:54:
Check 'false' failed at src/plugins/intel_gpu/src/plugin/program_builder.cpp:168:
[GPU] ProgramBuilder build failed!
Exception from src/plugins/intel_gpu/src/runtime/ocl/ocl_memory.cpp:569:
[GPU] clWaitForEvents, error code: -58 CL_INVALID_EVENT
[2026-06-11 16:40:53.554][1702589][modelmanager][error][servable_initializer.cpp:471] Error during LLM node resources initialization: The LLM Node resource initialization failed
[2026-06-11 16:40:53.554][1702589][serving][error][llm_node_initializer.cpp:60] Failed to process LLM node graph gemma4-e4b-it-int4-ov
[2026-06-11 16:40:53.560][1702589][modelmanager][info][pipelinedefinitionstatus.hpp:59] Mediapipe: gemma4-e4b-it-int4-ov state changed to: LOADING_PRECONDITION_FAILED after handling: ValidationFailedEvent:
[2026-06-11 16:40:53.560][1702589][modelmanager][error][modelmanager.cpp:208] Couldn't start model manager
[2026-06-11 16:40:53.560][1702589][serving][error][servablemanagermodule.cpp:58] ovms::ModelManager::Start() Error: The LLM Node resource initialization failed
[2026-06-11 16:40:53.560][1702589][serving][info][grpcservermodule.cpp:201] GRPCServerModule shutting down
[2026-06-11 16:40:53.560][1702589][serving][info][grpcservermodule.cpp:211] GRPCServerModule shutdown
[2026-06-11 16:40:53.560][1702589][serving][info][httpservermodule.cpp:59] HTTPServerModule shutting down
[2026-06-11 16:40:53.561][1702589][serving][info][httpservermodule.cpp:64] Shutdown HTTP server
[2026-06-11 16:40:53.561][1702589][serving][info][servablemanagermodule.cpp:65] ServableManagerModule shutting down
[2026-06-11 16:40:53.563][1702589][serving][info][servablemanagermodule.cpp:71] ServableManagerModule shutdown
[2026-06-11 16:40:53.563][1702589][serving][info][capimodule.cpp:50] C-APIModule shutting down
[2026-06-11 16:40:53.563][1702589][serving][info][capimodule.cpp:52] C-APIModule shutdown
Error: exit status 1

strace Logs

No response

System Logs / dmesg Output

No response

Backtrace (if crash or hang occurred)

No response

Source Code / Reproducer

No response

Command Line / Application Details

No response

oneAPI Version (if applicable)

No response

Screenshots / Video

No response

Additional Notes

No response

[kgibala]

Hi @jpm-canonical,

Thank you for reporting this issue.

We will isolate this regression internally and post an update here as soon as something meaningful is discovered.

Thank you for your cooperation.

[jchabere]

Hi @jpm-canonical,

Thanks for the detailed report. I tried to reproduce the issue but ran into a couple of problems with the reproducer from the description.

1. The pr-48 channel doesn't exist

Following the steps, the install from latest/edge works fine:

$ sudo snap install gemma4 --channel latest/edge
gemma4 (edge) 2.0.0-beta.4+35da21a from Canonical IoT Labs✓ installed

$ gemma4 chat
Using server at http://127.0.0.1:8336/v3
Type your prompt, then ENTER to submit. CTRL-C to quit.
» 

But the refresh to the PR channel fails because the branch doesn't exist:

$ sudo snap refresh gemma4 --channel latest/edge/pr-48
error: requested a non-existing branch on latest/edge for snap "gemma4": pr-48

Could you confirm the correct channel/branch name, or push the build to a channel we can access?

2. Couldn't reproduce the crash with a newer driver

I also tried swapping in the GPU driver from my host machine, but with a driver newer than the one you flagged as first-failing (26.18.38308.1) the crash is not reproducible on my side.

If you have a script or method to build the gemma4 snap with the driver pulled from the host machine, please share it — that would make it much easier for us to reproduce and continue the analysis.

Thanks!

[jpm-canonical]

I made a new PR on our side, updating to OpenVINO Model Server 2026.2.1 and Intel Compute Runtime 26.22.38646.4.

sudo snap install gemma4 --channel latest/edge/pr-73
sudo snap logs -f gemma4

With this combination OVMS is still crashing with the same error message.

+ ovms --rest_port 8336 --rest_bind_address 127.0.0.1 --model_name gemma4-e4b-it-int4-ov --model_path /tmp/gemma4-e4b-it-int4-ov --pipeline_type VLM --task text_generation --target_device GPU
[2026-06-23 16:27:37.030][811556][modelmanager][info][modelmanager.cpp:180] Available devices for Open VINO: CPU, GPU
[2026-06-23 16:27:37.031][811556][serving][info][capimodule.cpp:40] C-APIModule starting
[2026-06-23 16:27:37.031][811556][serving][info][capimodule.cpp:42] C-APIModule started
[2026-06-23 16:27:37.031][811556][serving][info][grpcservermodule.cpp:110] GRPCServerModule starting
[2026-06-23 16:27:37.031][811556][serving][info][grpcservermodule.cpp:114] GRPCServerModule started
[2026-06-23 16:27:37.031][811556][serving][info][grpcservermodule.cpp:115] Port was not set. GRPC server will not be started.
[2026-06-23 16:27:37.031][811556][serving][info][httpservermodule.cpp:35] HTTPServerModule starting
[2026-06-23 16:27:37.031][811556][serving][info][httpservermodule.cpp:39] Will start 8 REST workers
[2026-06-23 16:27:37.032][811605][serving][info][drogon_http_server.cpp:157] Binding REST server to address: 127.0.0.1:8336
[2026-06-23 16:27:37.082][811556][serving][info][drogon_http_server.cpp:184] REST server listening on port 8336 with 8 unary threads and 8 streaming threads
[2026-06-23 16:27:37.082][811556][serving][info][http_server.cpp:249] API key not provided via --api_key_file or API_KEY environment variable. Authentication will be disabled.
[2026-06-23 16:27:37.082][811556][serving][info][httpservermodule.cpp:52] HTTPServerModule started
[2026-06-23 16:27:37.082][811556][serving][info][httpservermodule.cpp:53] Started REST server at 127.0.0.1:8336
[2026-06-23 16:27:37.084][811556][serving][info][server.cpp:461] Graph config created in memory from model_path: /tmp/gemma4-e4b-it-int4-ov
[2026-06-23 16:27:37.084][811556][serving][info][servablemanagermodule.cpp:51] ServableManagerModule starting
[2026-06-23 16:27:37.084][811556][serving][info][mediapipegraphdefinition.cpp:101] Graph queue globally disabled via OVMS_GRAPH_QUEUE_OFF=1 for mediapipe: gemma4-e4b-it-int4-ov
[2026-06-23 16:27:37.085][811556][serving][info][mediapipegraphdefinition.cpp:479] MediapipeGraphDefinition initializing graph nodes
[2026-06-23 16:27:37.085][811556][modelmanager][info][servable_initializer.cpp:467] Initializing Visual Language Model Legacy servable
[2026-06-23 16:27:40.104][811556][serving][error][servable_initializer.cpp:88] Error during llm node initialization for models_path: /tmp/gemma4-e4b-it-int4-ov exception: Exception from src/inference/src/cpp/core.cpp:117:
Exception from src/inference/src/dev/plugin.cpp:54:
Check 'false' failed at src/plugins/intel_gpu/src/plugin/program_builder.cpp:168:
[GPU] ProgramBuilder build failed!
Exception from src/plugins/intel_gpu/src/runtime/ocl/ocl_memory.cpp:569:
[GPU] clWaitForEvents, error code: -58 CL_INVALID_EVENT
[2026-06-23 16:27:40.104][811556][modelmanager][error][servable_initializer.cpp:471] Error during LLM node resources initialization: The LLM Node resource initialization failed
[2026-06-23 16:27:40.104][811556][serving][error][llm_node_initializer.cpp:60] Failed to process LLM node graph gemma4-e4b-it-int4-ov
[2026-06-23 16:27:40.106][811556][modelmanager][info][pipelinedefinitionstatus.hpp:59] Mediapipe: gemma4-e4b-it-int4-ov state changed to: LOADING_PRECONDITION_FAILED after handling: ValidationFailedEvent:
[2026-06-23 16:27:40.106][811556][modelmanager][error][modelmanager.cpp:208] Couldn't start model manager
[2026-06-23 16:27:40.106][811556][serving][error][servablemanagermodule.cpp:58] ovms::ModelManager::Start() Error: The LLM Node resource initialization failed
[2026-06-23 16:27:40.106][811556][serving][info][grpcservermodule.cpp:201] GRPCServerModule shutting down
[2026-06-23 16:27:40.106][811556][serving][info][grpcservermodule.cpp:211] GRPCServerModule shutdown
[2026-06-23 16:27:40.106][811556][serving][info][httpservermodule.cpp:59] HTTPServerModule shutting down
[2026-06-23 16:27:40.106][811556][serving][info][httpservermodule.cpp:64] Shutdown HTTP server
[2026-06-23 16:27:40.106][811556][serving][info][servablemanagermodule.cpp:65] ServableManagerModule shutting down
[2026-06-23 16:27:40.107][811556][serving][info][servablemanagermodule.cpp:71] ServableManagerModule shutdown
[2026-06-23 16:27:40.107][811556][serving][info][capimodule.cpp:50] C-APIModule shutting down
[2026-06-23 16:27:40.107][811556][serving][info][capimodule.cpp:52] C-APIModule shutdown
Error: exit status 1

The original PR with OVMS v2026.2 and Intel Compute Runtime 26.18.38308.1 has also been built and published. Installing with sudo snap install gemma4 --channel latest/edge/pr-48 should work now, and should show the error in the logs.

If you use sudo snap install gemma4 --channel latest/edge you will get OVMS v2026.2 with Intel Compute Runtime 26.01.36711.4, which works correctly on my side.

[jchabere]

@jpm-canonical Thanks — I was able to reproduce the error now using the pr-73 channel:

sudo snap install gemma4 --channel latest/edge/pr-73
sudo snap logs -f gemma4
Failing case (latest/edge/pr-73):

[2026-06-24 10:29:51.788][29320][serving][error][servable_initializer.cpp:88] Error during llm node initialization for models_path: /tmp/gemma4-e4b-it-int4-ov exception: Exception from src/inference/src/cpp/core.cpp:117:
Exception from src/inference/src/dev/plugin.cpp:54:
Check 'false' failed at src/plugins/intel_gpu/src/plugin/program_builder.cpp:168:
[GPU] ProgramBuilder build failed!
Exception from src/plugins/intel_gpu/src/runtime/ocl/ocl_memory.cpp:569:
[GPU] clWaitForEvents, error code: -58 CL_INVALID_EVENT
[2026-06-24 10:29:51.788][29320][modelmanager][error][servable_initializer.cpp:471] Error during LLM node resources initialization: The LLM Node resource initialization failed
[2026-06-24 10:29:51.788][29320][serving][error][llm_node_initializer.cpp:60] Failed to process LLM node graph gemma4-e4b-it-int4-ov
[2026-06-24 10:29:51.789][29320][modelmanager][info][pipelinedefinitionstatus.hpp:59] Mediapipe: gemma4-e4b-it-int4-ov state changed to: LOADING_PRECONDITION_FAILED after handling: ValidationFailedEvent:
[2026-06-24 10:29:51.789][29320][modelmanager][error][modelmanager.cpp:208] Couldn't start model manager
[2026-06-24 10:29:51.789][29320][serving][error][servablemanagermodule.cpp:58] ovms::ModelManager::Start() Error: The LLM Node resource initialization failed
...
gemma4.server[29053]: Error: exit status 1

Working case (latest/edge):

[2026-06-24 10:35:32.809][69312][modelmanager][info][servable_initializer.cpp:467] Initializing Visual Language Model Legacy servable
[2026-06-24 10:35:48.821][69312][modelmanager][info][pipelinedefinitionstatus.hpp:59] Mediapipe: gemma4-e4b-it-int4-ov state changed to: AVAILABLE after handling: ValidationPassedEvent:
[2026-06-24 10:35:48.822][69312][serving][info][servablemanagermodule.cpp:55] ServableManagerModule started
[2026-06-24 10:35:48.822][69476][modelmanager][info][modelmanager.cpp:1060] Started model manager thread

Question: I'd like to dig deeper using a debug build of the compute runtime. My plan was to point the wget commands in snapcraft.yaml#L119 at our custom (debug) builds. Once I've done that:

What's the recommended way to build and install the snap locally with those modified runtime artifacts (e.g. snapcraft + snap install --dangerous)?

Any pointers on the build/install/verify loop would help us continue the analysis.

[jpm-canonical]

@jchabere I trimmed down the snap on branch update-ovms+intel-cr (pr-73) to only include the Intel related runtimes and models. This reduces the size and speeds up the build.

Prerequisites:

sudo snap install snapcraft --classic

This should automatically install and configure lxd.

Clone and build the branch:

git clone https://github.com/canonical/gemma4-snap.git -b update-ovms+intel-cr
cd gemma4-snap/
./download-models.sh
snapcraft pack -v

Install the locally built snap and components:

sudo snap install --dangerous ./*.snap ./*.comp

Check the logs:

sudo snap logs -f gemma4

Switch to intel-cpu:

sudo gemma4 list-engines
sudo gemma4 use-engine intel-cpu

With intel-cpu chat works:

gemma4 chat

You can modify the file snap/snapcraft.yaml to point to other versions of OVMS or runtime. A simple rebuild using snapcraft pack -v and install should work. Sometimes snapcraft does not detect changes to the source, in which case you can clean the build environment with snapcraft clean before doing a build.

If you want to remove the snap: sudo snap remove gemma4

To do debugging inside the snap sandbox, you can shell into it:

$ sudo gemma4 use-engine intel-gpu
Engine changed to "intel-gpu".
Restart gemma4 to apply the changes? [Y/n] 
$ sudo snap stop gemma4
Stopped.
$ sudo snap run --shell gemma4.server

Inside the sandbox the interesting directories are $SNAP and $SNAP_COMPONENTS. Also check $LD_LIBRARY_PATH.

# ls $SNAP
LICENSE  bin  engines  etc  meta  models  runtimes  usr  var  webui

# ls $SNAP_COMPONENTS
model-e4b-it-int4-ov-1-of-2  model-e4b-it-int4-ov-2-of-2  openvino-model-server

# echo $LD_LIBRARY_PATH
/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/snap/gemma4/x1/usr/lib:/snap/gemma4/x1/usr/lib/x86_64-linux-gnu

Inside the snap you can use modelctl run to run command with the same environment as ovms would see:

# modelctl run env
...
OCL_ICD_VENDORS=/snap/gemma4/x1/etc/OpenCL/vendors
LD_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/snap/gemma4/x1/usr/lib:/snap/gemma4/x1/usr/lib/x86_64-linux-gnu:/snap/gemma4/components/x1/openvino-model-server/lib:/snap/gemma4/components/x1/openvino-model-server/usr/lib/x86_64-linux-gnu:/snap/gemma4/components/x1/openvino-model-server/usr/local/lib:/snap/gemma4/x1/usr/local/lib
SNAP_NAME=gemma4
PATH=/snap/gemma4/x1/usr/sbin:/snap/gemma4/x1/usr/bin:/snap/gemma4/x1/sbin:/snap/gemma4/x1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/gemma4/components/x1/openvino-model-server/bin
MODEL_NAME=gemma4-e4b-it-int4-ov
MODEL_DIR=/tmp/gemma4-e4b-it-int4-ov
MODEL_PATH=/tmp/gemma4-e4b-it-int4-ov

# modelctl run ovms
...

[jchabere]

Hi @jpm-canonical

After analysis and debugging, this turns out not to be a driver regression — it's a problem with how the driver is installed, and the current approach breaks GmmLib compatibility.

Root cause:
GmmLib downgrade caused by glob ordering. The override-build section of the cli-intel-opencl-icd part in snapcraft.yaml downloads both libigdgmm12_22.5.0 (legacy) and libigdgmm12_22.10.0 (latest) into the same working directory, then runs dpkg -i *.deb. The shell glob *.deb expands alphabetically, and because "22.10" sorts before "22.5" in ASCII comparison ('1' < '5'), dpkg installs 22.10.0 first and then overwrites it with 22.5.0 as a downgrade. NEO expects GmmLib 22.10.0 at runtime, so loading the mismatched 22.5.0 produces CL_INVALID_EVENT (-58).

This bug was latent in snapcraft.yaml: it never triggered on 22.9 because single-digit minor versions sort correctly ('5' < '9'). It only surfaced once the minor version reached two digits (22.10).

Second compatibility fix: put the IGC libraries on the loader path. The IGC .debs install libigc.so.2, libigdfcl.so.2 and libiga64.so.2 into /usr/local/lib, which is not on the snap's loader path (only /usr/lib/$ARCH_TRIPLET is). I move them into /usr/lib/$ARCH_TRIPLET so NEO can link them at runtime.

With these two changes I was able to start the server successfully on 26.22.38646.4.

As an interim fix I kept the legacy and latest drivers in the same part, but installed them in an explicit order (two separate directories + two dpkg calls) and ship GmmLib only once, at 22.10.0. Longer term, if the legacy driver is still required, I'd suggest splitting it into two separate build files (or two installation methods) — one for latest and one for legacy — so the two GmmLib versions never share a directory and can't downgrade each other.

Here is the modified snapcraft.yaml:

name: gemma4
base: core24
summary: Inference Snap for Gemma 4
description: See snapcraft.io/gemma4

adopt-info: version

grade: stable
confinement: strict
compression: lzo

assumes:
  - snapd2.68 # for components support

platforms:
  amd64:
  arm64:

slots:
  status:
    interface: content
    source:
      read:
        - &status-share $SNAP_DATA/share/status
  open-webui:
    interface: content
    content: open-webui-config
    source:
      read:
        - &owui-share $SNAP_DATA/share/open-webui

plugs:
  # To allow side-loading models into server (root daemon) from user home directory
  home:
    read: all

environment:
  SNAP_COMPONENTS: /snap/$SNAP_INSTANCE_NAME/components/$SNAP_REVISION
  ARCH_TRIPLET: $CRAFT_ARCH_TRIPLET_BUILD_FOR
  STATUS_SHARE: *status-share # Status directory shared via content interface
  OWUI_SHARE: *owui-share # Open WebUI config directory shared via content interface
  # This is to find the intel.icd file, containing the path to libigdrcl.so
  # Required for Intel GPU detection and communication
  OCL_ICD_VENDORS: $SNAP/etc/OpenCL/vendors

layout:
  # OpenCL looks in /etc/OpenCL/vendors for .icd files containing the paths to shared objects
  /etc/OpenCL/vendors:
    bind: $SNAP/etc/OpenCL/vendors
  # Intel icd files use an absolute path to Intel OpenCL shared objects. Use a layout to mount the so files at the expected location
  /usr/lib/$CRAFT_ARCH_TRIPLET_BUILD_FOR/intel-opencl:
    bind: $SNAP/usr/lib/$CRAFT_ARCH_TRIPLET_BUILD_FOR/intel-opencl
  # Under wsl the libraries are expected in /usr/lib/wsl
  /usr/lib/wsl:
    bind: $SNAP_COMMON/usr/lib/wsl

hooks:
  install:
    plugs: &install-plugs
      # For hardware detection
      - hardware-observe
      - opengl
  post-refresh:
    plugs: *install-plugs

parts:
  version:
    after: [cli]
    plugin: nil
    source: . # To regenerate the git hash on every change
    build-packages:
      - jq
      - git
    override-pull: |
      # Do nothing. This significantly reduces build time when sourcing large files.
    override-build: |
      set -eo pipefail
      cli_version=$($CRAFT_STAGE/bin/gemma4 version --format=json | jq .cli --raw-output)
      git_hash=$(git -C "$SNAPCRAFT_PROJECT_DIR" describe --always)
      craftctl set version="${cli_version#v}+${git_hash}"

  cli:
    source:
      - on amd64: https://github.com/canonical/inference-snaps-cli/releases/download/v2.0.0-beta.4/inference-snaps-cli-linux-amd64.tar.xz
      - on arm64: https://github.com/canonical/inference-snaps-cli/releases/download/v2.0.0-beta.4/inference-snaps-cli-linux-arm64.tar.xz
    plugin: dump
    override-build: |

      # For tab completion
      ln --symbolic ./modelctl bin/gemma4

      craftctl default

  cli-dependencies:
    plugin: nil
    stage-packages:
      - pciutils # lspci
      - clinfo # used for Intel GPU vRAM detection. It depends on intel-opencl-icd

  # Used for NVIDIA vRAM and CUDA Capability detection.
  # Only the binary is included to avoid staging libraries, which should instead be provided by the host through the opengl interface.
  cli-nvidia-smi:
    plugin: nil
    build-packages:
      - nvidia-utils-580
    override-build: |
      mkdir -p $CRAFT_PART_INSTALL/usr/bin
      cp /usr/bin/nvidia-smi $CRAFT_PART_INSTALL/usr/bin/

      license_dir=$CRAFT_PART_INSTALL/usr/share/doc/nvidia-utils-580/
      mkdir -p $license_dir
      cp /usr/share/doc/nvidia-utils-580/copyright $license_dir/

  # OpenCL ICD and libigdgmm are required for Intel GPU vRAM lookup with clinfo.
  #
  # DRIVER VERSION: "Latest" = Intel Compute Runtime 26.22.38646.4 (public GitHub
  # release). The "Latest" driver here MUST match the one bundled by the
  # intel-compute-runtime part, otherwise clinfo (host-side vRAM detection) and
  # OVMS (inference) would use different Compute Runtime versions.
  cli-intel-opencl-icd:
    plugin: nil
    build-packages:
      - wget
    override-build: |
      if [ "$CRAFT_ARCH_BUILD_FOR" == "amd64" ]; then
        # ==============================================================
        # Legacy support ICD (compute-runtime 24.35.30872.36).
        #
        # COMPATIBILITY / CONFLICT NOTE:
        #   Do NOT download the legacy libigdgmm12_22.5.0 into the same
        #   directory as the latest libigdgmm12_22.10.0. `dpkg -i *.deb`
        #   installs in glob (alphabetical) order and "22.10" < "22.5" in
        #   ASCII, so 22.10.0 would install first and 22.5.0 would then
        #   DOWNGRADE over it. The legacy ICD only requires libigdgmm12
        #   (>= 22.5.0), so the latest 22.10.0 already satisfies it - we
        #   ship gmmlib ONCE, at the latest version.
        # ==============================================================
        mkdir -p "$CRAFT_PART_BUILD/icd-legacy"
        cd "$CRAFT_PART_BUILD/icd-legacy"
        wget -nv https://github.com/intel/compute-runtime/releases/download/24.35.30872.36/intel-opencl-icd-legacy1_24.35.30872.36_amd64.deb

        # ==============================================================
        # Latest ICD + gmmlib (compute-runtime 26.22.38646.4).
        # gmmlib 22.10.0 is installed here ONCE and satisfies both the
        # legacy and the latest ICD.
        # ==============================================================
        mkdir -p "$CRAFT_PART_BUILD/icd-latest"
        cd "$CRAFT_PART_BUILD/icd-latest"
        wget -nv https://github.com/intel/compute-runtime/releases/download/26.22.38646.4/intel-opencl-icd_26.22.38646.4-0_amd64.deb
        wget -nv https://github.com/intel/compute-runtime/releases/download/26.22.38646.4/libigdgmm12_22.10.0_amd64.deb

        # Install legacy FIRST, then latest, so the latest libraries win any
        # file-level overlaps. Separate directories + two dpkg calls keep the
        # ordering explicit (never rely on *.deb glob order across versions).
        dpkg --root="$CRAFT_PART_INSTALL" --force-all -i "$CRAFT_PART_BUILD/icd-legacy"/*.deb
        dpkg --root="$CRAFT_PART_INSTALL" --force-all -i "$CRAFT_PART_BUILD/icd-latest"/*.deb

        # If a driver generation drops IGC libraries (libigc/libigdfcl/libiga64)
        # into /usr/local/lib, that path is NOT on the snap's loader path (only
        # /usr/lib/$ARCH_TRIPLET is). Move them so NEO can link them at runtime.
        arch_libdir="$CRAFT_PART_INSTALL/usr/lib/$CRAFT_ARCH_TRIPLET_BUILD_FOR"
        if [ -d "$CRAFT_PART_INSTALL/usr/local/lib" ]; then
          mkdir -p "$arch_libdir"
          mv -n "$CRAFT_PART_INSTALL"/usr/local/lib/lib*.so* "$arch_libdir"/ 2>/dev/null || true
          rmdir "$CRAFT_PART_INSTALL"/usr/local/lib 2>/dev/null || true
        fi

        # Installing the ICD may create an absolute etc/alternatives/ocloc
        # symlink (-> /usr/bin/ocloc-<ver>) that points outside the snap and
        # fails store review. Drop ONLY that symlink; keep the real ocloc binary
        # (NEO/IGC need ocloc at runtime) and repoint usr/bin/ocloc with a
        # RELATIVE link that stays inside the snap.
        rm -f "$CRAFT_PART_INSTALL/etc/alternatives/ocloc"
        ocloc_real="$(ls "$CRAFT_PART_INSTALL"/usr/bin/ocloc-* 2>/dev/null | head -n1 || true)"
        if [ -n "$ocloc_real" ]; then
          rm -f "$CRAFT_PART_INSTALL/usr/bin/ocloc"
          ln -sf "$(basename "$ocloc_real")" "$CRAFT_PART_INSTALL/usr/bin/ocloc"
        fi
      fi

      craftctl default

  webui:
    plugin: dump
    source: https://github.com/canonical/inference-snaps-webui/releases/download/v1.0.0/inference-snaps-webui.tar.xz
    organize:
      "*": webui/

  engines:
    source: engines
    plugin: dump
    organize:
      "*": engines/

  models:
    source: models
    plugin: dump
    organize:
      "*": models/

  runtimes:
    source: runtimes
    plugin: dump
    organize:
      "*": runtimes/

  scripts:
    source: scripts
    plugin: dump
    stage-packages:
      - jq
    organize:
      "server.sh": bin/
      "server-webui.sh": bin/
      "export-shared-configs.sh": bin/

  model-e4b-it-int4-ov-1-of-2:
    plugin: dump
    source: components/model-e4b-it-int4-ov/gemma4-e4b-it-int4-ov
    organize:
      "*": (component/model-e4b-it-int4-ov-1-of-2)/
    prime:
      - "-(component/model-e4b-it-int4-ov-1-of-2)/openvino_text_embeddings*"
      - "*"

  model-e4b-it-int4-ov-2-of-2:
    plugin: dump
    source: components/model-e4b-it-int4-ov/gemma4-e4b-it-int4-ov
    organize:
      "openvino_text_embeddings*": (component/model-e4b-it-int4-ov-2-of-2)/
    prime:
      - "(component/model-e4b-it-int4-ov-2-of-2)/openvino_text_embeddings*"
      - -*

  openvino-model-server:
    plugin: dump
    source: https://github.com/openvinotoolkit/model_server/releases/download/v2026.2.1/ovms_ubuntu24_2026.2.1_python_off.tar.gz
    stage-packages:
      - libxml2
      - libcurl4
    organize:
      # move everything, including the staged packages
      "LICENSE": usr/share/doc/openvino-model-server
      "*": (component/openvino-model-server)
    # This part is only for amd64
    override-pull: &amd64-only |
      [ "$CRAFT_ARCH_BUILD_FOR" == "amd64" ] && craftctl default || exit 0
    override-build: *amd64-only
    override-stage: *amd64-only
    override-prime: *amd64-only

  # For Intel GPU usage, amd64 only
  #
  # DRIVER VERSION: "Latest" = Intel Compute Runtime 26.22.38646.4 + matching
  # IGC 2.36.3. This MUST match the cli-intel-opencl-icd part's 26.22 driver.
  intel-compute-runtime:
    plugin: nil
    build-packages:
      - wget
    stage-packages:
      - on amd64:
          - ocl-icd-libopencl1
    override-build: |
      if [ "$CRAFT_ARCH_BUILD_FOR" == "amd64" ]; then
        # ==============================================================
        # Legacy compute stack (compute-runtime 24.35.30872.36 + IGC 1.0.x).
        # Installed FIRST so the latest can override any older libraries.
        # ==============================================================
        mkdir -p "$CRAFT_PART_BUILD/opencl-intel-legacy1"
        cd "$CRAFT_PART_BUILD/opencl-intel-legacy1"
        wget -nv https://github.com/intel/intel-graphics-compiler/releases/download/igc-1.0.17537.24/intel-igc-core_1.0.17537.24_amd64.deb
        wget -nv https://github.com/intel/intel-graphics-compiler/releases/download/igc-1.0.17537.24/intel-igc-opencl_1.0.17537.24_amd64.deb
        wget -nv https://github.com/intel/compute-runtime/releases/download/24.35.30872.36/intel-level-zero-gpu-legacy1_1.5.30872.36_amd64.deb
        dpkg --root="$CRAFT_PART_INSTALL" --force-all -i *.deb

        # ==============================================================
        # Latest compute stack (compute-runtime 26.22.38646.4 + IGC 2.36.3).
        #
        # COMPATIBILITY NOTE: the "-2" IGC packages (intel-igc-core-2 /
        # intel-igc-opencl-2) are a NEW package generation that co-installs
        # alongside the legacy "intel-igc-core" / "intel-igc-opencl" (different
        # package names, no conflict). intel-ocloc 26.22 and libze-intel-gpu1
        # 26.22 must come from the SAME 26.22 release as the ICD in the
        # cli-intel-opencl-icd part.
        # ==============================================================
        mkdir -p "$CRAFT_PART_BUILD/opencl-intel"
        cd "$CRAFT_PART_BUILD/opencl-intel"
        wget -nv https://github.com/intel/intel-graphics-compiler/releases/download/v2.36.3/intel-igc-core-2_2.36.3+21719_amd64.deb
        wget -nv https://github.com/intel/intel-graphics-compiler/releases/download/v2.36.3/intel-igc-opencl-2_2.36.3+21719_amd64.deb
        wget -nv https://github.com/intel/compute-runtime/releases/download/26.22.38646.4/intel-ocloc_26.22.38646.4-0_amd64.deb
        wget -nv https://github.com/intel/compute-runtime/releases/download/26.22.38646.4/libze-intel-gpu1_26.22.38646.4-0_amd64.deb
        dpkg --root="$CRAFT_PART_INSTALL" --force-all -i *.deb

        # Same IGC-libdir safeguard as the cli-intel-opencl-icd part: move any
        # IGC .so from /usr/local/lib onto the snap's loader path.
        arch_libdir="$CRAFT_PART_INSTALL/usr/lib/$CRAFT_ARCH_TRIPLET_BUILD_FOR"
        if [ -d "$CRAFT_PART_INSTALL/usr/local/lib" ]; then
          mkdir -p "$arch_libdir"
          mv -n "$CRAFT_PART_INSTALL"/usr/local/lib/lib*.so* "$arch_libdir"/ 2>/dev/null || true
          rmdir "$CRAFT_PART_INSTALL"/usr/local/lib 2>/dev/null || true
        fi

        cd "$CRAFT_PART_BUILD"
      fi # End of Intel Compute Runtime installation

      craftctl default
    organize:
      # move everything, including the staged packages
      "*": (component/openvino-model-server)

  notice:
    plugin: nil
    source: NOTICE
    source-type: file
    override-build: |
      license_dir=$CRAFT_PART_INSTALL/usr/share/doc
      mkdir -p $license_dir
      cp NOTICE $license_dir/
      cp $SNAPCRAFT_PROJECT_DIR/LICENSE $license_dir/
      cp $SNAPCRAFT_PROJECT_DIR/LICENSE-gemma4 $license_dir/

components:

  model-e4b-it-int4-ov-1-of-2: &model-e4b-it-int4-ov
    type: standard
    summary: Google Gemma 4 E4B INT4 OpenVINO format
    description: Model weights for Google Gemma 4 E4B (Efficient 4B) in OpenVINO format, quantized to INT4

  model-e4b-it-int4-ov-2-of-2:
    <<: *model-e4b-it-int4-ov

  openvino-model-server:
    type: standard
    summary: OpenVINO Model Server
    description: OpenVINO Model Server for serving models

apps:
  gemma4:
    command: bin/gemma4
    completer: bin/snap-completer.bash
    plugs:
      - hardware-observe
      - opengl
      - network
      - desktop # To open the webui in the browser
    environment:
      ADDITIONAL_FEATURES: chat, webui

  server:
    command: bin/server.sh
    daemon: simple
    plugs:
      - network-bind
      - hardware-observe
      - opengl
      - process-control
      - home

  server-webui:
    command: bin/server-webui.sh
    daemon: simple
    plugs:
      - network-bind

Could you please try these changes on your side and let me know whether they resolve the issue for you? I'd be happy to help with any further debugging if needed.

Thank you.

[kgibala]

Hi @jpm-canonical,

We’d like to know if this issue is still affecting you. If so, please provide an update or any additional information. Otherwise, we’ll close this issue after 30 days of inactivity. Your feedback is appreciated.