[dgryski]

Updates #6047. Based on #6060, so the first commit here is from that PR.

Summary

With -scheduler=cores on riscv-qemu, the reproducer from #6047 still hangs after #6060. It does not panic. The program stops with no output.

A hart can take the GC stop-the-world interrupt while it holds schedulerLock with interrupts enabled. It then parks in gcInterruptHandler with the lock held. If another hart is spinning on schedulerLock with interrupts off, that hart cannot take the interrupt, and the GC initiator waits for it forever. This needs at least three harts.

A hart can end up spinning with interrupts off in two ways seen in hung runs.

1. Channel operations. chanTrySend disables interrupts and then calls scheduleTask, for example from time.sendTime in a timer callback.
2. A task in Pause called from alloc through gcLock and Futex.Wait. In the captures this stack had mstatus.MIE clear. The next section explains why.

A hart also holds the lock with interrupts enabled after it wakes from schedulerUnlockAndWait. The checkpoint jump leaves the trap with mret, which restores interrupts, and the scheduler loop then runs with the lock held.

This PR adds schedulerSpinLock. Lock disables interrupts and Unlock restores the previous state, saved per hart. Every schedulerLock call site gets the change, including the re-lock after idle wake.

Interrupt state across task switches

This PR does not change how task switches handle interrupt state, but it matters for the hang. tinygo_swapTask carries the old context's mstatus.MIE into the new one. A task that pauses with interrupts off can leave the next task running with interrupts off. On riscv-qemu this is common. In the #6047 reproducer, alloc was entered with MIE clear in 32 to 64 of 135 calls per run, always after a resume with MIE clear. I think this deserves a separate issue.

How this was found

I looped the #6047 reproducer under qemu with the gdb stub open and attached lldb to hung runs. Each hang had one hart in gcMarkReachable, one or more harts parked in gcInterruptHandler, and one or more harts spinning in schedulerLock.Lock with mstatus.MIE clear. With a temporary owner field on the spinlock, the lock owner was a parked hart.

I then wrote Spin models. The first covers the lock, per-hart interrupt enable and the GC rendezvous. It deadlocks with 3 and 4 harts and not with 2. With interrupts disabled at every lock site it has no deadlock, checked exhaustively for 3 and 4 harts.

The second model adds the gcLock futex mutex, pause and resume, and idle wake. It shows the same deadlock with 3 harts. With the fix it has no deadlock, checked exhaustively for 3 harts. The variant that also allows alloc with interrupts off was only a partial search. Wrapping lockScheduler alone is not enough, because many sites call schedulerLock.Lock() directly.

I can share the model files if they are useful.

Testing

On the build with #6060 but without this change, the reproducer hung in 5 of 76 runs across five loops. Some of those builds had temporary debug fields. With this change it passed 300 of 300 runs.

I also wrote three stress programs that mix sleepers, mutexes, sync.Cond, wait groups, timers, tickers and runtime.GC(). Results over 100 runs each on riscv-qemu with 4 harts.

Program	Without this change	With this change
timers	52 pass, 48 hang	100 pass
mixed	94 pass, 6 hang	100 pass
allocgc	100 pass	100 pass

allocgc does not catch the bug. The timers program hangs about half the time without the change, so it is a good candidate for a regression test. I have not added any of these programs to the repo yet.

I also ran the reproducer and the stress programs with -tags runtime_asserts and a temporary check that schedulerLock is unlocked on the same hart that locked it. The check never fired. It passed 20 runs each of the reproducer, mixed and timers, and 5 runs of allocgc. The check is not part of this PR. It may be worth adding in the future.

Not done

• I have not run the full test suite.
• builder/testdata/binary-size.txt needs regenerating for riscv-qemu. The size test compares it exactly and this change grows the code.
• rp2 and esp32s3 are unchanged. They have two cores, so this deadlock is not reachable there.
• A Spin model of the rp2 cores scheduler, with the FIFO interrupt, timers, sleepTicksMulticore and flash-safe sections, found no deadlock. The model is simplified, with one task per core, so this does not prove the code is safe. One hypothetical remains. An interrupt handler that wakes a goroutine through scheduleTask would spin on schedulerLock held by the code it interrupted on the same core. I found no rp2 runtime or machine handler that does this. A user callback could.
• The interrupt state carry-over above is confirmed on riscv-qemu only. The Cortex-M swap does not touch PRIMASK, so the same carry-over should apply there, but I have not tested it.
• With schedulerAsserts enabled, runtimeFatal at scheduler_cores.go:68 now runs with interrupts off while holding the lock. Printing takes printLock, which a parked hart may hold, so the message could hang instead of printing. This only affects assert builds and I have not tried it.
• There is no progress yet on the Pico 2 hang. This change and the models do not explain it, and I have not reproduced it. The interrupt state carry-over is one candidate on that platform, but I have not tested it there.

[davecheney]

@dgryski thanks for this, I have a pico set up with the software debugger and I’ll test this.

[dgryski]

I don't think this will affect the pico2 crash. Claude tried to model the rp2 cores scheduler in Spin and couldn't reproduce this hang and didn't find any other issues. Also, this particular hang found by the Spin model required three cores, and the rp2 only has two.