TL;DR

Goal: run MobileNet on the RK3576’s NPU through the open rocket driver + Mesa Teflon — the same stack is byte-perfect on the RK3588, so the bug is RK3576-specific. A month of all-zero output later:

• The int8 convolution is byte-correct now. The “all-grey” wall was a fixed-point bug: the rescale multiplier went out at Q14 where the chip wants Q4 — 2¹⁰ too hot, so every pixel saturated. Fix that (plus a pad value and a bias term) and a single conv matches the CPU reference byte-for-byte.
• MobileNet end-to-end still returns zero, behind what looked like two walls but turned out to be one, living below the registers: the command stream I send is byte-identical to the vendor’s and the chip still behaves differently. The wall is multi-task dispatch — the compute units won’t re-arm themselves for each task the way the vendor’s do — and the depthwise, which I’d taken for a second separate wall, is just the first layer wide enough to be forced through it (I confirmed that by instrumenting the vendor’s own driver and watching its tiled depthwise run aligned and correct). Ordinary single-task convolutions compute fine on the same path.
• So the open driver is exonerated — every byte I hand the chip matches the vendor’s; the gap is in silicon state, below what software can observe on either side. I even booted the driver on top of a mainline OP-TEE to rule out the firmware — same failure. The way around (not through): the wall only bites multi-task jobs, so send each row-tile as its own single-task job, which the hardware runs. That’s the next build.

The rest is the long version — mostly me being wrong, in order.