Correction, 18 August 2026. Everything below is what I believed on 10 July, and the
conclusion it builds to is wrong. Chained layers do compute. MobileNet V1 now runs end to
end on the NPU through this stack and returns the CPU’s class. The wall I described as
living below the registers, in silicon state no software can observe, was a register field:
PC_TASK_CON carries a 16-bit task number on the RK3576 where the RK3588-derived header
assumes 12, so the driver asked the hardware for 28673 tasks. On the development fork this
post is mostly about, it was not even that. It was fifteen ordinary faults in register and
buffer generation, each one smaller than the resolution of the test I was checking with.
The phrase byte exact is withdrawn too: the convolution is correct to within one count,
which is not the same word.
I am leaving the post standing rather than editing the conclusion out of it, because how the
error survived is the more useful half. Version 2 of the preprint says all of this properly:
https://doi.org/10.5281/zenodo.21990992. There is a postscript at the bottom of this page
with the short version.
TL;DR
Goal: run MobileNet on the RK3576’s NPU through the open rocket driver + Mesa Teflon — the same
stack is byte-perfect on the RK3588, so the bug is RK3576-specific. A month of all-zero output later:
- The int8 convolution is byte-correct now. The “all-grey” wall was a fixed-point bug: the rescale
multiplier went out at Q14 where the chip wants Q4 — 2¹⁰ too hot, so every pixel saturated. Fix that
(plus a pad value and a bias term) and a single conv matches the CPU reference byte-for-byte.
- MobileNet end-to-end still returns zero, behind what looked like two walls but turned out to be
one, living below the registers: the command stream I send is byte-identical to the vendor’s and the
chip still behaves differently. The wall is multi-task dispatch — the compute units won’t re-arm
themselves for each task the way the vendor’s do — and the depthwise, which I’d taken for a second
separate wall, is just the first layer wide enough to be forced through it (I confirmed that by
instrumenting the vendor’s own driver and watching its tiled depthwise run aligned and correct).
Ordinary single-task convolutions compute fine on the same path.
- So the open driver is exonerated — every byte I hand the chip matches the vendor’s; the gap is in
silicon state, below what software can observe on either side. I even booted the driver on top of a
mainline OP-TEE to rule out the firmware — same failure. The way around (not through): the wall only
bites multi-task jobs, so send each row-tile as its own single-task job, which the hardware runs. That’s
the next build.
The rest is the long version — mostly me being wrong, in order.