Abstract
Small transformer language models are today able to run on microcontrollers, where an attacker with physical access can disturb the supply voltage during text generation. Fault injection studies of neural networks in the past have mostly targeted classifiers and modeled the attacker as flipping a chosen bit of a chosen parameter. A voltage fault injection is instead aimed at corrupting whatever the processor is executing and targets the execution of the model. This work characterizes voltage fault injection against a 4,192-parameter autoregressive transformer with a key–value (KV) cache on an Arm Cortex-M4. The language model is augmented with triggers that mark six stages of the forward pass, and the work searches each stage for timing offsets that cause faults. Every stage is found to have an offset at which 43.0% to 99.7% of voltage fault injection glitches produce an anomaly without a crash. The vulnerable offsets depend on the input: on one prompt an offset in the feed-forward network, or multilayer perceptron (MLP), faulted on 299 of 300 attempts, yet on a second prompt it produced no anomalies while other MLP offsets became vulnerable. A single glitch can also leave the current token unchanged and change a later one, consistent with corrupted state carried between tokens. Finally, this work evaluates temporal redundancy that recomputes each token and compares all logits, assuming the recomputation is not itself faulted. No glitch in 6,100 attempts changed the emitted token without an alarm, but at the MLP offset the check alarmed on 299 of 300 attempts that left the output unchanged. These results suggest that defenses for on-device language models should protect the computation in every stage and the state carried between tokens, not only the stored weights.
Anthony Etim
PhD Candidate
My research interests include Machine Learning Security, Hardware Security and FPGA Security.