Transformer-based language models are increasingly deployed on resource-constrained edge devices, raising new security concerns when attackers have physical access to the underlying hardware. This paper investigates the susceptibility of on-device transformer language models to voltage fault injection attacks. We present a stage-level characterization of crowbar voltage glitches against a 4,192-parameter autoregressive transformer with a key-value (KV) cache running on an Arm Cortex-M4 microcontroller. Using stage-selectable triggers, on-device cycle measurements, and automated fault-parameter exploration, we analyze the effects of voltage glitches across six regions of the model’s forward pass. Across 3,300 fault-injection attempts at selected timing offsets, each evaluated stage exhibits non-crash anomalies, with the highest observed anomaly rates ranging from 43.0% to 99.7% depending on the stage. Additional experiments characterize the impact of glitch strength and input-dependent execution timing, demonstrating that effective injection parameters can vary across prompts. We also evaluate a temporal redundancy-based countermeasure that restores the precomputation state, recomputes model outputs with fault triggers disabled, and compares the resulting logits. Across 6,100 evaluated defense attempts, no silent emitted-token changes are observed under the tested conditions. We further distinguish between detected anomalies, actual token-output changes, device crashes, and subsequent text-generation divergence. These findings highlight physical fault-injection risks in embedded language models and motivate lightweight fault detection and redundancy mechanisms for secure on-device inference.