Radar ·
Synthesized programs approximate attention heads
A post reports Python programs synthesized from attention maps, reaching 69–79% mean IoU on three models; replacing up to 30–40% of heads reportedly caused little decline in QA.
Published on July 1, 2026, the post describes synthesizing Python programs from attention maps to reproduce attention patterns between tokens. On GPT-2, TinyLlama, and Llama-3B, the best programs reached 69–79% mean IoU. According to the post, replacing up to 30–40% of heads caused little decline in QA.
The work explores whether some attention heads can be approximated by explicit, interchangeable programs rather than neural computations. To check the method, evaluation conditions, and context for these figures, consult the original post and verify its results and definitions; the available summary does not detail these points. If using AI to study or apply the material, avoid sending sensitive organizational data without authorization.
Original source: Substituição de attention heads por programas sintetizados