Daily bookmarks: relevant news, facts and discoveries.
Articles 1216
Engineering ·
Radar: a paper that reads standard attention as Entropic Optimal Transport with an implicit uniform prior and proposes GOAT, with a continuous trainable prior that stays compatible with optimized kernels such as FlashAttention.
Engineering ·
The publication claims cuDNN, FA4 and FlexAttention have precision problems in backward passes and presents KohakuFA as an alternative kernel, with a public GitHub repository for inspection.
Engineering ·
Radar briefing dated 3 October 2026 on a text-to-image model that reuses Transformer blocks at each denoising step, with the performance claim stated by the original post.
Engineering ·
Radar: the paper presents LoopCD, a training-free contrastive decoding method for recurrent Transformers that uses representations from earlier loops as weaker predictions to select tokens.
Engineering ·
Radar: LoopCD refines decoding in looped Transformers by using the gap between early and late predictions of recurrent states, with no extra training, and reports accuracy gains on AIME 2024 with the Ouro-2.6B-Thinking model.
Engineering ·
A post describes adding sampling to the post-training stack and claims SFT can compete with current methods, with better generalization and less forgetting in some cases.
Engineering ·
A guide describes a proxy that records token IDs and sampled logprobs to train models with RL across four agent harnesses, with reported score gains and a comparison with imitation learning.
Engineering ·
The article examines whether chains of thought do computational work in serial tasks; according to the post, removing them lowers accuracy to the shallow-circuit limit.
Engineering ·
Published on October 1, 2026, the text presents multi-teacher on-policy distillation (MOPD): a method that trains a student model on its own completions, using teachers’ per-token log probabilities and a reverse KL objective.
Engineering ·
Running a model close to data can change the architecture. It does not replace input processing, access control or real evaluation.
Engineering ·
A PyTorch blog post describes Jagged Flash Attention, an attention kernel behind Meta's Generative Ads Model, developed with TLX for NVIDIA Blackwell. The post reports better performance than FlashAttention-4 on the GEM jagged shapes.
Engineering ·
Published on October 1, 2026, the post says Looped-DiT reuses Transformer blocks during denoising and that a 260-million-parameter model beat one 6.5 times larger on text-to-image benchmarks, with 4.9 times less inference computation.
Engineering ·
A post dated October 1, 2026 reports 890 bytes of global KV cache per token and a persistent cache eight times smaller in a language model.
Engineering ·
A September 30, 2026 publication argues that factoring S = WWᵀ changes gradient flow in attention and removes an information-exponent bottleneck that can stall unfactorized models.
Engineering ·
Published on September 30, 2026, the item describes DeepEP-Ascend as a high-performance communication library for machine learning training and inference on Huawei Ascend NPUs.
Engineering ·
An article by Reiner Pope and coauthors discusses scaling Transformer inference efficiently, a relevant topic for engineers evaluating ways to serve these models.
Engineering ·
Published on September 30, 2026, ProRL Agent is open-source infrastructure for multi-turn LLM agent rollouts. Its rollout-as-a-service model separates trajectory generation, environment orchestration, and evaluation from policy optimization.
Engineering ·
A September 29, 2026 report says BF16 rounding in FlashAttention may break a softmax-gradient symmetry and cause gradients to explode near the end of training. The authors propose GProj as a correction.
Engineering ·
The article examines positional encodings for Transformers, notes RoPE's bias toward nearby tokens, and organizes evaluations of alternatives across distinct configurations.
Engineering ·
A September 29, 2026 preview presents a benchmark for comparing information-flow methods and assessing whether deeper Transformers use their layers effectively.
Looking for something else?DocumentationContact
