Daily bookmarks: relevant news, facts and discoveries.
Articles 1216
Engineering ·
Published on September 26, 2026, the bookmark says Transformer-Within-Transformer replaces contiguous groups of similar layers with substitute attention layers. In DINOv2, it reports about half the compute, with a small drop in accuracy.
Engineering ·
On September 26, 2026, a Xiaomi MiMo post pointed to reinforcement learning environments, training code, and a MiMo-V2.6 RL dataset for exploring language-model workflows.
Engineering ·
Published on September 25, 2026, the post describes an alternative to attention value projection: token-indexed memory specific to each layer, combined with contextual keys.
Engineering ·
A post published on September 25, 2026 describes a change to Transformer attention: replacing the learned value projection with a sum of contextual keys and token-indexed memory vectors specific to each layer. The memory can reside on the CPU.
Engineering ·
NVIDIA library with quantization, distillation, pruning, neural architecture search and speculative decoding, with checkpoints exportable to inference runtimes.
Engineering ·
A study investigates scalable pretraining from random initialization using a program generator and a learner, without training on real data.
Engineering ·
Launched on September 25, 2026, the open-source collection includes tasks, environments, evaluations, and training to optimize small models for programming and data science.
Engineering ·
Efficient memory can improve service capacity. The design must also prevent one session's context from appearing in another session.
Engineering ·
An article published on September 24, 2026 presents linear-complexity layers with expressive hidden states, updated through self-supervised learning.
Engineering ·
A September 24, 2026 bookmark points to an article about backing up valuable secret keys. The record notes the security concern but gives no specific methods or results.
Engineering ·
A September 24, 2026 publication describes an attention variant that subtracts two maps to reduce noise and highlight signal. It notes potential interest for engineers exploring alternatives to standard attention.
Engineering ·
A publication dated September 24, 2026, outlines policy-gradient methods used to train LLMs, from vanilla policy gradient and REINFORCE to PPO, GRPO, and GRPO variants.
Engineering ·
A preliminary analysis reports that the attention matrix K exponent α tracks memorization of random labels during training, with variation across layers.
Engineering ·
The paper presents techniques for speeding up attention on Hopper GPUs, exploring asynchronous execution, low-precision computation, memory traffic, and hardware utilization.
Engineering ·
Dated September 24, 2026, the summary describes an analysis of equal learning rates for LoRA matrices A and B and a proposal to use different rates in wide models.
Engineering ·
Published on September 24, 2026, the article explores theoretical links between state-space models such as Mamba and attention variants, using structured semiseparable matrix decompositions.
Engineering ·
An architecture designed for efficient pretraining and inference on long sequences aims to address Transformer limitations.
Engineering ·
Published on September 24, 2026, the summary introduces Memory Attention, a proposal replacing the Transformer's learned value projection and examined across several attention variants.
Engineering ·
Retrieval can combine information from several documents. Privacy must follow retrieval, context assembly and the answer.
Engineering ·
Published on September 24, 2026, the article presents the Retentive Network, a language-model architecture proposing parallel training alongside recurrent and chunkwise recurrent inference paradigms.
Looking for something else?DocumentationContact
