Engineering ·
Memory-Efficient Expert Routing for Distributed MoE Training
An arXiv paper, dated October 7, 2026, proposing methods to reduce peak memory in distributed Mixture-of-Experts training, focusing on the all-to-all dispatcher.
Radar
Daily bookmarks: relevant news, facts and discoveries.
Articles 1216
Engineering ·
On-policy multi-teacher distillation to combine specialist models
Radar: the post describes MOPD, a method in which a student model learns from its own rollouts and receives token-level supervision from domain-specialist teachers, aiming to combine the abilities of separately trained specialists.
Engineering ·
EmbeddingGemma 2 and multimodal retrieval in one shared vector space
Radar: a post claims EmbeddingGemma 2, from Google, maps text, code, images, video and audio into a single vector space and uses an Apache 2.0 license. The source gives no metrics or technical documentation.
Engineering ·
JEPA-Anything applies predictive learning to seven domains
Radar: the post presents JEPA-Anything, a domain-agnostic framework for world modeling based on orthogonal predictive factorization, which explores predictive representation learning across seven domains, from biology to climate.
Engineering ·
LiFT: shared DiT core for iterative image inference
Radar item: LiFT repeatedly applies a shared Diffusion Transformer core and, according to the publication, outperforms a dense DiT on ImageNet 256×256 with about 60% fewer parameters.
Engineering ·
Quantizing 10M EmbeddingGemma vectors to 1 bit with MRL cuts RAM
Radar: a report that 10 million EmbeddingGemma 2 vectors, quantized to 1 bit and reduced to 256 dimensions with MRL, dropped from about 30 GB to 0.4 GB of RAM, with a declared quality loss of about 5%.
Engineering ·
Paper presents Native Hybrid Attention (NHA) for long-context attention
Briefing on a paper presenting Native Hybrid Attention, an approach that seeks to balance the speed of Transformers in long contexts with the retention of precise details from linear attention.
Engineering ·
Astral GPU Indexes add support for PyTorch 2.13–2.14 and Python 3.15
Radar notes that Astral GPU Indexes now support PyTorch 2.13 and 2.14 and Python 3.15, with precompiled GPU-enabled wheels for FlashAttention, FlashAttention 3, causal-conv1d and DeepSpeed.
Engineering ·
How attention sinks and residual sinks rescale transformer components
Summary of a paper that investigates attention sinks and residual sinks in large language models and proposes that these outliers, along with softmax attention and RMSNorm, rescale other components of the architecture.
Engineering ·
Dust: training transformers without gradients through activation perturbations
Radar: the publication describes Dust, a zeroth-order optimization method that uses parallel perturbations of activations as a virtual population to pretrain transformers, with scale experiments and comparisons with backpropagation and evolution strategies.
Engineering ·
Shared Memory Reduces Memory Use in Recurrent Transformers
Radar: an article on sharing memory across recursions in recurrent Transformers, reporting a 76–79% reduction in context memory for models from 150M to 1B parameters, compared with standard Transformers.
Engineering ·
Pre-training a 30B-A3B MoE model scaled from 16 to 512 B200 GPUs
Report of scaling with nearly linear scalability and 35.3% MFU, using small-scale tests to narrow the search space before adding more GPUs.
Engineering ·
Switching Linear Attention alternates between multiple linear maps
Radar: the SwiLA paper, from COLM 2026, proposes switching between multiple linear maps and discusses the trade-off between attention expressiveness and KV cache memory cost.
Engineering ·
Symmetric memory for NCCL communication
Briefing on a section of the Machine Learning Engineering Open Book that describes symmetric memory in NCCL and PyTorch for small and medium payloads and for overlapping communication and computation.
Engineering ·
Google releases 4B Gemma model for speaker diarization
Radar: according to a note dated October 4, 2026, Google released a 4B Gemma model, fine-tuned to correct speaker labels and reduce word-level diarization errors on four benchmarks.
Engineering ·
BF16 gradients in FlashAttention-3 may grow late in training
A reported 450-million-parameter transformer trained with BF16 FlashAttention-3 saw its gradient norm rise 1,000× after 25 billion tokens, ending with higher loss than FP32 attention and no NaNs.
Engineering ·
Projection sampling for rewriting SFT training data
Paper proposes rewriting expert demonstrations as trajectories that are correct for the task and likely under the target model, before using them in standard SFT.
Engineering ·
RL setup for calibrated probabilities in decisions
Radar briefing on a reinforcement learning setup for decision models that return calibrated probabilities instead of generated text. Publication date: October 4, 2026.
Engineering ·
VeriHarness verifies agent rollouts in long-horizon tasks
The paper presents VeriHarness, which checks agent claims in long tasks using a fixed base model, without reference answers or rubrics during testing.
Engineering ·
Distillation study: per-token KL direction matters more than rollout policy
A systematic study reports that the direction of per-token KL matters more than the rollout policy in on-policy versus off-policy distillation, questioning the idea that on-policy rollouts are inherently better.