Papers worth reading
Popular and classic papers with a short AI overview. Open any of them in the reader for full translation, explanations and chat.
Trending
- OpenHands: An Open Platform for AI Software Developers as Generalist AgentsOpenHands is an open-source platform uniting sandboxed execution, web browsing, multi-agent delegation, and 15 benchmarks for AI developers.
- TradingAgents: Multi-Agents LLM Financial Trading FrameworkTradingAgents simulates a trading firm with specialized LLM agents and multi-perspective debates, delivering superior risk-adjusted returns.
- Harness2: Recursive Agent Harnessing for an Open WorldHarness2 enables open-world recursive agent improvement across tasks and meta-harnessing levels without ground-truth verifiers.
- Unlimited OCR Works: Welcome the Era of One-shot Long-horizon ParsingUnlimited OCR introduces Reference Sliding Window Attention to enable constant-KV-cache decoding for one-shot long-horizon document transcription.
- Very Large-Scale Multi-Agent Simulation in AgentScopeAgentScope scales multi-agent simulations to 1M LLM agents using an actor-based distributed mechanism, configurable diversity, and web management.
- AgentScope 1.0: A Developer-Centric Framework for Building Agentic ApplicationsAgentScope 1.0 delivers a modular, ReAct-based framework with parallel tool calling, MCP integration, visual studio tracing, and sandboxed runtimes.
- Kronos: A Foundation Model for the Language of Financial MarketsKronos discretizes financial candlestick data into hierarchical tokens for autoregressive pre-training across 12 billion multi-market records.
- Hy-Nomas: Towards Free-form Interactive World GenerationHy-Nomas is an 8B video world model supporting interleaved free-form text instructions and continuous camera control across multi-turn interactions.
- MOSS-Image Technical ReportMOSS-Image pairs a 5B diffusion transformer with an agentic prompt enhancer to generate complex, layout-rich graphics with long bilingual text.
- The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing PredictionEdge0 streams 35B MoE weights from SSD at 20 tok/s in 3 GiB RAM on consumer hardware using 1-token-ahead routing prediction and recovery LoRA.
- Prism: Dynamic Sparse Attention for Native 2K Joint Video-Audio Generation Model TrainingPrism enables native 2K joint video-audio diffusion training with dynamic sparse attention guided by visual variance and audio cross-attention norms.
Classics
- Attention Is All You NeedIntroduces the Transformer, replacing recurrence and convolutions with multi-head self-attention for faster, state-of-the-art sequence modeling.
- BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingBERT pre-trains deep bidirectional Transformer representations using masked language modeling, advancing state-of-the-art across 11 NLP tasks.
- Deep Residual Learning for Image RecognitionDeep residual networks use identity shortcut connections to solve the degradation problem, training networks over 100 layers to state-of-the-art accuracy.
- Language Models are Few-Shot LearnersScaling autoregressive models to 175B parameters enables competitive few-shot NLP performance purely via in-context learning without fine-tuning.
- Generative Adversarial NetsIntroduces Generative Adversarial Networks (GANs) using a minimax game between a generator and discriminator, avoiding Markov chains during sampling.
- Auto-Encoding Variational BayesIntroduces the reparameterization trick and Auto-Encoding Variational Bayes (AEVB) for scalable inference in continuous latent variable models.
- Denoising Diffusion Probabilistic ModelsDemonstrates high-quality image generation using diffusion probabilistic models trained via a simplified score-matching objective.
- High-Resolution Image Synthesis with Latent Diffusion ModelsLatent Diffusion Models shift diffusion training into pretrained autoencoder latent spaces, slashing compute while enabling high-resolution synthesis.
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleVision Transformer (ViT) applies a standard Transformer directly to image patches, matching or beating CNNs when pre-trained at scale.
- Learning Transferable Visual Models From Natural Language SupervisionCLIP scales contrastive image-text pre-training on 400M web pairs, achieving zero-shot image classification competitive with supervised models.
- Training language models to follow instructions with human feedbackInstructGPT aligns GPT-3 to human instructions using RLHF, matching 175B performance with a 1.3B model while reducing hallucinations and toxicity.
- LoRA: Low-Rank Adaptation of Large Language ModelsLoRA freezes pre-trained LLM weights and injects trainable rank decomposition matrices, cutting trainable parameters up to 10,000x with zero latency.
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsChain-of-thought prompting provides step-by-step demonstrations, unlocking multi-step reasoning in large language models without fine-tuning.
- LLaMA: Open and Efficient Foundation Language ModelsMeta presents LLaMA, a suite of foundation models up to 65B parameters trained on up to 1.4T tokens of publicly available text.
- Llama 2: Open Foundation and Fine-Tuned Chat ModelsMeta releases Llama 2 (7B–70B), open pretrained and dialogue-aligned LLMs rivaling closed systems via iterative RLHF and Ghost Attention.
- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksCombines pre-trained parametric seq2seq models with dense non-parametric retrieval to set state-of-the-art results on knowledge-intensive NLP tasks.
- Adam: A Method for Stochastic OptimizationAdam combines adaptive learning rates and momentum with bias-corrected moment estimates for efficient, robust stochastic optimization.
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftBatch Normalization stabilizes internal layer input distributions, enabling faster training, higher learning rates, and state-of-the-art ImageNet accuracy.
- Scaling Laws for Neural Language ModelsEmpirical power laws show language model loss scales predictably with compute, parameters, and data, favoring large models over training to convergence.
- Training Compute-Optimal Large Language ModelsDemonstrates that LLMs are undertrained; compute-optimal scaling requires scaling model parameters and training tokens in equal proportions.
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessFlashAttention uses tiling and recomputation to avoid writing the $N \times N$ attention matrix to GPU HBM, speeding up Transformers up to 3x.
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelDPO aligns language models directly on human preference data using binary cross-entropy, eliminating explicit reward models and RL fine-tuning.
- Mamba: Linear-Time Sequence Modeling with Selective State SpacesMamba introduces selective state space models with a hardware-aware parallel scan, achieving Transformer-level quality with linear sequence scaling.
- ReAct: Synergizing Reasoning and Acting in Language ModelsReAct combines verbal reasoning traces and environmental actions in LLMs, reducing hallucinations and boosting decision-making task performance.
- Segment AnythingIntroduces SAM and SA-1B (1B masks, 11M images), establishing promptable foundation models for zero-shot image segmentation.
- DeepSeek-V3 Technical ReportDeepSeek-V3 is an open 671B MoE LLM (37B active) trained on 14.8T tokens for under $6M, matching top frontier closed models.
- Proximal Policy Optimization AlgorithmsProximal Policy Optimization introduces a clipped surrogate objective that simplifies trust region methods while improving sample efficiency.
- Playing Atari with Deep Reinforcement LearningDeep Q-Networks learn end-to-end control directly from Atari 2600 pixels using experience replay, surpassing human expert performance on three games.
- RoFormer: Enhanced Transformer with Rotary Position EmbeddingRoFormer introduces Rotary Position Embedding (RoPE) to encode relative token positions via orthogonal rotations within self-attention.
- A Simple Framework for Contrastive Learning of Visual RepresentationsSimCLR simplifies contrastive visual representation learning without specialized architectures or memory banks, matching supervised ResNet-50 on ImageNet.
- NeRF: Representing Scenes as Neural Radiance Fields for View SynthesisSynthesizes photorealistic novel views of complex 3D scenes by optimizing continuous 5D neural radiance fields with differentiable volume rendering.
- U-Net: Convolutional Networks for Biomedical Image SegmentationU-Net introduces a u-shaped encoder-decoder network with skip connections for fast, precise biomedical segmentation from very few annotated images.
- 3D Gaussian Splatting for Real-Time Radiance Field Rendering3D Gaussian Splatting uses anisotropic 3D Gaussians and tile-based rasterization to achieve real-time, high-quality novel-view synthesis at 1080p.
- Neural Machine Translation by Jointly Learning to Align and TranslateIntroduces soft-attention mechanisms into neural machine translation, allowing decoders to focus dynamically on relevant source words.
- Efficient Estimation of Word Representations in Vector SpaceIntroduces CBOW and Skip-gram to learn high-quality word vectors from billions of words rapidly while preserving linear semantic and syntactic analogies.
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts LayerSparsely-gated Mixture-of-Experts layers scale neural networks up to 137B parameters, achieving lower perplexity at reduced computational cost.
- Mixtral of ExpertsMixtral 8x7B introduces an open Sparse Mixture of Experts model using 13B active parameters per token to outperform Llama 2 70B and match GPT-3.5.
- The Llama 3 Herd of ModelsMeta presents Llama 3, open foundation models up to 405B parameters supporting 128K context, multilinguality, coding, reasoning, and tool use.
- GPT-4 Technical ReportGPT-4 is a large multimodal model demonstrating human-level exam performance, predictable scaling, and enhanced safety through RBRMs.
- Constitutional AI: Harmlessness from AI FeedbackConstitutional AI trains non-evasive, harmless language assistants using critique, revision, and AI feedback guided solely by written principles.
- Finetuned Language Models Are Zero-Shot LearnersFinetuning large language models on instruction-formatted datasets substantially boosts zero-shot generalization across unseen tasks.