Why Robotics Still Isn't Solved - But Could Be Soon | YC Paper Club
What this video is
⚡ a 84-minute video, readable in 60 secondsThis video documents "YC Paper Club: Robotics Edition," opening with a rundown of the paper "LeWorldModel" (Maes, Le Lidec, Scieur, LeCun, and Balestriero) before host Francois Chaubard, a PhD student and Visiting Partner at YC, frames four bottlenecks facing Physical AI and robotics in 2026. The rest of the session is a series of guest research talks: Marcel Torné on memory for vision-language-action models (MEM), Milan Ganai on self-supervised embodied chain-of-thought reasoning (R&B-EnCoRe), Tyler Ga Wei Lum and Kushal Kedia on a zero-shot dexterous tool-manipulation policy (SimToolReal), Niko West of Rerun on data infrastructure and business strategy for robotics startups, and General Instinct's Bill Jiao and Guanming Wang on infrastructure for world action models. The discussion's center of gravity is not a single debate but a shared problem space, memory, sim2real transfer, reasoning grounding, and compute cost, that each presenter tackles from a different technical angle, with no explicit disagreement voiced between speakers in the extracted material.
The video opens with a paper titled "LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels," by Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann LeCun, and Randall Balestriero. [00:00]
Key takeaways
- The paper's authors are affiliated with Mila and Universite de Montreal, New York University, Samsung SAIL, and Brown University. [00:01]
- LeWorldModel (LeWM) is presented as the first JEPA that trains stably end-to-end from raw pixels using only two loss terms: a next-embedding prediction loss and a regularizer enforcing Gaussian-distributed latent embeddings. [00:02]
- LeWM reduces tunable loss hyperparameters from six to one compared to the only existing end-to-end alternative. [00:02]
- LeWM has 15 million parameters and is trainable on a single GPU in a few hours. [00:02]
+ 65 more takeaways
- The paper claims LeWM runs up to 48 times faster than foundation-model-based world models while remaining competitive across diverse 2D and 3D control tasks. [00:03]
- The paper states LeWM's latent space encodes meaningful physical structure and that a surprise evaluation shows the model reliably detects physically implausible events. [00:03]
- Host Francois Chaubard joked that people said robotics would be solved in 2016 after AlphaGo became the first computer program to defeat a world champion. [00:27]
- Chaubard said MuJoCo enabled training a robot to walk in about 3,000 iterations by 2018, which again fueled predictions that robotics was about to be solved. [00:39]
- Chaubard cited Mobile ALOHA in 2023 as another breakout moment that raised expectations for near-term robotics solutions. [00:54]
- Chaubard framed four critical bottlenecks in Physical AI and Robotics for 2026, starting with physical real-world modeling. [02:40]
- Chaubard said the first bottleneck, physical real-world modeling and the sim2real gap, includes difficulty predicting contact dynamics and deformable objects, causing sim-trained policies to fail in reality. [02:51]
- Chaubard illustrated the sim2real gap with an example of a video-model-trained driving policy where a simulated car drives into a Whole Foods that magically becomes a highway without crashing. [03:15]
- Chaubard said the unresolved sim2real gap involves estimating the transition function from state t to state t+1, which becomes much harder when conditioned on action. [03:38]
- Chaubard said representation for the action space is a completely unsolved problem in this domain. [04:24]
- Chaubard described embodiment drift as a real-world robotics problem where actuators accumulate dust, corrosion, and rust over time, changing how much force a given action produces. [05:56]
- Marcel Torné, a Stanford PhD student who interned at Physical Intelligence, presented "MEM: Multi-Scale Embodied Memory for Vision Language Action Models." [08:01]
- Torné said the internship policies were tested on "robot Olympic" tasks like unlocking blocks, folding inside-out clothes, and making a peanut butter sandwich, with the longest task lasting about 2 minutes. [08:19]
- Torné listed what's missing for long-horizon robot tasks: keeping track of task progress, keeping track of time, reliable dexterity, and capacity to adapt in-context. [09:17]
- Torné argued memory is necessary to solve long-horizon tasks, noting most robot policies like Pi0.5 and GR00T have no memory. [09:41]
- Torné said without memory, a demo robot washing dishes never stops washing, and another cooking a grilled cheese burns it because it has no sense of elapsed time. [10:13]
- Citing his own prior paper "Learning Long-Context Diffusion Policies with Past Token Prediction" (2025), Torné named two problems with adding memory: effectiveness (distribution shifts and lack of data) and efficiency (higher resource cost). [10:44]
- Torné's proposed solution is compression: splitting the robot policy into high-level and low-level policies, and splitting memory into short-context dense frames and long-context compressed language. [11:15]
- Torné said the short-term visual memory method uses a new ViT-based encoder with added temporal attention layers, dropping all tokens except the current image. [12:03]
- Torné said MEM uses a recurrent textual memory scratchpad predicted by the high-level policy and fed back into itself to track what has happened. [13:40]
- Torné said text-based memory is much more compressed than image-based memory since text uses far fewer tokens, which helps training. [14:07]
- Torné demonstrated MEM solving long-horizon tasks like preparing all ingredients for a recipe in an unseen kitchen, shown at 4x speed and taking up to tens of minutes. [14:25]
- Torné showed memory-less policies get stuck repeating the same mistake, such as failing to open a fridge or pick up a chopstick, while memory-equipped policies recognize and correct the mistake. [15:23]
- In Q&A, Torné said high-level policies are trained with SFT (supervised fine-tuning). [16:48]
- Milan Ganai, a Stanford CS PhD student with research experience at AWS and Waymo, presented on robots that teach themselves how to reason. [20:20]
- Ganai described a recent trend of leveraging embodied reasoning, similar to chain-of-thought for LLMs, to improve action prediction in VLAs. [21:32]
- Ganai discussed his RSS-conference paper, "Self-Supervised Bootstrapping of Action-Predictive Embodied Reasoning" (R&B-EnCoRe), co-authored with Katie Luo, Jonas Frey, Clark Barrett, and Marco Pavone. [22:31]
- Ganai said good embodied reasoning is limited by a grounding problem: there is no labeled data grounding physical actions with reasoning and no physically grounded model to validate or generate that reasoning data. [22:47]
- Ganai raised a verbosity problem, questioning whether reasoning at every step is action-predictive or just distracting given robotics' latency constraints. [23:31]
- Ganai's method treats reasoning as an unobserved latent variable for the observed context and action, leveraging variational inference. [24:35]
- Ganai said the validator score has three parts, concision, non-triviality, and action predictiveness, with action predictiveness described as the most important of the three. [25:39]
- On LIBERO-90 manipulation evals, Ganai reported R&B-EnCoRe reached a 79.5% success rate with 129.3 average generated tokens, versus 75.9% for No Reasoning, 78.6% for All Primitives, and 76.5% for Random Primitives. [26:37]
- Treating the object list as a latent variable, Ganai reported R&B-EnCoRe achieved an 80.3% success rate and 25.02% object criticality rate, versus 76.1%/0.03% for List of All Objects and 77.0%/3.43% for List of Random Objects. [26:55]
- Ganai reported R&B-EnCoRe improved collision rate across 1s/2s/3s/average time horizons to 0.05/0.17/0.70/0.30, compared to No Reasoning's 0.10/0.25/1.11/0.49. [27:50]
- Ganai's first takeaway was that selective reasoning is more important than exhaustive reasoning, since even valid reasoning is not necessarily useful. [28:22]
- Ganai said the approach generalizes across embodiments (manipulation, navigation, driving) and across VLA model sizes from 1 billion to 30 billion parameters. [28:38]
- Ganai's main claim was that reasoning introduces priors like co-training data, and their "action forcing" technique lets you drop reasoning at inference time to avoid latency problems while keeping the benefits. [31:18]
- Tyler Ga Wei Lum, a Stanford CS PhD student, introduced "SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation," with co-authors Kushal Kedia, Jeannette Bohg, and C. Karen Liu. [33:42]
- Lum said all demo clips are a single policy working zero-shot, meaning it never saw the tools or tasks during training. [33:53]
- Lum said the policy runs at 60 Hz and simultaneously controls a 22 degree-of-freedom hand and 7 degree-of-freedom arm. [34:17]
- Lum said the policy is trained entirely in simulation, with GPU-accelerated simulation running tens of thousands of robots in parallel generating experience about a thousand times faster than real time. [35:00]
- Lum said the key insight is unifying dexterous tool manipulation as goal-reaching, where the policy only needs to move an object from its current pose to a desired goal pose. [36:47]
- Lum said at inference time, FoundationPose and SAM extract goal poses from an RGB-D human video demonstration, which the frozen RL policy tracks at 60 Hz, and the video is not used to train or fine-tune the policy. [37:48]
- Lum said the frozen policy was evaluated zero-shot across 12 unseen tools and 24 target behaviors, with weaker performance on heavier tools and smaller, more occluded objects. [38:52]
- Lum said the Fixed Grasp baseline could grasp the object but had to rotate it using only the arm, causing a table collision that highlighted the importance of in-hand reorientation. [40:02]
- Lum said the Kinematic Retargeting baseline imitated human fingertip motion but didn't reason about contact forces, so it failed to even grasp the object. [40:18]
- Lum said when the robot drops a hammer, RL-trained recovery lets it immediately regrasp the object, and in Q&A explained this emerged from domain randomization rather than explicit training. [40:45]
- Lum said all code, assets, and policy weights for SimToolReal are open-sourced at simtoolreal.github.io, including an in-browser interactive demo that also works on mobile. [40:50]
- Lum said roughly 60% of failures were attributed to pose tracking errors, worse on smaller, more occludable objects than larger ones with more features. [50:29]
- Niko West described the Rerun SDK as an open source library for logging, storing, querying, visualizing, and training on multi-rate, multimodal data. [51:36]
- West said a new category of company is emerging, "robotics application companies" or "neo integrators," that own solving a business problem end-to-end with minimal custom hardware. [52:52]
- West stated his personal belief that robotics application companies will become the new SaaS, driving transformation of the physical-world economy, stated without evidence given. [53:34]
- West proposed a 4-step pattern to start a robotics application company: select a customer problem, solve it with teleoperation and off-the-shelf hardware, set up the basics for learning, then iterate and scale. [54:09]
- In a hypothetical paper-plane-factory example, West said producing over 1,000 perfect planes per day would be needed for viability, and a proper paper tray could reduce failure rate by 50%. [55:29]
- West said "hello world" for setting up learning basics means fine-tuning an open model like Pi on a few hours of teleop demonstration data. [56:49]
- West said physical robotics data differs from web data by being multimodal, multi-rate, episodic, and having complex 3D semantics, making it hard to store in table-oriented systems like Postgres. [01:00:48]
- West pitched that someone should start a robotics application company since the market is enormous, base models keep improving, and real-world friction creates defensible moats. [01:04:21]
- A General Instinct presenter said the team distilled the VAE encoder-decoder and the DiT part, making the DiT smaller. [01:14:33]
- A General Instinct presenter said they split one shared transformer into two separate DiTs, using cross attention from the video transformer to the action transformer so future frames don't need to be decoded. [01:14:45]
- A General Instinct presenter said they distilled the auto-regressive flow matching sampling from about 50-100 steps down to one or two steps, a roughly 50x speedup. [01:15:10]
- A General Instinct presenter said using their infrastructure, the world action model runs at 500 milliseconds per chunk covering 16 actions on Jetson Thor. [01:16:12]
- In Q&A, a General Instinct presenter confirmed distillation cut sampling from about 50 steps to two or three steps without performance drops. [01:17:41]
- A General Instinct presenter distinguished VLA models, which use only the current observation to predict the current action, from world-action models, which predict future imagination to explicitly learn future kinematics. [01:18:34]
- A General Instinct presenter said the stated reason for introducing video prediction into world-action models is to teach the model future kinematics at the pixel level, on the hypothesis that future dynamics help action generation. [01:19:09]
- Asked why build infrastructure for world action models rather than an end-to-end robotics application company, a General Instinct presenter said they believe in building world-model understanding first, aiming to be first to build infrastructure that runs these models on real robots and the edge in real time. [01:21:54]
How this brief was shaped: Discourse (interview / podcast / video essay) · confidence Low
floor demotion: 'lecture_explainer' 0.62 < 0.65 floor -> classifier fallback 'discourse'. The transcript at 21-22 minutes explains named research concepts (embodied reasoning, chain of thought for robotics) and OCR shows a specific academic paper abstract with named authors including Yann LeCun, while the opening is host banter framing a recurring paper club format rather than a personal narrative or product review.
The lens sets this brief's structure, never its facts — every claim is held to the same citation and fact-check standard.
Jump to a moment
Their links, sorted & clickable
🤝 Work with them1
Apply to Y Combinator🔗 Other links1
Work at a startupWas this brief useful?