The PokeBot robot

PokeBot: Physics-Ready Digital Twins from a Phone and a Poke

Yuze Cai, Zehao Wen, Justin Ely, Homanga Bharadhwaj

Johns Hopkins University

Paper Video Code

Abstract

Visual reconstruction tools for real2sim recover object shapes [1, 2], but appearance alone does not uniquely determine interaction physics properties like friction or mass distribution. We introduce PokeBot, a cheap and accessible pipeline which turns human-driven pokes and phone capture into interaction physics rich simulator assets that are useful for real2sim2real manipulation. PokeBot provides the missing interaction measurements: a person pushing objects with a US$20 force probe and a phone/tablet with LiDAR depth. Our key insight is that foundation vision models can now recover object geometry and motion from a phone's colour video and depth [1, 3], while humans can choose informative pushes from our strong object interaction intuitions. Together, these enable system identification [4] with a single force sensor, without motion capture, force plates or robots [5, 6, 7, 8]. PokeBot scales with complex environments and objects. Paired with recent advances in monocular agentic scene-reconstruction methods [9], PokeBot identifies and produces entire structured scenes with sliding and articulated objects, such as a kitchen counter or an office desk. Across 379 held-out pokes on 31 objects and articulated parts, 21 of them in 3 scanned scenes, the fitted twins cut the unseen interaction motion error by a median of 65% compared with assets whose physics come from VLM guesses. We also show the downstream application of PokeBot generated assets in two distinct real2sim2real applications with physical robots.

Scenes

Click on each simulator scene to see how PokeBot made them!

Desk Bedroom Kitchen

Objects not part of a scene

Click on an object and see how PokeBot turns it into a simulator-ready asset.

JLCPCB box Barbecue bottle Scissors Charging deck 3M tape Mechanical keyboard Mechanical keyboard on the wooden table MacBook on the wooden table PLA+ filament box Free-swing microwave

Method

Probe

PokeBot consists of a handheld 3D printed probe housing a force-measuring load cell. [10, 11, 12, 13]

Scene scan

The pipeline starts with a RGB + LiDAR picture of a scene captured from a phone [14]. The depth gives the 3D points of the desk.

Agentic segmentation

A VLM agent prompts SAM 3 to segment every item in the scene. The VLM agent then sorts each item: a sliding object, a jointed part, or both.

Close-up recording

For a sliding object, a human records an up close RGB + LiDAR clip. Multiple pokes are performed at varying speeds, magnitudes, and locations on the object.

Reconstruction and tracking

SAM 3D gives our object mesh and placement. FoundationPose++ then tracks the object mesh through every frame.

Physics fit

In MuJoCo [15], we model the tracked object as a rigid body with 32 contact points following Coulomb friction. A VLM mass estimate bounds the overall object mass search range. A CEM search [16] fits these mass/friction values until the simulated mesh follows the tracked mesh in the simulator.

Camera videoFitted simulation
Grey shoe, push 11, held out

Held-out check

For each push, the fit uses all the other pushes. Then the simulation predicts the push that was left out.

Scene build

Once we individually fit each object's physics in the scene, an agentic pipeline structurally places items together in one simulation scene.

Robot trials

How far can pokes take us? We test PokeBot assets in two real-robot poke2sim2poke tasks with YAM arms limited to pokes planned in MuJoCo.

Laptop retargeting

We have two YAM arms move a laptop through the poses of a human demonstration, turning it and closing its lid, limited to poking motions. The planner splits the demonstration into goal poses, shown in green, and plans each poke with PokeBot-identified physics.

Human demonstrationRobot

Microwave door closing

A human demonstrates how fast to close a microwave, and a YAM arm plans a poke on the door to match the demonstration speed. Planned with PokeBot-identified physics, the door closes. Planned with VLM-inferred physics instead, the door rebounds to 31 degrees open.

Human demonstration, topPokeBot-identified physics, bottom leftVLM-inferred physics, bottom right

BibTeX

@misc{pokebot2026,
  title  = {PokeBot: Physics-Ready Digital Twins from a Phone and a Poke},
  author = {Cai, Yuze and Wen, Zehao and Ely, Justin and Bharadhwaj, Homanga},
  year   = {2026}
}

References

  1. SAM 3D Team, Xingyu Chen, Fu-Jen Chu, Pierre Gleize, Kevin J. Liang, Alexander Sax, Hao Tang, Weiyao Wang, Michelle Guo, Thibaut Hardin, Xiang Li, Aohan Lin, Jiawei Liu, Ziqi Ma, Anushka Sagar, Bowen Song, Xiaodong Wang, Jianing Yang, Bowen Zhang, Piotr Dollár, Georgia Gkioxari, Matt Feiszli, Jitendra Malik. SAM 3D: 3Dfy Anything in Images. arXiv:2511.16624, 2025. arxiv.org/abs/2511.16624
  2. Qiuhong Shen, Xingyi Yang, Xinchao Wang. Anything-3D: Towards Single-view Anything Reconstruction in the Wild. arXiv:2304.10261, 2023. arxiv.org/abs/2304.10261
  3. FoundationPose++ Contributors. FoundationPose++: Simple Tricks Boost FoundationPose Performance in High-Dynamic Scenes. GitHub repository, 2025. github.com/teal024/FoundationPose-plus-plus
  4. Daniel Rho, Jun Myeong Choi, Matthew Thornton, Biswadip Dey, Roni Sengupta. MonoPhysics: Estimating Geometry, Appearance, and Physical Parameters from Monocular Videos. Advances in Neural Information Processing Systems (NeurIPS), 2026. arxiv.org/abs/2605.30320
  5. Tim Engelbracht, René Zurbrügg, Mayank Mittal, Marco Hutter, Marc Pollefeys, Hermann Blum, Zuria Bauer. ForceTwin: Physics-informed Digital Twins for Robotic Manipulation from Instrumented Human Interaction. arXiv:2609.21751, 2026. arxiv.org/abs/2609.21751
  6. Kuan-Ting Yu, Maria Bauza, Nima Fazeli, Alberto Rodriguez. More than a Million Ways to Be Pushed: A High-Fidelity Experimental Dataset of Planar Pushing. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2016. arxiv.org/abs/1604.04038
  7. Nicholas Pfaff, Evelyn Fu, Jeremy Binagia, Phillip Isola, Russ Tedrake. Scalable Real2Sim: Physics-Aware Asset Generation Via Robotic Pick-and-Place Setups. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2025. arxiv.org/abs/2503.00370
  8. Shaoting Peng, Litian Liang, Yixuan Wang, Ming Yang, Katherine Driggs-Campbell, Mark Cutkosky, James Jingxi Xu. HapticWorld: An Interactive World Simulator with Real-time Torque Feedback. arXiv:2609.31924, 2026. arxiv.org/abs/2609.31924
  9. Xiaoxuan Ma, Jiashun Wang, Nicolas Ugrinovic, Yehonathan Litman, Kris Kitani. REST3D: Reconstructing Physically Stable 3D Scenes from a Single Image. arXiv:2605.30338, 2026. arxiv.org/abs/2605.30338
  10. Adafruit. Strain Gauge Load Cell, 4 Wires, 5 kg (product 4541). adafruit.com/product/4541
  11. Nuvoton. NAU7802 24-bit ADC for bridge sensors. nuvoton.com, NAU7802SGI
  12. TDK InvenSense. MPU-6050 six-axis motion sensor. product.tdk.com, MPU-6050
  13. Espressif. ESP32-WROOM-32 datasheet. documentation.espressif.com, ESP32-WROOM-32
  14. Marek Šimonik. Record3D, 3D Videos and Point Cloud (RGBD) Streaming for iOS. record3d.app
  15. Emanuel Todorov, Tom Erez, Yuval Tassa. MuJoCo: A Physics Engine for Model-Based Control. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2012. doi.org/10.1109/IROS.2012.6386109
  16. Pieter-Tjerk de Boer, Dirk P. Kroese, Shie Mannor, Reuven Y. Rubinstein. A Tutorial on the Cross-Entropy Method. Annals of Operations Research 134(1), 2005. doi.org/10.1007/s10479-005-5724-z