But this raises an important question: What kind of simulation is truly valuable?
The answer is simple: not all simulations are created equal.
In this installment of SynReal Decode, we continue this discussion: if simulation is the gateway for AI to enter the real world, then what makes a simulation truly effective? When AI begins to engage with the real world, it quickly becomes clear that simulating a world that merely looks realistic is not difficult. However, simulating a world that behaves correctly is far more challenging. This difficulty increases dramatically when objects transition from rigid bodies to deformable materials such as fabrics and soft objects.
In this article, we unpack why deformable object simulation is emerging as a key dividing line in the development of physical AI.
1. Three Levels of Simulation
From a technical perspective, simulation is not a single concept but can be divided into different levels. In current AI and robotics research, it can be broadly categorized into three layers, each increasing in both difficulty and value.
Level 1: 3D Environment and Space Simulation
The first level focuses on simulating the three-dimensional structure of the physical world. The core objective of this level is to reconstruct the spatial structure and three-dimensional geometric relationships of the real world, enabling AI systems to develop fundamental 3D perception and spatial understanding, and thereby make reasonable decisions within their environment.
For example, video generation models often produce inconsistencies in object size or spatial relationships. This stems from the model’s lack of a genuine understanding of three-dimensional spatial structure. By training models within virtual 3D environments—where they can observe, reason, and learn—these issues can be significantly reduced, allowing models to develop a more stable understanding of spatial perceptions.
In robot navigation and autonomous driving, virtual cities, roads, and indoor environments enable AI systems to learn localization, obstacle avoidance, and path planning. Once AI acquires an understanding of three-dimensional space, it can be characterized as exhibiting “spatial intelligence.”
Level 2: Rigid Body Physical Simulation
The second level is the physical simulation of rigid objects. While the first level deals primarily with static environments, this level focuses on how interactions affect object states and positions.
In reality, robots themselves are composed of rigid components connected by joints. Walking, climbing, and balancing all involve physical interactions with rigid surfaces such as the ground. The ability of modern robots to traverse complex terrain is largely due to the success of combining rigid body simulation with reinforcement learning.
Similarly, robotic arms can be trained to grasp objects such as cups or smartphones using rigid body simulation. This level of simulation is relatively mature and has already seen widespread adoption in both academia and industry.
Level 3: Deformable Object and Complex Physics Simulation
While the first two levels have achieved significant progress, the third level presents far greater challenges, and remains a major focus of current research.
This level involves simulating deformable objects and complex physical phenomena such as fluids. In the real world, most objects are not rigid, from clothing, curtains, and sheets to plastic bags, packaging materials, plants, and even water and air. Strictly speaking, fluids do not have a fixed shape and can be regarded as an extreme form of deformable objects.
Unlike rigid objects, deformable objects do not have a single stable form. Their shape continuously changes depending on forces, environment, and contact conditions. This makes them significantly more difficult for AI systems to perceive, understand, and manipulate.
This challenge is evident in real-world AI systems. Take cloth folding as an example: although robots today can perform this task, most rely heavily on large volumes of real-world demonstration data through imitation learning. In essence, these systems generalize within a limited range based on thousands of human demonstrations. Once faced with more complex garment styles, or even a change in fabric material, the original model may fail to function properly.
From Simulation to Physical Intelligence
We believe that only by enabling AI to truly understand the physical world and its underlying laws, can it interact reliably within complex environments, thereby achieving higher levels of intelligence. Only AI with such capabilities can be considered as physical AI.
In this process, deformable object simulation plays a crucial role. It represents the highest level of simulation and data synthesis capability, directly determining how complex and realistic the physical behaviors AI can learn, and ultimately defining the upper limit of physical AI.
2. Where Does Deformable Object Simulation Come From?
If deformable object simulation is so important, where does it originate?
It is not a standalone invention, but rather the result of the convergence of two major research domains.
Source 1: Computational Physics and Engineering Simulation
The first source lies in the numerical simulation technology widely used in traditional engineering disciplines, such as computational physics and mechanical engineering. These methods are primarily used to simulate deformation under force, helping evaluate structural safety, material reliability, and design feasibility.
As a result, numerical accuracy is the top priority. Simulation outputs such as stress, deformation, and stability often directly determine the success or failure of engineering outcomes, leaving little room for error.
However, from the perspective of artificial intelligence and data synthesis, these types of simulation also exhibit notable limitations. First, they typically do not prioritize visual realism, and often present results in a simplified form. For example, in engineering mechanics, analyses are frequently conducted on two-dimensional cross-sections rather than fully simulating three-dimensional appearance. Second, such simulations often do not adequately handle complex contact and collision interactions. More importantly, engineering simulation is not typically designed with throughput as a primary objective, resulting in high computational costs and relatively slow performance.
For data synthesis, however, the core advantage of simulation lies precisely in its cost-effectiveness. If a single simulation requires substantial computational resources and time, it becomes difficult to support large-scale data generation.
Source 2: Computer Graphics
The second source is computer graphics, originally developed for film, animation, and gaming. The realistic cloth, hair, smoke, and fluid effects we see in films today are all made possible by physical simulation techniques in computer graphics.
The defining feature of this approach is its emphasis on visual realism. As long as the result looks convincing, some simplifications in physical accuracy are acceptable. This aligns well with AI training needs, where perceptual realism often matters more than engineering-grade precision.
On the other hand, computer graphics has long been constrained by performance limitations. Achieving the highest possible level of realism under limited computational resources while reducing cost and improving efficiency has been a central challenge in the field.
An extreme yet classic example is the video game, where real-time physics must run under strict computational limits, which forces researchers to continuously balance simulation fidelity against computational cost. Many efficient simulation methods emerged from this context. For instance, NVIDIA’s PhysX engine—widely used in robotics simulation today—was originally designed for real-time physics in games.
The Convergence of Two Paths
It’s important to know that these two sources are not isolated, instead, they have continuously influenced each other. From a practical application perspective, physical simulation techniques developed in computer graphics align closely with the needs of AI for simulation and data synthesis.
More broadly, computer graphics and computer vision—an important branch of AI focused on understanding images and video—can be seen as two sides of the same coin: on one hand, computer graphics is dedicated to constructing virtual worlds and generating images and video; on the other, computer vision seeks to understand the real world from images and video.
At the intersection of these fields, deformable object simulation has emerged as a key bridge between virtual and physical reality, forming essential infrastructure for physical AI.
3. The Challenges of Deformable Object Simulation
If deformable object simulation is so critical, why does it remain so difficult?
In practice, the core challenges can be summarized in two words: accuracy and speed.
Challenge 1: Accuracy
Simulation itself aims to replicate the real world as closely as possible. The gap between simulation and reality, commonly referred to as the Sim-to-Real Gap, is a primary challenge.
Real-world materials exhibit complex properties. Differences in fabric type, thickness, elasticity, and friction significantly affect the motion and appearance of deformable objects. These properties are difficult to measure and even harder to model precisely. In many cases, simulation can only approximate behavior at the model level, inevitably introducing discrepancies with the real world.
On the other hand, deformable objects are often accompanied by a large number of complex contact and friction relationships. These interactions continuously appear and disappear as shapes change. For example, during cloth folding, the fabric comes into contact not only with the table but also forms multiple layers of self-contact. Handling such dynamic, multi-point, multi-layer contact without instability or penetration remains one of the most difficult problems in simulation. Even in rigid body simulation, contact handling is already highly complex; in deformable scenarios, this complexity is further amplified.
Challenge 2: Speed
The second challenge is efficiency. In film visual effects, it is indeed possible to achieve highly realistic results, but often without regard for cost. However, once simulation is applied to large-scale data synthesis, efficiency becomes an unavoidable core challenge. Only when simulation runs sufficiently fast can data be generated at scale.
Against this backdrop, efficiency-related challenges begin to emerge. A common trade-off is sacrificing accuracy for speed: using coarser time steps or simplified models. While this improves performance, it reduces physical realism.
For example, NVIDIA PhysX, based on Position-Based Dynamics (PBD), is highly efficient for small-scale problems and widely used in real-time applications such as video games. However, it has inherent limitations in physical accuracy, and its performance degrades as problem complexity increases.
Balancing Accuracy and Efficiency
Ultimately, accuracy and efficiency are fundamentally in tension. The evolution of deformable object simulation is essentially a continuous search for balance between the two.
For a long time, computer graphics has been dedicated to achieving a better trade-off between realism and efficiency. This balance continues to evolve alongside advances in computational power. Over the past decade, the rapid development of parallel computing technologies, particularly GPUs, has significantly accelerated deformable simulation in terms of both speed and scale. Designing simulation algorithms that better adapt to parallel hardware such as GPUs, and thereby achieve higher data throughput, has become an important research direction in deformable object simulation.
4. Our Work: Toward Scalable Deformable Object Simulation for Physical AI
In the preceding sections, we discussed why deformable object simulation is important and analyzed why it is so challenging. Against this backdrop, the work of the three of us—Huamin Wang, Fanfu Jiangchen, and Yin Yang—aims to address a concrete and practical question: can we build a deformable object simulation system that is sufficiently accurate, sufficiently efficient, while also being able to scale reliably, and truly serve the training and learning needs of artificial intelligence?
Our research spans computer graphics, physical simulation, and AI, converging toward a unified goal: transforming deformable object simulation from a research prototype into a foundational infrastructure for physical AI.
Accuracy: Physical Accuracy from Models to Materials (Huamin Wang)
The realism of deformable object simulation depends largely on the underlying materials. Different fabrics, rubbers, and soft materials exhibit vastly different behaviors in bending, stretching, and friction. If the material model deviates significantly from the real world, then even a numerically stable simulation will struggle to provide truly valuable training data for artificial intelligence.
Hence, one of our key research directions is physical material modeling, as well as the acquisition and modeling of real-world material parameters. This work focuses on two main aspects: first, how to design reasonable and controllable models that capture the key behavioral differences of various materials under force and contact; second, how to obtain material parameters from the real world through experiments and measurements, and map them into simulation models.
The goal of this process is not to achieve engineering-level precision, but to ensure that materials in simulation behave realistically at the behavioral level, while remaining sufficiently controllable and generalizable. Only in this way can simulated data remain meaningful for AI learning across different objects and material types.
Speed: Why Deformable Object Simulation Must Be GPU-Based (Huamin Wang, Yin Yang)
For artificial intelligence, simulation is not a tool that is run occasionally, but a data engine that must operate continuously, at high frequency, and at large scale. This means that traditional CPU-centric simulation methods, designed for single-run high-precision computation, are difficult to meet practical requirements.
Therefore, from the very beginning, we placed the implementation focus of deformable object simulation on GPUs. Built around GPU parallel architectures, we redesigned the entire simulation pipeline to significantly improve throughput while maintaining basic physical plausibility, thereby enabling large-scale data generation.
The focus of this direction is not on achieving maximum numerical precision, but on running sufficiently large-scale and fast simulations under an acceptable level of realism, thereby making simulation practically usable and capable of keeping pace with the continuous growth of AI model scale.
Stability: Achieving Stability in Complex Contact Through IPC (Huamin Wang, Yin Yang, Fanfu Jiangchen)
In deformable object simulation, failures are most often caused not by deformation itself, but by contact. For robot training, the ability to steadily handle friction and contact between robotic arms and deformable objects is a critical factor in determining whether reliable manipulation skills can be learned.
This is especially true in scenarios involving clothing and fabrics, where objects not only interact with the environment but also undergo extensive self-contact. If contact handling is unstable, simulation can easily result in penetration, oscillation, or even complete failure.
To address this, we jointly contributed to the development of contact handling methods based on the idea of IPC (Incremental Potential Contact). A key advantage of IPC is that it strictly prevents penetration at the numerical level, while maintaining simulation stability even under frequent multi-point, multi-layer, and self-contact scenarios.
By integrating IPC with GPU-based deformable object simulation, we achieved a better balance between stability and efficiency, which is particularly critical for long-running data synthesis tasks.
Scalability: Introducing MPM for Large Deformations and Complex Material Behavior (Fanfu Jiangchen)
In addition to mesh-based deformable object models, we have also systematically explored the potential of continuum methods such as the Material Point Method (MPM) in deformable object simulation. This direction primarily focuses on more steadily handling large deformations, material separation and recombination, as well as complex contact behaviors.
Comparing with traditional mesh-based methods, MPM tends to be more stable under extreme deformations, folding, and topological changes. This makes it particularly suitable for simulating cloth, granular materials, soft bodies, and even certain fluid behaviors. We further explored how to combine MPM with GPU-based parallel computing, enabling it to maintain physical plausibility while achieving sufficient efficiency for large-scale simulation and data generation.
A Unified System
It is important to emphasize that we do not view material modeling, GPU-based simulation, IPC, or MPM as isolated technical modules. The key question is: whether they can be integrated into a unified, practical, and scalable deformable object simulation system.
In such a system:
-
Material models and real parameter collection provide physical realism;
-
GPU-based deformable object simulation offers scalable data throughput capabilities;
-
IPC ensures stability under complex contact conditions;
-
MPM extends the range of deformable object simulations and material types.
The shared goal of these components is to enable artificial intelligence to repeatedly interact with, understand, and manipulate the most complex and common flexible objects in the real world within a simulation environment.
Conclusion
Returning to the original question, for artificial intelligence to truly enter the real world, the key is not just “understanding,” but whether it can reliably interact with the physical world. And in this process, deformable objects are everywhere, and they are the most challenging to handle.
This is why the physical simulation of deformable objects is an indispensable key step towards physical intelligence. Only when we are able to simulate these flexible objects in a sufficiently realistic, efficient, and stable manner can simulation data truly support artificial intelligence in moving from perception to action, and from understanding to practice.
This is not the endpoint, but the construction of an infrastructure.
The future of physical intelligence is gradually unfolding, starting with deformable object simulation.
Stay tuned for SynReal Decode, we will continue discussing embodied intelligence in the next article.
