Essay 02 · Immersive media
The past, present and future of the digital twin dream
The magic of effortless AR is one step closer
Immersive media was sold to the public as entertainment. Its more durable job is turning out to be quieter: letting a company walk through something that does not exist yet, test it, break it and fix it, and only then build it for real. That object is the digital twin, and AI is what finally makes it useful.
1. The headset, from cockpit to glasses
To see why this moment matters, follow the hardware. The idea of putting information in front of a person's eyes began in the military. Head up displays appeared in combat aircraft in the 1950s and 1960s so a pilot could read critical data without looking down, and helmet mounted displays followed in fighter aircraft. In 1968 the computer scientist Ivan Sutherland built an early head mounted display that showed simple wireframe graphics, a research prototype so heavy it hung from the ceiling.
- 1950s to 1960sThe military HUD. Flight data is projected into the pilot's line of sight, first on the aircraft and then on the helmet. The lesson is that information is most useful when it is already where you are looking.
- 2012 to 2016Consumer VR returns. The Oculus Rift campaign in 2012 and its sale to Facebook in 2014 restart the market. Microsoft unveils HoloLens in 2015 and ships a developer edition in 2016, the first widely seen see through computer for work.
- 2019 to 2020Standalone headsets. Meta's Quest line removes the cable and the PC, and brings VR to a mass audience at a consumer price.
- 2023 to 2024Spatial computing. Meta launches camera and audio Ray-Ban glasses in 2023. Apple announces Vision Pro in 2023 and releases it in February 2024 at $3,499, positioning it as a computer you wear. Microsoft is reported to have ended HoloLens 2 production in 2024.
- 2025The HUD returns, as glasses. Meta Ray-Ban Display goes on sale on September 30, 2025 at $799, with a small display in the right lens and a wristband that reads muscle signals for control.
The pattern is the same one that ran through computing: from the room, to the desk, to the pocket, to the face. Each step made the device smaller and cheaper and put information closer to the moment of use. The most recent step matters most, because glasses that look like glasses can be worn all day. The military's insight, that the right fact in the line of sight changes what a person can do, is now reaching everyone.
2. Great hardware, no killer app: Kinect, HoloLens and Magic Leap
It is worth asking why the companies that arrived first did not win. Microsoft's story is the clearest, because it happened twice.
Kinect: a hit with nothing to do
Microsoft launched the Kinect motion sensor for the Xbox 360 in November 2010, and Guinness World Records recognized it as the fastest selling consumer electronics device in history, with 8 million units sold in its first 60 days. It was a remarkable piece of technology: a depth camera that could track a whole body in a living room. But very few games or apps used it well beyond a handful of titles. Just Dance is the one I remember, and I still have fond memories of dancing my pants off with my little sister. Microsoft stopped making Kinect in October 2017, after roughly 35 million units sold.
The part that frustrated those of us watching was that the obvious next step never came. We expected Microsoft to build it into Windows PCs and laptops, so you could use your electronics without a keyboard, with just your eyes and your face. It stayed a gaming accessory.
HoloLens: a winner for work, with no market behind it
I saw the same pattern again with HoloLens. When HoloLens 2 had just come out, I was working with Microsoft's Mixed Reality team, and we thought they had a complete winner, especially for enterprise, manufacturing and design. The device was impressive and the use cases were real. In February 2025 Microsoft confirmed that it was getting out of HoloLens hardware entirely, with support for HoloLens 2 continuing through 2027 and the military program passing to Anduril. A cool product, but not a market large enough to carry it.
Magic Leap: money without wonder
Magic Leap raised billions of dollars, beginning with a round led by Google in 2014 and later backing from investors including Alibaba, AT&T and Saudi Arabia's sovereign investment fund. It had raised more than $2 billion by 2018. In April 2020 it announced it would cut about 1,000 people, around 46 percent of its staff by one estimate, and turn away from consumers toward enterprise customers, at a valuation reported near $1 billion after reports that it had failed to find a buyer.
The image most associated with Magic Leap's early years is the infamous photo that helped get it funded: a researcher peering into a prototype in the company's optics lab, with glowing wires arcing overhead. The lit up wires did not serve any purpose. It was a picture that sold a dream, long before there was a product to sell.
The skepticism began on day one. On December 20, 2017, as the Magic Leap One Creator Edition was unveiled, Bob Burrough called it a day-one failure, arguing that a product sold as a developer tool was being shown off with web browsing and general purpose computing, and that it was ugly.
The post mortems agree on the cause. One analysis blamed limited consumer interest, high prices, thin content and weak marketing. Bloomberg's account of the collapse put it more bluntly: the hype exceeded the product's capabilities. Capital was never the shortage. The shortage was a reason for an ordinary person to put the product on every day, and a public that was wowed.
The missing piece was generative AI
Looking back, I believe the missing piece was generative AI. A device that can see your face and your surroundings is only impressive if it can understand them and respond with an instant intelligence layer, so the experience feels fun, fluid and effortless, like magic. Without that layer, a headset is a screen strapped to your face and an app developer's problem. With it, the same hardware can read the scene, answer questions and act on your behalf.
The market reaction shows the difference. The chess video made with Meta's glasses and a simple trick played on strangers drew more than 13 million views in less than a week. It was not an enterprise sale or a polished advertisement. It was a person, a pair of ordinary looking glasses and an AI that did something startling. Those are numbers Microsoft and Magic Leap would have killed for, and they came from showing the public a moment of magic, not a specification.
The lesson I take is that hardware does not create a market. A capability that people can feel in the first ten seconds does, and in this generation that capability is intelligence.
3. What builders are asking for, and whether Meta delivered
The people who build with spatial sensing are specific about what they want, and a wish list I came across from one developer sums it up in three points.
| The wish | What it means |
|---|---|
| A. Long range | It has to work in large factories and outdoors, at long distances, and not only in a small living room. The developer notes that phone based AR platforms, Apple's ARKit and Google's Android AR, work only out to about twenty feet, so strapping a phone to a machine is not realistic. |
| B. Open to any system | It should not be tied to one operating system, so it can be dropped into a Raspberry Pi or an NVIDIA Jetson as easily as a phone. |
| C. Hobby priced | Roughly $1,000 to $2,000, so students, tinkerers and small teams can afford to experiment. |
The frustration behind the list is that the field went backward. The Kinect line was cancelled, and Microsoft's Mixed Reality platform was wound down, which removed the cheap and capable spatial sensors that many builders had relied on. Hobbyists and small companies lost the tools that let them experiment.
I think Meta may have pulled it off. The $799 price of the Ray-Ban Display, wristband included, sits below the hobby ceiling in the list. More importantly, Meta has opened its glasses to outside developers. Its Wearables Device Access Toolkit gives apps access to the glasses' camera, microphone and speakers, with Ray-Ban Meta and Ray-Ban Display among the supported models, and early partners have included Twitch, Microsoft, Logitech and Disney. Reports in 2026 describe developer access to the display itself.
That is the combination the Kinect crowd lost: a capable device that looks like ordinary glasses, at a price a hobbyist can reach, with a door open for anyone to build on it. The builders who made Kinect a hacker favorite have a new platform, and I expect them to push it into the long range, run it on anything uses they have been asking for. Whoever gives them that platform will own the next generation of spatial apps.
4. From spectacle to workbench
The first wave of virtual and augmented reality was a consumer story: games, film, social worlds and a great deal of hype. The community that grew around it, the founders, developers and investors trading lists of grants, accelerators and conferences, was built on the belief that headsets would become the next screen.
That belief has been tested, and the verdict is mixed. The consumer market has been slower than the early projections suggested. What has grown steadily is the use of immersive tools inside companies: training, remote assistance, design review and operations. The common thread in the successful uses is not entertainment. It is that a person can see and manipulate a faithful model of something complex, from inside it, instead of reading a drawing or a spreadsheet.
A faithful model that stays connected to the real thing has a name, and it is the subject of this essay.
5. What a digital twin is, and what AI changed
A digital twin is a virtual copy of a physical asset or system that is fed by real data and used to simulate, monitor and plan. A factory, a hospital ward, a pipeline or an organ can each have one. The idea is decades old and was once the preserve of aerospace and heavy industry, because building and maintaining a twin was expensive and slow.
AI changes three things.
It makes twins cheaper to build
Capture techniques such as photogrammetry and, more recently, Gaussian splatting reconstruct a three dimensional scene from photographs and video. Reality capture and digital twins now have dedicated tracks at the industry's main spatial computing conference, AWE, which tells you where the attention has moved. Instead of modeling a building by hand, a company can scan it.
It makes simulation fast enough to be interactive
Physics simulation used to run for hours and stay in the engineering department. Trained AI models can approximate those simulations in minutes or seconds, which turns an overnight job into something a planner can try live, in a headset or on a screen, and adjust.
It keeps the twin honest
A twin is only valuable if it tracks reality. Machine learning can reconcile sensor data with the model, flag where the two diverge and predict failures before they happen. A static model becomes a living one.
Immersive media is the way people meet all of this. It is the interface on top of the twin.
6. Three case studies
Case 1
BMW: the factory built before the factory
BMW Group has used NVIDIA's Omniverse platform to build factory twins and a planning tool that lets engineers test robot placement and layouts inside the virtual plant before anything is installed. NVIDIA reports that BMW achieves gains of up to 30 percent in efficiency when planning new, greenfield facilities this way. The same source describes real time collaboration between planners in different locations working in one shared model.
Lesson: the biggest savings come before construction, when changing a design costs a conversation instead of a rebuild. The figure is the vendor's own, so I treat it as a direction, not an audited result.
Case 2
Foxconn: simulating heat and moving plants
Foxconn, one of the world's largest electronics manufacturers, uses digital twins to validate how its factories will behave. In an NVIDIA case study, the company reports AI based thermal simulations running about 150 times faster than conventional computational fluid dynamics, cutting hours to minutes. It also reuses twin assets to move production lines between plants, and for a facility in Houston it validates mechanical, electrical and plumbing systems virtually before construction starts.
Lesson: once a twin exists it becomes an asset that can be copied. A line that works in one place can be reproduced in another without starting over. The 150 times figure is also vendor reported.
Case 3
Healthcare: the twin of a heart, and a liver
Dassault Systemes describes its Living Heart project as the first commercial human heart simulator, used to study cardiac function and to test devices such as valves and stents in silico, before a patient is involved. In early 2025 the company announced an AI powered version that can be configured for individual patients or populations, and the project has extended to organs including the liver, lungs and brain. Siemens Healthineers is working toward a full patient twin built from a person's medical history and records, and has developed a digital liver model with INRIA intended to help surgeons judge whether an operation is safe, as well as a heart model used at Heidelberg University Hospital.
Lesson: the same idea that plans a factory can plan a surgery. The patient twin is still largely a goal rather than a finished product, but organ level twins are real and in clinical use. This is also the part that connects most directly to my interest in regenerative healthcare.
7. The data center is the next twin
My own field offers the clearest case for building something twice. A modern AI data center is a tightly coupled system of power, cooling, networking and compute, and an error in layout or capacity is expensive to correct once steel is up and equipment is installed. NVIDIA has published a blueprint for building digital twins of data centers and what it calls AI factories, and software vendors including those that sell power and cooling equipment are building similar tools.
In my opinion this becomes standard practice. Before an operator commits capital to a site, it will model the electrical one line diagram, the thermal behavior of the hall, the construction sequence and the delivery schedule of long lead equipment, and then walk investors and partners through the result in an immersive view. The twin also keeps working after the build, as the operating model of the facility. I have not seen independent, quantified results for data center twins yet, so this is a prediction rather than a finding.
8. How big is the market?
Market research firms agree that the digital twin market is growing quickly, and disagree on how large it is. One widely cited forecast puts it at about $21 billion in 2025 and about $150 billion in 2030. Others land between roughly $49 billion and $196 billion for 2030.
A spread of four to one is a sign that the firms are measuring different things. Some count only software, others include services, sensors and integration. I would not anchor on any single number. What matters is that every firm points up, and steeply.
9. Life through the lens
A headset that sees what you see, knows what you are doing and can answer in real time is a platform for applications nobody has built yet. The ones I find most promising follow below.
A chess demo, and what it points to
A recent TikTok by a creator known as SourGrape02 showed a man in a park playing a stranger at chess while smart glasses fed him moves. The reported setup was simple: the glasses streamed the board, an AI model read the pieces, a chess engine chose the move and the answer came back as audio. The woman across the board was visibly stunned by an opponent who played like a grandmaster. The video drew skepticism, including from well known chess commentators, who called it more performance than breakthrough, and the setup was reproduced in a public project, Chess hacks for Meta Glasses, a macOS command line tool that follows the same pipeline: the glasses video call a Mac, a vision model reads the board, the Stockfish engine picks the reply and a voice speaks it into the call. Its page does not name a license, and it says the tool was tested on recordings and not yet on a live call. It is also a reminder that the same tool is a cheating device in a tournament.
I include it because it is funny and because it is a clean preview. A phone sized brain, a camera and a voice in your ear already turn an ordinary person into a world class player of a closed game. Replace chess with anything else that has rules and a state you can see.
Apps that read the scene
Once the glasses can see, developers can build tools that check the scene for you. A quality inspector can scan a part and have the headset highlight oddities in manufacturing. A security guard can have a tool that flags unusual behavior or objects in a public space, which is useful and also the point where privacy and misuse must be taken most seriously. A traveler can have live translation shown as subtitles under the people they are talking to. These are all existing capabilities, packaged into something you wear.
From tracking to coaching
Wearables such as WHOOP, the Oura Ring and the Apple Watch have taught millions of people to measure their sleep, recovery, heart rate and training load, and they have helped many of them become healthier. But they are mirrors. They tell you what happened and leave you to decide what to do.
Now imagine the best personal trainer in the world coaching you every day in your HUD. Picture someone like Arnold Schwarzenegger, the bodybuilder who defined the sport, talking you through your session. The glasses see your form and count your reps. Your ring and band report how recovered you are, so the coach knows whether today is a day to push or to hold back. It adjusts the weight, the tempo and the rest time as you go, and tells you when to stop. The data from the wearables becomes the coach's memory, and the headset becomes its eyes and voice.
This is the licensed expertise idea in its most practical form. The trainer is not a novelty character. It is a scarce, expensive skill delivered to anyone, at the moment they need it, with their own body's data behind it. It would of course require the person's consent and a proper license to use their name, voice and likeness, and a medical safeguard for people with injuries or conditions. Done properly, I think it is one of the first immersive applications that millions of people will pay for every month.
Five directions I expect
1. The gamification of life. You will be able to play the game of life. If you lift weights, as I do, the display could show points or coins for doing three more reps than your last session, and a friend could join your workout as a digital avatar and compete with you from another city. Effort becomes visible and scored, and motivation becomes ambient.
2. Collaboration in the factory. A worker on the floor can video call a manager or colleague who sees the same thing. A flat two dimensional design turns into a three dimensional model standing in front of both of them, and either can change it with voice and gaze. It connects directly to the digital twin: the twin is the shared object, and the headset is how both people reach into it.
3. Augmented commerce. You see an outfit you like, and the system already knows your size, pays for it and ships it to your home. Or you pick it off the rack and skip the fitting room, the search for a fit and the queue at the counter. The whole transaction can be a wink. This is where the first essay on this site meets this one: the glasses are the interface, and an agent pays on your behalf using machine payments, within limits you set.
4. Licensed expertise. A coach, a teacher or a celebrity could scan themselves and their knowledge and license a digital version to people they will never meet, who bring it into the problem in front of them. I would love to have Albert Einstein or Richard Feynman advise me on my daily challenges. That cannot happen without the permission of their estates and a framework for likeness and ownership, which does not yet exist. For living professionals it is closer: the agentic version of a person solving problems and giving guidance at a scale one person cannot match.
5. A metaverse that is the world. The metaverse Meta set out to build was imagined as a place you go to. I expect it to arrive as a layer on the place you already are, built by developers who write their own apps for what the glasses can see, hear and understand.
This is the world science fiction has described for decades, with knowledge and information in front of us, wherever we look. The industry that builds it will have to grow up quickly, with better tools for developers, clear rules for privacy and consent, and business models that reward creators and professionals rather than only platforms.
10. Where agents come in
This essay connects to my earlier one on agentic payments. A twin is a place where decisions can be tried at no cost, which is exactly where autonomous agents are most useful. An agent can run thousands of layouts, schedules or treatment scenarios inside a twin and bring back the best, and it will need to buy simulation compute, data and specialist models along the way. Machine payments are how that purchasing happens. The twin supplies the sandbox, the agent supplies the search, and the human supplies the goal and the final signature.
11. What can go wrong
- Fidelity. A twin that drifts from reality gives confident wrong answers. Keeping it calibrated is ongoing work.
- Vendor numbers. Most published results come from the companies selling the platforms. Independent measurement is scarce.
- Interoperability. Twins built on different platforms and standards do not easily talk to each other, and that limits their reuse.
- People. A twin of a person is among the most sensitive data there is. Who owns it, who may use it and who profits are open questions that regulation has barely begun to address.
- Cost of capture. Scanning and sensing a large asset is still a real expense, even as it falls.
12. A builder's checklist
| Question | Why it matters |
|---|---|
| What decision would a twin let you make earlier? | Value comes from moving a decision before the money is spent. |
| Can you capture the asset instead of modeling it by hand? | Scanning has become cheaper than drawing. |
| Which simulation is your slowest, and can a trained model approximate it? | Speed turns analysis into exploration. |
| How will the twin stay in sync with the real asset? | A stale twin is worse than none. |
| Who sees it, and in what interface? | The headset or screen is how non specialists understand the model. |
| Who owns the data in the twin? | Settle ownership before the first scan, especially for people. |
Conclusion
The early immersive media community assembled directories of grants, tools, conferences and courses because it believed a new medium was being born. It was, though not the one most of them expected. The medium that is taking hold is the twin: a copy of the world that can be entered, tested and changed at no cost, with AI doing the heavy lifting. Headsets are becoming glasses, and glasses are becoming the way people reach the twin, play with friends, buy things and borrow expertise. The companies that learn to build things twice, once as a model and once as reality, will make fewer expensive mistakes. Whether it is a factory, a data center or a heart, I expect that to become the normal way to build anything that matters.