The quest to save Notre Dame brought together unlikely game giants

Notre Dame is cleaner than we’ve ever seen it before. That’s what stood out to Maxime Durand, Ubisoft’s resident historian, when he saw photos of the newly restored nave.

“It’s the first time that we can actually see the full monument with this splendor,” he told Polygon. “Since the industrial revolution, the air pollution has affected the color of any landmark in Europe, and Notre Dame was part of that. It’s the only, and first, time we can actually see the full monument with this marble-white kind of color — well, limestone — but it’s very unique and we just got to appreciate that.”

After a 2019 fire, the cathedral — which took 182 years to build and was officially completed in 1345 — is finally open to the public again. The restoration efforts weren’t only about repairing damage from the fire, though. This was an unprecedented opportunity to learn more about the monument, its artistry, and the techniques that were used to build it. The work mobilized 175 researchers with specializations in stone, woodworking, acoustics, metal, and glass.

It also took cutting-edge technology, with some unexpected ties to video games.

The real Notre Dame, and the Unreal Engine

It was December 2022 and below freezing when Jennifer Feltman first photographed Notre Dame.

“They gave me two jumpsuits to wear — you don’t get to keep any of your clothes. You have Tyvek jumpsuits, and some shorts and a T-shirt. It’s like you’re getting an MRI or something,” she said.

Feltman is an associate professor of medieval art and architecture at the University of Alabama. I first spoke to her in 2019, when I was researching the story behind Assassin’s Creed Unity’s Notre Dame, because I wanted to ask an expert how the game’s cathedral differed from the real thing, and learn more about why certain aspects of its design are under copyright.

At the time, Feltman was part of the research group studying Notre Dame de Paris, but she was also working with a team of developers on a VR model for a different famous Notre Dame — Notre Dame de Reims, which sits in the heart of champagne country, 90 miles from Paris.

But in 2022, she became one of the chosen few to visit the Paris worksite. She did photogrammetry of the cathedral’s facade, capturing the detailed stonework on famous features like Notre Dame’s Judgment Door.

The Judgment Door is the central and largest portal on Notre Dame’s western facade. The carvings show the dead rising from their graves as the archangel Michael weighs their souls. The damned are led to hell by a grimacing demon, while the chosen look up to heaven and Christ, enthroned above them. 

In photogrammetry, high-resolution images are used to create 3D models. These models end up the same quality or even better than scans done with laser technology, Feltman said. 

“With the laser, you can miss portions of the surface — you might have a gap. When you have that gap, either it’s a hole, or the software […] will interpret the gap and try to fill it in,” she said. This can create a warped image of the structure — not great for restoration work.

The same thing can happen with photogrammetry, but it’s the photographer’s job to take as many overlapping photos as possible, to prevent that from happening.

“It’s easily 10,000 images that I have acquired in five years,” Feltman says. “I know our photogrammetry model is at least 2,000 images just of the facade.”

Equipped with a pole-mounted Sony a7R camera with a 50mm lens, Feltman did the laborious work of photographing the world’s most famous cathedral.

“My colleague described it to me — it’s like doing a kata, a form, and you look really awkward and you could pull a muscle,” Feltman said. “I did it for eight hours.”

These photos are then stitched together using RealityCapture — Epic Games’ photogrammetry software, which pipes into Unreal Engine.

“Some people do photogrammetry to make machine parts or reverse engineer a car,” Feltman said. “Others use it to make models for video games. We’re using it to document cultural heritage and put it back together, possibly.”

Or to find out what color it used to be. The cathedral of the public imagination is a yellowy-gray limestone — it glows nearly white in sunlight, but over the centuries has developed a warm golden crust.

But a medieval worshipper showing up to Sunday service might have seen its sculptures in vivid color.

“Typically they did a coating, and it could be lead-white or could be lead-white with yellow ochre,” Feltman said. “But then on top of these, they would add different mineral pigments.”

Alongside her colleague Stephanie Duchêne, a chemist at the Historic Monuments Research Lab, Feltman gathered stratigraphic samples of paint fragments. The colors are so degraded that they can’t be assessed by the human eye. 

“They’re all jewel-tone type colors, like the stained glass. Beautiful blues, yellow, red, really deep green and vermilion,” Feltman said. Her team was able to paint digital textures onto the 3D model of the Judgment Door using Blender, the open-source 3D modeling software. The cathedral’s brilliant colors could be digitally restored, all while the building itself maintained its austerity.

Restoration work is always a tricky balancing act between preservation and future-proofing. Surely, if someone proposed painting the cathedral today, the public outcry would be enormous. This 3D technology allows us to better understand how the cathedral would’ve been seen when it was built, without sacrificing our modern conception of it. 

Notre Dame’s appearance hasn’t always been treated with such reverence. During the French Revolution, the cathedral was used as a storehouse, and statues of biblical kings on the facade were beheaded by revolutionaries. But sometimes it was the monarchy that was responsible for damaging the cathedral.

“Even before the revolution, in the baroque era, the central portal was cut in two with an opening so that a French king could be brought in in a fancy ceremony and not need to be lowered,” Feltman said. “They also removed the central column figure of Christ. We still don’t know where it is.”

For Feltman and her team, photogrammetry serves to document the cathedral as it is now — and to understand virtually how it might have looked once, when all the fragments are put back together. The computing power required for this work is massive.

“You have to have a high-end graphics card to run the software to begin with,” Feltman said. “And when you start feeding it really large files — and these are initially shot in RAW, but we convert them to TIF format — those are challenging for most machines to process.”

The software uses alignment points, often manually tweaked by the researchers, to stitch together the photos and create hi-res models. The whole process used to take 20 hours, but thanks to a new high-performance computing center at the University of Alabama, Feltman says that time is down to 20 minutes. 

“Essentially, we’ve been able to run the computer program on nodes that allow multiple machines to be accessing the data,” she said. “I’m frequently finding myself, as an art historian, hanging out with computer scientists and chemists and geologists and others. So that’s been a lot of fun.” When Feltman and I first spoke in 2021, none of this existed at her university.

“It’s becoming clear to me that this is important, for the humanities to collaborate with scientists for, frankly, our existence,” Feltman said. “But on the other hand, I’ve found that they like us too, because we make their projects more interesting and the public cares. There’s an emotional response to Notre Dame that you wouldn’t have if it was just a computer program.”

It’s also created a data management nightmare. 

“As it turns out, digital models are more fragile than stone in terms of their longevity, really,” Feltman said. “I asked some of my colleagues in [other institutions], Where are you storing? How are you backing up your material? And they’re like, Well, this is kind of a joke, but here, it’s on this USB key.”

All jokes aside (and she double-emphasized it was a joke), there are a lot of outstanding questions about what happens to these models and the data moving forward. How can they be stored and preserved? What happens to the RAW images? How can models and data be shared between institutions? 

“This is one of the issues with working on the edge of something, and also with working in the tech world, because it changes so rapidly,” Feltman said. She also hopes that someday the models may be used in museums.

Notre Dame in (virtual) reality

Epic’s software isn’t the only part of the restoration where the high-tech overlaps with the medieval. Feltman experienced this herself when, during one of her visits to Paris, she saw Notre Dame in VR.

“You start out with the cathedral as a point cloud model,” Feltman said. “We did a tour […] of this experience, and we’re all geeking out, right?”

The Cité de l’architecture et du patrimoine is a smooth stone curve of a museum that overlooks the Eiffel Tower. It’s a temple to architecture, housing not just plaster casts of monumental sculpture and stained glass, but also a school dedicated to restoring historical structures. While Notre Dame underwent restoration, it hosted the copper statues of apostles that had flanked Notre Dame’s spire. 

And one spring, while dying of jet lag, Feltman visited the virtual Notre Dame in the museum’s basement. 

“We’re looking at the vaults together, and I can see my colleague, […] but they look weird. You don’t see their real body,” Feltman said. “You just see kind of a representation. And at some point when they take [the VR gear] off, it looks like this head that’s kind of floating in space.”

This VR Notre Dame is built from point cloud data collected by the late professor Andrew Tallon of Vassar College. A point cloud is a bunch of data points laid out in 3D space — it’s surreal, unpolished, and yet incredibly detailed. 

“Andrew Tallon […] was really a pioneer in the field of using LiDAR to scan cathedrals in France and create highly accurate models,” Feltman told me back in 2021. “It’s his model that often is cited as the one that’s going to save Notre Dame.”

Tallon used a laser scanner to gather some billion points of data that have allowed scientists to better understand the flying buttresses that support Notre Dame’s walls, and the vault that soars over its central nave.

This vault was heavily damaged in the fires. Posthumously, Tallon helped to preserve it.

Models like this one can’t replace experiencing monuments in real life. But they’re still valuable for researchers. Tallon’s data, for example, revealed many tiny imperfections in Notre Dame — deviations that aren’t noticeable to anyone who isn’t measuring very, very carefully. In the case of restoration, a limited number of workers were actually allowed on the delicate site. Many more people could access and study a virtual cathedral, and they wouldn’t have to fly to Paris to do it.

Another helpful virtual model was created by Ubisoft, based on the very same cathedral that appears in Assassin’s Creed Unity.

Since Notre Dame closed for restoration, Unity has been one of the few ways to appreciate its splendor. Caroline Miousse, associate art director on Unity, worked with her team to make Unity’s Notre Dame as believable as possible, while also adhering to the strictures of copyright and the needs of gameplay.

It does not matter that the game is now over 10 years old. The light streaming through the rose windows, the sheen of the marble, the towering scale — it all hangs together perfectly.

Maybe too perfectly. I’ve written before, and at length, about the false belief that Ubisoft’s models of Notre Dame were used by scientists restoring it. At the time, I couldn’t reach Ubisoft for comment, but the company had made plenty of statements that added up to essentially what Caroline Miousse told me in our interview:

“That was totally rumors, actually! We were involved in a different way for the restoration,” she said.

Ubisoft donated €500,000 (around $540,000) to Notre Dame and made Assassin’s Creed Unity available for free on PC for a week following the fire. Meanwhile, at Ubisoft Paris, Mélanie de Riberolles’ team jumped into action. The team lifted Miousse’s Notre Dame from Assassin’s Creed Unity, then a 5-year-old game, and made it explorable in virtual reality.

It only took two months.

“We had never tried yet to extract assets from Assassin’s Creed games because it was supposed to be quite difficult,” de Riberolles said. She has been with Ubisoft since 1994 and now serves as director, new business, developing VR experiences for the company. “But we really felt called to do something in VR.” The end result is Notre-Dame de Paris: Journey Back in Time, which made its debut at UNESCO’s Heritage Days in 2019 and was later released on Meta Quest. 

In Unity, Notre Dame is being used to stockpile goods for the revolution, and patriotic banners hang from the columns. De Riberolles’ team stripped all this away. 

“We wanted something more contemplative, not religious, but not storytelling,” de Riberolles said. “It’s just being in the monument, and contemplating the beauty of the architecture and the views.”

Fortunately for de Riberolles and her team, Miousse and Durand had already made creative decisions that made this process easier.

“During the French Revolution, the revolutionaries would disfigure these statues […] because they thought these were the French kings,” Durand said. “But in the game’s version, they’re not desecrated, because I think it still plays a part in showing the magnificence of the monument.” 

“We were lucky also that the team chose to add the spire, even if the spire at the time was not the real one from Viollet-le-Duc,” de Riberolles said. Notre-Dame’s decrepit wooden spire had been torn down in the 1790s, and the cathedral went topless until the 19th-century renovations were made. Miousse and her team made the call to add the spire that players would recognize, even if it didn’t exist during the French Revolution.

But 18th-century Paris is still the backdrop of the VR experience. Players can stand on the roof and see monuments like the now-destroyed Bastille in the distance. The nave is decorated with massive paintings of the apostles — a series called “the Mays,” which in the modern day can be found in the chapels and the transept. The massive organ plays music from Bach.

“That was recorded six months before the fire on the real [Notre Dame] organ,” de Riberolles said. “And honestly, that adds a lot to the immersion and the emotion.”

From the start, de Riberolles was in contact with museums to see if they had any use for the VR cathedral. It eventually found a temporary home in the archaeological crypt underneath île de la Cité. It was part of an exhibit on the history of Notre Dame, in which the visitor is accompanied by digital tour guides Eugène Viollet-le-Duc and Victor Hugo. 

Hugo, the author of The Hunchback of Notre-Dame, popularized the then-decrepit cathedral in the imagination of Parisians in the 19th century. Once Hugo had made Notre Dame a symbol of French heritage, Viollet-le-Duc was one of the architects responsible for restoring it to glory. He’s the one who topped the cathedral with the iconic spire that burned in 2019.

De Riberolles also captured videos of the model for the official Notre Dame tourism app. They show a close-up view of the flying buttresses and the upper galleries where the choir would traditionally sing. 

“I put views of things that people cannot see when they are on-site,” de Riberolles said. Since 2019, the VR cathedral has toured the world as the French Institute uses it to highlight the work that’s being done. “The cathedral cannot travel, but our experience can travel abroad very easily.”

Before work started on Notre Dame, Ubisoft also demoed the virtual reality cathedral at an event for the workers. 

“The stonemasons, the architects, the carpenters, the stained glass makers, they were all gathered,” de Riberolles said. It was a standout moment for her, getting to see the emotional reactions of the people who would be on-site surveying the damage firsthand. “The work site was very sad. It was all burnt and dark, but our cathedral was lively in a nice environment,” de Riberolles said. “So I think it gave people even more desire to restore it.”

It’s at this event that, according to Durand and de Riberolles, an archbishop strapped on the VR headset and stepped into the virtual Notre Dame.

“That’s when the archbishop tried it and said, ‘Oh, could we do a mass? Could we hold a mass in it?’” de Riberolles said. “That was fun.”

“Maybe there was a generational gap there, or [because it was his] first time using, but the person was super struck and amazed by this monument that they know very well,” Durand said. “For me it shows that the fact that we can allow people to navigate through these spaces in 3D ways, it just is a totally different experience than actually going to the actual landmark. And I think they can both cohabit.”

As Notre Dame reopens to the world and the queues to see it stretch for hours, virtual reality and Assassin’s Creed Unity might still be one of the easiest ways to get inside.

Opening the doors of Notre Dame

“When you have this type of huge monument […] when you start to look at micro details, you discover a lot of story,” Miousse said. When de Riberolles was invited to visit the restored cathedral, it was the colors that caught her eye.

“You don’t imagine so many warm colors. But that’s what they got when they restored all the stained glass,” she said. “All these blues and reds. […] That’s the thing that’s really struck me the most. And of course, I think the warmth and joy of people who were happy to be there.”

For Jennifer Feltman, research on the cathedral will continue — but her time with the restoration remains special.

“Notre Dame was never my favorite cathedral that I’ve studied before, because it was always very difficult to get close to it,” she said. “Being in this isolated, closed off, essentially a fortress by myself with quiet time with my sculpture guys… I mean, that has been really precious. There’s just this kind of immediacy. It’s going to be hard to replace that feeling, that buzz.”