From a simple photo to a powerful algorithm developed by scientists at the University of Washington can animate a three-dimensional character in the photo and take it out of the frame of the picture.
Characters who leave the frame of their photo leaving the background behind them, it only exists with great reinforcements of special effects in the Harry Potter saga with the animated pictures. Along with other researchers, Chung-Yi Weng, a Ph.D. student at the University of Washington (USA), has managed to make this effect possible by developing a powerful algorithm. In their publication, they describe how they manage to extract the main subject of a photo by modeling it, then move it realistically without leaving a black cutting area in its original location on the photo. The result is impressive!
Researchers have shown several examples including the famous basketball player Stephen Curry evolving naturally out of his picture as if he had been filmed. The effect is also applied in 3D, which is even more surprising. In another example, it is a work of Picasso ( Girl with a Sailboat ) that comes to life, while the character walks out of his canvas. To achieve this result, the researchers relied on a program called SMPL, developed by Microsoft and the Max Planck Institute. In the end, the algorithm developed by the researchers was called “Photo Wake-Up.”
In reality, this technique is not new. Since 2007, researchers have managed to model a 2D character from a photo to make it evolve in 3D. The concern was that the proportions were not respected and at best, the character’s textures were simply applied to a generic 3D model that did not match the dimensions of the subject.
When the algorithm does not imagine what is hidden
It must be said that until the techniques developed by the doctoral student, the scientists always stumbled on an element that makes manipulation difficult — the estimate of the three-dimensional pose that the character takes on the 2D picture. For the computer, it is difficult to identify and generate the body elements hidden by each other. For example, on a simple photo, for the algorithm to know if the character has his arms crossed is complicated. It’s even worse for him when his legs are crossed.
It is on this specific point that Weng and his collaborators intervened. From the photo, a 2D mesh of the body of the subject is constituted. This process already makes it possible to identify the elements of the body in its environment. Then the different parts of the body, legs, arms, head, torso are precisely determined. The mesh of each of these elements is then converted and deformed into 3D. The team pays particular attention to the head, on which the eye tends to focus. Thus, their algorithm identifies the direction of gaze and the angle of the head to form a coherent mesh. In the end, the result is spectacular, since the 3D mesh is made from the 2D image whereas previously, the image of the subject was extracted and applied on a pre-existing 3D model.
Closing the Holes in the Background
Everything is automated, but it remains possible to manually adjust the orientation of the different members of the skeleton relative to the body to correct any animation errors. When the subject is moving, another algorithm fills the holes in the background filling it by interpolating the missing elements. The three-dimensional character can then come out of the picture, run, jump and sit down.
The process associated with the augmented reality principle may well change the way we interact with photos or the masterpieces in an art museum.