[ad_1]
As the car moves along a narrow city street, reflections from shiny paint or side mirrors of parked cars help the driver track things that would otherwise be hidden from view, such as a child playing on the sidewalk behind parked cars.
Based on this idea, researchers at MIT and Rice University have developed a computer vision technique that uses reflections to image the world. Their method uses reflections to turn shiny objects into “cameras,” allowing users to see the world as if they were looking through the “lenses” of everyday objects, such as a ceramic coffee mug or a metallic paperweight.
Using images of the object taken from different angles, the technique transforms the object’s surface into a virtual sensor that captures the reflections. The artificial intelligence system interprets these reflections in a way that allows it to estimate the depth of the scene and capture new views that would only be visible from the subject’s perspective. This technique can be used in corners or behind objects that block the observer’s view.
This method is particularly useful in autonomous vehicles. For example, it could allow a self-driving car to use reflections from objects it passes, such as lampposts or buildings, to see a parked truck.
“We have shown that any surface can be transformed into a sensor with this formulation, which transforms objects into virtual pixels and virtual sensors. This can be used in many different areas,” says Kushagra Tiwari, a graduate student in the Camera Culture Group in the Media Lab and co-lead author of a paper on this research.
Tiwari is joined on the piece by co-anchor author Akshat Dave, a graduate student at Rice University; Nikhil Behar, MIT Research Support Associate; Tzofi Klinghoffer, graduate student at MIT; Ashok Veeraraghavan, professor of electrical and computer engineering at Rice University; and senior author Ramesh Raskar, associate professor of media arts and sciences and leader of MIT’s Camera Culture Group. The research will be presented at the Conference on Computer Vision and Pattern Recognition.
Thinking about reflections
Crime TV show characters often “zoom in and enhance” surveillance footage to capture reflections—perhaps those caught in a suspect’s glasses—that help them solve crimes.
“In real life, using these reflections is not as simple as pressing the volume button. Extracting useful information from these reflections is quite difficult because the reflections give us a distorted view of the world,” says Dave.
This distortion depends on the shape of the object and the world the object represents, about which researchers may have incomplete information. In addition, a glossy object can have its own color and texture mixed with reflections. Additionally, reflections are a two-dimensional projection of a three-dimensional world, making it difficult to judge depth in reflected scenes.
Researchers have found a way to overcome these challenges. Their technique, known as ORCa (which stands for Objects Like Radiance-Field Cameras), works in three steps. First, they take pictures of the object from multiple vantage points, capturing multiple reflections on the glossy object.
Then, for each image from the real camera, ORCa uses machine learning to transform the object’s surface into a virtual sensor that captures the light and reflection hitting each virtual pixel on the object’s surface. Finally, the system uses virtual pixels on the surface of the object to model the 3D environment from the object’s point of view.
catch the rays
Imaging an object from multiple angles allows ORCa to capture a multiview reflection, which the system uses to estimate the depth between the glossy object and other objects in the scene, in addition to estimating the shape of the glossy object. ORCa models the scene as a 5D irradiance field, which captures additional information about the intensity and direction of the light rays that exit and strike each point in the scene.
The additional information contained in this 5D radiance field also helps to accurately estimate the depth of ORCa. And because the scene is represented as a 5D radiation field rather than a 2D image, the user can see hidden features that would otherwise be blocked by corners or obstacles.
In fact, once ORCa has captured this 5D radiant field, the user can place a virtual camera anywhere on the scene and synthesize what that camera will see, Dave explains. The user can also insert virtual objects into the environment or change the appearance of the object, for example, from ceramic to metallic.
Credit: Courtesy of researchers
“It was particularly difficult to go from a 2D image to a 5D environment. You have to make sure the map works and is physically accurate, so it’s based on how light moves through space and how light interacts with the environment. We spent a lot of time thinking about how we can model the surface,” says Tiwari.
accurate estimates
The researchers evaluated their technique against other methods that simulate modeling, which is a slightly different task than ORCa performs. Their method was good at showing true object color from reflections and outperformed baselines by extracting more accurate object geometry and texture.
They compared the system’s depth estimates with simulated ground truth data of the actual distance between objects in the scene and found ORCa’s predictions to be reliable.
“Consistently, with ORCa, it not only accurately estimates the environment as a 5D image, but to achieve this, in the intermediate steps, it also does a good job of estimating the shape of the object and reflecting the separation of the object from the texture.” Dave says.
Based on this proof of concept, the researchers want to apply this technique to drone imaging. ORCa can use faint reflections from objects the drone is flying over to reconstruct the scene from the ground. They also want to enhance ORCa so that it can use other cues, such as shadows, to recover hidden information, or combine reflections from two objects to render new parts of a scene.
“Estimating specular reflections is really important for seeing corners, and it’s a natural next step for using weak reflections in the scene around corners,” says Raskar.
“Typically, shiny objects are difficult for vision systems. This work is very creative because it turns the subject’s long-standing weakness of luster into an advantage. Using ambient reflections from a shiny object, the paper can not only see hidden parts of the scene, but also understand how the scene is lit. This enables applications in 3D perception that include, but are not limited to, the ability to composit virtual objects into real scenes in a way that looks seamless, even in challenging lighting conditions,” said Achuta Kadambi, assistant professor of electrical engineering and computer science. University of California, Los Angeles, which was not involved in the case. “One of the reasons others haven’t been able to use shiny objects in this way is that most previous work requires surfaces with known geometry or texture. The authors have adopted an intriguing new formulation that does not require such knowledge.
The research was supported in part by the Intelligence Advanced Research Projects Activity and the National Science Foundation.
[ad_2]
Source link
