This site may earn affiliate commissions from the links on this page. Terms of use.

Stanford Optical Electrical Hybrid Camera

Until recently, cameras were exclusively designed to create images for humans — for fun, for fine art, and for documenting history. With the rapid growth of robots besides every bit various other kinds of machines and vehicles that demand to observe and learn from their environment, many cameras are dedicated to machine vision tasks. Some of the most visible of those, similar autonomous vehicles, rely heavily on object recognition, which almost universally ways neural networks trained on commonly institute objects. One limitation on the deployment of car vision in many embedded systems, including electric vehicles, is the necessary compute and electrical ability. So it makes sense to re-imagine camera design and consider what is the ideal photographic camera compages for a detail awarding, rather than simply repurposing existing photographic camera models.

In this spirit, a team at Stanford University led by Banana Professor Gordon Wetzstein and graduate educatee Julie Chang has built a paradigm of a system that moves the beginning layer of an object recognition neural network directly into the camera's optics. This means that the first portion of the needed inferencing takes essentially no time and no power. While their current epitome is limited and bulky, information technology points the way for some novel approaches to creating lower-power, higher-performance, inferencing solutions in IoT, vehicle, and other embedded applications. The inquiry draws heavily from AI, imaging, and optics, so at that place isn't any way we can item the unabridged organization in ane article. But we'll take y'all through the highlights and some of the breakthroughs that make the prototype so intriguing.

Basic Object Recognition, Neural Network Style

Most current object recognition systems utilize a multi-layer neural network. State of the art systems oft include dozens of layers, merely information technology is possible to accost simple exam suites like MNIST, Google'south QuickDraw, and CIFAR-ten with only a layer or two. However deep the network the first layer or layers are typically convolution layers. Convolution is the procedure of passing a matrix (called a kernel) over the image, multiplying it at each location and summing the result to create an activation matrix. In unproblematic terms, the procedure highlights areas of the image that are similar to the kernel's pattern. Typical systems involve multiple kernels, each reflecting a feature constitute in the objects beingness studied. As the network is trained, those kernels should offset to look like those features, so the resulting activation maps will help later levels of the network recognize particular objects that include various examples of the features.

Later layers of the network are often fully connected, which are simpler to compute than convolution layers. The Stanford hybrid optical-digital camera doesn't address those but instead models replacing the computationally-expensive initial convolution layer with an optical culling, which the team refers to as an opt-conv layer. In that location isn't any simple way with traditional optics to perform a convolution with an arbitrary kernel on an epitome, permit lone multiple, simultaneous, convolutions. All the same, if the image is showtime turned into its frequency equivalent using a Fourier transform, fast convolution suddenly becomes possible — considering multiplying in the frequency domain is like performing a convolution in the traditional spatial domain.

To take advantage of this holding, the team draws from the techniques of Fourier Optics, by edifice what is called a 4f optical system. A ivf system relies on an initial lens to render the Fourier transform of the epitome. The system allows for processing the transformed prototype using an intermediate filter or filters, and then reverses the transform with a final lens and renders the modified result.

Fourier Optical system implemented in a 4f telescope including a phase mask to implement image convolution

Fourier Optical system implemented in a 4f telescope including a phase mask to implement image convolution

The Magic of Optically Calculating a Convolution Layer

There is a lot of pretty deep science that goes into the optical portion of Stanford'south prototype, only it basically chains together a few powerful techniques that nosotros tin describe (if not fully explain) adequately succinctly:

First, it is a well-known feature of a Fourier transform (which take a signal or image and renders it in terms of frequencies), that you tin also reverse it and go the original epitome back. Importantly, you tin can do this using a simple optical organization with just a couple lenses, chosen a 4f optical arrangement (this whole expanse of optics is chosen Fourier Optics).

2d, if you filter the Fourier transform of an image by passing information technology through a partially opaque surface, that is the same as performing a convolution.

Tertiary, you tin tile multiple kernels into a single filter and utilize them to a padded version of the original image. This mimics the behavior of a multiple kernel organisation that would normally produce a multi-aqueduct output past creating one that outputs a tiled equivalent as shown hither:

The multi-channel output of a traditional convolution layer can be mimiced using tiling in an optical system

The multi-channel output of a traditional convolution layer can be mimicked using tiling in an optical system

Then by calculating the desired kernels using traditional machine learning techniques, they tin can be used to create a custom filter — in the form of a phase mask of varying thickness — that tin can be added to the center of the ivf organization to instantly perform the convolutions equally the lite passes through the device.

Training and Implementing the Optical Convolution Layer

One limitation of the proposed optical system is that the hardware filter has to be made based on the trained weights. And then it isn't applied to use the system to train itself. Grooming is done past using a simulation of the system. In one case the needed final weights are determined they are used to fabricate a phase mask (a filter with varying thickness that alters the phase of the light passing through it) with 16 possible values, that can be placed in-line with the 4f optical pipeline.

The learned weights are used to create a mask template, which is then fabricated into a mask of varying thickness

The learned weights are used to create a mask template, which is and so fabricated into a mask of varying thickness

Evaluating Operation of the Hybrid Optical-Electronic Photographic camera System

The Stanford squad evaluated the functioning of their solution in both simulation and using their physical epitome. They tested information technology both every bit a fashion to create a standalone optical correlator using Google'due south QuickDraw dataset and as the starting time layer of a two-layer neural network, which was combined with a fully continued layer to practice basic object recognition using the CIFAR-10 dataset. Even after allowing for the limitation of an optical system that all weights need to exist non-negative, as a correlator, the system achieved accuracy betwixt 70 percentage and 80 percentage. That's similar to that of a more-traditional convolutional layer created using standard automobile learning techniques, but without needing to take powered calculating elements to perform the convolutions. Similarly, the two-layer solution using a hybrid optical-electronic first layer accomplished a performance of almost 50 per centum on CIFAR-10, nigh the same as a traditional 2-layer network, merely with a tiny fraction of the computing power — and therefore electrical power — of the typical solution.

While the current prototype is bulky and requires a monochrome calorie-free source as well every bit merely working with grayscale images, the team has already started thinking about how to extend it to work under more typical lighting conditions and with full-color images. Similarly, the 4f organisation itself could potentially be reduced in size by using flat diffractive optical elements to supplant the current lenses.

To learn more y'all can read the squad'southward full newspaper in Nature'southward Scientific Reports. The team has besides said that they'll be making the full source lawmaking for their system publicly bachelor.

At present Read: Autofocals may replace progressive glasses, Camera-simply Autonomous Vehicle Test Bulldoze, and IBM Aims To Reduce The Power Required For Network Training.