When we measure how a depth camera performs, we look at numerous metrics, including the range, the depth noise, the accuracy, temporal noise, the XY resolution, and the depth (or Z) resolution, to name a few.

Although it isn’t discussed as often, one of the most important features specific to stereo depth algorithms relates to the ability to detect and match features on a sub-pixel scale. The way a stereo algorithm works is to identify a feature in one image, and then search along a line of pixels to find the closest match it can in the image from the other sensor – this search scans 128 patches of pixels looking for the best scoring match, giving us the disparity for that patch of pixels that constitute a feature. Once this has run for every pixel in an image, a disparity map is created that can be used to create the depth map.

When it comes to the sub-pixel scale, if an algorithm is able to search across 128 disparities, it would have the ability to map all depth points to one of the 128 depth planes. If on the other hand, it was possible to have 1/32 subpixel resolution, then depth can be mapped to 4096 depth planes. Creating good subpixel algorithms is very challenging and the end result is usually a non-linear sub-pixel depth determination.

This means, for example, if you measure range to a wall as you move away from the wall, you would hope to see a smooth linear relationship. Instead, what you may observe is more step-like functions, with steps getting larger at longer ranges.

In this whitepaper, RealSense™ Technology CTO Anders Grunnet-Jepsen explains and explores improvement to the Subpixel Linearity algorithms for the RealSense™ depth camera D400 series with the addition of a new parameter called the A-factor in the RealSense SDK 2.0.

Read the full white paper here.