Near the start of the new year in 2004, “3D effect” started to become a popular topic of discussion on some of the online digital imaging message boards. There seemed to be some consensus that some images had more “presence” in that the images simply looked more like the actual scene. A few examples were posted where people claimed that the images simply looked more like you could “walk into them”. The term “3D effect” was most likely coined due the images appearing to allow a better or more accurate perception of depth.

The term “3D effect” started to become most frequently used on the Sigma SD9/SD10 forums as the SD9/SD10 users felt their cameras returned images with more depth than other cameras. After reviewing some samples, I did see a difference as it appeared that some of the full color images from the SD9/SD10 had more depth than images from even the latest crop of single-color-per- pixel cameras (Canon 10D, Nikon D100, Fuji S2, etc.). I set out to discover the reason for the apparent difference and to find a more technical spin on something that people were describing as a “feeling” when viewing images.

Perception of depth in a photo has three basic elements: (1) size of an object relative to other objects in the frame, (2) location of an object relative to other objects, and (3) focus of the object in relation to other objects and the depth-of-field of the photo. Setting aside (usually minor) lens distortions, the first two elements (size and location) are not likely to change or be different just because the sensor that captures the image uses different technology. I therefore focused my analysis on the third element: focus of objects in the photo.

The apparent sharpness/softness of object in the photo is a major contributor to the human perception of depth or 3D in an image. Our brains know that all objects in the same focal plane (same distance from the lens) in an image should have the same focus. The subject in the photo is usually sharp while (depending on depth-of-field) objects in front of or behind the subject appear soft or out of focus. If one were to disrupt this relationship between focus and the object’s actual distance from the camera lens, this would certainly degrade the perception of depth in the photo. For example, if one were to take a photo of two people standing side by side where both subjects are in focus and artificially/intentionally blur the subject on the right, some of the depth in the photo may be lost because the brain wouldn’t know what to do with the information. Is the subject on the right actually taller but standing further from the photographer? Maybe the subject on the right is smaller but is closer to the camera? In a sense, you have disturbed the relationship of distance/focus and therefore have diluted the 3D effect in the image.

Having written software to deal with raw images from single color capture (Bayer type) cameras, I was aware of the pitfalls of single color capture methods and wondered if one of these pitfalls might be causing some images to have less apparent depth. The Bayer type sensors used in typical dSLR’s like the Nikon D100, Canon 10D, Kodak 14N, etc. capture a single color at each photo site in a pattern such as this:

To get the full color image from this type of “dithered” pattern, the missing two colors must be “predicted” at each photo site. The interpolation process by which the missing colors are predicted can be relatively complex. For a black and white image such as a resolution chart, things work pretty well because each pixel will have roughly the same brightness for blacks, whites, and grays (red, green, and blue values are equal). If the red, green, and blue values in a particular region are the same, the missing colors will be roughly the same brightness as the actual (measured) color so resolution for grays can at least approach that of the physical resolution of the sensor.

After studying the layout of a Bayer type sensor, however, it becomes apparent that different colors can only be captured at a fraction of the total sensor resolution. For example, an object that is pure red (pure as far as the sensor is concerned) and only “excites” the red photo sites will have roughly ¼ the resolution of the sensor because the blue and green sensors will have little or no information to contribute. The red channel at blue and green photosites will therefore have to be interpolated by simply “stretching” the ¼ resolution data. Blues have the same problem because pure blues will also be captured at roughly ¼ resolution. Since green is more abundant on the sensor, pure greens will have ½ the sensor’s resolution.

The “effective resolution” for each color is therefore variable and will depend on the weighting of how much data is available at each photo site. Gray colors where each photo site is contributing an equal amount of information will achieve the highest effective resolution while the worst cases will only capture ¼ of the sensor resolution, not forgetting all the combinations in between. Pure yellow, for example, will excite the red and green sensors in the grid but the blue sensors will be nearly useless for interpolation. Since the green sensors occupy ½ of the sensor and the red sensors ¼, you’ll be left with ¾ effective resolution for yellow.

Note that it is difficult/rare to get “pure” colors such as a red that has absolutely no effect on the sensor’s green and blue sensors. The color need not be “pure” however, to reduce the effective resolution of the image. The less information contributed by a photo site, the less “weight” the data at that site can contribute to the interpolation. The lower the effective resolution of a particular color, the more interpolation (stretching of the data) must be done, and the more you stretch data, the less detailed and softer that data will appear.

This inconsistency in sharpness across colors has shown up in my own work. I have noticed, for example, that when shooting a bright red flower with gray spots, the gray spots on the flower petals often look much sharper or more in focus than the details (veins) in the red petals even though both are in the same focal plane. The gray details simply have a higher effective resolution than the red details in the flower petals. This sharpness inconsistency can disturb the 3D effect in images and also may shift focus to something in the photo that is not your main subject. In theory, full color sensors like the Foveon sensor in the SD9/SD10 should not have this problem since all colors are captured at nearly the full resolution of the sensor. If my theory is correct, the phenomenon should show up in some well chosen tests of a camera like the Canon 10D versus the Sigma SD10.

To test my “sharpness inconsistency” theory, I ran a test shooting the same subject with the Canon 10D and Sigma SD10, trying to keep all variables (like aperture, lighting conditions, shutter speed, etc.) the same. The results of my test showed that the Sigma SD10 captured bright reds at about the same level of sharpness as gray colors, while the 10D captured red details much softer than gray details in the same photo. The example is shown below:

The Canon 10D crop appears on the top in the above sample while the Sigma SD10 crop is on the bottom. Note that the SD10 crop was upsampled to a size comparable to the 10D shot since the 10D returns a 6 MP image versus the SD10’s 3.4 MP image. The top 10D shot shows the red details in the taillight quite blurred/soft while the gray pinstripe at the bottom appears quite sharp. The SD10 shot on the bottom shows both the red dotted pattern in the taillight and the gray pinstripe beneath at an acceptable (and comparable) level of sharpness.

Having at least preliminarily confirmed my hypothesis that different colors may appear at different levels of sharpness when captured via a Bayer type (single color) sensor, I set out to try to find a way to equalize sharpness levels in Bayer based images. If inconsistency in sharpness was causing a lack of (or detriment in) 3D effect in Bayer based images, perhaps we could make a filter that increased sharpness only where needed. If we could make a filter that knew it had to increase sharpness of the reds in the above crop for example, while leaving grays untouched, we might be able to bring some balance back to overall sharpness and improve the “presence” of Bayer based images.

If we wrote a sharpening filter that could “detect” the effective resolution based on how many photo sites on the sensor contributed information, we could apply extra sharpening proportional to the amount of information “lost” by the imbalance of color samples. For example, we would apply a lot of sharpening to pure red pixels or pure blue pixels because effective resolution is down to ¼ in those regions. We’d apply a moderate amount to pure greens because effective resolution is ½ for pure greens. Finally, we’d apply no sharpening at all for pure grays. After testing such a filter and finding that it had a very noticeable effect on the depth of photos, I implemented the filter and made it available in Qimage. Here is the result of application of the filter to the above sample:

The filtered 10D crop is on the top and the SD10 crop on the bottom. Note how the filtered 10D shot has been improved: sharpness of red details is drastically improved while the sharpness of the gray pinstripe beneath has been affected very little.

The above shows how the new filter (called “sharpness equalizer” in Qimage) can equalize sharpness of different colors in the image, but how does this affect the overall 3D effect or perception of depth in photos. Take a look at the following example. The same filter as above was applied to a “real life” image.

The above shows an original crop from a Canon 10D on the top and the filtered image on the bottom. Looking at the original on the top, notice how the more vibrantly colored bird appears softer or more out of focus than the water and rocks in the same focal plane. The filtered image on the bottom shows how the sharpness has been equalized and the bird now appears to be in the same focal plane as the rocks. The bird in this case needed more sharpening than the rocks. The result is a better feeling of depth in the filtered image.

Note that while “sharpness equalization” does appear to “fix” problems with depth in Bayer based photos, it cannot bring back information that has been lost due to interpolation deficits in some colors. In other words, equalizing sharpness across colors in Bayer based images certainly improves their appearance and does appear to bring back depth lost by the Bayer interpolation process, but nothing can recover the actual detail that might be missing in some cases due to the loss of effective resolution in some colors.

Perhaps camera manufacturers will address this issue in camera firmware in the future. This issue (and the fix) appear to be basic enough that it seems like the best place for a solution would be in the firmware that is used to translate your camera’s raw sensor data into a final image. Manufacturers could use data related to a camera’s specific hardware and interpolation algorithms to select the proper amount of “sharpness equalization” to apply to the image based on the type of antialias filter used, the radius of the interpolation algorithm, etc. For now, sharpness equalization filters appear to offer users of Bayer sensor cameras an opportunity to make some pretty significant improvements to their images.

As a final note, the actual layout of the sensor need not be known when applying sharpness equalization filters. We apply the filter to the final image and that image has already had its full color data interpolated (spread over the image) as best as possible so there is no need to know what color was actually captured at each pixel. Also note that for the filter to work, the images need not be strictly Bayer images. All that is required is that the sensor only capture a single color at each pixel and that the green pixels account for ½ the total pixels on the sensor, red pixels ¼, and blue pixels ¼. The Fuji S1, S2 and all cameras using the Super CCD for example still use a Bayer pattern; it is simply rotated 45 degrees. For the purists who shoot in raw capture mode, be aware you can achieve slightly better performance by applying the filter to the image after it has been interpolated to full color (all three colors at each pixel) but before converting to a color space. In other words, there is some benefit to applying the filter to the color space used by the sensor since the distribution of RGB values is changed slightly once you convert to Adobe RGB, sRGB, etc. for the final image.