There are four common ways to do color detection with electronic image capture systems:

TIME SEQUENTIAL COLOR

the same photodetector is used three times in sequence to capture a red, green, and blue image. Used today in some studio camera setups for still life scenes, and some scanners make use of this principle as well. Used in the early days of film photography to get some of the first quality color photographs, see example below. Unable to capture motion, and even still life images can have some form of subtle motion.

OPTICAL COLOR SEPARATION

Refractive or reflective optics are used to create separate red, green and blue optical paths that are exposed on separate image sensors. Used in some high end video cameras, an example follows. Optics are expensive and have more constraints than for a single optical path.

3) LATERAL COLOR FILTER. An array of transparent color filters is arranged over the photodetector array, so that different photodetectors sample different colors, as well as different spatial locations. The most common form of color detector used in consumer electronic imagers today, often called "Bayer pattern" after its inventor (see US patent number 3,971,065). The normal output file for this type of pixel interpolates color values for all the physical photosites, requiring many assumptions about the local image characteristics, as well as complex calculations.

Advantages of lateral color filter sensors:

1. many images have a significant amount of detail for which luminance information only is sufficient. For example, an image of newsprint will generate a more or less equivalent light / dark response in all three color channels, because the white paper background contains red, green, and blue light. So the black and white image of the characters on the paper can be captured with good accuracy even though it is being reconstructed from asymmetric color sensors.
2. The human eye is also a lateral color filter detector, and is less sensitive to color transitions than black and white (luminance) transitions. So the Bayer pattern matches what the eye tends to see. But not necessarily what is in the scene.
3. Image sensor read circuitry overhead is low for the lateral color filter sensor relative to the size of the output file, because there is only one color sample per output pixel.
4. The color filter characteristics can be optimized independently from the underlying detectors.

Disadvantages of the lateral color filter sensor:

1. only about 1/3 of the photons that reach the top surface of the sensor, actually reach the photodetectors in the silicon, because of absorption in the overlaying color filter layers. For example, a red filter passes only red light on to the detector, and absorbs blue and green light.
2. each photosite only captures information about one of the three colors, and the other two colors are interpolated from surrounding data. So 2/3 of the color information in every output pixel is derived using complex software algorithms, instead of measured. As shown in figure 2 below, this can lead to interpretation errors when spatial luminance response is confused with color response.

Interpretation errors such as those illustrated in Figure 2 can cause distracting color Moire patterns in some images. To minimize these Moire patterns, some digital cameras have "Anti Aliasing" filters in the optical path to blur out the high spatial frequency patterns (see the discussion of sampling in Digital Camera Tech Notes #2 Pixel Size) , and thereby reduce Moire patterns. This approach is illustrated in figure 3 below.

Unfortunately, blurring of color information to suppress the color artifacts reduces image sharpness as well. Figure 4 below shows the trade-off between blurring and Moire patterns in a typical lateral color filter image sensor. As the aperture size is reduced (increasing f#) diffraction blurring is increased, reducing the color Moire artifact.

VERTICAL COLOR FILTER

Each photosite measures all three colors, using a vertically stacked arrangement of color filters and detectors. Used in color film for over 70 years, and is also possible to implement in electronic image sensors, as shown in figure 5.

Advantages of vertical color filter sensors:

1. the wavelength selection process is also the detection process, so all nearly all the incident photons are converted into a useful electrical signal in the detectors.
2. having all the color information available at each photosite reduces the confusion between spatial and color information, allowing for capture of single pixel color features and reducing the complexity of the computations, relative to the lateral color filter.
3. having all the color information available also enables more effective image enhancement signal processing, such as sharpening, noise reduction, or correction of lens aberrations.
4. communication bandwidth and computer memory is not wasted storing interpolated data, all the information stored and transmitted is measured data.
5. an "anti-Aliasing" filter is not required to suppress color aliasing, reducing camera cost and increasing image contrast.
6. because all the photosites are identical, it is easier to combine them to create a multi resolution image sensor, so that the trade-off between ISO and resolution can be more easily made available to the digital camera user. Also, better monochrome conversions are possible because all the photosites are identical.
7. the color filter characteristics depend on a fundamental physical characteristic of the best characterized material in the world, single crystal silicon. This makes the color filters more repeatable, more reliable, and more manufacturable then superficial plastic color filters.
8. the red, green, and blue color detectors of the silicon vertical color filter each have a response to all wavelengths in the visible spectrum. This allows for more precise determination of color than is possible with a color filter that has little response to colors outside each passband, as illustrated diagrammatically in figure 6a and 6b. However there is some penalty for the greater color accuracy, which is an increase in noise generated during the mathematical conversion of the measured color response to the corrected output color response. A detailed discussion of this topic is to be found in the Journal of The Society of Photographic Science and Technology of Japan (2003) Vol. 66, No. 1: 57-60. However, the important result for photographers is improved color accuracy for the silicon vertical color filter technology.

Photographic subjects in the real world are made up of myriad light reflecting (or emitting) objects each of which can have a different color. Lateral color filter sensors can have difficulty detecting abrupt color transitions, as described earlier in this article. To prevent more noticeable color errors, uncertainty in the area of color edges is usually rendered as a neutral grey or some blend of surrounding colors. This effect is shown in figure 7a below.

As seen in figure 7b, the same color transitions are rendered with less use of neutral color when the vertical color filter sensor is used, so the resulting image has more of the real color that was present. In scenes of objects with many color transitions, such as grains of sand on a beach or tree leaves, this reduction of incorrect colors near edges can amount to a significant amount of the scene content, enough to cause a noticeable difference in subjective color perception for the image.

CONCLUSION

We have described four basically different techniques for doing color separation in digital cameras, each of which has advantages and disadvantages which make it more, or less, suitable for different applications. Of these techniques, only the single chip lateral or vertical color filter method is practical for consumer cameras. The vertical color filter has the basic physics advantage of capturing more information per unit sensor area than the lateral color filter sensor. But the lateral color filter sensor has less overhead, in cases where having all color information in each photosite available does not contribute to image quality. In the ultimate (future) camera design with diffraction limited resolution, and photon capture cross section limited signal to noise, then the vertical color filter will have an advantage.