Demosaicking Algorithm for the X-Trans Color Filter Array

In my last post I showed that ACDSee´s demosaicking algorithm for X-Trans images has a few issues. Well, interpolating the X-Trans color filter array (CFA) is not as easy as interpolating a Bayer CFA, simply because the X-Trans CFA is much more complex. However, Fuji (inventor of the X-Trans CFA) claims that their layout can outperform the Bayer CFA in terms of image quality. Obviously this is only true when applying a demosaicking algorithm which is sophisticated enough to really handle the CFA.
Some time ago in 2013 I wrote my own demosaicking algorithm for the X-Trans CFA as part of my master thesis. Actually the goal of the thesis was to compare various CFAs which is not so easy because the quality of the resulting image does not only depend on the used CFA but also on the used demosaicking algorithm. But I don´t want to bother you with this here. Instead, I want to share my algorithm. I have written it in Matlab. Below you will find the links to the source files.

XTrans_to_RGB.m
RGB_to_XTrans.m
Test.m

I have to say I´m sorry for the missing comments in the source files. Let me briefly explain how it works. Best is to start with the test.m file. You can use it to load an RGB image and convert it into an X-Trans image (RGB_to_XTrans) or to load a tiff which contains the raw sensor data. The program is not ably to directly read *.RAF files. I used DCRaw with the options '-E -T -4' to convert the *.RAF into a 16bit TIFF.
After loading the image file, it will be converted into a full RGB image. No color correction will be applied to the image, this is why RAW files will appear dark after they have been processed. If you use RGB images for testing, however, then the result should be fine.
Below you'll find some processed test images (top) and the original image (bottom) to compare.




Well, decide for your own, but I think the quality is pretty good and for sure better than what ACDSee can achieve. But it´s not perfect. Let me discuss a few issues:

• The images have a strange pattern on their edges. This is not really a problem of the algorithm. It´s simply because my algorithm is just a prototype and I didn´t care about the edges. Maybe this is also what happens in ACDSee´s processing (but theirs is not a prototype). It could easily be fixed by adding a few pixel rows/columns before demosaicking and crop them afterwards.
• Loss of detail, e.g. in the grass in the first image. Well, of course. I can´t do magic.
• Strange colors on fine details, e.g. on the trousers of the driver in the foreground in the last image or in the water in front of the boat in the second image. The X-Trans CFA has a rather low color resolution. Without pre-filtering the image we should expect to see things like these. Maybe it is possible to fix it by tweaking the algorithm (e.g. increase the number of runs for the alias canceling), however, there´s always a tradeoff. Fixing this issue will most probably result in bleeding colors (like in the 'numbers' image in my last post).

Let me add a few words on how the algorithm works. I basically use the main principles of Hirakawa´s 'Adaptive Homogeneity-Directed Demosaicing Algorithm'. First, I interpolate the green channel in for directions (horizontally, vertically, and 2x diagonally). After that I interpolate the red and blue channels in the same way. For this I exploit the interchannel correlation. The result are four candidate images. In the next step I rate at every pixel location from which candidate image to take the RGB values to construct the final image. After that I add some alias canceling to suppress color seams. That´s it.

Feel free to play around with my algorithm. I will be happy about every feedback. Please don´t use it commercially!

X-Trans RAW files in ACDSee Pro 9

Recently ACDSee Pro was released in version 9. One of the new features is the support for Fuji cameras with an X-Trans sensor layout. Knowing that even Adobe did a pretty poor job with their first demosaicking algorithm for this kind of color filter array, and knowing that ACDSee still does a poor job with their demosaicking algorithm for Bayer color filter arrays I was curious about the performance of this new feature. So I downloaded some sample RAW files taken with a Fuji X-Pro1 and converted them using ACDSee Pro 9, Lightroom 5 and DCRaw to compare the results.

One of the first things that I figured out was that the comparison of the demosaicking performance would not be as easy as I thought. While ACDSee and Lightroom produce image files with 4896 x 3264 pixels, DCRaw created files with 4952 x 3288. It seems that both ACDSee and Lightroom crop the image, maybe to allow some distortion correction while still maintaining the same resolution (which is actually the same resolution as the camera itself delivers when shooting Jpeg). Only DCRaw seems to convert the whole captured image data. In my opinion it is a pity that both commercial products do not provide access to the whole captured data.

The above image is a 100% crop of the three converted images stacked and aligned in three layers in Photoshop. In order to enhance the images´ borders I added colored frames to them. The blue frame indicates the image from ACDSee, the green one is Lightroom and the red one is DCRaw. ACDSee´s and Lightroom´s images have the same size but they use a different crop of the original, that´s why the blue and green  frame don´t match. 

Another problem with comparing the images is that obviously Lightroom applies some sort of distortion correction to some images, even though lens corrections are disabled.

But anyway, let´s have a closer look to some critical details in the converted images and see if we can find some artifacts.

Test scene 1:

The image below shows a 200% crop of a detail of an image shot with the X-Pro1. The RAW file was converted with Lightroom 5, ACDSee Pro 9 and DCRaw (top to bottom). Because of the scaling factor it is pretty easy to identify artifacts. Beside that it is also easy to recognize that DCRaw has a different way of handling the colors (but this is nothing that I care about).

• In the DCRaw (bottom) you can clearly see awkward colored pixels around the numbers, something that is called zippering. This was pretty common in simple demosaicking algorithms in former times. Modern algorithms for the Bayer array typically don´t produce much visible zippering artifacts. Due to the fact that X-Trans is much more complicated for demosaicking, zippering seems to be an issue again.
• In the Lightroom conversion (top) there is no zippering at all. However, you can see that around the numbers the background is not as saturated as it should be. Lightroom applies a filter to reduce color artifacts but the price for that is washed out colors. Well, I have to admit that this is something that is barely visible at 200% and not at all visible at any zoom level lower than 100%.
• The ADCSee conversion (middle) has a similar problem as the one from Lightroom, only here it is much stronger. There is a yellow border around the white numbers. Obviously they have applied a little bit too much false color reduction.

Test scene 2:
The next crop shows a scene from which you might think that it should be pretty easy to handle. It shows a part of the rails of a tram - these are typically straight and not fringed.

• Lightroom (top) has no problems with this situation. The rails are perfectly smooth.
• ACDSee (middle) fails. There are strong artifacts along the rail.
• DCRaw (bottom) seems to do better than ACDSee. However, the image shows less contrast. Chances are good that artifacts would be more visible after some color correction and contrast enhancement.

Test scene 3:
A similar scene than the one before. The rear of a car with the rear window on top and some silver metal at the bottom. The differences to the scene before are that here we have

• two solid areas (before it was just a bright line on dark background)
• these areas are separated almost horizontally (the line before was skewed)
• and the areas have high contrast

These are three factors which should make demosaicking easier.

• Lightroom (top) perfectly separates the two areas. Nothing to claim here.
• ACDSee (middle) fails again. Strong zippering is visible.
• DCRaw (bottom) also shows strong zippering.

Test scene 4:

This time I only show a crop of an ACDSee image. It is taken from the very left side of the processed Raw file. As you can see there are extremely strong artifacts on the left side. Actually the border of the whole image looks like this (also check the image on top). There is a border around the image which is totally unusable. So, in fact you would have to crop the outer 16 pixels from all your converted images (I have tested this with some more images, it is the same with all of them)

In this case I wouldn´t even call it a bad algorithm but rather a bug which they should fix as soon as possible.

Conclusion:

I could continue with hundreds of examples, all showing the same thing: ACDSee´s Raw converter is not competitive. One could say that what I do is pixel peeping and not really relevant for photographers. Actually I agree that it is pixel peeping. Designing image processing algorithms is one of my hobbies and I believe it should be the ultimate ambition of companies like Adobe and ACD Systems to create the best performing algorithms. ACD should not be satisfied with being able to process X-Trans files, they should rather do their best to get the best out of these files. Well, it seems that they have different priorities. Regarding the relevance for photographers I have to say that I believe that image quality (even on a pixel level) is important for them. Why would I decide to by a 24 megapixel camera if I don´t care about the pixels and why would I decide to shoot in Raw if I then use a tool which does not get the best out of these files? If I didn´t care, I could use my smart phone...

Comparison of Color Filter Arrays

Color filter arrays are widely used in digital cameras. Such arrays allow for every pixel on a sensor array to capture only one of the primary colors red, green and blue. The missing color values need to be interpolated. The arrangement of color filters on the sensor array affects the quality of the reconstructed image. The study of alternative color filter arrays is therefor an important task to improve image quality. In this master thesis four color filter arrays are compared. These are the traditional Bayer-pattern, Fuji's X-Trans-pattern, the pattern that Sony uses in it's F65 camera as well as a pattern for hexagonal grids. For each of them specialized demosaicing algorithms have been implemented to achieve the best possible quality. Several sample images have been processed to show the different properties of the color filter arrays. PSNR and SSIM were measured to determine their quality. Based on these measurements and observations no clear recommendation for a CFA can be made. However, advantages and disadvantages of the CFAs are highlighted and discussed.

This is the abstract of my master thesis. I will add more of it in future posts. If you can´t wait to find out more feel free to contact me.