Rao. C. R¹, Mandal S. K²

¹Department of Electrical Engineering, Maulana Abul Kalam Azad university of Technology, Kolkata, West Bengal, India.

²Department of Electrical Engineering, National Institute of Technical Teachers' Training & Research (NITTTR), Kolkata, West Bengal, India.

¹email: crrao@bopter.gov.in, ²email: skmandal@nitttrkol.ac.in

Corresponding author email: crrao@bopter.gov.in

 

Received  : 17/07/2024             Accepted  : 24/09/2024            Published : 12/11/2024

 

------------------------------------------

Abstract:

Demosaicking is the procedure of re-constructing full colour information at each pixel from a CCD sensor, which captures only one colour component red or green or blue per pixel. Various demosaicking techniques are employed to perform this reconstruction. However, many studies overlook a detailed analysis of image quality, particularly with regard to artifacts that may appear in the edges and textures of the images. These drawbacks in the existing methods, the re-constructed image seems to be of poor in quality with  less PSNR values. To address the limitations of existing methods, this paper proposes a new demosaicking technique.  The technique proposed is designed with two major stages (i) CFA demosaicking (ii) Enhancement by Levenberg-Marquardt technique. The given input CFA image green, red, blue pixel values are interpolated to extract the full-colour image. This full-colour image seems to be less quality and has more distortion in the edge and texture of the image. So the output full colour image is enhanced by the Levenberg-Marquardt technique. Hence, the image can be demosaicked more effectively by reaching higher PSNR and MSE ratio compared to the conventional demosaicking algorithms. The comparison of results shows that the proposed technique extracts high-quality demosaicked images than the state-of-the-art methods, in terms of PSNR.

Keywords: Artifact, CCD Sensor, Demosaicking, edge, enhancement, texture, Interpolation, Levenberg Marquardt, Colour Filter Array (CFA), Peak signal-to-noise (PSNR), Pixel.

 

1.  INTRODUCTION

A full colour digital image  [2] comprises of 3 primary colours namely Red (R), Green (G) and Blue (B). However, due to technical limitations, a commercial digital camera [1] produces images with only one colour i.e G or R or B at each pixel location. This image in general referred to mosaic.  Demosaicking is a process which enhances the images captured by these commercial digital cameras to a full colour image.

The Commercial digital camera is mounted with a single CCD sensor [3] which hs the ability of measuring only one colour per pixel [5]. A camera would need three separate sensors to completely measure the image [6],[7],[8]. In a three-sensor colour camera, incoming light is divided and directed onto each of the spectral sensors. Each sensor requires dedicated driving electronics, and precise alignment of the sensors is essential for accurate performance. This increases the cost of the camera and not suitable for commercial applications. To reduce cost, manufacturers of digital camera [4]  use a single CCD/CMOS sensor [16]  with a colour filter array (CFA) [9] [10].

A demosaicking problem can be viewed as a, estimation or interpolation problem [11], where missing colour information at each pixel is estimated based on neighboring pixel values. The interpolation, however, often results in colour artifacts essentially at object boundaries [12], [13]. The demosaicking problem involves interpolating colour data to generate a fully coloured image from the Bayer pattern. Specifically, it focuses on reconstructing full colour image (I) from a sub-sampled image (D) of its pixel values. State of the art demosaicking methods fail when the local geometry cannot be inferred from the neighboring pixels [14],[15]. The re-constructed image is typically accurate in uniform-coloured areas, but has a loss of resolution [17] A huge number of demosaicking algorithms [18],[19],[20] have been described below: i) Nearest-Neighbor Interpolation: Simply replicates the neighboring pixel of the same colour channel (2x2 neighborhoods) [21]. ii) Bilinear Interpolation: Bilinear interpolation often produces noticeable artifacts, particularly along edges and other high-frequency areas of the image.  iii) Cubic Interpolation: Pixels that are farther from the current pixel are assigned lower weights. iv) Gradient-corrected bilinear interpolation: In this High-quality linear interpolation algorithm the assumption is that the chrominance components don’t vary much across pixels [22].  v) Smooth Hue Transition Interpolation: The assumption is that the hue changes smoothly across the surface of an object. vi) Pattern Recognition Interpolation: Classifies and interpolate three different boundary types in the green colour plane [23]. vii) Adaptive Colour Plane Interpolation: The assumption is that, within a sufficiently small neighborhood, the colour planes are perfectly correlated. viii) Directionally Weighted Gradient Based Interpolation: Improves the edge detection power of the adaptive colour plane method [24]. These techniques, referred to as demosaicking or colour interpolation algorithms, are critical in influencing the overall quality of the resultant image.

The remainder of this article is ordered as follows: Section 2 provides a concise review of recent works in image demosaicking research. Section 3 presents the proposed image demosaicking technique. Section 4 discusses on the results and comparative analysis, while Section 5 concludes the work.

 

2.  REVIEW OF IMPORTANT DEMOSAICKING ALGORITHMS

Some important recent studies on image demosaicking are reviewed here.

Qichuan Tiana et al. [27] proposed physical structure of colour image sensor in the camera. The proposed method involves processing images captured by image sensors equipped with a CFA comprising different channel filters. This algorithm, based on adaptive region demosaicking, is designed specifically for the Bayer format to minimize colour artifacts. This algorithm enhances image quality by improving the PSNR, sharpening textures and edges, and overall boosting visual fidelity.

Xiaofei Yang et al. [28] have proposed a physical structure of the colour image sensor with commercial digital cameras acquired images mounted with single sensor overlaid with a CFA. The proposed method employed a general interpolation approach to address texture and colour artifact edges. Gradient operators and a weighted average technique were utilized, adhering to constant-hue principles for reconstruction. The final output was optimized by leveraging the correlation of image details, resulting in improved PSNR, enhanced texture sharpness, edge clarity, and overall image quality.

Shan Baotanga et al. [29] have proposed a novel near-lossless compression method. The compression performance of sub-smpled images acquired requires improvement. To address this, a channel-separated filtering was proposed by the author. The evaluation demonstrated that the remainder set algorithm achieved superior performance, yielding higher PSNR for the reconstructed CFA images.

Wen-Jan Chen et al. [30] have proposed demosaicking method to prevent the generation of artifacts. The proposed demosaicking method is an image processing technique specifically designed for commercial cameras and addresses issues such as noise and blurred edges that can arise during image reconstruction, ultimately improving the quality of the reconstructed image. The results demonstrated that the proposed demosaicking method effectively reduced colour artifacts and enhanced overall image quality.

Yu Zhang et al. [31] proposed a method makes significant contributions in the following areas: i) It combines Linear Minimum Mean Squared Error (LMMSE) with statistical calculations in the wavelet domain. ii) It establishes a verified mathematical relationship between the adjacent pixels, particularly those located on the same edge in the CFA data. Experimental results establish that this method beats existing techniques both in terms of appearance and computational cost, suitable for enhancing the noisy CFA data.

In this Section, some of the important recent articles related to the demosaicking process are studied. These exiting methods mainly perform the interpolation on the images with the CFA data. After that, the interpolated data were utilized in compression, denoising and image refinement processes. To accomplish those processes in the interpolated data, the existing techniques mainly concentrate on the gradient techniques. The reconstructed image from the gradient techniques does not produce accurate result because, the reconstructed images have less image quality and more artifacts present in the edge and texture of the image. So, the application of gradient degrades the demosaicking performance. Hence to overcome this drawback, a novel image demosaicking technique is proposed which can obtain the improved PSNR ratio of image with higher efficiency.

 

3. CFA DEMOSAICKING VIA LEVENBERG-MARQUARDT OPTIMIZATION METHOD:

The proposed demosaicking technique comprised of two stages namely, (i) CFA demosaicking (ii) Image enhancement by Levenberg-Marquardt technique. These two stages are consecutively applied to the different images and obtain the demosaicked image as a result.  The proposed image demosaicking technique structure is illustrated in Figure 1.

Figure 1: Block Diagram of the Proposed Demosaicking Algorithm

 

3.1 CFA Demosaicking

In CFA demosaicking, initially, the green pixels are interpolated prior to the red and blue pixels are interpolated form the green pixels and get the interpolated image as a result.  The sampling frequency of the G channel is higher than that of the R and B channels. As a result, the G channel retains significantly more structural information compared to the other two colour channels. Typically, a better reconstruction of the G channel leads to improved reconstruction of the R and B channels. Initially, we interpolate the missing G pixel values based on the pixel positions of the R and B channels.

 

 

Figure 2: Bayer CFA Pattern

G-plane Interpolation: 

First the missing green pixel values are interpolated from the image R and B positions. Consider Bayer CFA pattern is shown in Figure 2. Initially the green pixel value at location R₇ can be computed as,

 

(1)

 

Moreover we compute colour difference along the four points surrounding R₇, i.e. G₃, G₆, G₈ and G₁₁ (four directions). The colour difference is calculated as follows,

 

(2)

 

(3)

 

(4)

 

(5)

 

After that the gradients at R₇ is calculated along the four directions by compute the optimal weights at those directions. The weights are estimated at four directions is inversely proportional to the gradient along that direction. The weights computation along the four directions is described as,

(6)

 

(7)

 

(8)

 

(9)

 

Where the gradients values at the four direction is stated as,

 

(10)

 

(11)

 

(12)

 

(13)

 

By using these gradients and the weights values, the colour difference value in Equation (1) can be estimated as,

 

(14)

 

In Equation (14), is the normalized weight value of the direction n. the missing green pixel at location can be computed as,

(15)

 

The same procedure is followed for all R and B positions to estimate the missing green pixel values.

R and B plane Interpolation: 

After the green pixels interpolation, the missing blue and red pixel values are interpolated from B, R and G plane components. First the B pixels are interpolated from R positions and then from G positions. Initially the blue pixel value at location G₇ can be computed as,

 

(16)

 

We estimate the colour difference between B and G along the four diagonal directions at R₇ is described as,

(17)

 

(18)

 

(19)

 

(20)

 

The gradients and weights are computed at four directions and then the in Equation (16) and missing blue pixel can be computed as,

 

(21)

 

(22)

 

Once the B values at the R locations are interpolated as defined above, consequently interpolate the B values at all the other G locations. Similarly, the missing red pixel values are interpolated form the B and G channels. After the G, R and B channels interpolation process the enhancement procedure is consecutively applied to the channel, which is discussed below. 

 

3.2 Image Enhancement by Levenberg Marquardt

In image enhancement process, the channels quality is enhanced by the Levenberg-Marquardt technique. The channels enhancement process is accomplished by initially applying gradients on the channels separately. Each channel gradients results are represented as ,  and . Next, the weight matrix is generated within the interval [0, 1] with the image size of . The Levenberg-Marquardt function is described for the G, R and B channels is stated as,

 

(23)

 

(24)

 

(25)

 

The gradients value is computed as,

 

(26)

 

Similarly, the gradients values are also computed for R and B channels.

By using this Levenberg-Marquardt function the enhanced G, R and B channels are obtained. We select the image channel from the abovementioned process, which satisfies the objective function is described below,

(27)

 

In Equation (27), where an image  is divided into blocks. , and  are the max, min and center pixel values in every block. The image enhancement function, is employed to improve the contrast of the G (green), B (blue), and R (red) channels. The objective function is computed for each channel Levenberg function at define number of iterations .  The channels G, R and B are selected which have the minimum value of  . Afterward, the channel pixel values are improved by selecting a large window is the size of the  from the channels G, R and B, is denoted as and smaller window is the size of is centered on window . From the larger window , small patches are selected with the size of , is computed by,

(28)

 

Based on the value, the patches  are selected from the window . The selected patches from the window are converted to row values and by we create the matrix with the size of  , where , . Moreover, the centered small window   is stored in row-wise manner and the smaller window values are repeated in  times for creating another one matrix . After that we compute the difference between the two matrices  and by,

 

(29)

 

The computed difference matrix is denoted by and computes the mean value for each row in the difference matrix . These computed mean values are sorted in ascending order and compared with the threshold value . The means values that satisfy the threshold value  are stored in the variable . The pixels in the window  progress process is explained below,

 

(i)         To improve the pixel values in window , initially compute

(30)

(ii)                  Find the beat row value by,

 

(31)

 

(iii)       Based on the value of , the corresponding index row value is extracted from  is represented as .

 

(iv)                Compute new pixel value by,

 

(32)

 

(33)

The process for all pixel values in the window  get repeated and after that a new window is created and replaces the pixel values with new pixel values. The same process is repeated until the all-pixel values in the channels G, R and B are replaced by the new pixel values. Now, we obtain the absolute enhanced channels ,  and .

 

4.  RESULTS

The proposed image demosaicking technique is evaluated by conducting experiments on two different datasets. In this work two datasets, namely, Kodak dataset and McMaster dataset are utilised.

 

The Kodak image dataset [26] is widely used as a standard dataset in demosaicking techniques (available at http://www.researchandtechnology.net/pcif/kodak_benchmarks.php?i=1). The Kodak dataset consists of 24 images with a spatial resolution of 768×512 pixels. Another dataset, the McMaster dataset, is used for evaluating colour demosaicking (CDM) algorithms. It includes 8 high-resolution colour images (2310×1814 pixels), which were originally taken on Kodak film and digitized. The sample images from both datasets are illustrated in Figure 3.

 

 

(i)

 

 

(ii)

Figure 3: Sample Colour Images from (i) Kodak Dataset (ii) McMaster Dataset

 

To obtain a full colour image, the dataset images are given to the colour interpolation and image enhancement process. These processes are consecutively applied to the dataset images and acquire demosaicked images as a result. The proposed image demosaicking technique performance is estimated by comparing it with the existing demosaicking LDI-NAT technique. The original and demosaicking images results of our proposed technique are shown in Figure 4

.

 

                                                   (i)                        (ii)                 (iii)

 

Figure 4: Results from Proposed Image Demosaicking Technique (i) Original Image (ii) Interpolated Image (iii) Full Colour Image

 

The proposed and LDI-NAT technique performance is illustrated in Figure 4. shows that our proposed image demosaicking technique has achieved good image results i.e. our proposed technique has obtained high quality image. The image quality measurement PSNR is calculated for our proposed and LDI-NAT technique are tabulated in Table 1 and Table 2.

 

Table 1: Our proposed image demosaicking technique PSNR value of Kodak and McMaster datasets images

 

 

Table 2: LDI-NAT technique PSNR value of Kodak and McMaster datasets images

 

 

 

 

 

 

(i)

 

(ii)

Figure 5: Graphical Representation of the Average PSNR by our Proposed and Existing Techniques on the (i) Kodak and (ii) McMaster Datasets

 

As can be seen from Figure 5, our proposed image demosaicking technique has achieved high performance in reconstruct the full colour image. In Figure 5(i) shows the proposed and LDI-NAT techniques performance in Kodak dataset images. In Kodak image dataset our proposed technique average PSNR value of the R, G and B channels acquire high performance than the existing LDI-NAT technique. Our proposed and LDI-NAT technique performance in McMaster Dataset is shown in Figure 5(ii).  In Figure 5(ii), our proposed technique R channel average PSNR value is lower than the LDI-NAT technique. But this low performance not degrades the demosaicking process because the channels G and B have given high PSNR value than the PSNR value of the LDI-NAT technique.

 

5.  CONCLUSIONS

The proposed image demosaicking algorithm presented in detail, along with the implementation results. The proposed methodology involves demosaicking the image using CFA interpolation, followed by an image enhancement process utilizing the Levenberg-Marquardt technique. This combined approach significantly improved the performance of the demosaicking process. The results demonstrated that the proposed image demosaicking technique, incorporating the Levenberg-Marquardt method, achieves higher PSNR values compared to existing demosaicking techniques. Therefore, the proposed method delivers superior performance in demosaicking images with a higher PSNR ratio. The proposed technique was also compared with existing methods to validate its enhanced demosaicking performance.

In future, deep learning algorithms may be used in demosaicking so that better results can be achieved.

 

DECLARATIONS:

Acknowledgments	:	Not applicable.
Conflict of Interest	:	The author declares that there is no actual or potential conflict of interest about this article.
Consent to Publish	:	The authors agree to publish the paper in the Global Research Journal of Social Sciences and Management.
Ethical Approval	:	Not applicable.
Funding	:	Author claims no funding was received.
Author Contribution	:	Both the authors confirms their responsibility for the study, conception, design, data collection, and manuscript preparation.
Data Availability Statement	:	The data presented in this study are available upon request from the corresponding author.

 

REFERENCES:

[1]     G. C. Holst, “CCD arrays, cameras, and displays,” 1998.

[2]    Seisuke Yamanaka, “Solid State Color Camera,” 1977 Accessed: Aug. 19, 2022. [Online]. Available: https://patents.google.com/patent/US4054906A/en

[3]    A. El Gamal and H. Eltoukhy, “CMOS image sensors,” IEEE Circuits and Devices Magazine, vol. 21, no. 3, pp. 6–20, May 2005, doi: 10.1109/MCD.2005.1438751.

[4]   K. Hirakawa and T. W. Parks, “Adaptive homogeneity-directed demosaicing algorithm,” IEEE Transactions on Image Processing, vol. 14, no. 3, pp. 360–369, 2005, doi: 10.1109/TIP.2004.838691.

[5]   A. Buades, B. Coll, J. M. Morel, and C. Sbert, “Self-similarity driven color demosaicking,” IEEE Transactions on Image Processing, vol. 18, no. 6, pp. 1192–1202, 2009, doi: 10.1109/TIP.2009.2017171.

[6]    S. Ferradans, M. Bertalmio, and V. Caselles, “Geometry-Based Demosaicking,” IEEE Transactions on Image Processing, vol. 18, no. 3, pp. 665–670, Mar. 2009, doi: 10.1109/TIP.2008.2010204.

[7]     B. K. Gunturk, J. Glotzbach, Y. Altunbasak, R. W. Schafer, R. W. Schafer, and R. M. Mersereau, “Demosaicking: color filter array interpolation,” IEEE Signal Process Mag, 2005, doi: 10.1109/msp.2005.1407714.

[8]    H. Schneiderman and T. Kanade, “A statistical method for 3D object detection applied to faces and cars,” in Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662), IEEE Comput. Soc, 2000, pp. 746–751. doi: 10.1109/CVPR.2000.855895.

[9]   M. Mancuso and S. Battiato, “An introduction to the digital still camera technology,” ST Journal of System Research, vol. 2, no. 2, pp. 1–9, 2001.

[10]  R. Lukac and K. N. Plataniotis, “Data adaptive filters for demosaicking: a framework,” IEEE Transactions on Consumer Electronics, vol. 51, no. 2, pp. 560–570, 2005, doi: 10.1109/TCE.2005.1468002.

[11]  Q. Jin, Y. Guo, J. M. Morel, and G. Facciolo, “A mathematical analysis and implementation of residual interpolation demosaicking algorithms,” Image Processing On Line, vol. 11, 2021, doi: 10.5201/IPOL.2021.358.

[12]  R. L. Ã, K. N. Plataniotis, and D. Hatzinakos, “A new CFA interpolation framework,” vol. 86, pp. 1559–1579, 2006, doi: 10.1016/j.sigpro.2005.09.005.

[13]   R. Lukac, K. N. Plataniotis, D. Hatzinakos, and M. Aleksic, “A novel cost effective demosaicing approach,” IEEE Transactions on Consumer Electronics, vol. 50, no. 1, pp. 256–261, 2004, doi: 10.1109/TCE.2004.1277871.

[14]  J. Mairal, M. Elad, and G. Sapiro, “Sparse Representation for Color Image Restoration,” IEEE Transactions on Image Processing, 2008, doi: 10.1109/tip.2007.911828.

[15]   Y. Hel-Or, “The impulse responses of block shift-invariant systems and their use for demosaicing algorithms,” in IEEE International Conference on Image Processing 2005, IEEE, 2005, pp. II–1006. doi: 10.1109/ICIP.2005.1530228.

[16]  M. I. Faruqi, F. Ino, and K. Hagihara, “Acceleration of variance of color differences-based demosaicing using CUDA,” Proceedings of the 2012 International Conference on High Performance Computing and Simulation, HPCS 2012, pp. 503–510, 2012, doi: 10.1109/HPCSim.2012.6266965.

[17]  Greg Pass, Ramin Zabih, and Justin Miller, “Comparing Images Using Color Coherence Vectors,” in MULTIMEDIA ’96: Proceedings of the fourth ACM international conference on Multimedia, Boston: ACM Inc., Nov. 1997, pp. 65–73.

[18]  Chatla Raja Rao and Soumitra Kumar Mandal, “A Survey of Image Demosaicking Algorithms,” International Journal of Research and Analytical Reviews, vol. 11, no. 1, pp. 117–134, Feb. 2024.

[19]  T. M. Lehmann, C. Gönner, and K. Spitzer, “Survey: Interpolation methods in medical image processing,” IEEE Trans Med Imaging, vol. 18, no. 11, pp. 1049–1075, 1999, doi: 10.1109/42.816070.

[20]  X. Li, B. Gunturk, and L. Zhang, “Image demosaicing: a systematic survey,” Visual Communications and Image Processing 2008, vol. 6822, p. 68221J, 2008, doi: 10.1117/12.766768.

[21]   Alexander Reshetov, “Morphological antialiasing,” in HPG ’09: Proceedings of the Conference on High Performance Graphics 2009, Stephen N. Spencer, David McAllister, Matt Pharr, Ingo Wald, David Luebke, and Philipp Slusallek, Eds., New York-: ACM Transactions, Aug. 2009, pp. 109–116.

[22]  H. S. Malvar, L. W. He, and R. Cutler, “High-quality linear interpolation for demosaicing of Bayer-patterned color images,” ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, vol. 3, pp. 2–5, 2004, doi: 10.1109/icassp.2004.1326587.

[23]   O. Ben-Shahar and S. W. Zucker, “Hue geometry and horizontal connections,” Neural Networks, vol. 17, no. 5–6, pp. 753–771, Jun. 2004, doi: 10.1016/j.neunet.2004.03.011.

[24]  C. Yin, P. J. Kellman, and T. F. Shipley, “Surface Completion Complements Boundary Interpolation in the Visual Integration of Partly Occluded Objects,” Perception, vol. 26, no. 11, pp. 1459–1479, Nov. 1997, doi: 10.1068/p261459.

[25]   X. Li, “Demosaicing by successive approximation,” IEEE Transactions on Image Processing, vol. 14, no. 3, pp. 370–379, 2005, doi: 10.1109/TIP.2004.840683.

[26]   S. Battiato, M. I. Guarnera, G. Messina, and V. Tomaselli, “Recent Patents on Color Demosaicing,” Recent Patents on Computer Science, vol. 1, no. 3, pp. 194–207, Jan. 2010, doi: 10.2174/1874479610801030194.

[27]   Q. Tian, X. Yang, L. Zhang, and Y. Yang, “Image reconstruction research on color filter array,” Procedia Eng, vol. 29, pp. 2204–2208, 2012, doi: 10.1016/j.proeng.2012.01.288.

[28]  X. Yang, Q. Tian, and M. Tian, “Adaptive Region Demosaicking Using Multi-Gradients,” Procedia Eng, vol. 29, pp. 2199–2203, 2012, doi: https://doi.org/10.1016/j.proeng.2012.01.287.

[29]  S. Baotang, S. Tingzhi, and W. Weijiang, “A Remainder Set Near-Lossless Compression Method for Bayer Color Filter Array Images,” Phys Procedia, vol. 25, pp. 1794–1801, 2012, doi: 10.1016/j.phpro.2012.03.313.

[30]  W. J. Chen and P. Y. Chang, “Effective demosaicking algorithm based on edge property for color filter arrays,” Digital Signal Processing: A Review Journal, vol. 22, no. 1, pp. 163–169, 2012, doi: 10.1016/j.dsp.2011.09.006.

[31]  Y. Zhang, G. Wang, J. Xu, Z. Shi, and D. Dong, “Novel color demosaicking for noisy color filter array data,” Signal Processing, vol. 92, no. 2, pp. 455–464, Feb. 2012, doi: 10.1016/j.sigpro.2011.08.009.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Citation:Rao.C.R. et.al. “Demosaicking of Colour Filter Array (CFA) Data via Levenberg-Marquardt Optimization Method” Ci-STEM Journal of Digital Technologies and Expert Systems, Vol. 1(2), pp. 98-110, 2024, doi: 10.55306/CJDTES.2024.1205

Copyright: ©2024 Rao.C.R.This is an open-access article distributed under the terms of the Creative Commons Attribution License which grants the right to use, distribute, and reproduce the material in any medium, provided that proper attribution is given to the original author and source, in accordance with the terms outlined by the license.(https://creativecommons.org/licenses/by/4.0/).