Restoring Images with AI: Testing the Waters of Photo Recovery

Old photographs hold memories of the past, especially those of our four-legged friends. Over time, these cherished images can suffer from scratches, noise, loss of details, and blurriness, degrading their quality.

This is the original photo we took for the experiment, dated around 1970

Pictures of dogs, especially those with intricate fur patterns like Japanese Spitzes, present a unique challenge for restoration. The complexity of their fur, with its fine details and varying textures, makes it difficult to restore these images accurately.

Using AI tools to tackle such a detailed and nuanced task offers an exciting opportunity to see just how far technology has come. By focusing on these challenging images, we can truly test the capabilities of modern AI in preserving the delicate details that make these photos so special.

AI image restoration: Project overview

The main goals of our project were focused on bringing new life to old photographs of Japanese Spitzes:

• Colorize black and white photographs to restore their original vibrancy.
• Remove artifacts, scratches, and noise to clean up and improve image quality.
• Enhance sharpness and detail, particularly in the dogs’ fur, to make the images more lifelike.
• Upscale the images to higher resolutions for better clarity and display.

We worked with low-quality photographs that presented various challenges. The collection consisted of both old black and white and color photographs of Japanese Spitzes taken before 1970.

Many of these images suffered from defects such as artifacts, scratches, noise, and blurriness, all of which required careful correction to restore their original quality.

Choice of tools and approaches

1. Using ready-made AI tools

We decided to start with ready-made tools to save time and resources. These tools offered a range of features, from basic noise reduction to advanced detail enhancement.

Here are the tools that made our list:

• OpenCV
• Real-ESRGAN
• BOPBL

2. Discarding some methods

However, not all methods yielded satisfactory results. Some approaches struggled with the intricate details of the dogs’ fur, leading to unnatural textures and loss of authenticity. 

Experiments with AI photo recovery

With all of this in mind, we started experimenting. Which tool will give more cohesive results?  

We’ll walk you through our experiments, explaining the techniques we used and showing the results we got.

Preliminary processing: OpenCV

To kick off our restoration process, we used OpenCV, an open-source computer vision library. OpenCV provided essential tools for preliminary processing like noise removal and sharpness enhancement. These initial steps were crucial in preparing the old photographs for further enhancement with AI tools.

• Filtering: cv2.bilateralFilter for noise removal.
• Morphological operations: cv2.morphologyEx to detect small defects.
• Inpainting: cv2.inpaint for restoring areas with defects.
• Sharpening: cv2.filter2D for enhancing sharpness.

The use of certain methods led to some improvement in image quality. However, they had significant drawbacks. These methods couldn’t completely remove artifacts, resulting in unsatisfactory outcomes. Additionally, distortions appeared in the images and worsened during the upscaling process, further reducing the overall quality.

OpenCV didn’t show the great results and caused distortions of the image.

Testing BOPBL

Bringing Old Photos Back to Life (BOPBL) is a project by Microsoft Research Asia that uses AI, specifically deep learning, to fix issues and make photos look new again.

The process involves several steps. First, the AI scans the photo to detect any scratches or blemishes. Then, it enhances the faces in the photos, making them clearer and more detailed. Finally, the AI restores the overall quality of the image, bringing back vibrant colors and sharpness. 

BOPBL successfully removed scratches and enhanced image sharpness

Testing Real-ESRGAN

Real-ESRGAN is particularly effective at enhancing low-resolution images, restoring details, and reducing noise, making it a popular tool for tasks like photo restoration, video enhancement, and improving the quality of digital art. This technology is based on deep learning models that have been trained on a wide variety of images, allowing it to produce high-quality results across different types of images.

With Real-ESRGAN, we saw significant improvement in detail without noticeable artifacts

But what if we overlay several tools?

This is what we got with the combination of OpenCV and Real-ESRGAN. Unfortunately, it worsened the image quality.

We used BOPBL first to get the images ready, and then processed them with Real-ESRGAN. This combination showed the greatest effect.

Evaluation of the results

After conducting our experiments and applying various AI tools and techniques, we carefully evaluated the results to determine the effectiveness of each method.

For objective quality assessment, we used the following metrics:

• Structural Similarity Index (SSIM): SSIM measures the structural similarity between images and can be used as a benchmark metric to identify global artifacts.
• Peak Signal-to-Noise Ratio (PSNR): PSNR measures the signal-to-noise ratio difference between images, but a high PSNR value can correspond to blurred images that lack sharpness, as the pixel values are close to the original.
• Learned Perceptual Image Patch Similarity (LPIPS): Perceptual difference between images.
• CIEDE2000 (ΔE₀₀): Assessment of color differences.
• Frechet Inception Distance (FID Score): FID evaluates the distribution of generated and real images by comparing feature representations from an Inception network. Lower scores indicate that the generated images closely mimic real-world image distributions, which correlates with higher image realism.

So here’s the breakdown of the metrics:

Combination of BOPBL + Real-ESRGAN – optimal balance between restoration and detail enhancement. In the BOPBL_rESRGAN row, SSIM is 0.7452 and PSNR is 17.47, demonstrating balanced image structure restoration with a focus on details. However, the sharpness quality is inferior to other methods, as seen from the low PSNR.

CV – effective noise removal. In the CV row, PSNR is 23.96, indicating a good level of noise removal. However, subjective image quality assessment shows that the restoration quality is visually worse than other methods. This is because metrics like PSNR and SSIM do not perform well in conditions with artifacts that this method could not fully eliminate.

BOPBL – excellent restoration quality. In the BOPBL row, SSIM is 0.7717 and PSNR is 17.99. This indicates good image structure restoration with a focus on maintaining quality, but without significant enhancement of details.

CV_rESRGAN – detail enhancement, minimization of artifacts. In the CV_rESRGAN row, PSNR is 25.50 and SSIM is 0.7950, confirming a high degree of noise removal and texture enhancement. As with the previous CV result, it showed the highest metrics, but is visually clearly inferior to other results.

Real-ESRGAN – high-quality detail enhancement, preservation of texture and sharpness. In the rESRGAN row, SSIM is 0.7600 and PSNR is 17.80, suggesting that this technology focuses on preserving fine details. However, the result slightly falls short in terms of the number of restored fine textures, as indicated by the lower PSNR and SSIM values compared to other methods.

At first glance, it seems that the CV method shows weak results if judged only by subjective assessment, despite high numerical metrics. This is because metrics such as PSNR and SSIM often do not account for the impact of visual artifacts. The visual quality deteriorates due to the specifics of the algorithm and the insufficient adaptation of metrics to account for these features.

The average values of each quality metric calculated across various technologies or models

These averages provide an overall benchmark, highlighting the general effectiveness and limitations of the selected tools. By comparing individual technology scores against these averages, we can identify which tools perform above or below the average, helping inform future decisions on tool selection.

Here are all the results of the photo restoration experiment side by side:

Conclusion and future plans

Through our experiments, we arrived at the following results:

• Optimal processing sequence for restoring old photographs of Japanese Spitzes is to use BOPBL for restoration, and then Real-ESRGAN for enhancing details. Alternatively, to avoid excessive denoising, you can rely solely on Real-ESRGAN. 
• Discarding some methods, such as U-Net with denoising and GFPGAN, allowed us to save time and resources without sacrificing quality.
• Using metrics enabled us to objectively assess the quality of various methods and select the best one.

To objectively assess image restoration quality, we plan to create a unified metric system that considers visual features and artifacts. This system will combine existing metrics with new methods for evaluating textures, sharpness, noise removal, and artifacts.

With this comprehensive approach, we aim to avoid discrepancies between metric values and actual visual quality, providing a clearer picture of algorithm performance.

The system will help compare technologies and improve them using real data, leading to higher quality in image restoration tasks.

We love AI challenges here at Furnets. Have an interesting one? Drop us a line to discuss future collaboration.