Denoising Diffusion null-space model (DDNM)

August 3: Setting the stage

We started making a Jupyter Notebook on August 3rd, with big plans in mind:

Evaluate super-resolution (SR) possibilities: Aiming to compare SR with single image super-resolution (SD) upscaling, we sought to assess the model’s capabilities.
Enhance SD upscaler results: Exploring the possibility of improving SD upscaler outcomes by denoising and coloring input images.
Background and pose transformation: Experimenting with inpainting techniques to change backgrounds and poses in dog images.

August 4: Building the foundation

We created the base notebook, laying the groundwork. We then added configuration details and key methods to assess coloring and super-resolution processes.

August 7: Unraveling challenges and discoveries

As the journey progressed, we faced new challenges. Forced conversion to grayscale did not stabilize colorization, leading to significant artifacts. Despite these challenges, initial attempts at colorization showed promising results.

Force conversion to grayscale doesn’t make the colorization more stable. One good result per 5 generations.
Re-colouring slightly improves the result while introducing significant artefacts:

First coloring. Better than nothing.

Second

Third

Denoising and inpainting are not supported in this model.
Inference time is much higher than with SD: 8 minutes for x4 upscaling of (384,256)
Colored images have better super-resolution results.
A lot of conditional models operate by using imagenet class id. Unfortunately, there is no class for Japanese Spitz dogs, so it leads to the TASK: identify the closest to Spitz dog breed.

Super-resolution:

'PSNR': 29.034648895263672, 'SSIM': 0.6728171706199646

Compared to SD, there are fewer artefacts (better SSIM), but an image is also much less detailed.

'PSNR': 29.806194305419922, 'SSIM': 0.669299304485321

{'PSNR': 29.892139434814453, 'SSIM': 0.6751380562782288}

Summary: Inference takes too long, and the results are too bad. We’d like to find something else for denoising. Maybe it’s because of the old version of pre-trained model, but there is no newer one, or because of downscaling under the hood.

BUT results with a very low-res images are better than with SD (SD is unusable below 128px)

The only application for that model is coloring of gray photos to make the results of SD upscaling slightly better, OR initial upscaling of images with dims <128 with a minimum distortion.

August 10: Cracking the code

A breakthrough occurred on August 10th when a stable method for colorization was discovered. By evaluating several similar dog breeds, the model was fine-tuned for Japanese Spitz, with the Samoyed class proving to be the most effective. This discovery opened new avenues for stable colorization, marking a significant milestone in the journey.

We found a way to make colorization stable. After reviewing ImageNet dog breed classes, I found a few similar to Japanese Spitzes: