Stable Diffusion Upscalers

Jupyter Notebook: https://www.kaggle.com/code/antonbogomazov/stable-diffusion-sr

Model: stabilityai/stable-diffusion-x4-upscaler · Hugging Face

19 Jul:

• Notebook created
• [Problem] Can’t run upscaling of 512*512 image due to low GPU memory (>20GB required)
• Metrics shouldn’t be calculated for test images. For evaluation, I need both the original and the predicted image.
• The noise increases along with the object
• A lot of artefacts on the upscaled old photo. To try: colouring, denoising, using another prompt.
• Questions: how to use the noise level parameter? How to reduce the number of artefacts?

20 Jul:

• Paperspace has P6000 instances with 24GB GPU in Pro Plan (8$/month) and cost of ~1$ per hour. Clouds (eg SageMaker) provide instances with the same configuration for the same price. Image generation time for SD upscaler is a few minutes, so it will be cheap enough to run.
• I spent a day reading about metrics and how to correctly evaluate them and as a result, wrote some code:

Here are the original image (1024×1024) on the left, and the generated on the right with the same resolution. Original was downsampled to 256×256. The metrics are: PSNR=28.671602249145508, SSIM=0.20393137633800507
PSNR considered as decent (higher is better) and looks OK, but I can’t explain why SSIM is that low (1 is perfect similarity). I want to read more about it more: https://arxiv.org/pdf/2006.13846.pdf

• noise_level adds the random noise. It was a hypothesis checking, and after evaluation author concluded, that the noise worsened the results. But I want to check the other parameters of the model and how it could change the results.

21 Jul:

• Yesterday I was using wrong methodology of evaluating the metrics: I compared the original full-resolution image and the upscaled, but the correct way is to compare the actual input of the model and downsampled upscaled image.
• I’ve tried to use a few different prompts: ‘dog’, ‘white fluffy dog’, ‘photo of white fluffy Spitz dog’. Surprisingly, a found the better PSNR/SSIM with a less detailed prompt – ‘dog’ (for grey old photo SSIM=63 vs 61.5). Its results also had fewer artefacts and the only one, that generates the wool from nothing.

• Expectedly, the worst results of upscaling were with an old grey photo (PSNR=30.87, SSIM=0.633) and the best with a photo of a ‘whole’ dog (PSNR=31.677, SSIM=0.84)
• I’d like to refactor it a bit and perform an evaluation with a new test dataset I’ll create from photos Yulia sent me using the prompt ‘dog’.
• Also, I want to find a way of getting SSIM>0.8 with old photos. I’ll try colouring and denoising.

24 Jul:

• Separated stable diffusion specific code from common, and another refactoring
• Improved result visualisation
• Added private test dataset with Julia’s pictures
• Now it works with non-square images

25 Jul:

Today I was testing how much better images get as their resolution increases and experienced a memory leaks, which I don’t want to fix right now. [tech debt: memory optimization]

• Conclusion: as the size of the original image increases, the quality of the result increases. The decreasing SSIM can be explained by the generated details that were not present in the original image. The wool was also becoming more natural.
• Subjective assessment: the minimum image size for acceptable results is 320px.
• To improve the experiment, multiple image clusters should be used and multiple generations should be performed for each cluster.
• Are there metrics to eval the aesthetic(?) qualities of the image?

27 Jul:

• Solved out of memory issues; python keep referencing the displaying in notebook images, so gc collector can’t get memory back.
• Added saving images to files and proper metrics collection.
• [TODO] Visualisation of metrics for evaluated batch of images