Search and compare upscaling models to enhancing photo resolution

Have you ever wondered which technology currently leads in dramatically enhancing photo resolution by ‘filling in’ missing details through Machine Learning? I’m familiar with technologies like ESRGAN, DDNM, HAT, GigaGAN, GFPGAN, but their performance seems somewhat underwhelming. It raises the question: why isn’t the progress in this field more remarkable? Considering the abundance of high-resolution images, especially faces, one would think there’s ample data for training these systems. This makes the relatively modest improvements, like the ESRGAN’s results on the first blurry image in the comments, quite puzzling

ESGRAN is often touted as the leading solution in image enhancement technology. While effective, training these systems can be a time-consuming and costly endeavor. For basic image sharpening, sophisticated tools like neural networks aren’t always necessary. A simple solution involves employing a HighPass filter and blending the processed layer using LinearLight or Overlay mode for a subtler effect.

However, the real challenge emerges when images are enlarged, say, by a factor of two. In such cases, the task shifts from merely restoring sharpness to actually reconstructing details. This involves generating four pixels from one original pixel, a significantly

The challenges in AI-generated imagery, especially in facial recognition, are well-documented. Take, for instance, the intricate project at Daniel Boschung’s gallery featuring super-detailed photographs of faces. Despite decades of development, AI often struggles with rendering eyes realistically, frequently resulting in an unsettling ‘zombie look’ or misaligned eyeballs. The core issue seems to lie in the AI’s handling of the pupils. While the AI has been trained on millions of images and can easily determine the position of a pupil, synchronizing the pupils’ positions in both eyes poses a significant challenge. This is not due to a lack of understanding of facial symmetry, as AI systems are well-trained in this aspect. The problem arises when the AI encounters variations in pupil positions across different images, even when the head’s position remains constant. These variations can disrupt the AI’s learning process, leading to inconsistent and often unrealistic images. Understanding and addressing this specific issue may be key to advancing AI’s capabilities in producing more lifelike and accurate facial images

There are a lot of pre-trained models for human faces with impressive results SR3: Image Super-Resolution via Iterative Refinement , so, it’s not working well with dogs. Another disadvantage is inference time; it’s ~10min for 64->512 upscaling on standard Colab hardware.

Something to try. It comes to mind, why not make several phases, at each of which the result of the previous phase is put in order. Some researchers already try to do that CodeFormer – propose a Transformer-based prediction network, named CodeFormer, to model global composition and context of the low-quality faces for code prediction, enabling the discovery of natural faces that closely approximate the target faces even when the inputs are severely degraded.

Some of our finding from experiments

1. Built of a demo stand with one of them (Real-ESRGAN) and instruction https://github.com/xinntao/Real-ESRGAN/blob/master/docs/Training.md – @Anton Bogomazov
2. After evaluation of Swin2SR we could confidently say that unconditional upscalers are not good enough for us – @Anton Bogomazov
3. Best Result for now with DDNM colouring (preparation to match imagenet dataset) → sd conditional upscaling (add detailed wool) works much more stable with a colored by DDNM image . We could try to upscale it a few times, maybe in smaller steps, to produce even better results, because we have a more detailed picture for that time- @Anton Bogomazov
4. Implementation SRGAN – work in progress GitHub – Deviousrage/img_upscaling_06062023: image upscaling to higher resolution, test for GAN scheme – @Egor Zyryanov

Next idea

Recently, while experimenting with Real-ESRGAN, Swin2SR, DDNM (Anton: AID-57/AID-64) and SRGAN (Egor AID-57) and waiting for the final implementation/trained GigaGan from GitHub – lucidrains/gigagan-pytorch: Implementation of GigaGAN, new SOTA GAN out of Adobe. Culmination of nearly a decade of research into GANs, we came across a groundbreaking new study, detailed at Generative Powers of Ten project. The researchers have unveiled an innovative application of diffusion techniques, demonstrating the ability to achieve a 10x zoom into any image. This zoom isn’t just a simple enlargement; it creatively ‘invents’ the content within the zoomed area, potentially enabling more than a 10x magnification. In essence, this represents a tenfold increase in image resolution. Currently, the practical model isn’t available yet, but the methodology is thoroughly described, complemented by intriguing video examples that showcase its potential.

Reference

Papers with Code – Image Super-Resolution – There is a compilation of research papers by topic with extracted metrics for models. Some models are not mentioned in the Image Super-Resolution category (e.g. Gigagan upscaler).