The Foundation for Reviving Old Images: A Six-Stage Pipeline

Old, heavily degraded photographs pose multiple restoration challenges: coarse noise, printing artifacts, cracks, missing parts, a general lack of detail, and so on. It’s tempting to think one or two magical super-resolution methods or filters might solve it all, but reality proves otherwise.

Each artifact can require specialized handling, done in a carefully orchestrated order.

Previously, we tried a simpler pipeline: RealESRGAN → Bilateral Filter → CLAHE. We saw that it sometimes injected artificial details or further accentuated artifacts. 

From that experience, we learned we need a modular “foundation” with separate steps for denoising, segmentation, upscaling, inpainting, and final contrast adjustments. In other words, a step-by-step approach that allows us to adjust or swap tools at each stage until we achieve the best result.

We’ve arrived at the following six-step pipeline that we consider our “main skeleton”:

1. Segmentation
2. Denoising
3. Initial upscale
4. Inpainting
5. Final upscale
6. Final tonal/contrast adjustments

Let’s break down each step, explain why it’s important, and suggest ways to make it better.

Please note: this article is only an introduction to our pipeline. We will further refine each step in future iterations.

Step 1: Segmentation (SegGPT)

Segmentation is key to an adaptive restoration approach. By separating the image into zones (dog’s head vs. background), we can apply denoising or upscaling more accurately.

⚠️ SegGPT may incorrectly classify edges like fur and fuzzy background. These subtle misclassifications can harm the next steps. 

Head segmentation with artifacts

💡 We plan to refine segmentation by either tweaking thresholds or combining with manual corrections. The better the segmentation, the less risk we have of messing up the subject’s shape in inpainting.

Step 2: Denoising (BM3D & Non-Local Means)

Highly noisy images can sabotage any super-resolution method, so we remove grain and artifacts early. We tested:

• Non-Local Means: Decent for moderate noise, somewhat faster if carefully implemented, but can leave residual speckles on very degraded photos. 
• BM3D: Often best for heavy noise, producing a crisp, consistent base for the next steps.

BM3D typically outperforms Non-Local Means in severely damaged images, making it a top choice for big restoration jobs. 

💡 Segmentation-aware? Not by default, but we plan to incorporate segmentation masks so we can treat fur differently from the background.

Step 2 is one of the best-performing parts of our pipeline. We’re satisfied with how it sets up the image for minimal artifact growth in the following stages.

Step 3: Initial upscale (RealESRNet)

We prefer RealESRNet here instead of RealESRGAN, because RealESRNet adds fewer “invented” details. It also ensures the image is gently enlarged so the inpainting and final upscale have more pixels to operate with.

However, if the original is too noisy or extremely low-res, RealESRNet might produce soapy or blotchy areas. That’s why the prior denoising step is so critical. 

We might experiment with other models, but for now RealESRNet is our default gentle upscaler.

RealESRNet upscale

💡 So far, RealESRNet is adequate for initial scaling, but we suspect we can still fine-tune parameters for certain textures.

Step 4: Inpainting (Stable Diffusion)

After upscaling, some images still have partial damage, missing details, or fur that looks unnatural. Stable Diffusion inpainting is our tool to fill these gaps and refine texture. 

However, in practice, we have the following:

1. Boundary errors: If segmentation misplaces the dog’s edges, inpainting might create square or flattened spots on the fur. 
2. Over-inventing: With higher strength or guidance scale, the model drastically changes shape or style, showing results that are worse than we hoped. 
3. Random artifacts: Even with small masks, we occasionally see strange blobs or smudges, indicating we need to tweak the pipeline’s negative prompts, mask size, or SD’s parameters.

💡 We’re not satisfied with the inpainting results right now. While it can do wonders for small cracks or subtle improvements, the shape of the dog’s head sometimes gets altered too much. We plan to return to Step 4 in future experiments.

Stable Diffusion inpainting. This result turns out bad because of the modified head

Step 5: Final upscale (RealESRGAN)

When the image is relatively cleaned and upscaled once, we can apply RealESRGAN to add a sharper, more photorealistic finish. RealESRGAN is known for inventing convincing textures, which can be beneficial as a final step.

We can anticipate the following risks:

• Overly “plastic” or “GAN swirl” artifacts if the content below is still flawed. 
• Might intensify any leftover segmentation inaccuracies or inpainting artifacts.

However, for images that have turned out well in earlier steps, RealESRGAN can produce a striking final pass – especially on fur edges and background textures.

💡 We might try SwinIR or skip this step entirely if minimal invention is desired.

RealESRGAN

Step 6: Final tonal and contrast balancing (CLAHE, Curves)

The last stage is to ensure the photo looks vivid and cohesive. We can do it with these tools:

• CLAHE (Contrast Limited Adaptive Histogram Equalization) can bring out local detail without overstating highlights. 
• Curves adjustments let us refine the global brightness/contrast or color balance.

Conclusion: A skeleton for adaptive restoration, with plans to improve

This six-step pipeline forms our foundation for enhancing old, damaged images:

1. Segmentation → We see it’s crucial but not always perfect. Future refinements or manual corrections needed. 
2. Denoising → Currently the best step for us, as it reliably sets up a clean base. 
3. Initial upscale → A conservative approach that’s “good enough” but might benefit from parameter tuning. 
4. Inpainting → Our biggest disappointment so far. Shapes get distorted if segmentation or mask settings are suboptimal. We aim to revisit this thoroughly. 
5. Final upscale → Potentially brilliant final enhancement, but can highlight earlier mistakes. 
6. Final tonal/contrast → Consistently positive finishing touch.

At the moment, Steps 2 and 6 show the most stable, satisfying results, while Steps 1, 3, 4, and 5 remain on our “to improve” list. Step 4 in particular, where inpainting can degrade shapes more than help. 

Despite these issues, we believe in the core skeleton of an adaptive pipeline:

• Each phase solves specific problems (noise, resolution, missing details). 
• We can swap or tweak the “instrument” (BM3D vs. Non-Local Means; lower or higher inpainting strength) to get a better output. 
• Over time, we’ll refine each stage’s parameters, update segmentation accuracy, and select the best combination for each image’s unique defects.

By iterating on each step, we hope to perfect the entire chain – turning severely damaged photos into pleasing, naturally detailed images, one stage at a time.