Photometric Enhancements in Low-Light Segmentation: Why Transformers Don’t Care

—

3–4 minutes
Photometric Enhancements in Low-Light Segmentation: Why Transformers Don’t Care

Image segmentation – the process of dividing images into meaningful regions – forms a critical foundation for computer vision applications ranging from medical diagnostics to autonomous driving. 

When working with grayscale images in low-light conditions, a natural assumption might be that enhancing brightness and contrast would improve segmentation results. Our recent research challenges this intuition, revealing why transformer-based models remain remarkably indifferent to these adjustments.

Initial hypothesis

Our study began with a straightforward premise: enhancing grayscale images through brightness and contrast adjustments should improve segmentation by emphasizing edges and textures. To test this, we systematically evaluated both Retinex-based algorithms and 11 manual Adobe Photoshop enhancement tools.

Key findings

Despite creating visually improved images with better object visibility, our experiments showed a surprising result: segmentation metrics remained virtually unchanged. 

Transformer models like SegGPT maintained consistent Intersection over Union (IoU) scores of approximately 0.75 regardless of the enhancements applied.

Why transformers remain unaffected

Our analysis revealed three fundamental reasons why transformer-based segmentation models ignore photometric adjustments:

1. Normalization erases global shifts

Input normalization effectively cancels out brightness and contrast adjustments, which are essentially linear operations. When images are standardized during preprocessing, these global shifts become irrelevant.

2. Augmentation-induced invariance

Modern vision transformers are typically trained with extensive data augmentation, including random brightness and contrast variations. This training approach teaches models to treat such variations as noise rather than meaningful features.

3. Structural primacy

Vision Transformers (ViTs) primarily analyze relative relationships between image patches rather than absolute brightness values. Global adjustments like those made with Levels uniformly scale gradients without altering the underlying structural relationships that transformers use for segmentation.

Technical deep dive: enhancement methods tested

Retinex algorithms (e.g., single-scale SSR) improved visibility but introduced artifacts (unnatural contrasts, halo effects). 

SegGPT’s IoU remained unchanged (0.75 → 0.75), prompting a shift to controlled manual adjustments.

So we moved on to the Adobe tools, validated against their theoretical formulations:

comparison of Adobe tools

1. Levels

Effect: Stretches the dynamic range to [0, 255], expanding the histogram endpoints. Dark areas become lighter, bright areas darker, increasing overall contrast.

levels
levels mask

2. High pass

Effect: Emphasizes sharp transitions (edges) by subtracting the blurred (low-frequency) component. The histogram centers around mid-gray (0.5), enhancing textures and contours.

high pass
high pass mask

3. Contrast

Effect: Compresses mid-tones and stretches histogram extremes. Enhances differences between light and dark areas, sharpening object boundaries.

contrast
contrast mask

4. Gradient

Effect: Highlights object boundaries by computing the difference between maximum and minimum intensity in a local neighborhood (e.g., 3×3). The histogram concentrates in mid-tones, visualizing object silhouettes.

gradient
gradient mask

5. Exposure

Effect: Linear brightness scaling shifts the histogram to the right; gamma correction non-linearly adjusts tones (γ < 1 lightens shadows, γ > 1 darkens). Risk of detail loss in overexposed areas.

exposure
exposure mask

6. Curves

Effect: Point-wise brightness correction. An S-shaped curve increases mid-tone contrast; concave/convex curves smooth the histogram. Enables flexible detail visibility control.

curves
curves mask

7. Brightness

Effect: Adds a constant offset to all pixels. The histogram shifts right (brighter) or left (darker) without altering its shape. Does not create new structural features.

brightness
brightness mask

8. Sharpen

Effect: Enhances high frequencies (edges) by adding the difference between the original and blurred image. A Laplacian filter (kernel: [-1, 5, -1]) increases sharpness but may amplify noise.

sharpen
sharpen mask

9. MixerColor (Channel mixer)

Effect: Converts color images to grayscale using human luminance perception (BT.601 standard). Preserves relative object brightness.

channel mix
channel mix mask

10. AutoFunc (Auto-contrast)

Effect: Automatically stretches the histogram by clipping the extreme 0.1-2% of pixels. Boosts contrast, but risks information loss in over-/underexposed regions.

auto contrast
auto contrast mask

11. Layers

Effect: Blends layers with different brightness/contrast. Lightens shadows or darkens highlights, balancing illumination. The histogram combines distributions from multiple layers.

layers
layers mask

Conclusion and future directions

We conducted an extensive analysis of the impact of manual brightness/contrast adjustments on transformer-based segmentation quality. Despite visual improvements in object visibility, segmentation metrics (IoU, Dice) showed no enhancement.

Thus, photometric transformations – simple increases in brightness/contrast – fail to improve segmentation quality in the studied approach. Normalization and training augmentations negate the effects of such adjustments.

Based on our findings, we propose shifting focus to:

  • Edge features and gradient descriptors: Using algorithms like Canny or HOG to emphasize object shape rather than intensity levels
  • Segmentation loss modification: Introducing regularization on object boundaries or adopting weighted metrics to prioritize precise edge delineation
  • Contour-based approaches: Extracting structural or contour-based object characteristics rather than relying on global photometric corrections

Leave a Reply