When pictures are dark, it’s hard to see all parts of objects clearly. New AI tools like SegGPT can identify objects in many different types of images without special training.
But even these advanced tools have trouble with small or thin parts when the lighting is poor. Our tests showed that in low-light photos, SegGPT often missed things like animal tails or thin limbs.
We tried fixing the problem by manual photometric enhancement of input images – tweaking brightness and contrast. Our thinking was simple: improve visibility, and the model should segment more completely. We boosted brightness/contrast and turned grayscale images into full RGB channels to match what the model needed.
Unfortunately, these photometric enhancements barely helped. Transformer models typically ignore uniform brightness changes because they focus more on the structure of objects rather than their lighting. So simply making the image brighter or adding color didn’t improve the transformer’s output much.
Next, we tried masks post-processing – fixing them after they were created using techniques like morphological closing (which fills small gaps in the mask) and other filter-based cleanups. These methods also failed to recover the missing parts.
Our segmentation quality measurements (IoU, Dice coefficient) stayed basically the same across all these attempts. This matches what others have found – even advanced preprocessing only improves IoU by about 1-2% at best.
The real problem was the model’s structure-focused logic – it makes decisions based on learned object shapes and context, so our pixel-level adjustments didn’t make much difference.
Early failures
We tried fixing the problem in two separate stages, which we explain below to show why these early attempts didn’t work well:
Phase 1 (Photometric and Basic Ops)
We took dark single-channel images and converted them to RGB(A) format to meet the model’s input needs. We then manually improved their photometric qualities through intensity scaling and denoising.
For the output masks, we used binary closing (a morphological operation) to try connecting separated parts, hoping this would attach faintly detected tails or legs to the main body of objects.
These efforts didn’t work well. The transformer outputs stayed basically the same – parts that were missing before remained missing, and our measurement metrics (IoU/Dice) showed no real improvement.
Phase 2 (Advanced Post-Processing)
We tried more complex refinement methods. We tested different interpolation methods when scaling images and masks. For example, we compared adaptive nearest-neighbor against high-quality Lanczos filtering for resizing to keep fine details intact.
We added mask labeling to find and track connected components (so if a tail showed up as a separate small mask, we could identify it). We set fixed random seeds to make sure our results could be reproduced. We also created logic to enforce expected object counts (if we knew there was one dog in the image, we would merge or keep masks accordingly).
Despite these technical improvements, we didn’t see a breakthrough. Sometimes we’d get better results in one case, but worse in another. Our IoU/Dice metrics stayed within a few percentage points of our starting point, showing that the basic segmentation problems weren’t being solved.
Table 1 summarizes the performance of these early approaches. As shown, neither the photometric enhancements nor the morphological post-processing provided a meaningful boost over the baseline SegGPT output.
The Intersection-over-Union (IoU) and Dice coefficients remained roughly constant, underscoring the ineffectiveness of purely image-level or mask-level tricks in altering the transformer’s segmentation behavior.
| Method | IoU (↑) | Dice (↑) |
| Baseline (SegGPT only) | 0.78 | 0.87 |
| + Photometric enhancement | 0.79 | 0.88 |
| + Morphological postproc | 0.80 | 0.89 |
| + Advanced interp/logic | 0.80 | 0.89 |
| Hybrid SegGPT + SAM | 0.90 | 0.95 |
Table 1. Segmentation performance across methods. A significant jump in IoU/Dice only occurs with the proposed hybrid pipeline.
Hybrid pipeline innovation
Confronted with the failure of brute-force enhancements, we devised a hybrid segmentation pipeline that combines the strengths of two state-of-the-art models:
- SegGPT provides a rapid initial guess of the object
- SAM fine-tunes that guess with high precision. .
The pipeline consists of three steps:
1. Coarse mask generation (SegGPT)
We first run SegGPT on the image to obtain a coarse binary mask of the target object. This mask is generated in a single forward pass – one of SegGPT’s advantages is its speed and ability to segment without task-specific training. The output at this stage correctly localizes the general region of the object (e.g., the dog’s body), but as noted, it may miss thin or subtle parts (tail, leg).
2. Mask conversion to SAM logits
To feed this coarse mask into Meta’s Segment Anything Model, we convert the binary mask into SAM-compatible logits. In practice, we map the SegGPT mask into a continuous-valued mask by assigning a high positive value (e.g. +1e4) to pixels inside the predicted region and a large negative value (e.g. -1e4) to pixels outside.
This creates a sharp probability map that SAM can interpret as a mask prompt for refinement. Essentially, we’re telling SAM, “this is roughly the object.” By providing the mask in this form, we leverage SAM’s ability to take mask or region prompts as guidance for segmentation.
3. SAM focused refinement
We also extract the bounding box of the coarse SegGPT mask and supply that to SAM as a spatial prompt. By giving SAM the tight bounding box around the SegGPT mask, we focus its attention on the relevant area of the image.
With the mask logits as an initial guide and the box to zoom in, SAM then generates a refined segmentation mask. SAM’s segmentation model – a ViT-based architecture trained on billions of masks – excels at boundary detail and can segment object parts with unprecedented precision.
The result is a mask that retains SegGPT’s general object coverage but with much sharper edges and, crucially, previously missing parts now included. Thin appendages, weakly contrasted sections, and intricate boundaries are captured in the SAM-refined output.
Through this hybrid prompt refinement, we effectively combined SegGPT’s context-driven coarse prediction with SAM’s detail-oriented segmentation. SegGPT’s job is to quickly propose “where” the object is, and SAM’s job is to perfect “what” the exact object outline is, within that suggestion.
This division of labor plays to each model’s strengths – the transformer’s broad understanding and the promptable segmenter’s precision.
Results
The hybrid pipeline was a major breakthrough in our tests, working where other methods failed. In qualitative terms, it was the first approach that consistently found the missing object parts.
For example, with a difficult low-light photo of a dog partly hidden in clutter, standard SegGPT would segment the dog’s torso and head but miss its thin tail and one hind leg. The hybrid SegGPT+SAM model, however, accurately recovered the dog’s tail and the hidden leg in the segmentation mask. This accomplishes our main goal: making sure no part of the target object gets left out of the segmentation.
Quantitatively, we can see the improvement in the much higher IoU and Dice scores (shown in Table 1). The hybrid method’s IoU on our test images increased to about 0.90, up from about 0.80 with our previous best attempt.
Similarly, Dice similarity rose from around 0.88 to 0.95, showing much better agreement with ground truth masks. These metrics confirm that our improved masks overlap much more accurately with the true object regions because they now capture those previously missed parts.
We also looked at how well the system performed on specific challenging parts (the tail and leg in our example). Table 2 shows the per-part segmentation accuracy. With the baseline and early approaches, the IoU for the tail and missing leg regions was basically 0 – meaning these parts weren’t segmented at all.
With the hybrid pipeline, however, these values improved dramatically (tail IoU around 0.70, leg IoU around 0.80), showing that most of these regions are now correctly included in the mask. In practical terms, this difference is huge: the final mask changed from an incomplete outline to an almost perfect silhouette of the dog.
| Approach | Tail IoU (↑) | Leg IoU (↑) |
| Baseline (SegGPT only) | 0.00 | 0.05 |
| + Photometric enhancement | 0.00 | 0.06 |
| + Morphological postproc | 0.01 | 0.05 |
| + Advanced interp/logic | 0.02 | 0.10 |
| Hybrid SegGPT + SAM | 0.70 | 0.80 |
Table 2: Detection of subtle object parts. Only the hybrid approach succeeded in segmenting the dog’s tail and missing leg (signified by the substantial IoU for those parts), whereas earlier methods effectively missed them entirely.
The results above make it clear that our SegGPT+SAM hybrid not only improves overall segmentation overlap with the truth, but specifically addresses the prior blind spots (tails, thin limbs, and other fine structures in difficult images).
Discussion
Why does our hybrid pipeline work when photometric and morphological methods failed? The answer is in how it combines structure awareness with detail refinement.
SegGPT, as a transformer model, is great at understanding high-level context: it recognizes which pixels generally belong to “the dog” versus background, based on learned visual semantics. However, its coarse mask shows it misses certain parts, probably because it focuses on the most obvious structures (like the dog’s torso and head) while treating thin extensions as separate or unimportant in its attention map.
Photometric enhancements (making the image brighter or adjusting contrast) didn’t help because the model’s attention weighting and learned features don’t really respond to these global changes. Essentially, the transformer “doesn’t care” about increased pixel intensity if it doesn’t change the object’s basic shape.
Similarly, using morphological operations for post-processing works only at the 2D pixel level without any understanding of object semantics, so it can’t create a missing tail if the model never identified it as part of the object to begin with.
The hybrid approach works because it combines two complementary systems. SegGPT’s coarse mask provides a strong spatial prior – basically an initial guess of the object’s structure. This guess might be incomplete, but it narrows down the problem area.
This is where SAM comes in: given that approximate region (plus the mask prompt showing “foreground versus background”), SAM uses its powerful image encoder and prompt-driven decoder to find any object boundaries in that area. SAM was trained on an extremely diverse set of objects and parts, learning to outline objects with fine accuracy from minimal prompts.
By guiding SAM with SegGPT’s output, we take advantage of this capability: SAM can sharpen edges and add fine segments that the coarse mask missed. In our example, once SAM is focused on the dog’s hindquarters (guided by the box) and knows which region should be the object, it can identify the outline of the tail and leg and include them in the mask.
The result is a segmentation that captures the global structure (thanks to SegGPT) while also including the local details (thanks to SAM).
Unlike our hybrid approach, the purely photometric or morphological attempts didn’t have this powerful combination of understanding. They either tried to brute-force the input image (which doesn’t change how the model fundamentally perceives things) or brute-force the output mask (which can’t add new object recognition).
Our hybrid pipeline is structure-aware: it recognizes that to improve a segmentation model’s output, you need another model that understands object boundaries, not just simple pixel adjustments. This finding highlights a broader lesson in deep learning segmentation – combining multiple models or stages often works better than single-stage quick fixes, especially for challenging situations like low-light or partially hidden object parts.
Conclusion & future work
We created a hybrid prompt-refinement segmentation approach that significantly improves transformer-based segmentation results on difficult images. By combining SegGPT’s quick coarse segmentation with SAM’s precise mask refinement, we reached our main goal: previously missed object parts (like thin tails and hidden legs) are now correctly segmented.
The hybrid pipeline solves the limitations we found in purely transformer-driven segmentation for low-light or complex scenes. We discovered that using a targeted second-stage refinement works well where photometric enhancements and post-processing hacks don’t help.
Future work will focus on making this two-step approach more efficient and powerful. One immediate plan is to combine the hybrid pipeline into a single model – for example, fine-tuning SAM (or a similar segmentation network) with SegGPT’s outputs as additional inputs, essentially learning the refinement process end-to-end.
Another direction is to improve how we handle the mask logits and prompts: currently we use a simple high/low value scheme, but learned prompt embeddings could represent coarse masks more effectively. We also plan to test this approach on more objects and lighting conditions to make sure it works beyond our initial examples.
The success of this hybrid method creates new opportunities for combining generalist transformers with specialist models for segmentation tasks that need both broad context understanding and precise detail. These promising results encourage more research into multimodel pipelines, bringing us closer to segmentation systems that truly capture everything – down to the last tail and toe.
Leave a Reply