Traditional methods for evaluating image quality use familiar metrics like variance, entropy, and gradient-based measures to assess sharpness, noise, and structure. However, these numbers often don’t fully reflect what people actually appreciate in an image.
At the other extreme, advanced machine learning models like contrastive language-image pretraining (CLIP) can understand images at a high level but often miss the finer details needed for specific tasks, such as evaluating the subtle texture of an animal’s fur or getting the right exposure in tricky lighting.
In our research, we’ve developed an image quality evaluation system that combines traditional metrics with insights from human experts. Instead of just using standard metrics or general models, we brought in experts familiar with dog breed standards and image aesthetics to label a specially selected dataset.
Using a machine learning model called Random Forest Regressor, trained on data labeled by experts, we’ve created a “super metric.” This new metric aligns well with how humans perceive image quality and outperforms traditional and some advanced methods in providing specific quality scores for different domains.
The core problem: Beyond conventional metrics
Traditional metrics are the basic tools for assessing image quality. Here’s a breakdown:
- Variance of Laplacian: This measures how quickly image intensity changes to assess sharpness. High values indicate sharp edges, but it can’t tell the difference between important details like fur strands and noise.
- Image entropy: It shows the complexity and variety of pixel intensities, suggesting texture richness. However, random noise also increases entropy, so it’s not enough on its own to judge quality.
- Dynamic range metrics: These look at the range of light and dark areas to capture details in highlights and shadows. But a wide range might result from poor exposure or scanning issues, not just good image features.
- GLCM-derived features: Metrics like contrast, homogeneity, and energy describe texture patterns. They’re useful for texture analysis but need careful interpretation since high contrast might mean rich texture or just noise.
- No-reference metrics (e.g., NIQE): These predict quality without needing a reference image by modeling “naturalness.” However, they might miss specific details, like the fine fur of a Spitz, leading to inaccurate quality assessments.
Individually, these metrics are like isolated instruments playing different tunes. They offer bits of information about image quality but can’t come together to create a complete “score” that matches how humans view an image. It’s especially true when the image has important details, like assessing a dog’s features for breed standards.
Involving human experts
To close this gap, we brought in expert knowledge to improve the evaluation process. We created a special dataset, including photos of Japanese Spitz dogs, showing different conditions like being underexposed, slightly blurred, or having detailed highlights and shadows.
Professionals who understand photographic quality and breed standards annotated these images. Instead of making broad aesthetic judgments, they used a structured rating system. This system included criteria similar to those in a CLIP-based evaluation but focused more on human perception and relevance to the domain. For example:
- Proper exposure criterion: Points were given based on whether the image avoided overly bright highlights and overly dark shadows.
- Fur texture and detail: Evaluated whether the fur texture was visible both in overall fluffiness and silhouette clarity, as well as in individual strands.
- Structural clarity and focus: Assessed whether important features, like the animal’s eyes or specific coat patterns, were sharp and clear without blur or pixelation when zoomed in.
Each image received an “Expert mark,” a numerical score reflecting human-driven image quality specific to our domain.
Combining traditional metrics with human judgments
With a collection of images and expert scores, we could now create a model that predicts human perception of image quality. The various calculated metrics for each image, like sharpness and texture details, were used as input features, while the expert scores acted as the target we aimed to predict.
We used a Random Forest Regressor, a reliable and easy-to-understand machine learning model, to train our system. The main steps involved were:
1. Data preprocessing
We filled in any missing data by using average values, and standardized all features with a tool called StandardScaler to keep the model’s performance balanced. Then we divided the dataset into training and testing parts to check how well the model works in general.
2. Model training and validation
Using the training data, the Random Forest model learned the complex connections between the calculated metrics and human preference scores. The test data was kept separate to ensure the model didn’t just memorize the training data, but actually understood the underlying patterns.
3. Evaluation and feature importance
After training, we checked the model’s predictions using measures like Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), and R². These showed the model was very accurate, with low MAE and RMSE and high R², meaning it closely matched expert judgments. We also looked at which features were most important, finding that metrics related to sharpness and texture consistency were more influential than those measuring overall brightness.
Comparing our approach against other methods
Versus traditional metrics alone
A single traditional metric can show us if an image is sharp, but it doesn’t tell us if that sharpness actually improves how we perceive the image’s quality. Our combined approach evaluates each metric in context. For instance, a high Tenengrad score, which indicates sharpness, is more meaningful when it’s paired with a balanced dynamic range and minimal blockiness, matching what experts prefer.
Versus CLIP and semantic models
CLIP is a strong model that connects text and images, helping to identify if an image “looks like a Spitz.” However, it doesn’t naturally pick up on subtle quality differences, like slight underexposure or slightly blurred fur. While CLIP can confirm that the image shows a Spitz, it can’t easily determine if it’s a”well-exposed, detailed Spitz image.”
Our model, trained with expert feedback on these subtle preferences, does a better job than CLIP at recognizing fine quality differences. Essentially, CLIP can say “this is a Spitz,” while our model can say “this is a high-quality, well-captured Spitz image.”
Our model rating score
| Metric | Score | Details | Similarity |
| Total score | 7 / 17 | Image Quality: Fair quality | – |
| Fur texture | 4 / 4 | Fur is clearly visible, fluffy and thick coat typical of Spitz breeds. | 30.99% |
| Fur details in highlights and shadows | 2 / 2 | Fur details are visible in both brightly lit areas and shadowed parts. | 27.11% |
| Proper exposure | 0 / 1 | Image has overexposed or underexposed areas. | 24.92% |
| Zoom test for pixel quality | 0 / 2 | At 300%-400% zoom, the image is pixelated or lacks detail. | 23.95% |
| Smooth textures | 1 / 1 | Fur appears soft and detailed. | 27.20% |
| Crisp focus on eyes | 0 / 4 | Eyes are blurry or unclear with no visible details. | 22.44% |
| Crisp focus on fur texture | 0 / 1 | Fur texture is blurry or indistinct. | – |
Our model rating score
| Metric | Score | Details | Similarity |
| Total score | 9 / 17 | Image quality: Good quality | – |
| Fur texture | 4 / 4 | Fur is clearly visible; a fluffy and thick coat typical of Spitz breeds. | 32.53% |
| Fur details in highlights and shadows | 0 / 2 | Fur details are not visible in both bright and dark areas. | 27.04% |
| Proper exposure | 0 / 1 | Image has overexposed or underexposed areas. | 24.67% |
| Zoom test for pixel quality | 1 / 2 | At 300%-400% zoom, pixel quality is maintained with clear details. | 24.07% |
| Smooth textures | 1 / 1 | Fur appears soft and detailed. | 27.72% |
CLIP evaluation
Our model rating score
| Metric | Score | Details | Similarity |
| Total score | 11 / 17 | Image Quality: Good quality | – |
| Fur texture | 4 / 4 | Fur is clearly visible; fluffy and thick coat typical of Spitz breeds. | 29.26% |
| Fur details in highlights and shadows | 0 / 2 | Fur details are not visible in both bright and dark areas. | 24.96% |
| Proper exposure | 1 / 1 | Image has proper exposure throughout with no overexposed or underexposed areas. | 23.84% |
| Zoom test for pixel quality | 2 / 2 | At 300%-400% zoom, pixel quality is maintained with clear details. | 23.61% |
| Smooth textures | 1 / 1 | Fur appears soft and detailed. | 25.45% |
| Crisp focus on eyes | 2 / 4 | Eyes are partially in focus with some details like reflections or outlines visible, but overall sharpness is lacking. | 25.22% |
| Crisp focus on fur texture | 1 / 1 | Fur texture is clear and distinct. | – |
CLIP evaluation
Versus no-reference IQA metrics (NIQE, BRISQUE)
Modern blind metrics can assess general image quality but don’t focus on specific details relevant to particular domains. Our model uses these metrics along with additional signals, learning to understand them based on what people see as “good.” This allows us to outperform standard no-reference metrics in tasks that need subjective and domain-specific evaluations.
What makes our approach unique?
Our method is not just a theoretical exercise in metric fusion. It is a practical, human-centric system. By grounding our predictions in expert feedback, we ensure relevance, adaptability, and reliability in specialized applications:
This is how we assigned feature importance and importance value in our system
Domain-specific customization
Although we’re currently focused on evaluating Spitz images, our approach can be applied to other areas too. If we need to assess microscopic images of cells, museum paintings, or scanned archival photos in the future, we can collect expert opinions for those specific areas and retrain the model. This adaptability is a big advantage compared to fixed, one-size-fits-all metrics.
Interpretability and iteration
Random Forest models allow us to inspect feature importances and refine the metric set. If some features don’t match well with human judgment, we can remove them and try new metrics or ways to process the data. Additionally, getting more expert feedback can keep enhancing the model, creating a cycle of continuous improvement.
Integrating expert feedback to train the model and derive metric weights
Instead of viewing traditional metrics as separate evaluations, we use expert ratings combined into a single “Expert_mark” for each image to create a supervised learning system. We started with a detailed table of metrics for each image, including sharpness, texture, dynamic range, and artifact detection scores. Our experts had already rated these images based on important criteria like proper exposure and fur texture quality. By matching these expert scores with the objective metrics, we trained a Random Forest Regressor to understand how each metric relates to expert preferences.
The Random Forest method is used both to predict and understand data. While training, it spots patterns and relationships, figuring out how much each metric affects the final quality score. For instance, if the Tenengrad sharpness metric often matches higher expert ratings, the model gives it more weight in its calculations. On the other hand, if a metric like raw dynamic range doesn’t align well with human judgments when other factors are present, the model automatically lessens its impact.
This process lets us combine all the metrics into one overall score. The outcome is a single predictive quality number, created by blending all the input metrics, with their importance adjusted by the learning algorithm to closely match expert evaluations. Essentially, we’ve turned a variety of measurements into one expert-informed “super metric” that can accurately predict how a trained eye would judge an image’s quality.
Practical application: Improving Spitz photographs
With our advanced evaluation tool, we can assess images of Spitz dogs and pinpoint areas that need improvement. For instance, if we have restored historic black and white Spitz portraits, our model can spot issues like poorly defined fur texture or overly bright highlights. Using this feedback, we can adjust the necessary settings to enhance the image. This process helps us achieve a result that scores well with the model, aligning with expert criteria and resulting in a better-quality image.
Conclusion & future improvements
Our image quality evaluation system, driven by machine learning and focused on human preferences, is a big step forward in connecting automated assessments with human aesthetic judgment. By using expert labels, traditional metrics, and comparing our approach to models like CLIP, we’ve developed a “super metric” that is highly accurate and relevant.
Here’s what we’re planning to do next:
- Enhance metric suites. We can experiment with learned image descriptors or advanced perceptual embeddings to capture finer details of image quality, like tiny texture details or subtle tonal changes in the fur.
- Attract larger and more diverse expert panels. By involving more experts from various backgrounds, we can develop more reliable and widely accepted criteria. Using checks for consistency among different experts ensures our model doesn’t just reflect one person’s aesthetic preferences.
- Introduce adaptive thresholding and real-time feedback. We could integrate this model into image enhancement processes, offering dynamic suggestions like “Increase exposure by 0.3 stops” or “Slightly sharpen fur areas” to help photographers, designers, or archivists create images that meet both technical and human standards.
As we keep improving and expanding this system – by enhancing metrics, involving more experts, and creating adaptive feedback loops – it promises to revolutionize how we rate and improve images. This ensures that every detail, from the sparkle in a Spitz’s eye to the softness of its fur, meets the highest quality standards as judged by humans.
Leave a Reply