Select Page

Object Detection API

Object Detection API Analysis Results

Descriptive Statistics

Metric Clarifai Google Vision
Raw Accuracy 81.04% 94.00%
Mean Confidence (TP) 0.9059 0.4155
Median Confidence (TP) 0.9304 0.5959
Mean Confidence (FP) 0.7844 0.0291
Median Confidence (FP) 0.9056 0.0000
False Confidence (>0.8, FP) 85.71% 0.0%

This chart compares Clarifai and Google Vision by plotting their average (mean) and median confidence scores for True Positive (TP) and False Positive (FP) predictions. To emphasise error severity, confidence scores for incorrect labels (FPs) are shown as negative values. The plot reveals that Clarifai assigns high confidence even when wrong, with its median FP confidence nearly as high as its TP confidence—indicating overconfidence. Google, in contrast, maintains low confidence in its false positives and exhibits clearer separation between correct and incorrect predictions, reflecting more cautious and calibrated behaviour.

Figure 1. Mean and Median Confidence Scores by Model

Correlation Between Confidence and Accuracy

Correlation Clarifai(r) Google(r)
Confidence vs. Accuracy (TP) -0.003 0.887
Confidence vs. Accuracy (FP) -0.381 -0.145

Clarifai’s confidence scores are not strongly predictive of label correctness, suggesting potential overconfidence or semantic drift. Google, by contrast, demonstrates a robust positive correlation between confidence and accuracy — though it rarely returns confident predictions above 0.9.

Accuracy by Confidence Threshold

Confidence Bucket Clarifai (Pos) Clarifai (Neg) Google (Pos) Google (Neg)
> 0.9 80.57% 78.21% N/A N/A
0.5–0.9 76.25% 74.64% 98.12% 33.34%
< 0.5 85.0% 97.5% 11.11% 67.55%

Google Vision rarely returns high-confidence predictions above 0.9, and when it does, the number is too small to support a statistically meaningful accuracy estimate in that confidence band. For this reason, Google’s >0.9 entries are marked as “N/A” in the grouped accuracy analysis. In contrast, most of Google’s predictions cluster between 0.5 and 0.9, where it performs with very high precision. Clarifai, on the other hand, generates labels across the full confidence spectrum including frequent predictions above 0.9, but with a substantially higher false confidence rate in that range, highlighting its tendency to over-commit semantically.

Statistical Significance

Mann-Whitney U test

Comparision: Clarifai confidence scores correct labels (TP) vs incorrect labels (FP)

p-value = 0.0249 

Stastically significant – Clarifai assigns meaningfully different confidence scores depending on whether it was correct or not. Despite its overconfidence overall, the model’s confidence is still sensitive to correctness.

Paired t- test

Comparision: Confidence scores on correct predictions (TP) by Clarifai vs Google

p-value = 7.79 × 10⁻¹⁴ 

Stastically highly significant – Clarifai is consistently and substantially more confident than Google when both models are correct – confirming the models differ not just in what they predict, but how confidently they do so.

Interpretation

Google Vision API excels in precision and calibration, but is notably conservative in its label generation. While its raw accuracy is high (94.00%), the model exhibits low average confidence even when correct (mean TP score = 0.4155), and it avoids making confident incorrect predictions altogether (false confidence = 0.0%). This results in cautious outputs that under-detect naturalistic or symbolically ambiguous features — a reflection of its training bias toward structured, commercial image domains like product photography, signage, and well-lit indoor scenes.

Clarifai, by contrast, provides broader semantic coverage and high average confidence when correct (TP score = 0.9059), but this comes with a tradeoff: 85.71% of its false positives are scored above 0.8, indicating a tendency toward overconfidence. This reflects Clarifai’s architecture, which is optimised for flexibility and generalised content tagging across diverse media, including art, lifestyle, and conceptual domains. The model returns a much wider array of labels, often including symbolically resonant features such as: trees, sky, grass, sunset, water and symbolic concepts like freedom, outdoors, connection.

These qualities are not just incidental but deeply relevant to the aims of this research, which centres on evolutionary and ecological conditions for human thriving, themes that require interpretive, not just literal, recognition.

Conclusion

For tasks that prioritise calibrated confidence, precision, and the detection of concrete, well-defined objects (e.g., in urban, commercial, or technical settings), Google Vision is the stronger model. However, for research involving naturalistic imagery, symbolic content, and conceptual meaning, as is the case in this Capstone project; Clarifai provides greater semantic richness, even if that richness comes at the cost of over-labeling and reduced precision.

The distinct training priorities of each model explain these differences: Google is tuned for reliability in structured domains, while Clarifai is trained for breadth and contextual nuance. For conceptual image analysis rooted in developmental psychology, environmental symbolism, and visual culture, Clarifai is the more aligned tool, provided its outputs are filtered or interpreted through confidence thresholds or human calibration.

Threshold Analysis for Clarifai Confidence Scores

The graph displays four key performance metrics — precision, recall, F1 score, and Youden’s J statistic — across a continuous range of Clarifai confidence thresholds (0.01 to 1.0). Each curve shows how the model’s behaviour shifts depending on where the confidence cutoff is set. Several vertical dashed lines indicate key decision points, including:<br />
Best F1 Score: where the balance between precision and recall is maximised<br />
Best Youden’s J: where the model best separates correct from incorrect predictions<br />
Minimum Cost: assuming false positives are twice as costly as false negatives<br />
Precision ≥ 90%: the first threshold where Clarifai's outputs become highly reliable<br />
This visualisation illustrates a fundamental tradeoff: lower thresholds increase recall but introduce more noise, while higher thresholds reduce false positives but risk losing meaningful labels. Notably, all three optimal metrics (F1, Youden’s J, and minimum cost) converge around threshold = 0.87, suggesting it is the most effective cutoff point for this context. At this level, Clarifai achieves over 91% precision while maintaining 95.5% recall, making it well suited for conceptual image analysis where semantic trustworthiness is prioritised over exhaustive detection.
Figure 2: Clarifai Threshold Analysis

The threshold analysis graph plots four key performance metrics—precision, recall, F1 score, and Youden’s J statistic—across a range of Clarifai confidence thresholds from 0.01 to 1.0. This visualisation reveals how the model’s behaviour shifts as the cutoff for label inclusion increases, illustrating the tradeoff between including more data and maintaining label reliability. At lower thresholds, recall is high but false positives increase, lowering precision. As the threshold rises, precision improves while recall declines.

The point at which these metrics optimally balance occurs around threshold = 0.87, where Clarifai achieves over 91% precision, 95.5% recall, and a near-maximum F1 score of 0.933. This threshold also yields the highest Youden’s J value, indicating the best separation between correct and incorrect predictions, and the lowest cost under an error-weighting model where false positives are considered twice as damaging as false negatives. Given the symbolic and affective focus of this project, where false positives risk semantic distortion, this threshold was selected to ensure trustworthy label output without significant loss of meaningful content.

←→