“Benchmarking an AI model is not simply about measuring its final performance. It is about understanding how it behaves across different environments, scenarios and operational conditions.”

Within STORE, LIST leads WP9 – AI Benchmarking, developing a comprehensive framework to evaluate AI models beyond standard performance metrics. 🔍The work focuses on context-aware evaluation, taking into account factors such as scene complexity, distance, lighting and weather conditions. WP9 also explores adversarial benchmarking, challenging models with difficult scenarios such as fog, snow, storms or low-light environments. 🌧️

The objective is to establish a 360° benchmarking approach, combining model performance, hardware capabilities, latency, memory, energy consumption and human evaluation to support the development of robust and trustworthy AI for defence applications. 🤖🛡️