Search

This paper presents a LADAR recognition challenge problem. There are two unique components of this challenge problem: the focus on condence estimates and the explicit model of the recognition output as a function of operating conditions (OCs). To promote the development of exploitation algorithms that map OCs onto condence estimates, a set of synthetic data has been generated that explicitly samples specic OC dimensions: resolution, noise, aspect, obscuration, and library. Submitted algorithms will be evaluated based on their ability to correctly estimate the condence based on OC knowledge. Tools will be provided to generate performance metrics on data sets. Participants will submit their algorithms for evaluation on a sequestered set of data. The resulting performance metrics will be made available online so participants can evaluate their algorithms relative to their peers.

The target classification algorithm community is making a special effort to explicitly treat operating conditions (OCs) in classifier assessments and performance modeling. This is necessary because humans do not intuitively appreciate what makes classification difficult for computers-it just seems so easy to us. In analyzing OCs, some OCs are more direct or primitive while others are more abstract or integrating. These more abstract or "Derived OCs" provide an intermediate step between direct OCs and classifier performance. Similar to the target, sensor, environment partition of OCs, the AFRL COMPASE Center introduces the "Mossing 3" partition of derived OCs into "Clarity," "Uniqueness," and "Conformity." Clarity is primarily concerned with the relevant information content available in the sensor data. Uniqueness is about the inherent separability between the types of objects to be classified (i.e., the library) and between all those types and objects not known to the classifier. Conformity is about the relationship between the OCs of the test instances and the OCs represented in the library types or training data. Furthermore, by analyzing derived OCs from multiple perspectives, informative subpartitions of the Mossing 3 are created. Clarity measures are well developed, particularly as image quality metrics. The other partitions are less well developed, but relevant work exists and is brought into context. While derived OCs and the Mossing 3 partition are not a complete solution to performance modeling, they help bring in powerful existing technologies and should enrich and facilitate dialogue on classifier performance theory and modeling.

The modern battlespace is populated with a variety of sensors and sensing modalities. The design and tasking of a given sensor is therefore increasingly dependent on the performance of other sensors in the mix. The volume of sensor data is also forcing an increased reliance on sensor data exploitation and content analysis algorithms (e.g., detecting, labeling, and tracking objects). Effective development and use of interconnected and algorithmic (i.e., limited human role) sensing processes depends on sensor performance models (e.g., for offline optimization over design and employment options and for online sensor management and data fusion). Such models exist in varying forms and fidelities. This paper develops a framework for defining model roles and describes an assessment process for quantifying fidelity and related properties of models. A key element of the framework is the explicit treatment of the Operating Conditions (OCs - i.e., target, environment and sensor properties that affect exploitation performance) that are available for model development, testing data, and model users. The assessment methodology is a comparison of model and reference performance, but is made non-trivial by reference limitations (availability for OC distributions of interest) and differences in reference and model OC representations. A software design of the assessment process is also described. Future papers will report assessment results for specific models.