scope:

This report will cover the following aspects of evaluation:

Survey of existing evaluation metrics and standardized approaches and protocols, including automatic, semiautomatic, and manual approaches, for evaluation of the following language processing systems that are end-user-facing, as well as developer-facing.

a) Text processing: MT, information retrieval, information extraction, dialogue and conversation systems, question answering, and summarization.

b) Multimodal language processing.

Survey of existing evaluation testbenches across the 22 scheduled languages of India that will enable one to test language systems in an automated fashion.

Identify gaps in both #1) and #2) above.

The following aspects, though within the scope of evaluation in general, were not investigated:

Specifics of evaluation of speech processing, basic text processing (morphology, parsing, parts of speech (POS), sentiment, etc.), and script/input method evaluation. These were covered by the respective subcommittees. Here, on the other hand, the authors report the more general aspects of the evaluation of speech and text processing systems that cut across specific tasks and applications.

Evaluation of quality of datasets-such as noise in a corpus, interannotator agreement metrics, etc. These could be covered by respective subcommittees while documenting existing resources.

Reporting of numbers from evaluation drives and studies.

The authors primarily cover the evaluation of functional aspects (i.e., accuracy type metrics) of the systems. Although the important nonfunctional aspects (speed, user interface aspects, developer-friendliness, scalability, etc.) are documented, they may not be discussed in depth. Certain aspects (like fairness and transparency) might be discussed in depth, if the subcommittee feels that these are important aspects of language systems for which standards are required.