scope:

This Recommendation describes subjective evaluation methods providing information about the quality of services relying on text-based chatbots, as experienced by the users of such services. Text-based chatbots addressed by the Recommendation enable a text-based natural language interaction with a human user via a text interface on a turn-by-turn basis. They possess natural language understanding (NLU), dialogue management and natural language generation (NLG) capabilities. They may provide access to customer care or allow different types of transactions to be performed.

The evaluation methods described here address different aspects of quality from a user's point of view, taking the chatbot as a black box. Important quality aspects are the overall impression, the information provided by the system, the communication with the system, the system behaviour, the user's impression of the system and the overall acceptability. The described methods are based on laboratory experiments in which participants interact with the chatbot in order to perform a pre-defined, realistic task. The participant's opinion on perceptive quality dimensions are solicited with the help of questionnaires, and examples of such questionnaires are provided. The Recommendation describes the set-up and running of interaction experiments, relevant quality dimensions perceived by the user and methodologies that will provide information about these quality dimensions. Further guidance on subjective evaluation methods related to speech-based (in contrast to text-based) chatbots are given in [ITU-T P.851].