Next IT, the provider of conversational AI for the Global 5000, announced the findings of a computation and language study conducted by its own Research and Innovation Team. The study, “An Annotated Corpus of Relational Strategies in Customer Service,” is the first to investigate the effect of relational language on Intelligent Virtual Agents (IVAs).
To conduct the study, human-computer data from three live customer service IVAs in the domains of travel and telecommunications were collected, and researchers marked all text that was deemed unnecessary to the determination of user intention. The study then demonstrated that removal of relational language from task-based user inputs has a positive effect on IVA understanding by both an increase in confidence and improvement in responses.
Using data from large-scale, real-world IVA’s, the study highlights the challenges facing developers of chatbots and IVAs as people increasingly speak to machines in the same manner they speak to human agents. In addition to the findings of the study, the researchers are making the data set that was used available at no charge to the broader AI research community. Notably, this is the first publicly available commercial customer service corpus with annotated relational segments.
“We’re excited to share this research with the broader community,” said Ian Beaver, Lead Research Engineer at Next IT. “But perhaps more importantly, we’re thrilled to share the raw data with AI researchers. Conversation is, without a doubt, the next great human-machine interface. This research helps the entire industry prepare for one of its biggest challenges: determining intent as people evolve the way they talk to machines. We’re proud to help solve that problem, industry wide.”
To complete the study, researchers gathered 6,000 conversations between humans and AI-powered customer service agents. After merging the selections of multiple reviewers to create highlighted texts, a second round of annotation was conducted to determine the classes of language present in the highlighted sections, such as the presence of Greetings, Backstory, Justification, Gratitude, Rants, or Emotions.
The resulting corpus of annotated data is a valuable resource for improving the quality and relational abilities of IVAs. As well as discussing the corpus itself, researchers compare the usage of such language in human-human interactions on TripAdvisor forums to provide greater context.
“With one of the richest annotated data libraries around, our data represents how humans communicate with enterprise IVAs as well as how people relate to those businesses,” said Jen Snell, Next IT VP Marketing and Brand Strategy. “While we aptly handle thousands of user and business intents today, it’s important that we continue to lead the advancement of human-machine interactions as we are seeing those relationships evolve.”
To download the study and access the data, please visit: https://arxiv.org/abs/1708.05449