EMNLP 2021 Workshop | 首届对话系统评测Workshop征稿启事

 

Call for Papers - The First Workshop on Evaluations and Assessments of Neural Conversation Systems (EANCS) co-located with EMNLP 2021, will be held virtually on Nov 11, 2021.

Much progress has been made recently to improve conversation systems and chatbots using neural-based deep models. While the architectures of conversation models evolve quickly, evaluation techniques over the years remain unchanged. As a matter of fact, evaluating and assessing conversation models has been a decades long challenge. One part of the difficulty comes from the fact that human dialogues exist in a variety of forms. Existing approaches that compare generated conversations from a neural model with ground truth can easily incur large biases due to the lack of diversity. Another part of the difficulty comes from the interactive nature of conversations, which often requires an agent (usually a real person), to conduct the assessments. Human evaluations, on the other hand, can introduce a large amount of variance and are often impractical on a large scale. A third dimension of evaluation has to do with the fairness and reliability of the models, which has become an increasingly important issue for commercial use of neural based systems. Finally, transferability, transparency and ethical issues in evaluations are among some of the other important topics to explore.

Limitations of evaluation and assessment techniques can not only cause difficulties in comparing conversation models but also delay the developments of advanced learning algorithms. Recognizing the importance and the challenges of this topic, we put together this workshop to explore the domain of dialogue evaluation. In particular, we are interested in contributions in the following areas:

• settings, benchmarks, and metrics for automatic evaluation

• simulated agents for evaluation

• human evaluation metrics

• transferability of evaluation methods

• fairness of conversation models

• reliability of conversation models

• nterpretability and transparency of evaluations

• datasets for evaluation

• ethical issues in evaluations

For questions regarding the workshop submissions, please feel free to contact the organizing committee at [e-mail].

Shared Task

We release a new benchmark dataset that aims at facilitating the research of evaluating neural conversational models. In this dataset, a few deep neural models are assessed by human agents, each with a numerical score indicating the performance of the models. Dialogue evaluation methods are assessed by comparing the predicted scores to the actual human scores. Please refer to our shared task page (https://sites.google.com/view/eancs/shared-task) for details.

 

Important Dates

EANCS 2021 will adopt dual submission deadlines. Accepted submissions before the early deadline (before Sept 5, 2021) will be eligible (pending the approval of the authors) to be included in the ACL anthology. Accepted submissions after the early deadline will be included in our workshop arxiv. All the deadlines are valid anywhere around the world.

 

• Early Submission Deadline: Sun, Sept 5, 2021

• Early Submission Acceptance Notification: Fri, Sept 17, 2021

• Early Submission Camera Ready Deadline: Fri, Oct 1, 2021

• Regular Submission Deadline: Fri, Oct, 1, 2021

• Regular Submission Acceptance Notification: Fri, Oct 15, 2021

• Regular Submission Camera Ready Deadline: Fri, Oct 31, 2021

Submission Site

Submissions should be delivered to our workshop portal through SoftConf using the link below. Contact the organizers ([e-mail]) should you run into any issues with submitting.

https://www.softconf.com/emnlp2021/EANCS/

 

Author Guidelines

We welcome short papers as well as extended abstracts. All submissions should follow EMNLP 2021 templates (https://2021.emnlp.org/call-for-papers/style-and-formatting). Submissions should be either short papers (between 4 and 6 pages, excluding references) or extended abstracts (2 pages, excluding references). We welcome submissions that are work in progress, as well as submissions that are currently under review at other venues.

 

Commitment to Diversity

This workshop is committed to providing equal access and participation without regard to race, color, religion, gender identity, among other things. The organizing committee shares a strong commitment to promote diversity.

 

Useful Links

• Workshop Homepage: https://sites.google.com/view/eancs/home

• Call for Papers: https://sites.google.com/view/eancs/call-for-papers

• Submission Site: https://www.softconf.com/emnlp2021/EANCS/

• Shared Task: https://sites.google.com/view/eancs/shared-task

• Contact Workshop Organizers: [e-mail] 

Keynote Speakers

Dilek Hakkani-Tur (Amazon)

Jason Weston (Facebook)

Larry Heck (Samsung America)

Maxine Eskenazi (CMU)

Yejin Choi (University of Washington)

Percy Liang (Stanford)

 

Organizers

Program Chairs:

Wei Wei (Google Cloud)

Bo Dai (Google Brain)

Tuo Zhao (Georgia Institute of Technology)

Lihong Li (Amazon)

Diyi Yang (Georgia Institute of Technology)

Yun-Nung Chen (National Taiwan University)

Y-Lan Boureau (Facebook AI Research)

Asli Celikyilmaz (Microsoft Research)

Alborz Geramifard (Facebook AI)

 

Website Chair:

Aman Ahuja (Virginia Tech)

 

Shared Task Chair:

Haoming Jiang (Amazon)

 

·