WebDialogueBrowser

From Olympus
Jump to: navigation, search

by Antoine Raux (antoine@cs.cmu.edu)


Contents

Introduction

WebDialogueBrowser is a set of tools that generate html representations of dialogues with Olympus-based systems. These html logs can be used for browsing past dialogues or to annotate them with various turn- and dialogue-level tags. The generated logs are primarily aimed at being used through a web server, although local browsing (directly from a local hard drive) is also possible. Note that you do need a web server to perform annotation though, since the annotation tool is CGI-based. The tools are heavily relying on the Olympus log directory and file architecture and, therefore, do not lend themselves to logs generated by other systems (although we welcome any effort to generate the logs from different architectures). This manual documents three level of use of WebDialogueBrowser. First, assuming you already have the server set up and the html logs generated, we explain how to navigate and annotate the logs using any standard web browser. This first section is what you need if you are an annotator starting to work on labeling dialogues. Second, we'll see how you can set up a new set of logs on an existing server. This is what you need if you have just collected dialogues using a new system and want to browse and annotate them. Finally, we'll see how you can set up a new annotation server (based on the Apache http server). This is what you need if you're in charge of a site with your own server.

How to use the HTML logs

Since the dialogues are simple html pages, there is nothing really new about how to browse them. Just use hyperlinks and form buttons as you usually do. Specifically, there are two types of pages: daily indexes and dialogues. A daily index gives you the list of all the dialogues recorded on a particular day. Session numbers (at the left of the page) are linked to the corresponding dialogue pages. Other information follows each session number (from left to right): the time when the dialogue started, whether the user successfully achieved their goal with the dialogue or not (this can be manually annotated or estimated depending on the system), the duration of the dialogue (in ms) and the number of turns of the dialogue. Once you click on a dialogue number, you're taken to the corresponding dialogue page. This page contains a header with the id of the dialogue, which usually contains the name of the system, the date and the session number (e.g. "LetsGoPublic-20050304-015-gloss"), and some general information about the dialogue (start time, duration, etc). There are also links to raw log files (the Galaxy hub log, the Helios log, the RavenClaw log...) and to the recording of the full conversation (if available). Below are the dialogue level tags (usually with background of various colors, indicating the type of type). Next is the dialogue itself, with the system turns represented with a gray background and the user turns with a white background. Turn numbers are given on the left-most column, with shaded background indicating non-understandings (when the system failed to get anything from the user) and misunderstandings (when the system gets the wrong information). For the user turns, you can see the ASR result (usuall all caps), the transcription (if available) and the parse result (based on the ASR). There is also a link to the audio of this turn (labeled "speech"). To the right of each turn are the corresponding turn-level tags. By clicking on the "Change annotations" button at the top of the dialogue page, you switch to the annotation page. The layout of this page is the same as the dialogue page, the only difference being that dialogue- and turn-level labels are replaced by checkboxes, radiobuttons, and text boxes. Just above the dialogue itself is a series of checkboxes that let you select the aspect(s) of dialogue you wish to annotate. You need to allow cookies in your browser settings so that the browser remembers this information between dialogues (otherwise, you'll have to select the aspects every time). Once you're satisfied with your annotation of the dialogue, click on "Submit" and your labels will be stored on the server. You will then be redirected to the dialogue page, with the tags reflecting your new annotation.

How to define your own annotation tags

WebDialogueBrowser is mainly used as an annotation tool. However, there are many aspects that one can annotate in a dialogue: dialogue acts, understanding errors, speaking style, task completion, etc. In order to let annotators define their own set of tags, WebDialogueBrowser uses a markup language called DLTD (Dialogue Label Tag Definition). For each set of logs, WebDialogueBrowser uses a specific tag definition file usually called label_tags.xml. You can modify this file to allow the labeling and visualization of your own tags. Here is an example of such a file:

 <?xml version="1.0" encoding="ISO-8859-1"?>
 <DLTD>
   <GROUP NAME="task" LABEL="Task" EDITABLE="YES">
     <CHECKBOX SCOPE="DIALOGUE" NAME="misunderstood_scenario" EXPLANATION="Misunderstood scenario" LABEL="MIS-SCEN"/>
     <CHECKBOX SCOPE="DIALOGUE" NAME="task_fail" EXPLANATION="Dialogue failed to achieve goal" LABEL="FAIL" COLOR="#FF7700"/>
     <TEXTBOX SCOPE="DIALOGUE" NAME="task_fail_mode" EXPLANATION="Failure mode" LABEL="F-MODE"/>
 
   </GROUP>
   <GROUP NAME="user_response" LABEL="User resp." EDITABLE="YES">
     <CHECKBOX SCOPE="DIALOGUE" NAME="user_response_labeled" EXPLANATION="User response labeled" LABEL="U-RESP" COLOR="#D1FED0"/>
     <RADIOBUTTON SCOPE="TURN" NAME="user_response_label" VALUES="REP;RPH;CTR;CNG;OTH;NEW" COLOR="#D1FED0"/>
   </GROUP>
 </DLTD>

Let's look at the different XML tags in this file. The first line simply indicates this is based on XML 1.0. The whole file has to be included in a <DLTD> tag. Within this DLTD tag are a set of <GROUP> tags, each of which defines a set of tags that are related (usually because they concern the same aspect of dialogue). In the example above, the first group, called "task", contains tags describing whether the user successfully achieved their goal with the system. The second group ("user_response") concerns the type of responses given by the user to the system. <GROUP> has the following properties:

  • LABEL contains a short, human-readable description of what the group is about. This is used as the label of the checkboxes that turn on and off the editting of this group in the annotation page (see section 2).
  • EDITABLE indicates whether this particular group can be editted by a human annotator or not. In general, you want this property to be YES. If not, the group will not appear in the annotation page.

Within each group is a set of tags. There are three types of tags, depending on the way a human annotator would label them:

  • CHECKBOX are tags that have boolean value, i.e. YES/NO. For example, "task_fail" is a CHECKBOX that indicates if the user achieved their goal in the dialogue. "user_response_labled" indicates whether user responses have been labeled by this annotator for this dialogue.
  • RADIOBUTTON are tags that take one of a small set of values (given in the VALUES property, separated by semi-columns). For example "user_response_label" can be one of REP (user repeats what they previously said), RPH (user rephrases what they previously said), CTR (user contradicts the system), CNG (user changes what they are trying to convey), OTH (other), and NEW (user introduces starts a new topic).
  • TEXTBOX are tags that can contain any value. For example "task_fail_mode" contains a description of the cause of dialogue failure, which, in this case, the annotator can describe freely.

All types of tags share the following properties:

  • SCOPE indicates whether this tag concerns the full dialogue (DIALOGUE) or individual turns (TURN). For now these are the only two scopes supported.
  • NAME indicates the internal name of this tag, which is used in the ASCII labels file associated with each dialogue.
  • EXPLANATION contains a human-readable description of the meaning of the tag. It is displayed in the status bar of the browser when the annotator hovers over the tag.
  • LABEL is a shorter description which is displayed in the dialogue page when a CHECKBOX is checked (it does not concern other tag types).


How to generate HTML logs for local browsing

In some cases, you only want to generate html logs to make it easier to browse your data, without needing to label it. In this case, assuming you have direct access to the machine where the dialogues are stored, you can simply generate static html files containing the daily indexes and dialogue pages. You will then be able to view them without using a web server, but you won't be able to make any annotations this way. The script that generates the HTML pages is called process_hubs.pl. Its command line syntax is as follows:

 perl process_hubs.pl [options]
 Options:
    -l <local_path>             The path to your data folder
    -r <root_folder>            The path from the data folder to the root of the 
                                particular set of logs you want to generate HTML pages for
    -t <labels_definition_file> The XML annotation tag definition file
    -b <begin_date>             The date from which you want to start generating the logs
    -e <end_date>               The date until which you want to generate the logs
    -S <system_name>            The name of your system

For example, assume you have a set of logs from a system called WorldCupInfo, located under F:\Research\Dialogue\data\WorldCupInfo. To generate the corresponding HTML pages, you would execute the following command line:

 perl process_hubs.pl -l F:/Research/Dialogue/data 
                      -r WorldCupInfo 
                      -t F:/Research/Dialogue/data/WorldCupInfo/label_tags.xml 
                      -b 20060613 
                      -e 20060709
                      -S WorldCupInfo  


How to add a new set of logs to an existing server

Prerequisites

  1. An Apache Web Sever and access to its configuration
  2. Perl, with the CGI-Enurl package installed

Instructions

  1. Get access to the server machine. It should already be running Apache as a service and have a data folder where the logs of one or more dialogue systems are stored.
  2. Create a folder for your new system in the data folder
  3. If necessary, modify Apache's httpd.conf so that the path is mapped to the correct URL. This is the case on accent.speech.cs.cmu.edu, if you add a subfolder to G:\data. You need to create an Alias in httpd.conf as follows:
    Alias /data/MySystem "G:\data\MySystem"
    because the main URL /data/ is mapped to D:\htdocs\data (so without Alias, accent.speech.cs.cmu.edu/data/MySystem maps to D:\htdocs\data\MySystem).
  4. Copy .htaccess, label_tags.xml, and MySystem.stats from another system's folder.
  5. Modify .htaccess to provide access to the right people to your system's folder. NB: even if you don't care about privacy, user authentication is required for labeling purposes (so that each labeler can be identified).
  6. If necessary, create new users to the passwd file of your Apache distribution (probably located in C:\Program Files\Apache Software Foundation\Apache2.2\passwd) using htpasswd (in the bin folders of Apache). You can also create user groups (e.g. MySystemTeam), by adding to the exisiting usergroups file (also in passwd).
  7. Copy the dialogue logs to the server. This can be done with the following script:
 perl copy_olympus_logs.pl [options]
 Options:
       [-b <begin_date> -e <end_date>]
       -source <source_directory>
       -target <subpath_to_target_directory>
       -target_root <root_folder_of_target_directory>
       -log_url <root_url_for_system_logs>
       -system_name <system_name>
       

The begin_date and end_date indicate the period you want to collect logs/stats for. If you omit these two options, the script will run on yesterday's date (this is for use as an automatically scheduled daily statistics, see NOTE below). Source is the path to your source data (usually through a mapped network drive). The target directory is split in two, the root and the subpath. The subpath has to be the same as the subpath of the URL of the logs (this is the same thing as for process_hubs.pl - see above).

For example, if you want your logs to be stored in a folder called G:\data\Conquest and the corresponding URL is defined (in the Apache httpd.conf) as accent.speech.cs.cmu.edu/data/Conquest, the target_root would be "G:", the system_url "accent.speech.cs.cmu.edu", and the target "data/Conquest".

system_name should be a human-friendly form of the name of your system, for report purposes.

The following set of non-required options has the script send an email report on the given time period (number of dialogs, average length...) to a fixed list of recipients.

       [-emails <email_list_file> 
        -system_url <url_for_system_report_page>
        -system_email <email_contact_for_system_admin>]        

The email_list_file is a text file with one email address per line. The system_url is the url of the main page of your system, usually with longer term statistics as well (see below), which is included in the email.

The following option gives the path of an html page that contains global statistics for this system (e.g. total number of calls so far). The page should have specific sections so that the script knows where to put the numbers it computes. Look at D:\htdocs\LetsGoPublic\index.html for an example.

       [-report_html <html_file_containing_overall_statistics_info>]

This is a folder where the scripts will log the stdout of all the subscripts it runs (auto_label.pl, auto_label_nonunderstandings.pl, process_hub.pl, produce_olympus_report.pl)

       [-log <log_directory>]

NOTE: if your system is constantly running and getting calls/sessions, you might want to schedule this script to run everyday automatically.

==How to set up a new server==

Personal tools