N-best rescoring again

[Updated July 9]

I've just checked in new code that makes it possible to create training material for doing N-best rescoring on speech translation applications - the functionality is basically the same as what we already had for dialogue applications, but there were a number of details that had to be fixed. It seems that the potential for improving performance using N-best rescoring varies considerably between apps. So far, we've looked at the following cases:

• Calendar: can already almost halve error rate using rescoring, more should be possible.
• Ford app: almost no potential for improvement.
• Paideia app: considerable potential for improvement (don't currently have figures)
• English MedSLT: maximum possible improvement looks like about 10% relative.
• French MedSLT: maximum possible improvement about 15-20% relative.
• Japanese MedSLT: almost no potential for improvement.

The variation in behavior between the different apps is quite surprising. In particular, I don't yet have a good explanation for why the MedSLT languages should be so different.

"Paraphrase corpora" for estimating semantic error rates

I've implemented a first cut at the "paraphrase corpus" idea that I suggested in yesterday's post. So far, it only works for speech translation, but it's rather nice to see that we can now measure the effect that N-best rescoring has on semantic error rate in a way that's both much quicker and much more objective than what we were doing previously. On the whole of the Eng corpus (the only one I've tried so far), semantic error rate on this metric is reduced by N-best rescoring by about 4% absolute, or 8% relative.

My next task here is to extend the method to dialogue processing - this should be easy, I think. We will then be able to do dialogue N-best rescoring experiments using out-of-coverage as well as in-coverage data, which should open up several new possibilities.

Better ways to estimate semantic error rate

I've just added some code to automatically estimate semantic error rate for translation applications. It does more or less the same thing as the code we've had for a while in dialogue apps, and counts an example from a speech corpus as semantically correct if it produces the same interlingua as the transcription would have done.

Unfortunately, the problem with this definition is that it doesn't work for utterances that are in domain, but out of grammar coverage. For example, I was just looking though the results for the English MedSLT corpus. In one example, the transcription is "does the pain ache", which is out of grammar coverage. The first hypothesis which produces well-formed interlingua is "does the pain feel aching", which is a good paraphrase and is selected. So this should really be counted as semantically correct, but isn't.

I think we can address the problem by allowing the developer to declare a file of paraphrases, and say that the example is semantically correct if it gives the same result as either the actual transcription or one of its paraphrases. Then if the developer adds in-coverage paraphrases where they exist, things will work correctly. This should be easy to implement. Probably we want a warning if a paraphrase in fact is also determined to be out of coverage.

This paraphrase functionality should also be useful for the N-best rescoring work that Maria and I have been doing for dialogue apps. We have the same problem there - we want to be able to experiment with out of coverage examples, but currently get no figures.

Progress on N-best rescoring

Maria Georgescul and I have been doing some work over the last few days on N-best rescoring, using the Calendar application as a test-bed. The basic division of labor was for me to define features and transform N-best hypothesis lists into lists of feature vectors, while Maria fed these into an SVM-based learner to perform the actual rescoring. We did the experiments using a set of 459 recorded utterances. Rescoring now reduces semantic error rate from 19% to 11%, and WER from 11% to 10%.

I defined the features by looking at examples of N-best lists, and finding common examples of things which I felt intuitively should be penalized. The current set of features is as follows:

rank: Place in the N-best list

no_dialogue_move: Hypothesis produces no dialogue move

underconstrained_query: Query with no contentful constraints

non_indefinite_existential: Existentials with non-indefinite arg, e.g. "is there the meeting next week"

non_show_imperative: Imperatives where the main verb isn't "show" or something similar

indefinite_meeting_and_meeting_referent: combination of indefinite mention of meeting + available meeting referent

Printing N-best feature info at top level

If you're in dialogue mode, and have N-best preferences defined, you now get them printed out at top level. This is useful for debugging feature definitions. Here's an example from the Calendar application:

>> what was the last meeting

      Old state: [lf=[[whq,form(past,[[be,term(the_last,meeting,[]),[loc,where]]])]], 
                  referents=[record(meeting,meeting_10),attribute(meeting,meeting_10,where)]]
             LF: [[whq,form(past,[[be,term(the_last,meeting,[]),term(what,null,[])]])]]
    Resolved LF: [[whq,form(past,[[be,term(the_last,meeting,[]),term(what,null,[])]])]]
     Resolution: [trivial]
  Dialogue move: [tense_information=referent(past), utterance_type=whq, 
                  aggregate(last_n_meetings(1),[])]
  Resolved move: [tense_information=interval(datime(1980,0,0,0,0,0),datime(2008,5,28,18,27,24)),
                  utterance_type=whq, aggregate(last_n_meetings(1),[])]
     Paraphrase: list meetings in past the last meeting
Abstract action: say(referent_list([record(meeting,meeting_10)]))
Concrete action: tts(meeting at pierrette 's room on november 25)
      New state: [lf=[[whq,form(past,[[be,term(the_last,meeting,[]),term(what,null,[])]])]], 
                  referents=[attribute(meeting,meeting_10,where),record(meeting,meeting_10)]]

N-BEST FEATURES AND SCORES:

rank                                    -1.00 * 0.00 = 0.00
no_dialogue_move                        -50.00 * 0.00 = 0.00
underconstrained_query                  -10.00 * 0.00 = 0.00
inconsistent_tense                      -10.00 * 0.00 = 0.00
non_indefinite_existential              -10.00 * 0.00 = 0.00
non_show_imperative                     -50.00 * 0.00 = 0.00
definite_meeting_and_meeting_referent   3.00 * 0.00 = 0.00

Total score: 0.00

Dialogue processing time: 0.00 seconds

>> when did that meeting start

      Old state: [lf=[[whq,form(past,[[be,term(the_last,meeting,[]),term(what,null,[])]])]], 
                  referents=[attribute(meeting,meeting_10,where),record(meeting,meeting_10)]]
             LF: [[whq,form(past,[[start,term(that,meeting,[])],[time,when]])]]
    Resolved LF: [[whq,form(past,[[start,term(that,meeting,[])],[time,when]])]]
     Resolution: [trivial]
  Dialogue move: [query_object=start_time, referent_from_context=meeting, 
                  tense_information=referent(past), utterance_type=whq]
  Resolved move: [meeting=meeting_10, query_object=start_time, referent_from_context=meeting, 
                  tense_information=interval(datime(1980,0,0,0,0,0),datime(2008,5,28,18,27,35)),
                  utterance_type=whq]
     Paraphrase: start time for that meeting in past
Abstract action: say(referent_list([attribute(meeting,meeting_10,start_time)]))
Concrete action: tts(10 00 on november 25)
      New state: [lf=[[whq,form(past,[[start,term(that,meeting,[])],[time,when]])]], 
                  (referents
                   = 
                   [attribute(meeting,meeting_10,where), record(meeting,meeting_10), 
                    attribute(meeting,meeting_10,start_time)])]

N-BEST FEATURES AND SCORES:

rank                                    -1.00 * 0.00 = 0.00
no_dialogue_move                        -50.00 * 0.00 = 0.00
underconstrained_query                  -10.00 * 0.00 = 0.00
inconsistent_tense                      -10.00 * 0.00 = 0.00
non_indefinite_existential              -10.00 * 0.00 = 0.00
non_show_imperative                     -50.00 * 0.00 = 0.00
definite_meeting_and_meeting_referent   3.00 * 1.00 = 3.00

Total score: 3.00

Dialogue processing time: 0.01 seconds