Thursday 24 July 2008

Documentation for NUANCE_PARSER command

I've added some basic documentation for the new NUANCE_PARSER command. Here's what you get when you access it using DOC:

>> DOC NUANCE_PARSER
(Print documentation for command or config file entry)

NUANCE_PARSER
[Brief doc: Start new Nuance nl-tool process and use it as parser]

Start an nl-tool process, and use it to do parsing. Any old nl-tool processes
are first killed. The current config file needs to include either a
dialogue_rec_params declaration (for dialogue apps) or a translation_rec_params
declaration (for speech translation apps); the declaration must
contain definitions for 'package' and 'grammar'. The following is a
typical example of a suitable declaration:

regulus_config(dialogue_rec_params,
[package=calendar_runtime(recogniser), grammar='.MAIN',
'rec.Pruning=1600', 'rec.DoNBest=TRUE', 'rec.NumNBest=6']).

Notes:

- After NUANCE_PARSER is successfully invoked, nl-tool is used for
ALL parsing, including batch processing with commands like TRANSLATE_CORPUS
and Prolog calls to parse_with_current_parser/6.
- The Nuance parser only returns logical forms, not parse trees.

Thursday 17 July 2008

More improvements to Nuance documentation

You can now access documentation about config file entries from the Regulus top level. The new command HELP_CONFIG looks for information about config file entries, and DOC shows documentation for both commands and config entries. Here's an example:

>> HELP_CONFIG nuance
(Print help for config file entries whose name or description match the string)

7 config file entries matching "nuance":

ebl_nuance_grammar
nuance_compile_params
nuance_grammar
nuance_grammar_for_compilation
nuance_grammar_for_pcfg_training
nuance_language_pack
nuance_recognition_package

>> DOC nuance_grammar
(Print documentation for command or config file entry)

nuance_grammar
Points to the Nuance GSL grammar produced by the NUANCE command.

Wednesday 16 July 2008

Improvements to Regulus documentation

I have been doing some work on and off over the last few weeks to try and improve the Regulus documentation. It's one of those "important non-urgent" tasks that is very hard to schedule, because you always feel you have something that should take higher priority, but I do finally seem to have made some concrete progress.

There are three parts to the work, which are meant to be closely interlinked. First, I have created a directory under Regulus/doc called CommandDoc, which is supposed to contain one short file for each command and type of config file entry. I've so far populated it with the information in RegulusDoc.html, which certainly confirmed that RegulusDoc is badly out of date... I'm afraid half the files are currently empty.

Second, I have added a new top-level Regulus command called DOC. If you type DOC followed by the name of a command, you get the CommandDoc file printed out in a reasonably readable way. For example:

>> DOC LOAD_DIALOGUE
(Print documentation for command)

LOAD_DIALOGUE
[Brief doc: Load dialogue-related files]
Compile the files defined by the dialogue_files config file entry.

>> DOC EBL_NUANCE
(Print documentation for command)

EBL_NUANCE
[Brief doc: Compile current specialised Regulus grammar into Nuance GSL form]
Compile current specialised Regulus grammar into Nuance GSL form. Same
as the NUANCE command, but for the specialised grammar. The input is
the file created by the EBL_POSTPROCESS command; the output Nuance GSL
grammar is placed in the file defined by the ebl_nuance_grammar config
file entry.

>> DOC TRANSLATE_CORPUS
(Print documentation for command)

TRANSLATE_CORPUS
[Brief doc: Process text translation corpus]

Process the default text mode translation corpus, defined by the
translation_corpus config file entry. The output file, defined by
the translation_corpus_results config file entry, contains
question marks for translations that have not yet been judged. If
these are replaced by valid judgements, currently 'good', 'ok' or
'bad', the new judgements can be incorporated into the translation
judgements file (defined by the translation_corpus_judgements
config file entry) using the command
UPDATE_TRANSLATION_JUDGEMENTS.

TRANSLATE_CORPUS <Arg>
[Brief doc: Process text translation corpus with specified ID]

Parameterised version of TRANSLATE_CORPUS. Process the text mode
translation corpus with ID <Arg>, defined by the
parameterised config file entry
translation_corpus(<Arg>). The output file, defined
by the parameterised config file entry
translation_corpus_results(<Arg>), contains
question marks for translations that have not yet been judged. If
these are replaced by valid judgements, currently 'good', 'ok' or
'bad', the new judgements can be incorporated into the translation
judgements file (defined by the translation_corpus_judgements
config file entry) using the parameterised command
UPDATE_TRANSLATION_JUDGEMENTS <Arg>.


Third, and last, I have also arrange things so that the doc files are automatically included in the new Cookbook. This is still just a skeleton, but my plan is to start by completing the command and config-file section, so that the book will immediately be useful for something, and then work outwards from there. The PDF version is checked in as Regulus/doc/Cookbook/draft_cookbook.pdf.

Sunday 13 July 2008

Substitutable help classes

Following a discussion with Nikos last month, I've added a new feature to the help system, so that help examples can be modified to be closer to the recognition result. Recall that the help system assumes that the designer will have declared a set of help classes; each class C defines a set of phrases P(C). When choosing a help match, both the recognition result and the help examples are backed off so that, for each class C, phrases in P(C) are replaced by C.

The new functionality I've just added makes it possible to declare some help classes as "substitutable". Suppose that class C is defined as substitutable, that the phrase P1 in the recognition result is backed off to C, and that the phrase P2 in a matched help example H is also backed off to C. In this case, H will not be presented in its original form, but with P2 substituted by P1. Evidently, not all help classes can be defined as substitutable, since it's essential that all the words in a substitutable class have exactly the same syntactic properties.

There are however some important classes which can in general be made substitutable, in particular (at least in English) names for specific types of individual, plural numbers, days of the week and months of the year. I've tested the new functionality on the Calendar app, and it does indeed seem to give considerably more useful responses. Here's an example. Without substitutable classes, the sentence "what meetings has nikos been to" gets the help responses

#1 : "what meetings has pierrette attended in geneva"
"what meeting_noun has person_name attend_verb preposition loc_name" (backed off)
#2 : "which meetings has elisabeth attended"
"which meeting_noun has person_name attend_verb" (backed off)
#3 : "what meetings is pierrette going to attend in geneva"
"what meeting_noun is person_name going to attend_verb preposition loc_name" (backed off)
#4 : "what meetings is pierrette going to attend"
"what meeting_noun is person_name going to attend_verb" (backed off)
#5 : "what meetings have there been in geneva"
"what meeting_noun have there been preposition loc_name" (backed off)

With substitutable classes (person_name is one of them), the response is

#1 : "what meetings has nikos attended in geneva"
"what meeting_noun has person_name attend_verb preposition loc_name" (backed off)
#2 : "which meetings has nikos attended"
"which meeting_noun has person_name attend_verb" (backed off)
#3 : "what meetings is nikos going to attend in geneva"
"what meeting_noun is person_name going to attend_verb preposition loc_name" (backed off)
#4 : "what meetings is nikos going to attend"
"what meeting_noun is person_name going to attend_verb" (backed off)
#5 : "what meetings will nikos attend in geneva"
"what meeting_noun will person_name attend_verb preposition loc_name" (backed off)

I haven't done any systematic testing, but anecdotally I'm pretty sure that this seems to make help more responsive.

Friday 11 July 2008

MedSLT almost fully converted to AFF

I've made a lot of progress this week on converting the bidirectional English/Spanish version of MedSLT to AFF format. I've parameterized the Spanish system to support AFF representations, and added AFF versions of most of the necessary config file and scripts. In particular,
  • You can build a full AFF version of Spa by doing 'make role_marked' in Spa/scripts.
  • You can run interactive bidirectional AFF Eng/Spa text systems using the files load_bidirectional_role_marked.pl and load_bidirectional_restricted_role_marked.pl in EngSpa/scripts.
  • There are targets in EngSpa/scripts/Makefile for running AFF versions of the QA corpus, both plain and restricted, with the obvious naming conventions.
I've done preliminary testing, and everything should be checked in. There are still things missing, e.g. nothing so far in Spa/Spa for checking back-translation, but I figured it would be best at this point to hand over to Pierrette, so that she can refine the rules. I've only made some absolutely minimal changes, enough to check that a few sentences go through.

When this piece of work is finished, all of MedSLT should be available in AFF format, which means that we'll be able to retire the old linear version and only support one system. I'm hopeful that build times will then be low enough for us to go back to building and testing the system every night, as we used to do.

Tuesday 8 July 2008

N-best rescoring again

[Updated July 9]

I've just checked in new code that makes it possible to create training material for doing N-best rescoring on speech translation applications - the functionality is basically the same as what we already had for dialogue applications, but there were a number of details that had to be fixed. It seems that the potential for improving performance using N-best rescoring varies considerably between apps. So far, we've looked at the following cases:
  • Calendar: can already almost halve error rate using rescoring, more should be possible.
  • Ford app: almost no potential for improvement.
  • Paideia app: considerable potential for improvement (don't currently have figures)
  • English MedSLT: maximum possible improvement looks like about 10% relative.
  • French MedSLT: maximum possible improvement about 15-20% relative.
  • Japanese MedSLT: almost no potential for improvement.
The variation in behavior between the different apps is quite surprising. In particular, I don't yet have a good explanation for why the MedSLT languages should be so different.