EBL mode

Yet another thing I should have done years ago. As everyone knows who's tried to do it, writing operationality criteria is rather messier than it should be. One of the worst things is that you don't immediately get feedback on the effects of the changes you've made. You need to rebuild the whole specialised grammar, which typically takes a while, and then you have to go through it to find out what the effect of your changes was.

I've just added an enhancement that should substantially simplify the development process. The new command EBL_MODE puts the top-level in a mode where each input sentence is subjected to EBL analysis, using the current set of operationality criteria; the operationality criteria file is reloaded each time, in case it has changed. The system prints a list of derived rules, with the substring used to derive each rule. Here's an example from CALL-SLT/French:

>> EBL_MODE
(Do EBL processing on input sentences)

--- Performed command EBL_MODE, time = 0.00 seconds

>> puis je avoir un sandwich

Taking operationality criteria from d:/cygwin/home/speech/call-slt/fre/prolog/operationality_recognition.pl

--- Written compiled operationality file (37 items) d:/cygwin/home/speech/call-slt/fre/generatedfiles/fre_recognition_tmp_ebl_operational.pl
% compiling d:/cygwin/home/speech/call-slt/fre/generatedfiles/fre_recognition_tmp_ebl_operational.pl...
%  module tmp_ebl_operational imported into user
%  module lists imported into tmp_ebl_operational
%  module utilities imported into tmp_ebl_operational
%  module ebl_operational imported into tmp_ebl_operational
% compiled d:/cygwin/home/speech/call-slt/fre/generatedfiles/fre_recognition_tmp_ebl_operational.pl in module tmp_ebl_operational, 0 msec 6928 bytes

Rule of form ".MAIN-->utterance"
derived from [puis,je,avoir,un,sandwich]

Rule of form "utterance-->med_utterance"
derived from [puis,je,avoir,un,sandwich]

Rule of form "med_utterance-->vp"
derived from [puis,je,avoir,un,sandwich]

Rule of form "vp-->vbar,vbar,np,optional_pp"
derived from [puis,je,avoir,un,sandwich]

Rule of form "vbar-->verb,hyphen,pronoun,optional_adverb"
derived from [puis,je]

Rule of form "verb-->puis"
derived from [puis]

Rule of form "hyphen-->[]"
derived from []

Rule of form "pronoun-->je"
derived from [je]

Rule of form "optional_adverb-->[]"
derived from []

Rule of form "vbar-->verb,optional_adverb"
derived from [avoir]

Rule of form "verb-->avoir"
derived from [avoir]

Rule of form "optional_adverb-->[]"
derived from []

Rule of form "np-->spec,n"
derived from [un,sandwich]

Rule of form "spec-->un"
derived from [un]

Rule of form "n-->sandwich"
derived from [sandwich]

Rule of form "optional_pp-->[]"
derived from []

Regulus with Nuance 9, continued

Success, of sorts! I have a first version of the Nuance 9/server integration checked in, and Matthew tells me it runs correctly on his machine. (I don't yet have a full Nuance 9 installed here). Everything appears to do what it's supposed to: the original grammar is compiled into a Nuance 9 GrXML grammar, with the semantics transformed into string-concatenation semantics that put together a string representation of the semantic form. This is passed from the MRCP process to the dialogue server, which unpacks the strings, reconstructs the real semantic forms, and then passes them to downstream processing.

We were expecting that we'd save a non-trivial amount of time by skipping the parsing phrase in the server... but Matthew tells me that, as far as he can tell, it's slightly slower than when we were running the recogniser without semantics, and creating the semantic forms on the server side! I don't understand this at all. Will run some offline tests tomorrow and see if I can spot any obvious time-sink.

Regulus with Nuance 9

The last couple of weeks, I've been working with Matthew Fuchs (paideia.com) on getting Regulus to work with Nuance 9. We've had a bunch of problems, but we're making good progress and are nearly at the point of having things up and running.

When we started, we knew about two things right off. Nuance 9 doesn't support recursive grammars, and it doesn't allow GSL either; instead, it requires GrXML, where the semantics are done using ECMAScript (JavaScript). Our basic strategy was to carry on generating Nuance 8.5 GSL, and rely on the Nuance conversion tool, which is supposed to be able to convert GSL into GrXML. We also had to make sure that the grammars were non-recursive.

I added code to check for recursivity, and it turns out to be easy enough, at least in the cases we've looked at so far, to fix the operationality criteria in grammar specialisation so that the generated grammars are non-recursive. It wasn't so easy, though, to use the 8.5 to 9 conversion tool, since it turned out that it didn't handle the 'concat' operator, completely central to Regulus semantics.

We wondered for a while if we'd either have to give up on using semantics in Nuance 9, and parse everything in Regulus, or else have Regulus directly generate Nuance 9 grammars - possible, but non-trivial. But I thought of a cute work-around over the weekend, which seems to solve the problem for now. Instead of generating the actual semantics, we generate strings which encode the semantics, and put them together with 'strcat' rather than 'concat' - the strcat operator is handled by the conversion tool.

Another reason why we were reluctant to generate Nuance 9 semantics directly is that, as far as we can make out, Nuance 9 doesn't have a tool for doing PCFG tuning, which is essential to good performance. But, with the current scheme, we can generate Nuance 8.5 grammars, do the PCFG tuning in 8.5, and then translate into 9. The conversion tool correctly carries across the generated probabilistic weights.

It's a bit of a Frankenstein's monster, but it does all seem to work! Matthew just told me that he was able to run the Nuance 9 grammar successfully in MRCP. Now we just need to integrate everything with the dialogue server, and we'll have the first version fully running. More soon, I hope...

Screenshots from Japanese CALL-SLT

The system shows a French prompt, "DEMANDER DE_MANIERE_POLIE EAU"

The student speaks a corresponding sentence in Japanese, here "omizu onegai shimasu". The system does speech recognition and understanding, and shows the recognised sentence to the student in both Japanese and Roman orthography.

If the students wishes, they can click the help button, to get an example of a Japanese native speaker saying something that will work.

A Scandinavian/English grammar, continued

More progress on the Scandinavian/English grammar:

• I've now got quite reasonable initial rules for negation and lexically passivized verbs, two of the largest holes in the Swedish. Negation, in Swedish, is just an adverb. I did however have to add a feature which distinguished main clauses from subordinate clauses, since the negation adverb occurs after the verb in a main clause, and before it in a subordinate clause. It turned out to be easy to adapt the existing rules for passives to handle lexical passives as well: the grammar now allows passive versions of the present, imperfect, supine and infinitive forms. Thus for example den kan inte köpas här, "it can not buy-INF-PASSIVE here" = "it can not be bought here".
• I've improved the coverage in the Swedish version of MedSLT. I can now translate 92% of the combined interlingua corpus into Swedish, and translate back 99% of the results. This mostly involved adding new lexical items and transfer rules, though I also had to make a couple of minor adjustments to the grammar.

Scandinavian/English grammar now used for English MedSLT

I've now changed the config files for English MedSLT so that they use the shared Scandinavian/English grammar rather than old English-only grammar. I found a couple of small bugs, but now everything seems to be working fine again.

Some time soon, Nikos and I should get together and figure out how to add Swedish to the MedSLT demo and the nightly build. It shouldn't be at all hard.

Swedish MedSLT, continued

I temporarily broke off working on Interlingua to Swedish, and spent a day concentrating on the opposite direction. I built the recognition grammar by training on a corpus which was the union of the original recognition corpus and the generation corpus (this ensures that everything you can generate will also get recognized); then I did PCFG tuning using the set of translations produced from the combined Interlingua corpus. I also used the set of translations as the initial Swedish corpus for translation testing. All the corpora concerned are created on-the-fly as part of the make process, so the correspondences will stay up to date.

It was easy to get things working in Swe -> Int direction, and 98% of the translation corpus now produces well-formed interlingua. I compiled a Swedish recognizer, and hooked everything together to get a speech-to-speech system for Swedish -> English. Anecdotally, it's not bad.

The most urgent thing now is probably to add more Swedish coverage. There are several very common constructions that currently aren't in the specialized Swedish grammar.