EBL mode

Yet another thing I should have done years ago. As everyone knows who's tried to do it, writing operationality criteria is rather messier than it should be. One of the worst things is that you don't immediately get feedback on the effects of the changes you've made. You need to rebuild the whole specialised grammar, which typically takes a while, and then you have to go through it to find out what the effect of your changes was.

I've just added an enhancement that should substantially simplify the development process. The new command EBL_MODE puts the top-level in a mode where each input sentence is subjected to EBL analysis, using the current set of operationality criteria; the operationality criteria file is reloaded each time, in case it has changed. The system prints a list of derived rules, with the substring used to derive each rule. Here's an example from CALL-SLT/French:

>> EBL_MODE
(Do EBL processing on input sentences)

--- Performed command EBL_MODE, time = 0.00 seconds

>> puis je avoir un sandwich

Taking operationality criteria from d:/cygwin/home/speech/call-slt/fre/prolog/operationality_recognition.pl

--- Written compiled operationality file (37 items) d:/cygwin/home/speech/call-slt/fre/generatedfiles/fre_recognition_tmp_ebl_operational.pl
% compiling d:/cygwin/home/speech/call-slt/fre/generatedfiles/fre_recognition_tmp_ebl_operational.pl...
%  module tmp_ebl_operational imported into user
%  module lists imported into tmp_ebl_operational
%  module utilities imported into tmp_ebl_operational
%  module ebl_operational imported into tmp_ebl_operational
% compiled d:/cygwin/home/speech/call-slt/fre/generatedfiles/fre_recognition_tmp_ebl_operational.pl in module tmp_ebl_operational, 0 msec 6928 bytes

Rule of form ".MAIN-->utterance"
derived from [puis,je,avoir,un,sandwich]

Rule of form "utterance-->med_utterance"
derived from [puis,je,avoir,un,sandwich]

Rule of form "med_utterance-->vp"
derived from [puis,je,avoir,un,sandwich]

Rule of form "vp-->vbar,vbar,np,optional_pp"
derived from [puis,je,avoir,un,sandwich]

Rule of form "vbar-->verb,hyphen,pronoun,optional_adverb"
derived from [puis,je]

Rule of form "verb-->puis"
derived from [puis]

Rule of form "hyphen-->[]"
derived from []

Rule of form "pronoun-->je"
derived from [je]

Rule of form "optional_adverb-->[]"
derived from []

Rule of form "vbar-->verb,optional_adverb"
derived from [avoir]

Rule of form "verb-->avoir"
derived from [avoir]

Rule of form "optional_adverb-->[]"
derived from []

Rule of form "np-->spec,n"
derived from [un,sandwich]

Rule of form "spec-->un"
derived from [un]

Rule of form "n-->sandwich"
derived from [sandwich]

Rule of form "optional_pp-->[]"
derived from []

Regulus with Nuance 9, continued

Success, of sorts! I have a first version of the Nuance 9/server integration checked in, and Matthew tells me it runs correctly on his machine. (I don't yet have a full Nuance 9 installed here). Everything appears to do what it's supposed to: the original grammar is compiled into a Nuance 9 GrXML grammar, with the semantics transformed into string-concatenation semantics that put together a string representation of the semantic form. This is passed from the MRCP process to the dialogue server, which unpacks the strings, reconstructs the real semantic forms, and then passes them to downstream processing.

We were expecting that we'd save a non-trivial amount of time by skipping the parsing phrase in the server... but Matthew tells me that, as far as he can tell, it's slightly slower than when we were running the recogniser without semantics, and creating the semantic forms on the server side! I don't understand this at all. Will run some offline tests tomorrow and see if I can spot any obvious time-sink.

Regulus with Nuance 9

The last couple of weeks, I've been working with Matthew Fuchs (paideia.com) on getting Regulus to work with Nuance 9. We've had a bunch of problems, but we're making good progress and are nearly at the point of having things up and running.

When we started, we knew about two things right off. Nuance 9 doesn't support recursive grammars, and it doesn't allow GSL either; instead, it requires GrXML, where the semantics are done using ECMAScript (JavaScript). Our basic strategy was to carry on generating Nuance 8.5 GSL, and rely on the Nuance conversion tool, which is supposed to be able to convert GSL into GrXML. We also had to make sure that the grammars were non-recursive.

I added code to check for recursivity, and it turns out to be easy enough, at least in the cases we've looked at so far, to fix the operationality criteria in grammar specialisation so that the generated grammars are non-recursive. It wasn't so easy, though, to use the 8.5 to 9 conversion tool, since it turned out that it didn't handle the 'concat' operator, completely central to Regulus semantics.

We wondered for a while if we'd either have to give up on using semantics in Nuance 9, and parse everything in Regulus, or else have Regulus directly generate Nuance 9 grammars - possible, but non-trivial. But I thought of a cute work-around over the weekend, which seems to solve the problem for now. Instead of generating the actual semantics, we generate strings which encode the semantics, and put them together with 'strcat' rather than 'concat' - the strcat operator is handled by the conversion tool.

Another reason why we were reluctant to generate Nuance 9 semantics directly is that, as far as we can make out, Nuance 9 doesn't have a tool for doing PCFG tuning, which is essential to good performance. But, with the current scheme, we can generate Nuance 8.5 grammars, do the PCFG tuning in 8.5, and then translate into 9. The conversion tool correctly carries across the generated probabilistic weights.

It's a bit of a Frankenstein's monster, but it does all seem to work! Matthew just told me that he was able to run the Nuance 9 grammar successfully in MRCP. Now we just need to integrate everything with the dialogue server, and we'll have the first version fully running. More soon, I hope...

Screenshots from Japanese CALL-SLT

The system shows a French prompt, "DEMANDER DE_MANIERE_POLIE EAU"

The student speaks a corresponding sentence in Japanese, here "omizu onegai shimasu". The system does speech recognition and understanding, and shows the recognised sentence to the student in both Japanese and Roman orthography.

If the students wishes, they can click the help button, to get an example of a Japanese native speaker saying something that will work.

A Scandinavian/English grammar, continued

More progress on the Scandinavian/English grammar:

• I've now got quite reasonable initial rules for negation and lexically passivized verbs, two of the largest holes in the Swedish. Negation, in Swedish, is just an adverb. I did however have to add a feature which distinguished main clauses from subordinate clauses, since the negation adverb occurs after the verb in a main clause, and before it in a subordinate clause. It turned out to be easy to adapt the existing rules for passives to handle lexical passives as well: the grammar now allows passive versions of the present, imperfect, supine and infinitive forms. Thus for example den kan inte köpas här, "it can not buy-INF-PASSIVE here" = "it can not be bought here".
• I've improved the coverage in the Swedish version of MedSLT. I can now translate 92% of the combined interlingua corpus into Swedish, and translate back 99% of the results. This mostly involved adding new lexical items and transfer rules, though I also had to make a couple of minor adjustments to the grammar.

Scandinavian/English grammar now used for English MedSLT

I've now changed the config files for English MedSLT so that they use the shared Scandinavian/English grammar rather than old English-only grammar. I found a couple of small bugs, but now everything seems to be working fine again.

Some time soon, Nikos and I should get together and figure out how to add Swedish to the MedSLT demo and the nightly build. It shouldn't be at all hard.

Swedish MedSLT, continued

I temporarily broke off working on Interlingua to Swedish, and spent a day concentrating on the opposite direction. I built the recognition grammar by training on a corpus which was the union of the original recognition corpus and the generation corpus (this ensures that everything you can generate will also get recognized); then I did PCFG tuning using the set of translations produced from the combined Interlingua corpus. I also used the set of translations as the initial Swedish corpus for translation testing. All the corpora concerned are created on-the-fly as part of the make process, so the correspondences will stay up to date.

It was easy to get things working in Swe -> Int direction, and 98% of the translation corpus now produces well-formed interlingua. I compiled a Swedish recognizer, and hooked everything together to get a speech-to-speech system for Swedish -> English. Anecdotally, it's not bad.

The most urgent thing now is probably to add more Swedish coverage. There are several very common constructions that currently aren't in the specialized Swedish grammar.

Swedish MedSLT, continued

A few more days of messing around, and I'm now translating about 80% of the 1200-item combined interlingua corpus into Swedish. Elisabeth (a native speaker) looked at about half of the material, and made some suggestions in the direction of improving quality. After implementing them, she thinks that over 90% of the translations are clearly good. To do this work, I have had to make a few more improvements in the combined Scandinavian/English grammar: the most important of these is an initial treatment of lexically reflexive verbs.

I'll see if I can improve the numbers a bit more, and will then start on the Swedish-to-Interlingua direction.

Swedish MedSLT, continued

Encouraging progress on Swedish MedSLT: I can now translate a third of the interlingua corpus into Swedish. The translations aren't very good yet, but they are nearly all grammatical. I think it will be fairly easy to improve things.

Swedish MedSLT

I have just started on a Swedish version of MedSLT; this will give the new Scandinavian/English grammar a much more thorough workout. After an hour or two of messing around, I can parse one sentence, var har du ont ("where is your pain", literally "where have you pain").

The first thing I notice is that we need rules for impersonal and lexically reflexive verbs, which are very important in this domain. In particular, an expression that occurs all the time is det gör ont; "it hurts", literally "it makes pain".

More soon.

A Scandinavian/English grammar, continued

I've now changed the config files for English CALL-SLT so that they use the new Scandinavian grammar instead of the English-only one that it's based on. This will give us more of a chance to test how it works.

Next, I will make similar changes in the English part of MedSLT.

A Scandinavian/English grammar, continued

The Scandinavian grammar is working quite well, but it's certainly not complete yet. Here are some important things that still need to be added, all of which occur in Scandinavian but not in English:

• Negation. This is just an adverb in Scandinavian. The slightly non-trivial thing is that the position of this adverb (also some others) is different, depending on whether it's a main or a subordinate clause.
• Lexically reflexive pronouns. As in Romance, some Scandinavian verbs subcategorize for lexically reflexive pronouns, which have no semantic value.
• Lexical passive. Scandinavian verbs have a lexical passive form. I propose to do this in the morphotax.
• Definiteness. I don't yet have all the definiteness constraints. In particular, a definite singular NP has an implicit definite article, but a premodifying adjective requires an explicit definite article.

I don't think any of these things are particularly difficult to implement, or should require major changes to the grammar.

Swedish CALL-SLT

I must stop messing around with the Scandinavian grammar... it's really too much fun! Anyway, I now have a reasonable first cut at a Swedish version of CALL-SLT, working as usual in the restaurant domain. Elisabeth helped me add more material to the Swedish corpus last night; currently, it contains about 160 entries, of which about 90% work. I just tried out the live system, using Maria's GUI, and it runs fine. As I'd hoped, recognition picked up a good deal once I was able to switch on N-best rescoring. I'm getting performance in Swedish only slightly inferior to what I get in English, which seems reasonable given my relative abilities in the two languages.

If people want to try it out, you need to update both Regulus and CALL-SLT using the -d flag, and do a make in CALLSLT/Swe/scripts. Then run in the usual way. So far there is no spoken help, but Elisabeth has promised to record files over Christmas.

A Scandinavian/English grammar, continued

A bit more fiddling around, and I have two-thirds of the initial Swedish CALL-SLT corpus parsing. You can see it here. I've also compiled an initial recogniser. So far, it doesn't work very well, but if it's like the other languages it will improve considerably once I add N-best rescoring.

This is all progressing rather nicely! I need to do some work now on processing the results of our recent CALL-SLT experiments, but once I've done that I'll return to the Swedish. Elisabeth says she will act as our native speaker.

A Scandinavian/English grammar

This weekend, I finally started on a task that I've been meaning to do for ages, and put together a first version of a shared grammar that is intended to cover both English and the Scandinavian languages Swedish, Norwegian and Danish. I've checked it into Regulus/Grammar/Scandinavian. Initially, I'm developing using the CALL-SLT restaurant domain, in English and Swedish. The Swedish version can already handle a reasonable range of language. Here's an example: skulle jag kunna få en pizza = would I be-able-to get a pizza. This is a common way to ask for something in Swedish.

>> skulle jag kunna få en pizza
(Parsing with left-corner parser)

Analysis time: 0.34 seconds

3 possibilities:

----------------------------------------------------------------
Possibility 1
Return value: [(null=[action,få]), (object=[food,pizza]), (null=[modal,kan]),
         (null=[modal,skulle]), (agent=[pronoun,jag]), (null=[utterance_type,ynq]),
         (null=[voice,active])]

Global value: []

Syn features: []

Parse tree:

.MAIN [GENERAL_SCA:541-546]
top [GENERAL_SCA:552-558]
/  utterance_intro null [GENERAL_SCA:566-568]
|  utterance [GENERAL_SCA:615-620]
|     s [GENERAL_SCA:713-718]
|        s [GENERAL_SCA:817-826]
|           vp [GENERAL_SCA:1124-1137]
|           /  vbar [GENERAL_SCA:876-898]
|           |  /  v lex(skulle) [GEN_SWE_LEX:51-51]
|           |  |  np [GENERAL_SCA:1952-1960]
|           |  \     pronoun lex(jag) [GEN_SWE_LEX:200-200]
|           |  vp [GENERAL_SCA:1124-1137]
|           |  /  vbar [GENERAL_SCA:853-875]
|           |  |     v lex(kunna) [GEN_SWE_LEX:61-62]
|           |  |  vp [GENERAL_SCA:1317-1337]
|           |  |  /  vp [GENERAL_SCA:1042-1051]
|           |  |  |  /  vbar [GENERAL_SCA:853-875]
|           |  |  |  |     v lex(få) [CALLSLT_LEX:33-33]
|           |  |  |  |  np [GENERAL_SCA:2073-2091]
|           |  |  |  |  /  np [GENERAL_SCA:1907-1917]
|           |  |  |  |  |  /  d lex(en) [GEN_SWE_LEX:277-278]
|           |  |  |  |  |  |  nbar [GENERAL_SCA:2118-2130]
|           |  |  |  |  |  \     n lex(pizza) [CALLSLT_LEX:179-181]
|           |  |  |  \  \  post_mods null [GENERAL_SCA:1451-1457]
|           \  \  \  post_mods null [GENERAL_SCA:1451-1457]
\  utterance_coda null [GENERAL_SCA:597-599]

------------------------------- FILES -------------------------------

CALLSLT_LEX: d:/cygwin/home/speech/call-slt/swe/regulus/callslt_lex.regulus
GENERAL_SCA: d:/cygwin/home/speech/regulus/grammar/scandinavian/general_sca.regulus
GEN_SWE_LEX: d:/cygwin/home/speech/regulus/grammar/scandinavian/swedish/gen_swe_lex.regulus

Here are some of the issues I've encountered so far:

• Agreement is different in English and Swedish, so the range of possible values for the 'agr' feature has to be language-dependent. In English, agreement is by person and number. In Swedish, it's primarily by number and gender ("common" or "neuter"). However, you also need person, since reflexive pronouns agree with the subject in person.
• All verbs invert in Swedish, and there is no auxiliary "do".
• The range of verb inflections is different in the two languages. English verbs have five forms: base, third person singular present, imperfect, past participle, present participle. So, for example, go, goes, went, gone, going. Modern Swedish doesn't inflect by person or number in the present tense, but on the other hand distinguishes the imperative from the infinitive, distinguishes the "supine" (the form used for the perfect tense) from the past participle, and inflects the past participle by number and gender. So, for example, bryta (break) has the forms bryt (imperative), bryta (infinitive), bryter (present), bröt (imperfect), brutit (supine), brytande (present participle), bruten (past participle singular common), brutet (past participle singular neuter), brutna (past participle plural).
• Swedish nouns inflect for definiteness. So for example bord is "table", but bordet is "the table". Adjectives also inflect for definiteness, thus ett stort bord ("a big table") but det stora bordet ("the big table").
• Swedish possessives inflect for gender and number. So min bil ("my car", common/singular), mitt hus ("my house", neuter/singular), mina barn ("my children", plural).
• Swedish partitives are slightly different. In English, "a bottle of beer"; in Swedish en flaska öl (nothing corresponding to the "of").
• Swedish date and time grammar is slightly different. In English, "december fourteenth"; in Swedish, fjortonde december. In English, "nine thirty"; in Swedish, nio och trettio.
• Swedish negation is basically an adverb, e.g. jag beställde inte någon pizza = I ordered not any pizza. The negation adverb's position is after the verb in main clauses, before in subordinate clauses.

Most of this stuff seems easy, and I can adapt the treatments we implemented in the SLT grammar during the 90s. I am guessing that 85-90% of the rules in the final shared grammar will be common to English and Swedish. Judging from our experiences with SLT, Danish and Swedish overlap to 95% or better, and Norwegian should be similar.

New treatment of Japanese verbs

I've been discussing Japanese verbs with Yukie - we need new inflectional forms for CALL-SLT, in the particular the volitional (-tai) form, and the old system was getting out of hand. Japanese has extraordinarily regular morphology, with only two irregular verbs and some very straightforward sound-changes, so it seemed to me that we really ought to be able to get by without explicitly listing all the inflections of every verb we needed.

Yukie wrote down a table of inflections, and we discussed ways of splitting up inflected verbs into stems and affixes. Based on our discussion, I've implemented a first version of a new treatment, where you now only need to specify a single root form of the verb, and everything else is done by morphotax rules, where the affixes are treated by Nuance as separate words. I've tested by converting the Japanese Calendar lexicon to the new form, and compiling into a recognizer. Coverage is what it was, and recognition is anecdotally fine with my voice. I will dig out some Japanese Calendar data soon and run proper tests.

If anyone wants to look at the details, the morphotax rules are in $REGULUS/Grammar/Japanese/japanese_verb_morphology.regulus. The new version of the Japanese Calendar lexicon is at $REGULUS/Examples/Calendar/Regulus/japanese_calendar_lex_new.regulus.

Here's an example of a parse:

$ nanji ni owa ri mashita ka
(Parsing with left-corner parser)

Analysis time: 0.09 seconds

Return value: [[question,form(past,[[owaru],[ni,term(null,nanji,[])]])]]

Global value: []

Syn features: []

Parse tree:

.MAIN [JAPANESE_CORE_RULES:112-116]
  top [JAPANESE_CORE_RULES:117-119]
     utterance [JAPANESE_CORE_RULES:120-123]
     /  main_clause [JAPANESE_CORE_RULES:147-151]
     |     s [JAPANESE_CORE_RULES:155-162]
     |     /  comps [JAPANESE_CORE_RULES:190-195]
     |     |  /  pp [JAPANESE_CORE_RULES:414-423]
     |     |  |  /  np [JAPANESE_CORE_RULES:267-273]
     |     |  |  |     n lex(nanji) [JAPANESE_CALENDAR_LEX_NEW:84-84]
     |     |  |  \  p lex(ni) [JAPANESE_CALENDAR_LEX_NEW:274-284]
     |     |  \  comps null [JAPANESE_CORE_RULES:163-166]
     |     |  vbar [JAPANESE_CORE_RULES:249-253]
     |     |     v [JAPANESE_VERB_MORPHOLOGY:13-26]
     |     |     /  v_stem [JAPANESE_VERB_MORPHOLOGY:27-38]
     |     |     |  /  v_stem lex(owa) [JAPANESE_CALENDAR_LEX_NEW:227-237]
     |     |     |  \  stem_affix lex(ri) [JAPANESE_VERB_MORPHOLOGY:138-138]
     |     \     \  affix lex(mashita) [JAPANESE_VERB_MORPHOLOGY:80-83]
     \  lex(ka)

------------------------------- FILES -------------------------------

JAPANESE_CALENDAR_LEX_NEW:
d:/cygwin/home/speech/regulus/examples/calendar/regulus/japanese_calendar_lex_new.regulus
JAPANESE_CORE_RULES:     
d:/cygwin/home/speech/regulus/grammar/japanese/japanese_core_rules.regulus
JAPANESE_VERB_MORPHOLOGY:
d:/cygwin/home/speech/regulus/grammar/japanese/japanese_verb_morphology.regulus

"Abstract actions" and the dialogue server

I've had some productive discussions with Maria over the last few days, which have resulted in a couple of significant improvements to the dialogue server. Maria is going to build a Java GUI for the CALL-SLT system. She needs to be able to send requests to the dialogue server, and get back information that she will pass on to the user. Most often this will be in the form of screen-based output. The new functionality is motivated by this scenario, but is quite generic.

The first point Maria made was that she would prefer to use XML-formatted messages. Java finds it easy to manipulate XML; parsing Prolog messages, on the other hand, is a complete pain. So I added switches that allow the client to put the server into a mode where all messages are XML strings inside a minimal Prolog wrapper.

Yesterday, Maria made another very sensible request. In the first version of the application, the Prolog "output manager" module received abstract actions, and transformed them into concrete actions. Typically, concrete actions would involve printing strings. So, for example, suppose that the system has just given you the prompt

POLITE REQUEST TABLE outside

and you have correctly replied

i would like a table outside please

The abstract action produced is

display_matching_info('i would like a table outside please', correct, [2,1,2])

which means that the recognized words were 'i would like a table outside please'; they correctly matched the prompt; and the score is now 2 correct, 1 incorrect, with a positive streak of 2. This is converted by the output manager into the concrete action

print('I heard: "i would like a table outside please"

Correct!

Score: 2 right, 1 wrong (66.7%) Streak: 2')

This works fine for a text-based command-line interface; but, as Maria pointed out, the output processing isn't necessarily appropriate when you have a Java Swing GUI, in which case you probably would prefer to format it yourself. For instance, you might want to print the recognized words in one pane, render the "correct" as a green tick-mark in another one, and present the score graphically as three columns of different heights.

In general, the abstract action is going to be more useful to you than the concrete one. So I've just added a little more functionality to the dialogue server to handle that too. Here's a summary of the new messages, and what they do; they are also documented in the file itself, $REGULUS/Prolog/dialogue_server.pl.

• action(xml_messages). Format future messages in both directions in XML form. Each message will be of the form
  
  xml_message(XMLString).
  
  where XMLString is an XML encoding of the corresponding Prolog message produced using the predicate prolog_xml/2 in $REGULUS/PrologLib/prolog_xml.pl. The XML can be converted back into Prolog if necessary using the same predicate.
  
• action(prolog_messages). Format future messages in both directions in Prolog form (default).
• action(abstract_actions). Pass abstract actions to the client, so that the client can do its own output management.
  
• action(concrete_actions). Pass concrete actions to the client (default).
  

Warning: don't use SICStus 4.0.2

Maria and I have just spent two very frustrating days trying to figure out why CALL-SLT wasn't running correctly on her machine. In the end, it turned out that a few bits of Regulus functionality don't work correctly under SICStus 4.0.2, which is the version she was using... there appears to be something wrong with the SICStus/operating system interface.

So avoid this release! 4.0.4 is fine.

Dialogue server now accepts XML format messages

Another of those things I should no doubt have done years ago: after a discussion with Maria, I've now modified the dialogue server so that it can also run in a mode where all messages are XML-formatted.

Details (this is documented in $REGULUS/Prolog/dialogue_server.pl):

- Initially, the server is in Prolog mode.

- To put the server into XML mode, send the message

xml_messages.

- Subsequent messages are of the form

xml_message(XMLMessage).

where XMLMessage is an XML encoding of the corresponding Prolog message produced using the predicate prolog_xml/2 in $REGULUS/PrologLib/prolog_xml.pl.
The XML can be converted back into Prolog if necessary using the same predicate.

I have tested by converting the CALL-SLT Prolog client to use XML-flavor messages, and  it all works fine.

The routines in $REGULUS/PrologLib/prolog_xml.pl should in general be useful for translating Prolog into XML form in a reversible way. Look at the file for documentation and an example.

CALL-SLT and Japanese

I have just added a little more coverage to the Japanese version... it now has about a dozen sentences for the student to practice on. I tried it, and so far it still recognizes everything I say. This is probably more because vocabulary is so small than because I have a wonderful Japanese accent :)

Yukie and I should talk about how to proceed here. The first step will be to add material to the Japanese corpus.