Regulus for DORIS

When I was visiting Melbourne Uni last week, I talked with Patrick Ye about the possibility of using Regulus to provide speech recognition for the DORIS project, which, as he pointed out, is in fact quite similar to SHRD2; in both domains, the basic idea is to find things, pick them up, and move them around. I did indeed find it very easy to use the examples corpus and vocabulary that Patrick sent me to adapt the existing SHRD2 resources, and in just a few hours put together an initial Regulus grammar that could be compiled into a recogniser. The current version of the grammar covers a bit more than 80% of Patrick's corpus, and the recogniser can turn spoken sentences in Australian-accented English into either strings of words or scoped logical forms. For example, here's the representation it produces of "the red book is on the desk":

[[dcl,
  quant(def_sing, A, [[book,A],[color,A,red]],
        quant(def_sing, B,
              [[desk,B]],
              quant(exist,C,[[be_on_loc,C,A,B],[tense,C,present]],true)))]]

If people want to look at the details, the files are checked in at http://regulus.cvs.sourceforge.net/viewvc/regulus/Regulus/Examples/Doris/. The interesting ones are the lexicon, at http://regulus.cvs.sourceforge.net/viewvc/regulus/Regulus/Examples/Doris/Regulus/doris_lex.regulus, and the corpus, at http://regulus.cvs.sourceforge.net/viewvc/regulus/Regulus/Examples/Doris/corpora/doris_corpus.pl

Update on SHRD2 paraphrasing

I extended the paraphrasing capabilities for SHRD2, and we now get paraphrases for about 45% of the 180-ish examples in the current corpus. Beth Ann and I did a little experiment for the paper we presented at the SETQA-NLP workshop in Colorado last week. We each tried judging the corpus examples for correctness, using both the paraphrases and the underlying representations. In order to minimize learning effects, we permuted the order in which we did things.

Even though the "logic English" paraphrases seem very similar to the scoped logical representations, it in fact turns out that judging paraphrases is a lot faster. Even for me, knowing all the representations from having worked on them, judging paraphrases took 29 minutes, against 22 minutes for judging structures. For Beth Ann, who didn't know the datastructures previously, paraphrases were more than twice as quick. Part of the payoff comes from the fact that the paraphrase grammar acts as a filter; most ill-formed structures produce no paraphrase, hence don't need to be judged at all when paraphrases are used.

One person in the workshop audience said he was pleased to see a paper about software engineering which actually contained an experiment! Beth Ann was clearly right to insist that we do this, and work out the methodology.

First version of paraphrase grammar for SHRD2

I've just added a first version of a paraphrase grammar for SHRD2, where it tries to realize the scoped LFs in "logic English". Text example below. It doesn't do much more than this yet, but now that I have the basic structure working I'm hoping that it will be easy to add coverage. Planning to continue with this later today!

>> pick up a big red block

      Old state: []
             LF: [[imp, 
                   form(imperative, 
                        [[pick_up,term(pro,you,[]),term(a,block,[[size,big],[color,red]]),up]])]]
 Intermediate 1: [[imp, 
                   scoping_unit([modal, imperative, 
                                 term(event_exists, A, 
                                      [[pick_up, A, term(pro,B,[[you,B]]), 
                                        term(a,C,[[block,C],[size,C,big],[color,C,red]]), up]])])]]
 Intermediate 2: [[imp, 
                   quant(pro, A, [[you,A]], 
                         quant(a, B, 
                               [[block,B],[size,B,big],[color,B,red]], 
                               imperative(quant(event_exists,C,[[pick_up,C,A,B,up]],true))))]]
 Intermediate 3: [[imp, 
                   quant(pro, A, [[you,A]], 
                         quant(a, B, 
                               [[block,B],[size,B,big],[color,B,red]], 
                               imperative(quant(event_exists,C,[[pick_up,C,A,B]],true))))]]
  Dialogue move: [[imp, 
                   quant(exist, A, [[you,A]], 
                         quant(exist, B, 
                               [[block,B],[size,B,big],[color,B,red]], 
                               imperative(quant(exist,C,[[pick_up,C,A,B]],true))))]]
     Paraphrase: COMMAND there is an A SUCH THAT you are A AND there is a B SUCH THAT B is a block AND B is big AND B is red AND MAKE IT TRUE THAT there is a C SUCH THAT C is that A picks up B
Abstract action: say(i_dont_understand,present)

Multiple processing stages in dialogue top-level

When building dialogue applications, it's quite often the case that there are multiple stages in the process of converting the LF (the thing that comes out of the recognizer) into a dialogue move. I've just added some hooks so that you can pass back the intermediate levels of representation as an optional extra argument to lf_to_dialogue_move, and display them. This is useful for debugging.

Here's a text example from SHRD2:

>> is the large red block in the box

      Old state: []
             LF: [[ynq, 
                   form(present, 
                        [[be, term(the_sing,block,[[size,large],[color,red]]), 
                          [in_loc,term(the_sing,box,[])]]])]]
 Intermediate 1: [[ynq, 
                   scoping_unit(term(event_exists, A, 
                                     [[be, A, 
                                       term(the_sing, B, 
                                            [[block,B],[size,B,large],[color,B,red]]),
                                       [in_loc,term(the_sing,C,[[box,C]])]],
                                      [tense,A,present]]))]]
 Intermediate 2: [[ynq, 
                   quant(the_sing, A, [[block,A],[size,A,large],[color,A,red]], 
                         quant(the_sing, B, [[box,B]], 
                               quant(event_exists, C, 
                                     [[be,C,A,[in_loc,B]],[tense,C,present]], true)))]]
  Dialogue move: [[ynq, 
                   quant(the_sing, A, [[block,A],[size,A,big],[color,A,red]], 
                         quant(the_sing, B, [[box,B]], 
                               quant(event_exists,C,[[be_in_loc,C,A,B],[tense,C,present]],true)))]]
Abstract action: say(i_dont_understand,present)
Concrete action: tts(sorry, I don't understand)
      New state: []

Intermediate 1 is after addition of variables; Intermediate 2 is after scoping; and Dialogue move is after rewriting of lexical predicates. This last step is still very primitive.

"Pick up a big red block", revisited

OK, I did some more work on SHRD2, and, as of a few minutes ago, I managed to speak a sentence and get it turned into a logic-like representation. Here's a slightly edited trace (the less interesting parts of the output have been removed). I did indeed say "pick up a big red block"!

>> RECOGNISE
(Take next loop input from live speech)

Recognised: recognition_succeeded([rec_result(62,pick up a big red block,...)

      Old state: []
             LF: [[imp, 
                   form(imperative, 
                        [[pick_up,term(pro,you,[]),term(a,block,[[size,big],[color,red]]),up]])]]
  Dialogue move: [[imp, 
                   quant(pro, A, [[you,A]], 
                         quant(a, B, 
                               [[block,B],[size,B,big],[color,B,red]], 
                               imperative(quant(event_exists,C,[[pick_up,C,A,B,up]],true))))]]
Abstract action: say(i_dont_understand,present)
Concrete action: tts(sorry, I don't understand)
      New state: []

Dialogue processing time: 0.01 seconds

Obviously, there's plenty more left to do before we have anything resembling a complete system. In particular, there is as yet no reference resolution or dialogue management, so it can't react to the commands and questions in any way. But, all things considered, I think we're making reasonable progress.

SHRD2

I was talking with some people about Terry Winograd's book Understanding Natural Language, and how inspiring I found the SHRDLU system when I first read about it as an undergraduate. They asked me what the current equivalent would be. It bothered me that I couldn't come up with anything, and that no one really seemed to be building this kind of system any more.

Well... so why not do something about it? Stage 1 is to build a speech-enabled, and hopefully rather more robust reconstruction of SHRDLU, using Regulus. Some initial stuff is already checked in under $REGULUS/Examples/SHRD2. When we've satisfied ourselves that we can handle the Blocks World, Stage 2 will be to define a new and more ambitious domain - something which will hopefully demonstrate that there has in fact been significant progress since the 70s.

I'll be posting more about this soon. So far, SHRD2 has already turned up some important holes in the general English grammar, which I'm working on fixing.