Sunday 23 August 2009

CALL-SLT and Japanese

And we now have a skeleton Japanese system too. So far, it can only do one sentence,

hitori no teeburu wa arimasu ka

which is rendered in the Interlingua as

POLITE REQUEST TABLE 1 PERSON

Though it is nice that this goes all the way through: I can get the Interlingua as a prompt, speak the sentence, and be informed that I got it right. Not surprisingly, since the grammar doesn't cover anything else, it's very reliable when you say the one thing it knows!

Yukie and I need to get together and add more content. The first step will be for Yukie to flesh out the corpus, which currently only has a dozen or so examples.

Saturday 22 August 2009

CALL-SLT and French

We should now have a complete set of of scripts, config files etc for the French version of CALL-SLT. I added a couple of placeholder files, with enough translation rules to do the sentence "Je voudrais deux bières". The make appears to work, and I was able to run the initial version of the translation game in the server. Over to Pierrette to add some actual content!

Next, Japanese...

Progress on CALL-SLT

I have the generic CALL-SLT functionality packaged up so that it can be run inside the dialogue server - this involved extending the dialogue server a little bit, so that you can now call recognition from inside it. The interface between the client and the CALL-SLT-loaded dialogue server is consequently very simple. There are so far just three commands:
  • "Next prompt". The server generates a new prompt, using its current strategy, and returns it.
  • "Recognize and match". The server performs recognition, translates it to Interlingua, matches against the current prompt, and returns a string explaining what happened.
  • "Help". The server returns the current prompt, plus a text example illustrating one possible way to realize the prompt.
I have been testing with a minimal Prolog-based client. It should be easy to write a Java client which offers a nice GUI-style interface.

Thursday 20 August 2009

Progress on CALL-SLT

I have a first version of the translation game working for English! It's still very clunky indeed (I'm running it from the command-line in the development environment), but it's already kind of fun. Next step will be to package it up in a better way, using the dialogue server; this should be quite easy. Once I've done that, it'll be possible to add a Java GUI, so that we have a complete first version of the system.

Of course, what I really want to do is try it in another language... I already know how to order in English restaurants! I'll start sorting out the config files and scripts for Japanese. My restaurant Japanese is shaky, and I'm very curious to see if I can use CALL-SLT to improve it.

Wednesday 19 August 2009

Progress on CALL-SLT

I've spent some time messing around with the English to Interlingua translation rules and the Interlingua grammar. We're now able to translate 95% of the development corpus into sensible-looking interlingua - though the current surface form will probably be revised a bit at some point by Pierrette and Johanna.

At any rate, you can now run the speech-input English system from the Regulus command-line, setting it so that spoken inputs are parsed and translated into Interlingua. I don't think it's that much extra work to add code so that we have a complete first version of the system. There will initially be two commands. You can either ask for a new Interlingua prompt, or ask to speak. If you speak, it will translate what you say into Interlingua, compare with the current prompt, and score you. With any luck, I'll have this working before next week.

Here's the current English development corpus, with the Interlingua translations it produces:

i will take a beer REQUEST beer
give me a beer REQUEST beer
hello NEUTRAL-GREETING
good evening EVENING-GREETING
i would like a table for one POLITE REQUEST TABLE 1 PERSON
could i have a table for one POLITE REQUEST TABLE 1 PERSON
do you have a table for one POLITE REQUEST TABLE 1 PERSON
i would like a table for two POLITE REQUEST TABLE 2 PERSON
could i have a table for three POLITE REQUEST TABLE 3 PERSON
do you have a table for four POLITE REQUEST TABLE 4 PERSON
do you have a table outside POLITE REQUEST TABLE outside
is there a table outside POLITE REQUEST TABLE outside
could i have a table outside POLITE REQUEST TABLE outside
could we have a table by the window POLITE REQUEST TABLE by-loc window
do you have a table near the window POLITE REQUEST TABLE in-loc window
i 'd like a table in the smoking area POLITE REQUEST TABLE smoking
i 'd like a table in the non-smoking area POLITE REQUEST TABLE non-smoking
smoking please POLITE REQUEST TABLE smoking
non-smoking please POLITE REQUEST TABLE non-smoking
could i have a table in the corner POLITE REQUEST TABLE in-loc corner
can i have a table in the corner POLITE REQUEST TABLE in-loc corner
i would like a non-smoking table for one POLITE REQUEST TABLE 1 PERSON non-smoking
do you have a non-smoking table for one POLITE REQUEST TABLE 1 PERSON non-smoking
a non-smoking table for two please POLITE REQUEST TABLE 2 PERSON non-smoking
could i have a non-smoking table for three people POLITE REQUEST TABLE 3 PERSON non-smoking
i would like a table for three people in the smoking area POLITE REQUEST TABLE 3 PERSON smoking
i would like a table for four POLITE REQUEST TABLE 4 PERSON
do you have a table for four POLITE REQUEST TABLE 4 PERSON
i would like to reserve a table for seven o'clock POLITE REQUEST TABLE 19 00
could i reserve a table for seven thirty POLITE REQUEST TABLE 19 30
i 'd like to reserve a table for six forty five POLITE REQUEST TABLE 18 45
i would like to reserve a table for two for seven fifteen POLITE REQUEST TABLE 2 PERSON 19 15
could i reserve a table for two people for eight o'clock POLITE REQUEST TABLE 2 PERSON 20 00
do you have a table for two at seven thirty POLITE REQUEST TABLE 2 PERSON 19 30
could i reserve a table for seven o'clock tomorrow evening POLITE REQUEST TABLE 19 00 time-evening date-tomorrow
i would like to reserve a table for six forty five tomorrow please parsing_failed
could i reserve a table for two for tomorrow evening POLITE REQUEST TABLE 2 PERSON time-evening date-tomorrow
could i reserve a table for this evening POLITE REQUEST TABLE time-evening date-today
do you have a table for three tomorrow evening around seven o'clock POLITE REQUEST TABLE 3 PERSON 19 00 time-evening date-tomorrow
i have a reservation in the name of smith parsing_failed
i should have a reservation in the name of smith parsing_failed
could we see the menu POLITE REQUEST menu
i would like to see the menu POLITE REQUEST menu
could we get the menu POLITE REQUEST menu
could we have the bill POLITE REQUEST check
could we have the bill please POLITE REQUEST check
may i have the bill POLITE REQUEST check
may i have the check POLITE REQUEST check
could you give us the check please POLITE REQUEST check
could you give me a receipt POLITE REQUEST receipt
i would like a receipt please POLITE REQUEST receipt
could i get a receipt please POLITE REQUEST receipt
i would like a beer POLITE REQUEST beer
could we have two beers POLITE REQUEST 2 beer
could you give us two beers POLITE REQUEST 2 beer
could i have a latte POLITE REQUEST latte
i 'd like a medium latte POLITE REQUEST medium latte
a cup of tea please POLITE REQUEST cup tea
two cups of tea please POLITE REQUEST 2 cup tea
could we get two glasses of water POLITE REQUEST 2 glass water
i 'd like a glass of the house red POLITE REQUEST glass house-red-wine
could we have two glasses of the house red POLITE REQUEST 2 glass house-red-wine
i would like a large glass of white wine POLITE REQUEST large glass white-wine
i would like two small glasses of white wine POLITE REQUEST 2 small glass white-wine
i would like a pizza POLITE REQUEST pizza
i would like a hamburger POLITE REQUEST hamburger
i would like the soup POLITE REQUEST soup
i would like two pizzas POLITE REQUEST 2 pizza

Tuesday 18 August 2009

Progress on CALL-SLT

I've now got enough stuff working in CALL-SLT that it's possible to translate a few simple sentences from spoken English to Interlingua. Here's an example:

>> i would like two beers

Source: i would like two beers
Target: POLITE REQUEST two beer
Other info:
n_parses = 1
parse_time = 0.047
source_representation = [(agent=[pronoun,i]), (null=[action,like]), (null=[modal,would]),
(null=[utterance_type,dcl]), (null=[voice,active]),
(object=[drink,beer]), (object=[spec,2])]
transfer_to_source_discourse_time = 0.0
source_discourse = [(null=[utterance_type,dcl]), (agent=[pronoun,i]), (null=[voice,active]),
(null=[modal,would]), (null=[action,like]), (object=[spec,2]),
(object=[drink,beer])]
resolved_source_discourse = [(null=[utterance_type,dcl]), (agent=[pronoun,i]),
(null=[voice,active]), (null=[modal,would]), (null=[action,like]),
(object=[spec,2]), (object=[drink,beer])]
resolution_processing = trivial
resolution_time = 0.0
transfer_to_interlingua_time = 0.0
interlingua = [(arg2=[drink,beer]), (arg2=[number,2]), (null=[politeness,polite]),
(null=[utterance_type,request])]
interlingua_surface = POLITE REQUEST two beer
interlingua_checking_time = 0.0

--- Performed command i would like two beers, time = 0.05 seconds

We are not far from having everything we need to be able to build and run an initial version of the CALL-SLT server. Originally, it will prompt in the surface interlingua. but I'll leave in a hook so that we can use the picture interlingua as soon as we have something available.

Sunday 16 August 2009

CALL-SLT corpus

For people who don't already know, we are collecting our initial corpus for CALL-SLT on a Google Docs document. If you don't already have access to this document and would like to contribute, please get a Google account and mail me your Google address.

All contributions very gratefully received - and don't worry that you might break something accidentally, Google Docs has excellent facilities for tracking revisions and if necessary reverting to earlier versions.

Here's a sample extract showing what the corpus looks like:

Fre: auriez-vous une table sur la terrasse (SVP) ?
Fre: auriez-vous une table en terrasse ?
Eng: do you have a table outside
Eng: is there a table outside
Eng: could i have a table outside
Int: POLITE REQUEST LOCATION-TABLE LOCATION-OUTSIDE [note: is "en terrasse" different from "outside"?]

Fre: auriez-vous une table près de la fenêtre (SVP) ?
Fre: auriez-vous une table à côté de la fenêtre (SVP) ?
Jap: mado gawa no seki wa ari masu ka
Eng: could we have a table by the window
Eng: do you have a table near the window
Int: POLITE REQUEST LOCATION-TABLE NEAR WINDOW

Fre: auriez-vous une table non-fumeur/fumeur (SVP)?
Fre: auriez-vous une table dans la zone non-fumeur /fumeur (SVP)?
Fre: auriez-vous une table dans la section non-fumeur/fumeur (SVP)?
Jap: kinenseki wa ari masu ka
Eng: i 'd like a table in the smoking area
Eng: i 'd like a table in the non-smoking area
Eng: smoking please
Eng: non-smoking please
Int: POLITE REQUEST LOCATION-TABLE IN SMOKING-AREA
Int: POLITE REQUEST LOCATION-TABLE IN NON-SMOKING-AREA

Fre: auriez-vous une table dans le coin (SVP) ?
Jap: oku no seki wa ari masu ka
Jap: oku no teeburu wa ari masu ka
Eng: could i have a table in the corner
Int: POLITE REQUEST LOCATION-TABLE IN CORNER

First CALL-SLT meeting

We've had our first CALL-SLT meeting, which has done a lot to clarify our immediate goals for the project. We're going to start by building a simple version of the system, constructed in such a way that it will be easy to upgrade it by successively replacing simple modules by more complex ones. Initially, we will be working in the tourism/restaurant domain, using the languages English, French and Japanese. When we have those working well enough, we'll also start on Chinese; this is a language none of us know, so it will give us intuitions about what it's like to be an elementary-level student trying to use the system to get some fluency in a language.

The initial prototype will work as follows. At each turn, the system prompts the student with a description of what they are supposed to say, formulated in a version of the Interlingua. The student will attempt to speak it in the L2 (the language they are trying to learn). The system applies speech recognition to the student's utterance, then tries to translate the result into the interlingua. Finally, it compares the translated interlingua with the one used to prompt the student, and gives them feedback on how they did. Here are more details:
  • Prompting in Interlingua. In the first version, the interlingua will be shown to the student in a text-based form, using the methods we've developed under MedSLT. So for example, the system might show the student

    POLITE REQUEST TABLE 3 PERSON TIME 19:30

    expecting the student to say something like

    I would like to reserve a table for three people at seven thirty

    or whatever the equivalent is in the L2 they are using.

  • As soon as we've figured out a good way to do it, we would like to be able to present the interlingua prompt in graphical form. So here, we might have a picture that could be described as

    Scene:
    Client is talking to waiter.
    Speech bubble from client.
    Inside speech bubble:
    three chairs around a restaurant table;
    large clock in background shows 19:30
  • All speech input to the system will be logged in the usual way. We will have a registration process which allows us to associate each recorded utterance with meta-data which in particular will specify whether or not the utterance was recorded by a native speaker, and whether or not speech recognition got it right.

  • When the system has compared the student's interlingua with the prompt interlingua, there are two simple ways for it to give helpful feedback. The first is to present both versions of the interlingua, highlighting the elements that are different. For instance, in the example above, if the system recognized

    Could I have a table for two people at seven thirty

    then the system would present the prompt and recognized interlinguas roughly as follows:

    POLITE REQUEST TABLE *3* PERSON TIME 19:30
    POLITE REQUEST TABLE *2* PERSON TIME 19:30

    The second way to give help will be to play an example of a native speaker saying some version of the sentence in the L2, if such an example already exists.

  • The prompt selection module will have hooks allowing specification of a strategy. A simple strategy we will implement soon is to choose the prompt from a list of examples where there is a recorded example of a native speaker saying the prompt in the L2, possibly with some other constraints. This will make it easy for a teacher to create a lesson. They will first interact with the system in the L2, to create a set of recorded examples which work correctly. When the student logs on, the system will then be set to select prompts matching the teacher's examples.

  • The functionality will be bundled up as a Prolog-based server, which does most of the processing, and will connect to a lightweight Java-based GUI which presents a client view. The server will initially handle two messages: (1) NEXT_EXAMPLE, returning a new interlingua prompt with associated information, and (2) RECOGNISE, prompting the student to speak, carrying out recognition, and returning the pieces of information produced by carrying out the interlingua comparison process.

Monday 10 August 2009

Initial English recognizer for CALL-SLT (Part 2)

I have done some more work on the English CALL-SLT recognizer, and we now have about 170 surface words. You can order food and drink in various ways, e.g.

i would like two beers
could i have a pizza

There is language for reserving tables, e.g.

do you have a table for two at seven thirty
could i have a non-smoking table for three people

I've also added a few more things like asking for the menu and the check. If I could say all this stuff in a language I didn't already know, say Chinese, I'd really feel I'd learned something useful!

Initial English recognizer for CALL-SLT

We can now build an initial English recognizer for CALL-SLT too. The domain is the same, ordering in restaurants. The training corpus currently contains about 60 examples, about 95% of which parse. Vocabulary is about 130 surface words. Recognition is anecdotally quite good with my voice, though it will of course be more interesting to see how foreign voices do.

If you want more details, here are the corpus and the domain lexicon.

Friday 7 August 2009

Initial French recognizer for CALL-SLT

Pierrette's now checked in some real (as opposed to placeholder) material, and I was able to compile an initial French recognizer for CALL-SLT. It covers some very basic tourist French, so far all about ordering in restaurants. Simple as it is, I was already able to use it to improve my pronunciation of "Je voudrais un verre d'eau". As everyone who's heard me speak French knows, my version of the "r" sound is terrible. Well, at least I can roll my eyes.

If people want to look at the CALL-SLT files, they're at http://callslt.cvs.sourceforge.net/viewvc/callslt/CALL-SLT/

Tuesday 4 August 2009

Adding Romance to Regulus (part 2)

I've just checked in initial French files under CALL-SLT/Fre. Update CALLSLT with the -d flag to get them.

So far, the CALL-SLT files are pretty much the same as the MedSLT files they were adapted from. I'm assuming Pierrette will make the necessary changes! I checked that you can get as far as building a Nuance grammar... it all worked fine for me, but let me know if there are problems.

Monday 3 August 2009

Adding Romance to Regulus

Following discussions with Pierrette, I've now moved the shared Romance grammar from MedSLT/Rom to the new directory Regulus/Grammar/Romance. I've also moved the domain-independent French grammar and lexicon files to Regulus/Grammar/Romance/French. This mirrors the directory structure in Jen's Germanic grammar.

I've adjusted the MedSLT config files for Fre, Spa and Cat to point to the new files. All three languages still appear to build correctly in the AFF (role-marked semantics) versions. I have not done anything with the old (linear semantics) versions, which I am now assuming are obsolete.

All three languages appeared to build fine when I tested, but Pierrette will probably want to check things more carefully. If there are problems, please me know - they should be easy to fix.

The payoff is that I will now be able to construct a config file for the French version of CALL-SLT. Coming up next.