MedSLT almost fully converted to AFF

I've made a lot of progress this week on converting the bidirectional English/Spanish version of MedSLT to AFF format. I've parameterized the Spanish system to support AFF representations, and added AFF versions of most of the necessary config file and scripts. In particular,

• You can build a full AFF version of Spa by doing 'make role_marked' in Spa/scripts.
• You can run interactive bidirectional AFF Eng/Spa text systems using the files load_bidirectional_role_marked.pl and load_bidirectional_restricted_role_marked.pl in EngSpa/scripts.
  
• There are targets in EngSpa/scripts/Makefile for running AFF versions of the QA corpus, both plain and restricted, with the obvious naming conventions.

I've done preliminary testing, and everything should be checked in. There are still things missing, e.g. nothing so far in Spa/Spa for checking back-translation, but I figured it would be best at this point to hand over to Pierrette, so that she can refine the rules. I've only made some absolutely minimal changes, enough to check that a few sentences go through.

When this piece of work is finished, all of MedSLT should be available in AFF format, which means that we'll be able to retire the old linear version and only support one system. I'm hopeful that build times will then be low enough for us to go back to building and testing the system every night, as we used to do.

Interlingua corpora for multiple domains

Following a discussion with Pierrette last week, I have added two more MedSLT Interlingua corpora, for the chest pain and abdominal pain domains. I've also added all the associated config files, scripts etc for the currently relevant language pairs (EngInt, JapInt, IntEng, IntFre and IntJap), so it should now possible to do systematic interlingua-centered development for all three domains. I have only built AFF versions, since we're planning to retire the linear formalism soon.

The naming conventions are the usual ones. I hopefully managed to check everything in, but let me know if files that you expected to find are missing. Pierrette should at some point do some work tidying up IntFre and FreInt and Yukie should do the same for IntJap and JapInt. Further down the line, we should really add coverage for these domains in the missing languages.

AFF version of Catalan

I've added initial versions of all the files needed for the AFF version of Catalan in MedSLT. Naming conventions are the usual ones, and I was able to build all the AFF Cat resources by doing

make role_marked

in the Cat/scripts directory. There should now be config files for all 5 x 5 = 25 pairs of languages in {Ara, Cat, Eng, Fre, Jap} - this involved adding a few new pairs. I only tested Interlingua to Catalan and Catalan to Interlingua. We get currently translations for about 75% of the sentences in IntCat, and about 20% in CatInt. Hopefully it will be easy to improve these figures.

Over to Pierrette and Bruna to debug the rules. Note that I have macrotised the Cat lexicon to make the AFF version work. It should be mostly OK, but there were a few cases (in particular, WH+ PPs) where I wasn't quite sure how to do the macrotisation - people who actually know Catalan should review the entries.

Interlingua corpora

Over the last few months, we have been moving MedSLT development towards a new way of doing things, which is based on the idea of an "Interlingua corpus". We present the basic picture in our LREC 2008 paper, but that's already somewhat out of date, and doesn't give any low-level details.

We now have four interlingua corpora, representing the cross-produce of {linear, AFF} x {plain, combined}. The linear/AFF distinction is concerned with the type of semantics used. "Linear" is the old MedSLT semantics; AFF semantics is explained in the paper by Pierrette, Beth Ann, Yukie and myself which has just been accepted for COLING 2008, and which will soon be appearing on the Geneva website.

The plain/combined distinction says what information has been incorporated in the corpus. The "plain" corpus is created by merging results of translating FROM each source language into interlingua, so each interlingua form lists the source language results that translate into it. The "combined" corpus contains all the information in the "plain" corpus, plus also the results of translating TO each target language.

At the moment, we use the plain corpus for developing translations rules that go from Interlingua to target languages. The combined corpus is used for creating help resources.

All the scripts used to build interlingua corpora are referenced in $MED_SLT2/Interlingua/scripts/Makefile.

Building help resources from the combined interlingua corpus (2)

Considerable progress on this task today:

1. I've added French to the AFF interlingua corpora, including the New York material as requested by Pierrette. The corpora are remade and checked in.
2. The help resources for Eng and Ara (the languages where we have help class definitions) are now made from the combined interlingua corpus. A separate help file is made for each of the six pairs EngAra, EngFre, EngJap, AraEng, AraFre, AraJap, reflecting the different levels of coverage. You can make the help resources for all of these pairs by doing 'make help_resources' in $MED_SLT2 (i.e. at the top level in the MedSLT directory), and it only takes a few minutes.