Warning: don't use SICStus 4.0.2

Maria and I have just spent two very frustrating days trying to figure out why CALL-SLT wasn't running correctly on her machine. In the end, it turned out that a few bits of Regulus functionality don't work correctly under SICStus 4.0.2, which is the version she was using... there appears to be something wrong with the SICStus/operating system interface.

So avoid this release! 4.0.4 is fine.

Regulus 2.9.0 released

Nikos has just created and uploaded the new 2.9.0 release of Regulus. I tried downloading and running a couple of simple tests in text and speech mode (under SICStus 4.0.3), and Toy1 at least appears to work fine. Please mail me if you notice problems.

Here are the release notes:

MAIN CHANGES TO REGULUS BETWEEN 2.8.0 AND 2.9.0

A large number of new features have been added to Regulus since
version 2.8.0. Most importantly, Regulus now runs under Sicstus 4;
it is possible to use speech input directly from the top-level;
N-best processing is supported in both dialogue and translation mode;
and a new semantics for translation applications has been added.

The new features are listed below in more detail. Not all of them are
fully documented yet, but we are giving priority to adding the
necessary documentation.

- Support for Sicstus 4
  - Regulus runs under Sicstus 4.
    - It has been thoroughly tested under 4.0.2.
    - Some testing has been done under 4.0.3, but this has not yet been carefully
      verified. NOTE: under 4.0.3, it is necessary to load the patch files in
      Prolog/SicstusPatches/4.0.3
  - Regulus still runs under Sicstus 3, and has been thoroughly
    tested under 3.12.5.

- Top-level
  - Errors are now written to stderr
  - There is a version of regulus_batch with an extra argument, which returns
    the list of error outputs created when running the commands.
  - It is possible to compile Nuance grammars from the Regulus top-level
    using the NUANCE_COMPILE command.
  - It is possible to perform speech recognition directly from the top-level
    - The LOAD_RECOGNITION command starts defined speech resources, including
      a license manager, recserver and Regserver
    - After loading resources using LOAD_RECOGNITION, the RECOGNISE command
      takes live speech input and passes it to the current application.
    - Wavfiles are automatically logged by RECOGNISE. The WAVFILES  command
      lists the  most recent recorded wavfiles.
    - When speech resources are loaded, text input of the form

      WAVFILE: 

      performs recognition on , and passes the result to the
      current application

- Java GUI
  - The Java GUI has been greatly improved, and many bugs have been fixed.
  - The GUI supports direct speech input, similar to the Prolog top-level
    described above
  - It is possible to run multiple copies of the GUI at the same time.

- Stepper
  - The commands LOAD, LOAD_GENERATION, EBL_LOAD and EBL_LOAD_GENERATION
    can be invoked from within the stepper.

- Support for spoken dialogue applications
  - When speech resources have been loaded from the command line,
    dialogue corpora can contain items of the form wavfile().
    This makes it possible to test corpora containing a mixture of speech
    and non-speech inputs.
  - Batch processing of speech input in dialogue mode produces figures
    for semantic error rate. An utterance is deemed semantically correct if
    it produces the same dialogue move as the transcription would have done.
  - A timeout has been added in batch dialogue processing, so that processing
    gives up after 10 seconds.
  - If N-best preferences are defined, preference info is printed in
    dialogue mode.
  - Allow dialogue server to take XML-formatted requests

- Generation
  - When the declaration

    regulus_config(prolog_semantics, yes).

    is included, generation grammars can contain arbitrary Prolog structures.

- Translation
  - There is extensive support for translation using both the original
    "linear" semantics, and also the new "Almost Flat Functional" (AFF)
    semantics. AFF is described in our COLING 2008 paper, which will soon
    posted on the Regulus website. Some initial documentation will be added
    to RegulusDoc.htm.
  - It is possible in a translation config file to define an interlingua
    as either a source or a target language. There are many examples
    in the MedSLT project directory.
  - Batch translation produces output files for judging both in Prolog
    and in CSV form. There are new commands for updating judgements from the
    CSV files.
  - When speech resources have been loaded from the command line,
    translation corpora can contain items of the form wavfile().
  - A simple version of N-best processing has been added for applications
    that use interlingual translation with an interlingua grammar. In N-best mode,
    the first utterance producing well-formed interlingua is selected.
  - Interlingua expressions ambiguous according to the interlingua grammar
    are flagged in translation mode.
  - If performing batch translation from Source to Target through Interlingua,
    combine available Source -> Interlingua and Interlingua -> Target
    judgements into Source -> Target judgements if possible.
  - Show average number of generated target language surface forms when
    doing batch translation.
  - Translation conditions can include elements of the form

    context_below()

    This matches an  in a clause.

- Grammar specialisation
  - Fix bug in processing of include_lex declarations.

- Help
  - When defining intelligent help for translation applications, help resources
    can be built from an interlingua corpus.

- Extension to Regulus grammar formalism
  - Allow =@ as synonym for = @
  - Add runtime support for GSL functions strcat/2, add/2, sub/2, neg/1, mul/2, div/2

- English grammar
  - Rules for dates including years have been added.

- Other
  - Tool added to perform random generation from PCFG-trained GSL grammars

Problems with SICStus 4.0.3 resolved

The SICStus people were as usual very responsive, and we now seem to be OK for running under 4.0.3. However, (this is IMPORTANT), you need to install a couple of patch files if you are using that version of SICStus. So, if you're using 4.0.3, do the following:

• Update Regulus from CVS, using the -d option to get new directories.
• Copy the files from Prolog/SicstusPatches/4.0.3 to C:/Program Files/SICStus Prolog 4.0.3/library, or wherever you have your copy of SICStus.

I will set my default version of SICStus to 4.0.3, which means I'll no doubt test it a fair amount over the next few days. I would not recommend people to switch over to 4.0.3 until I've run with it a while and reported on how it's working.

Problems with SICStus 4.0.3

We are unfortunately still having problems with SICStus 4. Things have been more or less stable with 4.0.2, but there were a few rather ugly patches - the SICStus people said things would be better in the next version. Sad to tell, I have just downloaded 4.0.3 and tried it out, and in fact, at least as far as Regulus is concerned, it's gone backwards. Due to new incompatibilities in the operating system interface libraries, it's not currently possible to run Regulus in speech mode with 4.0.3 - there may also be other problems. I can presumably implement a workaround, but the idea of having to patch the code after every new SICStus release makes me very nervous.

For Prolog people who want the low-level details, here is part of the mail I just sent to the SICStus team:

Unless I am misunderstanding something important, SP4.0.3's version of the
system3 library is still not  downward-compatible with SP3's system library, and is in fact rather
less downward-compatible than SP4.0.2's system3. The problem is now in system/1.
In SP4.0.2, system/1 is defined as follows:

system(Cmd) :-
  system_binary(Binary, DashC),
  proc_call(Binary, DashC, Cmd, exit(0)).

so it's possible to make calls like the following, running under Cygwin:

| ?- system('dir > tmp_dir.txt').
      1      1 Call: system('dir > tmp_dir.txt') ?
      2      2 Call: system3:environ('COMSPEC',_790) ?
      2      2 Exit: system3:environ('COMSPEC','C:\\WINDOWS\\system32\\cmd.exe') ?
      3      2 Call: system3:process_create('C:\\WINDOWS\\system32\\cmd.exe',['/C','dir > tmp_dir.txt'],system3:[process(_1437)]) ?
      3      2 Exit: system3:process_create('C:\\WINDOWS\\system32\\cmd.exe',['/C','dir > tmp_dir.txt'],system3:[process('$process'('$ptr IEDNJP'))]) ?
      4      2 Call: system3:process_wait('$process'('$ptr IEDNJP'),exit(0)) ? s
      4      2 Exit: system3:process_wait('$process'('$ptr IEDNJP'),exit(0)) ?
      1      1 Exit: system('dir > tmp_dir.txt') ?

Under SP4.0.3, system/1 is defined thus:

system(Cmd, Status) :-
      shell_exec(Cmd, [], exit(Status)).

and the corresponding call looks like this:

| ?- system('dir > tmp_dir.txt').
      1      1 Call: system('dir > tmp_dir.txt') ?
      2      2 Call: system3:system('dir > tmp_dir.txt',0) ?
      3      3 Call: system3:process_create('dir > tmp_dir.txt',[],system3:[commandline(true),process(_1119)]) ?
      3      3 Exit: system3:process_create('dir > tmp_dir.txt',[],system3:[commandline(true),process('$process'('$ptr ALJLOO'))]) ?
      4      3 Call: system3:process_wait('$process'('$ptr ALJLOO'),exit(0)) ?
      4      3 Fail: system3:process_wait('$process'('$ptr ALJLOO'),exit(0)) ?
      2      2 Fail: system3:system('dir > tmp_dir.txt',0) ?
      1      1 Fail: system('dir > tmp_dir.txt') ?

The problem, as far as I can see, is that process_create requires the first arg
of process_create to be a program, which it isn't here.

Unfortunately, we have people running Regulus under at least 3.12.5, 4.0.2 and 4.0.3.
Maintaining the code so that it runs under all these different versions is
becoming quite difficult - the operating system interface primitives are
absolutely essential. Advice appreciated.