#### Recognizer: Whisper transcribe/translate
Produced By: Whisper: OpenAI, ELAN extension: TLA, Max Planck Institute for Psycholinguistics, Nijmegen

### How to use `Whisper transcribe` and `translate` from within ELAN (locally)

Version 1.2, May 2024

Whisper by OpenAI (https://openai.com/blog/whisper/) is an automatic speech recognition (ASR) system trained on many hours of data collected from the web. This extension implements ELAN's Recognizer API and allows to call a local installation of Whisper to transcribe or translate a media file linked in the current document.

Prerequisite is that Whisper (and everything it depends on) is properly installed and is working correctly when invoked from the command line. If that is the case, the Whisper extension in ELAN's user interface allows to configure the parameters and to start the recognition process.

Extensions based on the Recognizer API are installed in a sub-folder of ELAN's extensions folder. The `whisper` folder contains:

- this `readme.html` file
- an icon image `.png` file
- a `.jar` file with the actual implementation
- a `recognizer.cmdi` file identifying the extension and its parameters to ELAN

The main parameters and parameter categories are:

Parameter type | Parameter id | Description
--- | --- | ---
`<input>` | audio | ELAN automatically preselects the first suitable media file of your current annotation session, but you can change that to other supported files belonging to the session. WAVE audio files are preferred but since the media file is processed by `FFmpeg` other audio or video files probably work as well.
`<textparam>` | run-command | by default the command line command is set to `whisper`. Depending on the platform and on how Python and Whisper etc. are installed and configured this might not work and e.g. the full path to the executable might need to be entered. See the "known issues" below.
`<textparam>` | --task | by default the task to perform is set to `transcribe`. This can be changed to `translate`.
`<textparam>` | --model | by default the model to use is set to `base`. This can be changed to one of the other generic or English language models listed.
`<textparam>` | --language | by default the language is set to `None`, which triggers Whisper to auto-detect the language. This can be changed to one of the languages listed.
`<textparam>` | --word_timestamps | by default this is set to `False`. When set to `True` Whisper (version 03-2023) extracts word-level timestamps and refines the results based on those timestamps (experimetal). See the "known issues" below.
`<numparam>` | --*** | several numerical parameters, please refer to the Whisper documentation for descriptions.
`<output>` | --output_dir | a folder where Whisper should store resulting output files (`.srt`, `.txt`, `.vtt`). If not specified, the files will be stored in whatever the "current directory" is. If the folder does not exist, Whisper will try to create it. Depending on the platform it might be necessary to specify a path with a trailing slash, e.g. `C:\Temp\`.

In case a parameter you would like to use (i.e. would like to change the default value of) is not in the list in `recognizer .cmdi`, you can add it there and it will become an option in the user interface.

There are a few known issues, some depending on the platform:

- If Python, Whisper and other components are installed using Anaconda, the command to run whisper might look like this (Windows example):
  `C:\anaconda3\python.exe C:\anaconda3\Scripts\conda-script.py run --no-capture-output whisper`
  The `--no-capture-output` option is to ensure that output is not captured by Anaconda. Without this option the results will only become available at the very end of the processing.
  If the output is not captured at a lower level, the Report window will be updated regularly with the results for every next 30 seconds of the audio. After clicking the Cancel button to stop the process, the results up to that moment are then still available for adding to the ELAN document.
- If Python, Whisper and/or other components have been installed with Homebrew on macOS (or Linux) and `whisper` can be run from the Terminal, it probably still is necesary to provide the full path as the run-command, e.g.
  `/opt/homebrew/bin/whisper`
  The reason for this is that the environment variables in the Java environment are usually a subset of the variables in the Terminal. This mismatch can also be the cause of the process to fail if `whisper` subsequently cannot find `ffmpeg`.
- If a clear error occurs the process will stop and a message will be shown. But some failures do not trigger the end of the process and the Report window has to be opened to see that something went wrong (and the Cancel button has to be pressed to stop the processing in ELAN).
- It's best to first run a test with the model you wish to use (base, medium, large etc.), on the command line at least once, for the first-time download of the model. It will probably work from within ELAN as well, but the download process will fill the report with hundreds of progress update messages.
- When setting `--word_timestamps` to `True` and the installed version of Whisper supports it (version 03-2023 or higher), the word level segmentation will only be available in some of the exported output files. ELAN currently uses the `.srt` file to extract the word level segmentation and add it as a second tier to the results. Therefore word level segmentation will not be available in partial results, after canceling the recognition process. Also, the `.srt` import function (via the `File->Import` submenu) does not support the Whisper-specific encoding of words in `.srt` yet.