AnnotationGuide
Contents
Guide to Annotation with TotalRecall
Introduction
TotalRecall is a program for marking word onsets when scoring free recall and recognition data. The program displays the wordpool below the speech waveform to facilitate this process. Below are instructions for how to properly annotate data. When annotating, it is crucial to remain consistent; therefore, refer to these instructions whenever you are unsure of a word onset.
Installation
The most recent version of the program can be found through our TotalRecall page. Download the `.dmg` file (if you are working on a Mac) and drag the app file into your Applications folder. (If you are not an Administrator on your machine, ask an RA to authenticate this action.)
Files
In normal operation, the three file types associated with audio annotation are as follows.
Kahana Lab members: A script called pinch.sh will download and organize the following types of files for each session you annotate. See this page for its operation.
Wordpool file
This must be a `.txt` file containing all possible normal vocal responses from the subject. All words found in the `.lst` files (see below) must be contained in this wordpool. Usually, this is all words that may be presented to any subject for a given experiment.
Kahana Lab: `pinch.sh` puts the wordpool for each subject in a given experiment in that subject's folder for said experiment; e.g., in `parse_files/ltpFR/LTP046/wordpool.txt`. See the pinch page for more information.
.wav files
These are the audio files that are fed into the annotation program.
Kahana Lab: For our current set of experiments, each audio file consists of a subject's vocal responses for one "trial", i.e., one list of words that they are recalling/recognizing. (An exception is `ffr.wav`, which is not present in all sessions. In this file, the subject is recalling every word presented to him/her throughout the session.)
.lst files
A `.lst` file displays the words that the participant viewed during the particular trial that you are scoring. There is a .lst file for every trial, every session. To be read correctly by TotalRecall, `.lst` files must have the same time as their corresponding `.wav` file; e.g., `15.lst` and `15.wav` would correspond to the same trial within a session. The `.wav` and `.lst` files must be in the same directory.
When you open an audio file in TotalRecall, the software highlights the words that are contained in the corresponding `.lst` file in bold in the "Wordpool" window.
.ann files
These are the output files of the annotation program. They display the words recalled with the respective times of recall (in milliseconds) within the recall period. .ann files can be edited like text documents, but it is best to avoid changing any of the text in a `.ann` file.
.tmp files
Before an annotated or partially-annotated file is marked complete, it exists as a `.tmp` file, with the same name as the `.wav` file. When you press "Mark Complete" in TotalRecall after annotating a `.wav` file, it will convert the `.tmp` file to a `.ann` file.
If onset-detection or word-recognition scripts are used (not described here for sake of generality/brevity), they should generate a `.tmp` file which is automatically loaded with the corresponding audio file.
Annotation Procedure
Starting an annotation session
There are several ways to load a multi-trial session into TotalRecall to begin annotation. This is one example.
- The examples that follow assume Mac keybindings. That is, they make reference to the Command key, ⌘. Use the modifier key appropriate to your operating system, e.g., Ctrl on Windows.
1. Open the application TotalRecall
- If you are using a Mac and followed the procedure in "Installation" above, it will be in your Applications folder.
2. Open a session folder with the command "Open Audio Folder..." in the "File" menu.
- You can also use the keyboard shortcut Option + Command + O, or ⌥⌘O.
- If you only have one audio file to annotate, you may also use the "Open Audio File..." command from the File menu, or use ⌘O.
3. Open the corresponding wordpool, from the File menu (or using Shift + Command + O, ⇧⌘O).
You are now ready to start annotating!
Marking Onsets
Select an audio file from the "Audio Files" list by double-clicking on it. You will see the waveform appear in the top half of the window, and it is now available for playback.
You must now find and mark the onset of each vocal response. The space bar starts and stops playback. When you get to a response, stop playback and navigate to the beginning of the word using the arrow keys.
Command + Z (⌘Z) is an extremely useful command. It will replay the 200 msec prior to the cursor position. When you move the cursor to the beginning of the word, press ⌘Z to check the position. Listen carefully: you should just barely hear the beginning of the word. What this sounds like exactly will depend on the first phoneme of the word in question (i.e., consonants will sound different from vowels, etc.). The cursor is in the correct position when you cannot hear any part of the word using ⌘Z, but when you press → then ⌘Z you can just barely hear the first phoneme of the word.
Once the cursor is in the correct position, type the vocal response. If it is in the wordpool, you may begin typing, then press "tab" to complete. Note that tab will complete with the top response in the list below the text field.
Rules for marking the word onset
- Mark the onset as soon as you hear a vocalization (even if it doesn't quite sound like a speech sound).
- The annotation mark should go 5 ms BEFORE the onset of the vocalization (if you press ⌘Z at the annotation mark, you should not hear the onset. If you move forward 5ms--one arrow key--you should hear it).
- If the person stutters "d..dog," mark the onset at the first "d.."
- If the word lasts audibly longer than 1 second, score the beginning of the word as usual, then put a vocalization mark 1 second (command + shift + forward arrow moves you 500ms) after the beginning of the word (even if the word is only just slightly past the 1 second mark).
- If the word lasts audibly longer than 1 second and is followed by a vocalization, score the beginning of the word, add a vocalization mark 1 second after, and another vocalization mark at the beginning of the vocalization.
- If the person says "ma...(as in man)...boy," score "ma" as a vocalization and mark the onset of the word at the first sound of "b."
- If the person says "man...no...boy," score "man" as an intrusion, "no" as a vocalization, and "boy" as a correct recall.
- If the person says "boy...or something like boy," score the first "boy" as a correct recall, "or something like" as a vocalization, and the second "boy" as a correct recall.
- In general, if the person says a word that is in the word pool, mark it, even if appears in a vocalization or a whisper (Ex: "farmer...no, not farmer...shepherd": if "farmer" is in the word pool, mark "farmer" as a recall twice, "no, not" as a vocalization, and also "shepherd" as a recall).
- If the person says two words so that the first word blends into the second, use your best judgement, but try to mark it such that both words sound like complete words (try not to cut off the first word by putting the mark for the second too early). An easy way to listen to each word is to click and drag over the area you want to listen to.
- If the person mispronounces a word and then corrects themselves, mark the first word (the mispronounced one) as an intrusion and the second as the corrected word. If the person repeats two words without mispronouncing them, however, mark both as the correct word.
Identifying the word
When a word is unclear (i.e. the word could be 'sat' or 'cat'), you should always err on the side of words that are part of the wordpool ('cat' in this example). If you are choosing between two words in the wordpool, err on the side of words that are in the bolded set of words (those are the words on the list that the participant is supposed to be recalling, whereas the non bolded words are from other lists in the session). If you are sure the word isn't in the wordpool, mark it as an intrusion (see "Rules for marking vocalizations and intrusions" below). Finally, if the word is really unintelligible and you have no idea what the participant said, mark it as a vocalization.
There are instances when a participant will pluralize a word when recalling it. In these instances, you should ignore the plural and just score it as being the singular version of the word in the wordpool. This also applies to words like 'phone' (i.e. if the person says 'telephone,' score it as 'phone' rather than an intrusion). This would only apply for longer words that contain the desired word. However, if the word was "coat" and the participant says 'jacket', 'jacket' would not count as a correct recall and should be scored as an intrusion.
Rules for vocalizations and intrusions
Intrusions are recalled words that are not in the participant's wordpool. If you think that someone made an intrusion, first check the .lst file and then look through the wordpool to see if there are any words that sound really similar to what you heard. You can easily do this by scrolling through the wordpool list or just by starting to type the word into the box. If the word really wasn't on the list, you can score it as an intrusion, by typing in the word you think the participant said and hitting Command + Shift + Enter (⌘⇧⏎).
Sometimes people will mumble to themselves or say things like "No...wait...wrong word" or "umm..." These should not be scored as intrusions, but as vocalizations. To do this, find the beginning of the vocalization (doesn't have to be quite as exact as a recalled word, but should be pretty close) and hit Command + Shift + Enter (⌘⇧⏎) without typing anything into the box. This should make a "<>" annotation mark. Do not ignore vocalizations because EEG data during vocalization periods should not be examined. For vocalizations lasting longer than 1 second, score a new '<>' at every full second of the vocalization. (If default settings are used, pressing ⇧⌘→ twice will move the cursor forward one second.)
Rules for Empty Lists
If a participant does not recall any of the words on the list, and does not make any vocalizations or intrusions (in other words, you haven't made any annotation marks for a `.wav` file), scroll forward to the last second of that list and make a vocalization mark ('<>'; shift-command-enter (⌘⇧⏎)). Unless you make this mark at the end of an empty list, Penn Total Recall will give you an error when you try to press "Mark Complete". In addition, make a note of any blank lists and tell the research specialist the participant number and session. Similarly, if a participant has multiple low recalls (less than 3 words per list) for a session, also tell the research specialist the participant number and session. If this becomes an ongoing problem for a certain participant they may need a reminder to try harder and if there still is no improvement, they may need to be booted from the experiment.
Finishing a Session
When all responses, intrusions, and vocalizations have been marked, and the cursor has moved to the very end of the file, you may mark the session as complete. Simply press the space bar to stop playback, then click the "Mark Complete" button. (You may also press Command + Shift + D, ⌘⇧D.)
Annotation commands
The following are commands central to using TotalRecall. Note that time intervals are default values, and may be changed by the user.
- Space: Start and stop playback. (Must also be pressed when the cursor reaches the end of the audio file before marking it as complete.)
- Enter: Commit text as normal annotation
- Command + Shift + Enter, ⌘⇧⏎: Commit text as intrusion or mark vocalization
- Arrow keys, ← and →: Move the cursor 5 milliseconds back and forward, respectively
- Command + arrow, ⌘← and ⌘→: Move the cursor 50 msec back and forward, respectively
- Shift + Command + arrow, ⇧⌘← and ⇧⌘→: Move the cursor 500 msec back and forward, respectively
- Shift + arrow, ⇧← and ⇧→: Move the cursor between existing annotation marks
- Command + Backspace or Command + Delete: Delete selected annotation (use ⇧← and ⇧→ to navigate and select)
- Command + L, ⌘L: Undo play, i.e., go to cursor position before playback
All other key bindings may be found in the program drop-down menu, under File > Edit Keyboard Shortcuts.
Annotation / TotalRecall FAQ
Have a question? Email `memory - software @ psych . upenn . edu` (remove all the spaces).