Need an interface for rapid selection without losing focus on primary task

In my application, the user needs to listen to a transcript of a conversation while taking notes. But the user ALSO needs to be able to indicate in some way when the speaker changes.

I have the note taking interface built. Basically, keeping her hands on the keyboard, the user listens and whenever a note is needed, she hits a hotkey to create a new note and place the keyboard focus there, and she can just type the note.

Now I need an interface to allow her to record when the speaker changes. The speakers are known in advance, so all the user has to do is pick one. There will be at most a handful of speakers, certainly not more than 9.

My best idea so far is to number the speakers 1 - 9 and establish a hotkey like Ctrl+N to assign the current speaker as speaker N.

But my concern is that the number is arbitrary and people will have trouble remembering the assignment between number and speaker, especially if the number of speakers is on the high side, like 6 or 7 people.

Is there a better way to do this that I'm missing?