Real time speech to text script - some questions #122
Replies: 9 comments 31 replies
-
|
Beta Was this translation helpful? Give feedback.
-
|
Addon to 1: |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the swift reply and answers! Regarding:
I have an example transcription below. The first is what I was reading and the second is what the script transcribed. I was using the base.en model for real time transcription (by changing the What I was reading:
What was transcribed:
Edit as I'm writing: Actually now that I'm testing it, I see the same behavior even when using the tiny.en model (using the version of the script in my original post which doesn't use a New edit as I'm writing: I was trying to collect some logs for you, however I was having trouble doing that so I combined both scripts into one and logging worked fine (UNI_v_1_0_1.zip). Anyway I noticed that I was getting an error about Anyway, below I'm posting the whole realtimesst.log from a new testing run trying to transcribe the paragraph that I posted above. New transcription:
realtimesst.log: |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for providing additional information. You can ignore the ffmpeg warnings. The amount of skipped / not transcribed text is quite high. Although in earlier versions it would skip a word here and then. But never multiple words in two consecutive sentences. I need to look into that thoroughly, might take me some time. |
Beta Was this translation helpful? Give feedback.
-
|
Some more info
|
Beta Was this translation helpful? Give feedback.
-
Okay, I'll try to reproduce.
I don't think so. Mic probs usually cause other problems in my experience.
Oh, that sounds interesting, I need to try that too.
Yes, that's true. It literally only ensures uppercase and period ends, but does not ensure lowercase in the other way round. After a pause whisper often decides the new "sentence" (or sentence fragment / lowercase) has to be uppercase, because it's the start of a new transcription. I don't see a way to safely determine if both the last sentence was finished and if not if the first word must be uppercase because it's written this way and can't be forced to lowercase even if the last sentence was not finished. |
Beta Was this translation helpful? Give feedback.
-
|
Just tried out the new 0.2.45 version. This specific issue persists. I'm attaching the logs from two test runs similar to the previous ones if you want to take a look. In the |
Beta Was this translation helpful? Give feedback.
-
|
Looked into the logs. 2024-10-02 18:09:38.690 - RealTimeSTT: root - DEBUG - Current realtime buffer size: 74752
2024-10-02 18:10:01.459 - RealTimeSTT: root - DEBUG - Current realtime buffer size: 74752Over 23 seconds no new frames were appended to the current recording audio buffer. 2024-10-02 18:10:02.256 - RealTimeSTT: root - INFO - voice deactivity detected at 18:09:37.161, time since silence start: 25.096 secondsAlso this log message was supposed to appear WAY earlier. Together with the "Audio queue size exceeds latency limit." it looks very much like a somehow blocked _recording_worker. This is only what I see at first glance. It won't set help us track down the problem. Maybe it's needed to write a special _recording_worker with great amount of extended logging to find the lines of code that block the worker. I'm somehow suspicious of the VAD components, but then again they really shouldn't block for like 23 seconds to process a single chunk. |
Beta Was this translation helpful? Give feedback.
-
|
Could you please replace this line: self._check_voice_activity(data_copy)by these lines: time_before_voice_activity_check = time.time()
self._check_voice_activity(data_copy)
logging.debug(f"voice activity checked in: {time.time() - time_before_voice_activity_check} seconds")Here it prints: So the 0.001 here is fine, but if it's greater than 0.05 we could run into realtime problems when checking every chunk for voice activity. |
Beta Was this translation helpful? Give feedback.
-
|
All right, tested it out. Most Again I noticed the full correct transcription appearing pretty early in the terminal, but it stayed white. Then I got the warning again and after that did the final transcription print in yellow. Note that the way I'm running the test is: starting the script, waiting for 'Say something' to appear, then saying "The quick brown fox jumps over the lazy dog.", then wait until the transcription turns yellow and finally typing Ctrl + C to exit the script - btw I noticed you improved the exit behavior, previously I was getting a bunch of errors when exiting. Terminal while waiting for the transcription to turn yellow: Full log: Oh and I've got to test how 0.2.45 works with the 'holes' in the middle of a paragraph. |
Beta Was this translation helpful? Give feedback.
-
|
Okay, I'll think about that. Sorry, I might poke around in the dark a bit right now, because I'm back to the "can't reproduce" point again unfortunately. |
Beta Was this translation helpful? Give feedback.
-
|
Making a new comment thread to say that version 0.2.45 [using What I was saying:
What it transcribed:
Pretty fast too 👍 Also another log because why not |
Beta Was this translation helpful? Give feedback.
-
|
Found something. On CPU if I really stress everything with large-v2 and talking while transcripting a lot a can run into a problem where the pipe seems to be blocked: 2024-10-02 22:31:21.046 - RealTimeSTT: root - DEBUG - Debug: early transcription request pipe send
2024-10-02 22:31:21.061 - RealTimeSTT: root - DEBUG - Realtime text detected: Hey there, this is a little test.
2024-10-02 22:31:21.178 - RealTimeSTT: root - DEBUG - Current realtime buffer size: 54272
2024-10-02 22:31:21.178 - RealTimeSTT: faster_whisper - INFO - Processing audio with duration 00:03.392
2024-10-02 22:31:21.200 - RealTimeSTT: faster_whisper - DEBUG - Processing segment at 00:00.000
2024-10-02 22:31:21.437 - RealTimeSTT: root - DEBUG - Realtime text detected: Hey there, this is a little test.
2024-10-02 22:31:21.538 - RealTimeSTT: root - DEBUG - Current realtime buffer size: 54272
2024-10-02 22:31:21.538 - RealTimeSTT: faster_whisper - INFO - Processing audio with duration 00:03.392
2024-10-02 22:31:21.562 - RealTimeSTT: faster_whisper - DEBUG - Processing segment at 00:00.000
2024-10-02 22:31:21.784 - RealTimeSTT: root - DEBUG - Realtime text detected: Hey there, this is a little test.
[...]
2024-10-02 22:31:26.451 - RealTimeSTT: root - DEBUG - Current realtime buffer size: 54272
2024-10-02 22:31:26.451 - RealTimeSTT: faster_whisper - INFO - Processing audio with duration 00:03.392
2024-10-02 22:31:26.475 - RealTimeSTT: faster_whisper - DEBUG - Processing segment at 00:00.000
2024-10-02 22:31:26.704 - RealTimeSTT: root - DEBUG - Realtime text detected: Hey there, this is a little test.
2024-10-02 22:31:26.810 - RealTimeSTT: root - DEBUG - Current realtime buffer size: 54272
2024-10-02 22:31:26.810 - RealTimeSTT: faster_whisper - INFO - Processing audio with duration 00:03.392
2024-10-02 22:31:26.834 - RealTimeSTT: faster_whisper - DEBUG - Processing segment at 00:00.000
2024-10-02 22:31:27.072 - RealTimeSTT: root - DEBUG - Realtime text detected: Hey there, this is a little test.
2024-10-02 22:31:27.184 - RealTimeSTT: root - DEBUG - Current realtime buffer size: 54272
2024-10-02 22:31:27.184 - RealTimeSTT: faster_whisper - INFO - Processing audio with duration 00:03.392
2024-10-02 22:31:27.207 - RealTimeSTT: faster_whisper - DEBUG - Processing segment at 00:00.000
2024-10-02 22:31:27.432 - RealTimeSTT: root - DEBUG - Debug: early transcription request pipe send returnBetween "early transcription request pipe send" and "early transcription request pipe send return" only this line is executed: self.parent_transcription_pipe.send((audio, self.language))In my infinite naitivity about the nature of python I assumed this call to be nonblocking in every case. Turns out it isn't. Since I already had like 1 trillion problems around using pipes I am thinking about switchin all interprocess communication from pipes to using thread safe queues in the hope that this solves it. Will be another quite substantial refactoring session. Can't see any other solution right now. |
Beta Was this translation helpful? Give feedback.
-
|
Checked in another version of audio_recorder.py working with a polling thread for the pipe connection into dev branch. Looked good here in first test. |
Beta Was this translation helpful? Give feedback.
-
|
Issue persists in 0.3.0 test logs (hey I learned how to fork!) |
Beta Was this translation helpful? Give feedback.
-
|
Okay, let's look deeper. I currently can't see much in the logs. Could you set use_extended_logging=True to get the detailed log output? Also I would like to know if it works better with early_transcription_on_silence=0. |
Beta Was this translation helpful? Give feedback.
-
|
Sure, here are the logs. early_transcription_on_silence=0.2 One thing to note here is that in the README you specify
Curiously enough that's what o1-preview suggested as well. It didn't help though. |
Beta Was this translation helpful? Give feedback.
-
|
Looked into the early_transcription_on_silence=0 file. The only occasion where the _recording_worker was "blocked" for more than 0.1s was here: 2024-10-03 09:49:21.548 - RealTimeSTT: root - DEBUG - Debug: Entering inner try block
2024-10-03 09:49:21.548 - RealTimeSTT: root - DEBUG - Debug: Trying to get data from audio queue
2024-10-03 09:49:21.665 - RealTimeSTT: root - DEBUG - Debug: Queue is empty, checking if still running
2024-10-03 09:49:21.704 - RealTimeSTT: root - DEBUG - Debug: Continuing to next iteration
2024-10-03 09:49:21.708 - RealTimeSTT: root - DEBUG - Debug: Entering inner try block
2024-10-03 09:49:21.708 - RealTimeSTT: root - WARNING - ### WARNING: PROCESSING TOOK TOO LONGNo real reason for the 0.160s delay to see here in the logs. 2024-10-03 09:49:21.548 - RealTimeSTT: root - DEBUG - Debug: Trying to get data from audio queue
2024-10-03 09:49:21.665 - RealTimeSTT: root - DEBUG - Debug: Queue is empty, checking if still runningOnly call in between is this: data = self.audio_queue.get(timeout=0.01)Which should have been timeouted after 0.01s and not after 0.12s. Unsure why this happens, might just be some kind of "system blocked a bit due to some other operations" stuff. All in all the 0.160s processing time for this cycle should not have caused any significant issues. So the logging looks not bad so far, can't see any hints of code in _recording_worker that would cause gaps in transcriptions. I'm a bit lost currently, I need to think about that for a while. |
Beta Was this translation helpful? Give feedback.
-
|
Btw where is the main model being used? Is it only for static files? |
Beta Was this translation helpful? Give feedback.
-
|
Hm, even setting it to |
Beta Was this translation helpful? Give feedback.
-
|
Here's what I think is going on: the main model is responsible for the actual final transcription. I remember @WindyYam mentioning that real-time transcription is designed more for preview purposes and doesn't work well at the end of audio recordings [#120]. In the For my purposes, I think the best approach is to set Also, like with 0.2.45, there are no gaps mid-paragraph and the resource hit while it's running is minimal. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, you have a very solid understanding now of what's going on. I agree basically with all points and also with your conclusions about how to best deal with this situation. I depending on the time a single "tiny.en" transcription takes you might consider setting use_main_model_for_realtime to True and set realtime_processing_pause to like 1.0 to get some kind of 1x per second update on the currently spoken text without having to big hit on the performance. Also realtime transcription as you said is mostly because it adds visual response. But real-time transcription you also enables you to implement a more solid sentence end detection. Admitted, this only works with a fast realtime transcription. Let me show you what I mean and explain (maybe, some random other readers might find this valuable too): Sentence.End.Detection.mp4To achieve this, you can use the following code to finetune how the recorder handles speaking pauses (very simple, you might want to finetune): end_of_sentence_detection_pause = 0.4
unknown_sentence_detection_pause = 0.7
mid_sentence_detection_pause = 2.0
prev_text = ""
def realtime_text_detected(text):
global prev_text
sentence_end_marks = ['.', '!', '?', '。']
if text.endswith("..."):
recorder.post_speech_silence_duration = mid_sentence_detection_pause
elif text and text[-1] in sentence_end_marks and prev_text and prev_text[-1] in sentence_end_marks:
recorder.post_speech_silence_duration = end_of_sentence_detection_pause
else:
recorder.post_speech_silence_duration = unknown_sentence_detection_pause
prev_text = textThen you add a prompt for whisper with the initial_prompt parameter like this one: This encourages RealtimeSTT / Whisper to end mid-sentence detection with "..." which we can then detect in the realtime_text_detected method and increase the post_speech_silence_duration duration when that happens. With this changes the user gets more time when he makes a thinking pause mid-sentence while speaking. He can then finish his sentence before the system considers the sentence as being finished too early. |
Beta Was this translation helpful? Give feedback.
-
Yeah I agree.
Hadn't read anything about the prompts, thought they were only applicable to chatbot implementations. Also did some testing with the new test scripts in commit af15476. The new terminal look is great btw. So the updated
|
Beta Was this translation helpful? Give feedback.
-
|
This looks very ugly in the logging: I switched the realtime text color to bright yellow. My guess is you only see that one currently and no main transcription. Wonder what's going wrong here again. Prompts are not completely easy to implement, Whisper tends to put parts of the prompts into the final transcription and gets confused easily. I found the new terminal not working well if the console windows runs completely full of text. Might switch to Textualize/textual instead of rich, not sure. |
Beta Was this translation helpful? Give feedback.
-
Hi guys,
I've been trying to build a real time STT implementation for quite a while and thankfully stumbled upon this project. I'm a pretty amateur programmer with limited knowledge and most of my development has been done with ChatGPT. Despite this I’ve managed to create a script that works quite well for real-time transcription, which I've attached to this thread. Below are some questions to help me understand how the project works better.
System and Setup
realtimestt_test.py. It starts muted, with Ctrl + 1 unmuting the microphone and Ctrl + 2 muting it again. It also performs live typing usingpyautogui.realtimestt_test_v_5_1_0_5.zip
Questions:
Real-Time and Main Model Integration: When using the script in live transcription mode with the
'use_main_model_for_realtime'argument set to false, does the real-time transcription rely solely on the real-time model? Or does the main model contribute at any point? I think I recall reading somewhere that the real-time model handles the first pass, and the main model does the final transcription. How does this process work exactly?Audio Buffer Behavior: I’ve noticed that when speaking for an extended period, the transcription sometimes has gaps (to be more specific, reading a large paragraph to the microphone I'll notice small gaps within the paragraph, it's not specifically the tail-end of the recording that's lost). Even after changing the
ALLOWED_LATENCY_LIMITfrom 10 to 200, this behavior persists.[Note: When I modified the
audio_recorder.pyto change the limit, I directed my script to use the modified version instead of the default one from the RealtimeSTT library).]Could this be related to how the audio buffer is managed internally? Would introducing an external buffer (e.g. with
deque) to capture the microphone input help resolve the issue? My idea is to have the external buffer handle overflow when the internal buffer is full, thus preventing data loss.Model Versions: Is there a significant difference between using the tiny vs tiny.en models for real-time transcription? I’ve been using the .en versions for efficiency, but I'd like to know if there's any notable performance difference.
Script Components: As far as I can tell, the main work is handled by
audio_recorder.py. Is there any other file or process involved in the transcription pipeline, or isaudio_recorder.pythe only one responsible?I’m open to any suggestions or ideas you might have for improving my script. If there’s something I don’t understand, I’ll be happy to look it up myself.
Thanks in advance for your help and clarifications!
Beta Was this translation helpful? Give feedback.
All reactions