Added support for gpt4o-realtime models for Speect to Speech interactions

[sharananurag998]

This PR introduces real-time voice pipeline support for OpenAI’s gpt-4o-realtime-preview model, enabling seamless, low-latency speech-to-speech interactions in the Speect framework. The update brings a modern, streaming audio interface, integrated tool execution, and robust event handling—while maintaining full compatibility with the existing STT/TTS pipeline.

---

Key Features & Changes

• RealtimeVoicePipeline:

  • New pipeline for direct, continuous audio-to-audio conversations with OpenAI’s real-time models.
  • Handles streaming microphone input and speaker output at 24kHz, as required by the API.
  • Supports push-to-talk and half-duplex operation to prevent echo/feedback.

• Integrated Tool Calls:

  • Tools are registered with the pipeline and executed automatically when the model requests a function call.
  • Tool results are sent back to the model using the correct OpenAI Realtime API protocol.

• Event Handling & Debugging:

  • Full support for all major OpenAI Realtime API events, including:
    • Audio and text deltas
    • Tool call arguments (streamed and completed)
    • Transcription events (conversation.item.input_audio_transcription.delta and .completed)
    • Session and rate limit updates
  • Example logs all transcription events for easy debugging of what the model “hears.”

• Echo & Feedback Mitigation:

  • Implements a buffer window after assistant audio playback to prevent microphone echo from triggering new turns.
  • Optionally enables server-side noise/echo reduction via input_audio_noise_reduction in the session config.

• Sample Rate Fixes:

  • Ensures both input and output audio are always 24kHz PCM, as required by the OpenAI API (fixes “slow motion” audio bug).

• Backwards Compatibility:

  • All changes are fully compatible with the existing STT/TTS pipeline and configuration.
  • Legacy examples and workflows continue to work without modification.

• Documentation & Examples:

  • Updated docs/voice/pipeline.md with new real-time usage, configuration, and troubleshooting sections.
  • New example: continuous_realtime_assistant.py demonstrates push-to-talk, tool calls, and event handling.

---

🛠️ How to Use

• Realtime Pipeline:
  See the new example and documentation for how to use RealtimeVoicePipeline with your OpenAI API key and tools.
• Classic Pipeline:
  No changes required—existing STT/TTS flows are unaffected.

[dkundel-openai]

Thank you so much for the PR @sharananurag998! I'll try to look at the PR later this week. Thank you for your patience

[sharananurag998]

@dkundel-openai @rm-openai

I haven't found a way for native speech-to-speech integration with an agent, but we can define an agent and use it as a tool in the real-time speech pipeline, and it works!

The agent-as-tool approach provides better latency than the STT-TTS-based VoicePipeline.

Also this branch has Juspay specific MCP tool handling changes since we're using the fork as a python dependency, I'll move it to a separate branch so that main can be merged.

@dkundel-openai you can review the new pipeline and let me know of any changes I'll be happy to work on it.

[EmanueleTribi]

Hi everyone, any news on this pull request or in general timeline to integrate the realtime api? i'm very much interested in using it with the SDK agent and i was wondering if to write my own code or to wait it to be directly integrated. Thanks!
@dkundel-openai @sharananurag998

[github-actions]

This PR is stale because it has been open for 10 days with no activity.

[lobes]

Hi everyone, any news on this pull request or in general timeline to integrate the realtime api? i'm very much interested in using it with the SDK agent and i was wondering if to write my own code or to wait it to be directly integrated. Thanks! @dkundel-openai @sharananurag998

I second this! Would love to have realtime STS support in this SDK.

[github-actions]

This PR is stale because it has been open for 10 days with no activity.

[EmanueleTribi]

Hi everyone, any news on this pull request or in general timeline to integrate the realtime api? i'm very much interested in using it with the SDK agent and i was wondering if to write my own code or to wait it to be directly integrated. Thanks! @dkundel-openai @sharananurag998

I second this! Would love to have realtime STS support in this SDK.

No news but I think on top of that the new Voice Live API has been released (both still in beta) so i don't think they Will integrate unless they released these API in a stable version

[sibblegp]

Darn, I was waiting on exactly this. Does it also do handoffs or do you have to use agent.as_tool()?