feat: OpenAI realtime api

[vladimir-a-sap]

Context

AI/ai-sdk-java-backlog#386.

Adds OpenAI realtime API support

Feature scope:

• Implemented destination selection, connection and authorization logic
• Added heartbeat functionality to correctly manage conversation connections without unintended TCP resets
• Implemented SDK Realtime API operations
  • Text to Speech
  • Speech to Speech
  • Speech to Text (implemented with a workaround but not exposed because of low reliability, update on the BE side needed)
• Added sample app interactive examples
  • Text to Speech
  • Speech to Speech
  • Speech to Text (a blocker to be resolved on the BE side)
• Added smoke test
  • Text to Speech
  • Speech to Speech
  • Speech to Text

Definition of Done

• Functionality scope stated & covered
• Tests cover the scope above
• Error handling created / updated & covered by the tests above
• Aligned changes with the JavaScript SDK
• Documentation updated
• Release notes updated

[CharlesDuboisSAP]

What do we do about the previous realtime PR? Is it still valid? Why is it abandonned?

• #870

[vladimir-a-sap]

What do we do about the previous realtime PR? Is it still valid? Why is it abandonned?

• feat: PoC OpenAI Realtime API #870

My current understanding is that the mentioned PR cannot be completed to a SDK realtime solution. Here is the context and outcomes of my research why I think so:

There are three ways to implement realtime API with OpenAPI:

• WebRTC
• Websockets
• SIP (telephony)

The previous PR implements SIP - it is a way for client and server to communicate about a connection in order to pass data, but not to establish connection or pass data in fact. The data layer should be implemented separately from the SIP protocol.

OpenAPI SIP protocol by design works with client-side web hooks as a mean to pass data and accept incoming connections (calls). Outgoing calls (client initiated) are not seem to be possible at all using solely SIP API.

We cannot expect SDK environment to have public http endpoints accessible from the internet to expose hooks. AI Core does not implement hook functionality either AFAIK.

Given that WebRTC is difficult to support on the BE and, AFAIK, our backend only supports Websockets at the moment, Websockets seem to be the only option we have.