-
Notifications
You must be signed in to change notification settings - Fork 857
Realtime Client Proposal #7285
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
tarekgh
wants to merge
77
commits into
dotnet:main
Choose a base branch
from
tarekgh:RealtimeClientProposal
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Realtime Client Proposal #7285
Changes from all commits
Commits
Show all changes
77 commits
Select commit
Hold shift + click to select a range
4c04354
Realtime Client Proposal
d6d6671
Replace [Experimental("MEAI001")] with DiagnosticIds.Experiments.AIRe…
07947cc
Apply suggestions from code review
tarekgh f2fe352
Remove AsIRealtimeClient extension method
4f9f05d
Address PR review comments
ff14b19
Fix multiple issues found during code review
770fa28
Remove duplicate internal FunctionInvocationStatus enum
3cbb1d2
Remove stale CreateToolsMap calls before the loop
5511b8d
Remove FunctionInvocationResultInternal; use public FunctionInvocatio…
fbdc7cb
Add IAsyncDisposable to IRealtimeSession; store and await receive tas…
f33e790
Revert API baseline JSON changes
7dad147
Clarify audio/text token counts are subsets of Input/OutputTokenCount
d66e6fe
Replace ToolChoiceMode enum with ChatToolMode for cross-modality cons…
fc5341e
Unify TranscriptionOptions across Realtime and SpeechToText
77449ba
Use HasTopLevelMediaType instead of StartsWith for audio check
f7fe2ef
Add API compat suppressions for SpeechToTextOptions breaking changes
a6b4ccd
Replace string Eagerness with SemanticEagerness struct (ChatRole patt…
03d7648
Remove InternalsVisibleToTest from Microsoft.Extensions.AI.OpenAI
5845991
Add RawRepresentationFactory to RealtimeSessionOptions
63531b5
Remove OpenAI-specific tracing properties from RealtimeSessionOptions
f333eaa
Remove AIFunction/HostedMcpServerTool properties, use ChatToolMode
4a6947e
Improve XML docs for Usage on response and transcription messages
29bbdca
Remove leftover temp files
ab0e74d
Improve ConversationId XML docs on RealtimeServerResponseCreatedMessage
39a042d
Split Text/Audio properties on RealtimeServerOutputTextAudioMessage
46ae23d
Replace RealtimeServerMessageType enum with readonly struct
156b376
Move Parameter into ErrorContent.Details on error messages
f59afd6
Add RawRepresentation property to RealtimeContentItem
e8393c1
Rename Metadata to AdditionalProperties for consistency
6aae7d3
Improve MaxOutputTokens XML docs to clarify modality scope
c7f041b
Improve XML docs on RealtimeClientResponseCreateMessage properties
cde880a
Improve RealtimeClientResponseCreateMessage class-level XML doc
d2516c2
Rename EventId to MessageId for terminology consistency
1d15400
Remove blank line between using directives in audio buffer messages
0ca0a17
Rename RealtimeAudioFormat.Type to MediaType for consistency
03014ce
Refactor IRealtimeSession: rename InjectClientMessageAsync to SendCli…
02fd006
Make RealtimeSessionOptions and related types immutable with init-onl…
aa17989
Fix XML doc param order, add null validation to constructors
e24108b
Merge branch 'main' into RealtimeClientProposal
tarekgh c7dbcf8
OpenAI Realtime Provider using OpenAI SDK
756fd75
Fix typo: add missing space in comment ('IDsince' -> 'ID since')
5c61540
Make CreateSessionAsync return non-nullable IRealtimeSession
82e0b4c
Remove SemanticEagerness from Abstractions; move eagerness to Additio…
dfb684a
Include CompatibilitySuppressions for SpeechToTextOptions API changes
faa6cfb
Remove MCP-specific types from abstractions; add provider contract docs
1be4dba
Remove trailing blank line in RealtimeServerErrorMessage
381dc44
Clarify ErrorMessageId XML doc to distinguish from base MessageId
e186ab8
Rename ErrorMessageId to OriginatingMessageId for clarity
f752d83
Change ExcludeFromConversation from bool to bool? for provider-defaul…
8a6754d
Remove blank lines between Experimental attribute and class declaration
7c5eeb2
Add null validation to Content property setter
d8e8e5f
Make RealtimeAudioFormat.SampleRate non-nullable and remove misleadin…
621d49d
Rename IRealtimeSession to IRealtimeClientSession and SendClientMessa…
6d50fec
Fix null DefaultConversationConfiguration when ExcludeFromConversatio…
7269c15
Remove IDisposable from IRealtimeClientSession, keep only IAsyncDispo…
8843a12
Remove ConversationId from RealtimeServerResponseCreatedMessage
f372c42
Rename ResponseCreateMessage and ConversationItemCreateMessage to Cre…
0c8fe65
Rename RealtimeContentItem to RealtimeConversationItem
df8d2c0
Remove VoiceSpeed from RealtimeSessionOptions and use RawRepresentati…
0e9c3b8
Remove UpdateAsync from IRealtimeClientSession, use message-based app…
5ffae0c
Rename realtime session classes to include Client in name
ddd058b
Simplify DisposeAsyncCore to return ValueTask directly
0161bf7
Convert RealtimeSessionKind from enum to extensible string struct
d16fa7c
Remove NoiseReductionOptions from abstraction
a2dd558
Remove VoiceActivityDetection hierarchy from abstraction
0f160d2
Remove AnonymousDelegatingRealtimeClientSession and anonymous Use ove…
2f4a0b2
Fix JSON injection in MessageId by using JsonSerializer.Serialize
50a773c
Remove meaningless duration metric from SendAsync in OTel session
5932110
Refactor builder to operate on IRealtimeClient instead of IRealtimeCl…
39ad4b0
Rename message classes: move RealtimeClient/RealtimeServer before Mes…
1013c1d
Merge branch 'upstream-main' into RealtimeClientProposal
b8d5741
Fix cancellation tests to also accept WebSocketException on net462
51213de
Remove PreviousId from CreateConversationItemRealtimeClientMessage as…
056f19b
Document that ResponseId may be null and response lifecycle events ma…
2075355
Document that CreateResponseRealtimeClientMessage may be a no-op for …
712eb7e
Add RealtimeResponseStatus constants and document interruption/barge-…
046a71b
Remove session parameter from RealtimeSessionOptions.RawRepresentatio…
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
29 changes: 29 additions & 0 deletions
29
...rosoft.Extensions.AI.Abstractions/Realtime/CreateConversationItemRealtimeClientMessage.cs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| // Licensed to the .NET Foundation under one or more agreements. | ||
| // The .NET Foundation licenses this file to you under the MIT license. | ||
|
|
||
| using System.Diagnostics.CodeAnalysis; | ||
| using Microsoft.Shared.DiagnosticIds; | ||
| using Microsoft.Shared.Diagnostics; | ||
|
|
||
| namespace Microsoft.Extensions.AI; | ||
|
|
||
| /// <summary> | ||
| /// Represents a real-time message for creating a conversation item. | ||
| /// </summary> | ||
| [Experimental(DiagnosticIds.Experiments.AIRealTime, UrlFormat = DiagnosticIds.UrlFormat)] | ||
| public class CreateConversationItemRealtimeClientMessage : RealtimeClientMessage | ||
| { | ||
| /// <summary> | ||
| /// Initializes a new instance of the <see cref="CreateConversationItemRealtimeClientMessage"/> class. | ||
| /// </summary> | ||
| /// <param name="item">The conversation item to create.</param> | ||
| public CreateConversationItemRealtimeClientMessage(RealtimeConversationItem item) | ||
| { | ||
| Item = Throw.IfNull(item); | ||
| } | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets the conversation item to create. | ||
| /// </summary> | ||
| public RealtimeConversationItem Item { get; set; } | ||
| } |
125 changes: 125 additions & 0 deletions
125
...ries/Microsoft.Extensions.AI.Abstractions/Realtime/CreateResponseRealtimeClientMessage.cs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,125 @@ | ||
| // Licensed to the .NET Foundation under one or more agreements. | ||
| // The .NET Foundation licenses this file to you under the MIT license. | ||
|
|
||
| using System.Collections.Generic; | ||
| using System.Diagnostics.CodeAnalysis; | ||
| using Microsoft.Shared.DiagnosticIds; | ||
|
|
||
| namespace Microsoft.Extensions.AI; | ||
|
|
||
| /// <summary> | ||
| /// Represents a client message that triggers model inference to generate a response. | ||
| /// </summary> | ||
| /// <remarks> | ||
| /// <para> | ||
| /// Sending this message instructs the provider to generate a new response from the model. | ||
| /// The response may include one or more output items (text, audio, or tool calls). | ||
| /// Properties on this message optionally override the session-level configuration | ||
| /// for this response only. | ||
| /// </para> | ||
| /// <para> | ||
| /// Not all providers support explicit response triggering. Voice-activity-detection (VAD) driven | ||
| /// providers may respond automatically when speech is detected or input is committed, in which case | ||
| /// this message may be treated as a no-op. Per-response overrides (instructions, tools, voice, etc.) | ||
| /// are advisory and may be silently ignored by providers that do not support them. | ||
| /// </para> | ||
| /// </remarks> | ||
| [Experimental(DiagnosticIds.Experiments.AIRealTime, UrlFormat = DiagnosticIds.UrlFormat)] | ||
| public class CreateResponseRealtimeClientMessage : RealtimeClientMessage | ||
| { | ||
| /// <summary> | ||
| /// Initializes a new instance of the <see cref="CreateResponseRealtimeClientMessage"/> class. | ||
| /// </summary> | ||
| public CreateResponseRealtimeClientMessage() | ||
| { | ||
| } | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets the list of the conversation items to create a response for. | ||
| /// </summary> | ||
| public IList<RealtimeConversationItem>? Items { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets the output audio options for the response. | ||
| /// </summary> | ||
| /// <remarks> | ||
| /// If set, overrides the session-level audio output configuration for this response only. | ||
| /// If <see langword="null"/>, the session's default audio options are used. | ||
| /// </remarks> | ||
| public RealtimeAudioFormat? OutputAudioOptions { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets the voice of the output audio. | ||
| /// </summary> | ||
| /// <remarks> | ||
| /// If set, overrides the session-level voice for this response only. | ||
| /// If <see langword="null"/>, the session's default voice is used. | ||
| /// </remarks> | ||
| public string? OutputVoice { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets a value indicating whether the response output should be excluded from the conversation context. | ||
| /// </summary> | ||
| /// <remarks> | ||
| /// When <see langword="true"/>, the response is generated out-of-band: the model produces output | ||
| /// but the resulting items are not added to the conversation history, so they will not appear | ||
| /// as context for subsequent responses. | ||
| /// If <see langword="null"/>, the provider's default behavior is used. | ||
| /// </remarks> | ||
| public bool? ExcludeFromConversation { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets the instructions that guide the model on desired responses. | ||
| /// </summary> | ||
| /// <remarks> | ||
| /// If set, overrides the session-level instructions for this response only. | ||
| /// If <see langword="null"/>, the session's default instructions are used. | ||
| /// </remarks> | ||
| public string? Instructions { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets the maximum number of output tokens for the response, inclusive of all modalities and tool calls. | ||
| /// </summary> | ||
| /// <remarks> | ||
| /// This limit applies to the total output tokens regardless of modality (text, audio, etc.). | ||
| /// If <see langword="null"/>, the provider's default limit is used. | ||
| /// </remarks> | ||
| public int? MaxOutputTokens { get; set; } | ||
tarekgh marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| /// <summary> | ||
| /// Gets or sets any additional properties associated with the response request. | ||
| /// </summary> | ||
| /// <remarks> | ||
| /// This can be used to attach arbitrary key-value metadata to a response request | ||
| /// for tracking or disambiguation purposes (e.g., correlating multiple simultaneous responses). | ||
| /// Providers may map this to their own metadata fields. | ||
| /// </remarks> | ||
| public AdditionalPropertiesDictionary? AdditionalProperties { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets the output modalities for the response (e.g., "text", "audio"). | ||
| /// </summary> | ||
| /// <remarks> | ||
| /// If set, overrides the session-level output modalities for this response only. | ||
| /// If <see langword="null"/>, the session's default modalities are used. | ||
| /// </remarks> | ||
| public IList<string>? OutputModalities { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets the tool choice mode for the response. | ||
| /// </summary> | ||
| /// <remarks> | ||
| /// If set, overrides the session-level tool choice for this response only. | ||
| /// If <see langword="null"/>, the session's default tool choice is used. | ||
| /// </remarks> | ||
| public ChatToolMode? ToolMode { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets the AI tools available for generating the response. | ||
| /// </summary> | ||
| /// <remarks> | ||
| /// If set, overrides the session-level tools for this response only. | ||
| /// If <see langword="null"/>, the session's default tools are used. | ||
| /// </remarks> | ||
| public IList<AITool>? Tools { get; set; } | ||
| } | ||
68 changes: 68 additions & 0 deletions
68
src/Libraries/Microsoft.Extensions.AI.Abstractions/Realtime/DelegatingRealtimeClient.cs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| // Licensed to the .NET Foundation under one or more agreements. | ||
| // The .NET Foundation licenses this file to you under the MIT license. | ||
|
|
||
| using System; | ||
| using System.Diagnostics.CodeAnalysis; | ||
| using System.Threading; | ||
| using System.Threading.Tasks; | ||
| using Microsoft.Shared.DiagnosticIds; | ||
| using Microsoft.Shared.Diagnostics; | ||
|
|
||
| namespace Microsoft.Extensions.AI; | ||
|
|
||
| /// <summary> | ||
| /// Provides an optional base class for an <see cref="IRealtimeClient"/> that passes through calls to another instance. | ||
| /// </summary> | ||
| /// <remarks> | ||
| /// This is recommended as a base type when building clients that can be chained around an underlying <see cref="IRealtimeClient"/>. | ||
| /// The default implementation simply passes each call to the inner client instance. | ||
| /// </remarks> | ||
| [Experimental(DiagnosticIds.Experiments.AIRealTime, UrlFormat = DiagnosticIds.UrlFormat)] | ||
| public class DelegatingRealtimeClient : IRealtimeClient | ||
| { | ||
| /// <summary> | ||
| /// Initializes a new instance of the <see cref="DelegatingRealtimeClient"/> class. | ||
| /// </summary> | ||
| /// <param name="innerClient">The wrapped client instance.</param> | ||
| /// <exception cref="ArgumentNullException"><paramref name="innerClient"/> is <see langword="null"/>.</exception> | ||
| protected DelegatingRealtimeClient(IRealtimeClient innerClient) | ||
| { | ||
| InnerClient = Throw.IfNull(innerClient); | ||
| } | ||
|
|
||
| /// <inheritdoc /> | ||
| public void Dispose() | ||
| { | ||
| Dispose(disposing: true); | ||
| GC.SuppressFinalize(this); | ||
| } | ||
|
|
||
| /// <summary>Gets the inner <see cref="IRealtimeClient" />.</summary> | ||
| protected IRealtimeClient InnerClient { get; } | ||
|
|
||
| /// <inheritdoc /> | ||
| public virtual Task<IRealtimeClientSession> CreateSessionAsync( | ||
| RealtimeSessionOptions? options = null, CancellationToken cancellationToken = default) => | ||
| InnerClient.CreateSessionAsync(options, cancellationToken); | ||
|
|
||
| /// <inheritdoc /> | ||
| public virtual object? GetService(Type serviceType, object? serviceKey = null) | ||
| { | ||
| _ = Throw.IfNull(serviceType); | ||
|
|
||
| // If the key is non-null, we don't know what it means so pass through to the inner service. | ||
| return | ||
| serviceKey is null && serviceType.IsInstanceOfType(this) ? this : | ||
| InnerClient.GetService(serviceType, serviceKey); | ||
| } | ||
|
|
||
| /// <summary>Provides a mechanism for releasing unmanaged resources.</summary> | ||
| /// <param name="disposing"><see langword="true"/> if being called from <see cref="Dispose()"/>; otherwise, <see langword="false"/>.</param> | ||
| protected virtual void Dispose(bool disposing) | ||
| { | ||
| if (disposing) | ||
| { | ||
| InnerClient.Dispose(); | ||
| } | ||
| } | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.