本页面介绍了通过 Firebase AI Logic 使用 Live API 时可用的功能，包括：
访问其他页面，了解如何使用各种配置选项（例如添加转写或设置回答语音）自定义实现。您还可以了解如何管理会话。
输入模态
本部分介绍了如何向 Live API 模型发送各种类型的输入。原生音频模型始终需要音频输入（以及可选的其他文本或视频输入模态），并且始终以音频输出进行回答。
流式传输音频输入
|
点击您的 Gemini API 提供商，以查看此页面上特定于提供商的内容和代码。 |
Live API 最常见的功能是双向音频流式传输，即实时流式传输音频输入和输出。
Live API 支持以下音频格式：
- 输入音频格式：原始 16 位 PCM 音频，采样率 16kHz，小端字节序
输出音频格式：原始 16 位 PCM 音频，采样率 24kHz，小端字节序
支持的 MIME 类型：
audio/x-aac、audio/flac、audio/mp3、audio/m4a、audio/mpeg、audio/mpga、audio/mp4、audio/ogg、audio/pcm、audio/wav、audio/webm
如需传达输入音频的采样率，请将每个包含音频的 Blob 的 MIME 类型设置为类似 audio/pcm;rate=16000 的值。
Swift
如需使用 Live API，请创建 LiveModel 实例并将响应模态设置为 audio。
import FirebaseAILogic
// Initialize the Gemini Developer API backend service
// Create a `liveModel` instance with a model that supports the Live API
let liveModel = FirebaseAI.firebaseAI(backend: .googleAI()).liveModel(
modelName: "gemini-2.5-flash-native-audio-preview-12-2025",
// Configure the model to respond with audio
generationConfig: LiveGenerationConfig(
responseModalities: [.audio]
)
)
do {
let session = try await liveModel.connect()
// Load the audio file, or tap a microphone
guard let audioFile = NSDataAsset(name: "audio.pcm") else {
fatalError("Failed to load audio file")
}
// Provide the audio data
await session.sendAudioRealtime(audioFile.data)
var outputText = ""
for try await message in session.responses {
if case let .content(content) = message.payload {
content.modelTurn?.parts.forEach { part in
if let part = part as? InlineDataPart, part.mimeType.starts(with: "audio/pcm") {
// Handle 16bit pcm audio data at 24khz
playAudio(part.data)
}
}
// Optional: if you don't require to send more requests.
if content.isTurnComplete {
await session.close()
}
}
}
} catch {
fatalError(error.localizedDescription)
}
Kotlin
如需使用 Live API，请创建 LiveModel 实例并将响应模态设置为 AUDIO。
// Initialize the Gemini Developer API backend service
// Create a `liveModel` instance with a model that supports the Live API
val liveModel = Firebase.ai(backend = GenerativeBackend.googleAI()).liveModel(
modelName = "gemini-2.5-flash-native-audio-preview-12-2025",
// Configure the model to respond with audio
generationConfig = liveGenerationConfig {
responseModality = ResponseModality.AUDIO
}
)
val session = liveModel.connect()
// This is the recommended approach.
// However, you can create your own recorder and handle the stream.
session.startAudioConversation()
Java
如需使用 Live API，请创建 LiveModel 实例并将响应模态设置为 AUDIO。
ExecutorService executor = Executors.newFixedThreadPool(1);
// Initialize the Gemini Developer API backend service
// Create a `liveModel` instance with a model that supports the Live API
LiveGenerativeModel lm = FirebaseAI.getInstance(GenerativeBackend.googleAI()).liveModel(
"gemini-2.5-flash-native-audio-preview-12-2025",
// Configure the model to respond with audio
new LiveGenerationConfig.Builder()
.setResponseModality(ResponseModality.AUDIO)
.build()
);
LiveModelFutures liveModel = LiveModelFutures.from(lm);
ListenableFuture<LiveSession> sessionFuture = liveModel.connect();
Futures.addCallback(sessionFuture, new FutureCallback<LiveSession>() {
@Override
public void onSuccess(LiveSession ses) {
LiveSessionFutures session = LiveSessionFutures.from(ses);
session.startAudioConversation();
}
@Override
public void onFailure(Throwable t) {
// Handle exceptions
}
}, executor);
Web
如需使用 Live API，请创建 LiveGenerativeModel 实例并将响应模态设置为 AUDIO。
import { initializeApp } from "firebase/app";
import { getAI, getLiveGenerativeModel, GoogleAIBackend, ResponseModality } from "firebase/ai";
// TODO(developer) Replace the following with your app's Firebase configuration
// See: https://firebase.google.com/docs/web/learn-more#config-object
const firebaseConfig = {
// ...
};
// Initialize FirebaseApp
const firebaseApp = initializeApp(firebaseConfig);
// Initialize the Gemini Developer API backend service
const ai = getAI(firebaseApp, { backend: new GoogleAIBackend() });
// Create a `LiveGenerativeModel` instance with a model that supports the Live API
const liveModel = getLiveGenerativeModel(ai, {
model: "gemini-2.5-flash-native-audio-preview-12-2025",
// Configure the model to respond with audio
generationConfig: {
responseModalities: [ResponseModality.AUDIO],
},
});
const session = await liveModel.connect();
// Start the audio conversation
const audioConversationController = await startAudioConversation(session);
// ... Later, to stop the audio conversation
// await audioConversationController.stop()
Dart
如需使用 Live API，请创建 LiveGenerativeModel 实例，并将回答模态设置为 audio。
import 'package:firebase_ai/firebase_ai.dart';
import 'package:firebase_core/firebase_core.dart';
import 'firebase_options.dart';
import 'package:your_audio_recorder_package/your_audio_recorder_package.dart';
late LiveModelSession _session;
final _audioRecorder = YourAudioRecorder();
await Firebase.initializeApp(
options: DefaultFirebaseOptions.currentPlatform,
);
// Initialize the Gemini Developer API backend service
// Create a `liveGenerativeModel` instance with a model that supports the Live API
final liveModel = FirebaseAI.googleAI().liveGenerativeModel(
model: 'gemini-2.5-flash-native-audio-preview-12-2025',
// Configure the model to respond with audio
liveGenerationConfig: LiveGenerationConfig(
responseModalities: [ResponseModalities.audio],
),
);
_session = await liveModel.connect();
final audioRecordStream = _audioRecorder.startRecordingStream();
// Map the Uint8List stream to InlineDataPart stream
final mediaChunkStream = audioRecordStream.map((data) {
return InlineDataPart('audio/pcm', data);
});
await _session.startMediaStream(mediaChunkStream);
// In a separate thread, receive the audio response from the model
await for (final message in _session.receive()) {
// Process the received message
}
Unity
如需使用 Live API，请创建 LiveModel 实例并将响应模态设置为 Audio。
using Firebase;
using Firebase.AI;
async Task SendTextReceiveAudio() {
// Initialize the Gemini Developer API backend service
// Create a `LiveModel` instance with a model that supports the Live API
var liveModel = FirebaseAI.GetInstance(FirebaseAI.Backend.GoogleAI()).GetLiveModel(
modelName: "gemini-2.5-flash-native-audio-preview-12-2025",
// Configure the model to respond with audio
liveGenerationConfig: new LiveGenerationConfig(
responseModalities: new[] { ResponseModality.Audio })
);
LiveSession session = await liveModel.ConnectAsync();
// Start a coroutine to send audio from the Microphone
var recordingCoroutine = StartCoroutine(SendAudio(session));
// Start receiving the response
await ReceiveAudio(session);
}
IEnumerator SendAudio(LiveSession liveSession) {
string microphoneDeviceName = null;
int recordingFrequency = 16000;
int recordingBufferSeconds = 2;
var recordingClip = Microphone.Start(microphoneDeviceName, true,
recordingBufferSeconds, recordingFrequency);
int lastSamplePosition = 0;
while (true) {
if (!Microphone.IsRecording(microphoneDeviceName)) {
yield break;
}
int currentSamplePosition = Microphone.GetPosition(microphoneDeviceName);
if (currentSamplePosition != lastSamplePosition) {
// The Microphone uses a circular buffer, so we need to check if the
// current position wrapped around to the beginning, and handle it
// accordingly.
int sampleCount;
if (currentSamplePosition > lastSamplePosition) {
sampleCount = currentSamplePosition - lastSamplePosition;
} else {
sampleCount = recordingClip.samples - lastSamplePosition + currentSamplePosition;
}
if (sampleCount > 0) {
// Get the audio chunk
float[] samples = new float[sampleCount];
recordingClip.GetData(samples, lastSamplePosition);
// Send the data, discarding the resulting Task to avoid the warning
_ = liveSession.SendAudioAsync(samples);
lastSamplePosition = currentSamplePosition;
}
}
// Wait for a short delay before reading the next sample from the Microphone
const float MicrophoneReadDelay = 0.5f;
yield return new WaitForSeconds(MicrophoneReadDelay);
}
}
Queue audioBuffer = new();
async Task ReceiveAudio(LiveSession liveSession) {
int sampleRate = 24000;
int channelCount = 1;
// Create a looping AudioClip to fill with the received audio data
int bufferSamples = (int)(sampleRate * channelCount);
AudioClip clip = AudioClip.Create("StreamingPCM", bufferSamples, channelCount,
sampleRate, true, OnAudioRead);
// Attach the clip to an AudioSource and start playing it
AudioSource audioSource = GetComponent();
audioSource.clip = clip;
audioSource.loop = true;
audioSource.Play();
// Start receiving the response
await foreach (var message in liveSession.ReceiveAsync()) {
// Process the received message
foreach (float[] pcmData in message.AudioAsFloat) {
lock (audioBuffer) {
foreach (float sample in pcmData) {
audioBuffer.Enqueue(sample);
}
}
}
}
}
// This method is called by the AudioClip to load audio data.
private void OnAudioRead(float[] data) {
int samplesToProvide = data.Length;
int samplesProvided = 0;
lock(audioBuffer) {
while (samplesProvided < samplesToProvide && audioBuffer.Count > 0) {
data[samplesProvided] = audioBuffer.Dequeue();
samplesProvided++;
}
}
while (samplesProvided < samplesToProvide) {
data[samplesProvided] = 0.0f;
samplesProvided++;
}
}
流式文本 + 音频输入
|
点击您的 Gemini API 提供商，以查看此页面上特定于提供商的内容和代码。 |
如果需要，您可以发送文本输入以及音频输入，并接收流式音频输出。
Swift
如需使用 Live API，请创建 LiveModel 实例并将响应模态设置为 audio。
import FirebaseAILogic
// Initialize the Gemini Developer API backend service
// Create a `liveModel` instance with a model that supports the Live API
let liveModel = FirebaseAI.firebaseAI(backend: .googleAI()).liveModel(
modelName: "gemini-2.5-flash-native-audio-preview-12-2025",
// Configure the model to respond with audio
generationConfig: LiveGenerationConfig(
responseModalities: [.audio]
)
)
do {
let session = try await liveModel.connect()
// Provide a text prompt
let text = "tell a short story"
await session.sendTextRealtime(text)
var outputText = ""
for try await message in session.responses {
if case let .content(content) = message.payload {
content.modelTurn?.parts.forEach { part in
if let part = part as? InlineDataPart, part.mimeType.starts(with: "audio/pcm") {
// Handle 16bit pcm audio data at 24khz
playAudio(part.data)
}
}
// Optional: if you don't require to send more requests.
if content.isTurnComplete {
await session.close()
}
}
}
} catch {
fatalError(error.localizedDescription)
}
Kotlin
如需使用 Live API，请创建 LiveModel 实例并将响应模态设置为 AUDIO。
// Initialize the Gemini Developer API backend service
// Create a `liveModel` instance with a model that supports the Live API
val liveModel = Firebase.ai(backend = GenerativeBackend.googleAI()).liveModel(
modelName = "gemini-2.5-flash-native-audio-preview-12-2025",
// Configure the model to respond with audio
generationConfig = liveGenerationConfig {
responseModality = ResponseModality.AUDIO
}
)
val session = liveModel.connect()
// Provide a text prompt
val text = "tell a short story"
session.send(text)
session.receive().collect {
if(it.turnComplete) {
// Optional: if you don't require to send more requests.
session.stopReceiving();
}
// Handle 16bit pcm audio data at 24khz
playAudio(it.data)
}
Java
如需使用 Live API，请创建 LiveModel 实例并将响应模态设置为 AUDIO。
ExecutorService executor = Executors.newFixedThreadPool(1);
// Initialize the Gemini Developer API backend service
// Create a `liveModel` instance with a model that supports the Live API
LiveGenerativeModel lm = FirebaseAI.getInstance(GenerativeBackend.googleAI()).liveModel(
"gemini-2.5-flash-native-audio-preview-12-2025",
// Configure the model to respond with text
new LiveGenerationConfig.Builder()
.setResponseModality(ResponseModality.AUDIO)
.build()
);
LiveModelFutures model = LiveModelFutures.from(lm);
ListenableFuture<LiveSession> sessionFuture = model.connect();
class LiveContentResponseSubscriber implements Subscriber<LiveContentResponse> {
@Override
public void onSubscribe(Subscription s) {
s.request(Long.MAX_VALUE); // Request an unlimited number of items
}
@Override
public void onNext(LiveContentResponse liveContentResponse) {
// Handle 16bit pcm audio data at 24khz
liveContentResponse.getData();
}
@Override
public void onError(Throwable t) {
System.err.println("Error: " + t.getMessage());
}
@Override
public void onComplete() {
System.out.println("Done receiving messages!");
}
}
Futures.addCallback(sessionFuture, new FutureCallback<LiveSession>() {
@Override
public void onSuccess(LiveSession ses) {
LiveSessionFutures session = LiveSessionFutures.from(ses);
// Provide a text prompt
String text = "tell me a short story?";
session.send(text);
Publisher<LiveContentResponse> publisher = session.receive();
publisher.subscribe(new LiveContentResponseSubscriber());
}
@Override
public void onFailure(Throwable t) {
// Handle exceptions
}
}, executor);
Web
如需使用 Live API，请创建 LiveGenerativeModel 实例并将响应模态设置为 AUDIO。
import { initializeApp } from "firebase/app";
import { getAI, getLiveGenerativeModel, GoogleAIBackend, ResponseModality } from "firebase/ai";
// TODO(developer) Replace the following with your app's Firebase configuration
// See: https://firebase.google.com/docs/web/learn-more#config-object
const firebaseConfig = {
// ...
};
// Initialize FirebaseApp
const firebaseApp = initializeApp(firebaseConfig);
// Initialize the Gemini Developer API backend service
const ai = getAI(firebaseApp, { backend: new GoogleAIBackend() });
// Create a `LiveGenerativeModel` instance with a model that supports the Live API
const liveModel = getLiveGenerativeModel(ai, {
model: "gemini-2.5-flash-native-audio-preview-12-2025",
// Configure the model to respond with audio
generationConfig: {
responseModalities: [ResponseModality.AUDIO],
},
});
const session = await liveModel.connect();
// Provide a text prompt
const prompt = "tell a short story";
session.send(prompt);
// Handle the model's audio output
const messages = session.receive();
for await (const message of messages) {
switch (message.type) {
case "serverContent":
if (message.turnComplete) {
// TODO(developer): Handle turn completion
} else if (message.interrupted) {
// TODO(developer): Handle the interruption
break;
} else if (message.modelTurn) {
const parts = message.modelTurn?.parts;
parts?.forEach((part) => {
if (part.inlineData) {
// TODO(developer): Play the audio chunk
}
});
}
break;
case "toolCall":
// Ignore
case "toolCallCancellation":
// Ignore
}
}
Dart
如需使用 Live API，请创建 LiveGenerativeModel 实例，并将回答模态设置为 audio。
import 'package:firebase_ai/firebase_ai.dart';
import 'package:firebase_core/firebase_core.dart';
import 'firebase_options.dart';
import 'dart:async';
import 'dart:typed_data';
late LiveModelSession _session;
Future<Stream<Uint8List>> textToAudio(String textPrompt) async {
WidgetsFlutterBinding.ensureInitialized();
await Firebase.initializeApp(
options: DefaultFirebaseOptions.currentPlatform,
);
// Initialize the Gemini Developer API backend service
// Create a `liveGenerativeModel` instance with a model that supports the Live API
final liveModel = FirebaseAI.googleAI().liveGenerativeModel(
model: 'gemini-2.5-flash-native-audio-preview-12-2025',
// Configure the model to respond with audio
liveGenerationConfig: LiveGenerationConfig(
responseModalities: [ResponseModalities.audio],
),
);
_session = await liveModel.connect();
final prompt = Content.text(textPrompt);
await _session.send(input: prompt);
return _session.receive().asyncMap((response) async {
if (response is LiveServerContent && response.modelTurn?.parts != null) {
for (final part in response.modelTurn!.parts) {
if (part is InlineDataPart) {
return part.bytes;
}
}
}
throw Exception('Audio data not found');
});
}
Future<void> main() async {
try {
final audioStream = await textToAudio('Convert this text to audio.');
await for (final audioData in audioStream) {
// Process the audio data (e.g., play it using an audio player package)
print('Received audio data: ${audioData.length} bytes');
// Example using flutter_sound (replace with your chosen package):
// await _flutterSoundPlayer.startPlayer(fromDataBuffer: audioData);
}
} catch (e) {
print('Error: $e');
}
}
Unity
如需使用 Live API，请创建 LiveModel 实例并将响应模态设置为 Audio。
using Firebase;
using Firebase.AI;
async Task SendTextReceiveAudio() {
// Initialize the Gemini Developer API backend service
// Create a `LiveModel` instance with a model that supports the Live API
var liveModel = FirebaseAI.GetInstance(FirebaseAI.Backend.GoogleAI()).GetLiveModel(
modelName: "gemini-2.5-flash-native-audio-preview-12-2025",
// Configure the model to respond with audio
liveGenerationConfig: new LiveGenerationConfig(
responseModalities: new[] { ResponseModality.Audio })
);
LiveSession session = await liveModel.ConnectAsync();
// Provide a text prompt
var prompt = ModelContent.Text("Convert this text to audio.");
await session.SendAsync(content: prompt, turnComplete: true);
// Start receiving the response
await ReceiveAudio(session);
}
Queue<float> audioBuffer = new();
async Task ReceiveAudio(LiveSession session) {
int sampleRate = 24000;
int channelCount = 1;
// Create a looping AudioClip to fill with the received audio data
int bufferSamples = (int)(sampleRate * channelCount);
AudioClip clip = AudioClip.Create("StreamingPCM", bufferSamples, channelCount,
sampleRate, true, OnAudioRead);
// Attach the clip to an AudioSource and start playing it
AudioSource audioSource = GetComponent<AudioSource>();
audioSource.clip = clip;
audioSource.loop = true;
audioSource.Play();
// Start receiving the response
await foreach (var message in session.ReceiveAsync()) {
// Process the received message
foreach (float[] pcmData in message.AudioAsFloat) {
lock (audioBuffer) {
foreach (float sample in pcmData) {
audioBuffer.Enqueue(sample);
}
}
}
}
}
// This method is called by the AudioClip to load audio data.
private void OnAudioRead(float[] data) {
int samplesToProvide = data.Length;
int samplesProvided = 0;
lock(audioBuffer) {
while (samplesProvided < samplesToProvide && audioBuffer.Count > 0) {
data[samplesProvided] = audioBuffer.Dequeue();
samplesProvided++;
}
}
while (samplesProvided < samplesToProvide) {
data[samplesProvided] = 0.0f;
samplesProvided++;
}
}
请注意，您还可以在有效会话期间发送文本作为增量内容更新。
流视频 + 音频输入
提供输入视频内容可为输入音频提供视觉背景信息。
Live API 需要接收一系列离散的图片帧，并支持 1 帧/秒 (FPS) 的视频帧输入。
建议的输入：768x768 原生分辨率，1 FPS。
支持的 MIME 类型：
video/x-flv、video/quicktime、video/mpeg、video/mpegs、video/mpg、video/mp4、video/webm、video/wmv、video/3gpp
流式视频 + 音频输入是一种更高级的实现方式，因此请查看示例应用，了解如何实现此功能： Swift - 即将推出！| Android - 示例应用 | Web - 即将推出！| Flutter - 示例应用 | Unity - 即将推出！
不支持的功能
使用 Live API 时，Firebase AI Logic 尚不支持的功能，但很快就会支持！
处理中断
会话管理的一些方面，包括跨多个连接恢复会话、延长会话时长或压缩上下文窗口。不过，请注意，系统支持消失通知。
停用和配置语音活动检测 (VAD)
设置输入媒体分辨率
添加思考配置
启用共情对话或主动音频
在响应中接收
UsageMetadata
使用 Live API 时，Firebase AI Logic 不支持的功能，目前尚未规划。
服务器提示模板
混合推理或设备端推理
Firebase 控制台中的 AI 监控
您还可以做些什么？
使用各种配置选项（例如添加转写或设置回答语音）自定义实现。
了解如何管理会话，包括在会话期间更新内容以及检测会话何时即将结束。
通过让模型访问工具（例如函数调用和依托 Google 搜索实现接地），增强实现效果。有关如何将工具与 Live API 搭配使用的官方文档即将推出！
了解使用 Live API 的限制和规范，例如会话时长、速率限制、支持的语言等。