流式输出的体验价值很明显:首字更快、长回答不必等全部生成。但它的工程细节比非流式多得多,尤其是网络不稳定的时候。
服务端返回 text/event-stream,每个事件是一行 data: {...},最后以 data: [DONE] 结束。
data: {"choices":[{"delta":{"role":"assistant","content":""},"index":0}]}
data: {"choices":[{"delta":{"content":"向量"},"index":0}]}
data: {"choices":[{"delta":{"content":"检索"},"index":0}]}
data: {"choices":[{"delta":{},"finish_reason":"stop","index":0}],"usage":{...}}
data: [DONE]delta 不是 message。从非流式代码直接改过来,最常见的 bug 就是取错字段导致永远拿到空字符串。{"choices":[{"delta":{"cont。此时不能解析,继续等;解析异常就跳过该块而不是崩溃。def iter_sse(resp):
buf = ""
for raw in resp.iter_content(chunk_size=1024, decode_unicode=True):
if not raw:
continue
buf += raw
while "\n\n" in buf:
event, buf = buf.split("\n\n", 1)
for line in event.split("\n"):
if not line.startswith("data:"):
continue
data = line[5:].strip()
if data == "[DONE]":
yield {"done": True}
return
try:
obj = json.loads(data)
except ValueError:
continue # 半截 JSON,跳过
ch = (obj.get("choices") or [{}])[0]
delta = ch.get("delta") or {}
if delta.get("content"):
yield {"text": delta["content"]}
if ch.get("finish_reason"):
yield {"finish": ch["finish_reason"]}两种做法:浏览器直接读 fetch 的 ReadableStream,或后端转发成自己的 SSE。后者更可控(可以做鉴权、计费、脱敏),生产环境更推荐。
const resp = await fetch('/api/chat', {method: 'POST', body: JSON.stringify(req)});
const reader = resp.body.getReader();
const dec = new TextDecoder();
let buf = '';
while (true) {
const {value, done} = await reader.read();
if (done) break;
buf += dec.decode(value, {stream: true});
const parts = buf.split('\n\n');
buf = parts.pop();
for (const p of parts) {
for (const line of p.split('\n')) {
if (!line.startsWith('data:')) continue;
const s = line.slice(5).trim();
if (s === '[DONE]') return;
try {
const o = JSON.parse(s);
const t = o.choices?.[0]?.delta?.content;
if (t) appendToDom(t);
} catch (e) { /* 忽略半截块 */ }
}
}
}AbortController),并把已生成内容保留下来作为一条完整回答。Nginx 默认会缓冲上游响应,导致流式变成「等全部生成完再一次性返回」。必须关掉缓冲:
location /api/ {
proxy_pass http://127.0.0.1:8000;
proxy_buffering off;
proxy_cache off;
proxy_set_header X-Accel-Buffering no;
proxy_read_timeout 300s;
chunked_transfer_encoding on;
}