AI Transport: Add a guide for token streaming using the OpenAI SDK #3024

lawrence-forooghian · 2025-12-12T18:20:40Z

Description

This guide provides a concrete example of how to implement the message-per-token pattern that Mike documented in #3014.

I initially got Claude to generate this but replaced a fair chunk of its output. I trusted that its prose is consistent with our tone of voice and AI Transport marketing position (whether mine is, I have no idea) and in general trusted its judgements about how to structure the document. I would definitely welcome opinions on all of the above, especially from those familiar with how we usually write docs.

I have tried to avoid repeating too much content from the message-per-token page and have in particular not tried to give an example of hydration since it seems like a provider-agnostic concept.

Checklist

Commits have been rebased.
Linting has been run against the changed file(s).
The PR adheres to the writing style guide and contribution guide.

coderabbitai · 2025-12-12T18:20:46Z

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch AIT-token-streaming-OpenAI-SDK

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

lawrence-forooghian · 2025-12-12T18:24:06Z

src/pages/docs/guides/ai-transport/openai-sdk.mdx

+
+You should see publisher output similar to the following:
+
+```text


@GregHolmes is there some sort of collapsible component that I can use for this (like a <details> I guess)? I'd like to include this output but not force the the user to scroll through it all if just skimming.

Hey @lawrence-forooghian @rainbowFi , I'm afraid at this moment I don't think there is a collapsible component. I'm not entirely sure we need all of that output though. Could we not to a start, middle, and finish part. So it's cropped like:

f03945a: Created stream f03945a: Got event response.created f03945a: Publishing 'start' event for response resp_097628d5ede953e800693c497c30148194adc300e4ee412171 f03945a: Got event response.in_progress f03945a: Ignoring OpenAI SDK event response.in_progress for response resp_097628d5ede953e800693c497c30148194adc300e4ee412171 de6fbd3: Created stream de6fbd3: Got event response.created de6fbd3: Publishing 'start' event for response resp_0f89f403f4f5f71800693c497c319c8195acdf3676dbe32cf5 de6fbd3: Got event response.in_progress de6fbd3: Ignoring OpenAI SDK event response.in_progress for response resp_0f89f403f4f5f71800693c497c319c8195acdf3676dbe32cf5 ... Rest of log f03945a: Ignoring OpenAI SDK event response.output_text.done for response resp_097628d5ede953e800693c497c30148194adc300e4ee412171 f03945a: Got event response.content_part.done f03945a: Ignoring OpenAI SDK event response.content_part.done for response resp_097628d5ede953e800693c497c30148194adc300e4ee412171 f03945a: Got event response.output_item.done f03945a: Ignoring OpenAI SDK event response.output_item.done for response resp_097628d5ede953e800693c497c30148194adc300e4ee412171 f03945a: Got event response.completed f03945a: Publishing 'stop' event for response resp_097628d5ede953e800693c497c30148194adc300e4ee412171

Same would go for the other lengthy snippets?

@GregHolmes cropped following your suggestion, are you okay with how it's come out?

lawrence-forooghian · 2025-12-12T18:25:28Z

src/pages/docs/guides/ai-transport/openai-sdk.mdx

+This is only a representative example for a simple "text in, text out" use case, and may not reflect the exact sequence of events that you'll observe from the OpenAI API. It also does not handle response generation errors or refusals.
+</Aside>
+
+1. `{ type: 'response.created', response: { id: 'resp_abc123', … }, … }`: This gives us the response ID, which we'll include in all messages that we publish to Ably for this response. When we receive this event, we'll publish an Ably message named `start`.


@GregHolmes this list, and these JSON events, came out looking a bit cramped and ugly — any suggestions on what I could do to improve this?

I think having all the braces makes this look more complicated than it is and not helping the spacing. If you put it into a table, you could have a column with the heading type for the OpenAI messages and just put response.in_progress or response.output_item.added, then a column for any other interesting fields and a column with the Ably message mapping...?

Alternatively, keep the list format but just put the type name into the text and highlight important fields separately e.g.
3. response.output_item.added with item.type = 'reasoning': we'll ignore this event since we're only interested in messages

The more I think about this, the more I think we should skip this list. I love the technical detail, but it isn't reflected in the code (because we're handling only the specific events we need) and it doesn't aid understanding of Ably. So, I would suggest making the "Understanding Responses API events" section into a clear description of the flow of relevant events we get back from OpenAI, then have a table showing how we'll map those relevant messages to Ably events.

If we were to keep it, why couldn't we just format the list so you have:

1 - Description

// Prettied JSON

2 - Description

etc?

Also, 11 point list seems a bit overwhelming. Is it worth breaking this down into a table such as:

OpenAIEvent and Action columns.
Do we then need json or instead the first column would have the type. response.in_progress, second will be the action. Ignore. etc?

Link to the pending `/ai-transport` overview page.

[AIT-148] AI Transport example filter and product tile

Add intro describing the pattern, its properties, and use cases.

Includes continuous token streams, correlating tokens for distinct responses, and explicit start/end events.

Splits each token streaming approach into distinct patterns and shows both the publish and subscribe side behaviour alongside one another.

Includes hydration with rewind and hydration with persisted history + untilAttach. Describes the pattern for handling in-progress live responses with complete responses loaded from the database.

src/pages/docs/guides/ai-transport/openai-sdk.mdx

rainbowFi · 2025-12-16T17:00:51Z

src/pages/docs/guides/ai-transport/openai-sdk.mdx

+This is only a representative example for a simple "text in, text out" use case, and may not reflect the exact sequence of events that you'll observe from the OpenAI API. It also does not handle response generation errors or refusals.
+</Aside>
+
+1. `{ type: 'response.created', response: { id: 'resp_abc123', … }, … }`: This gives us the response ID, which we'll include in all messages that we publish to Ably for this response. When we receive this event, we'll publish an Ably message named `start`.


I think having all the braces makes this look more complicated than it is and not helping the spacing. If you put it into a table, you could have a column with the heading type for the OpenAI messages and just put response.in_progress or response.output_item.added, then a column for any other interesting fields and a column with the Ably message mapping...?

Alternatively, keep the list format but just put the type name into the text and highlight important fields separately e.g.
3. response.output_item.added with item.type = 'reasoning': we'll ignore this event since we're only interested in messages

src/pages/docs/guides/ai-transport/openai-sdk.mdx

rainbowFi · 2025-12-16T17:14:11Z

src/pages/docs/guides/ai-transport/openai-sdk.mdx

+This is only a representative example for a simple "text in, text out" use case, and may not reflect the exact sequence of events that you'll observe from the OpenAI API. It also does not handle response generation errors or refusals.
+</Aside>
+
+1. `{ type: 'response.created', response: { id: 'resp_abc123', … }, … }`: This gives us the response ID, which we'll include in all messages that we publish to Ably for this response. When we receive this event, we'll publish an Ably message named `start`.


The more I think about this, the more I think we should skip this list. I love the technical detail, but it isn't reflected in the code (because we're handling only the specific events we need) and it doesn't aid understanding of Ably. So, I would suggest making the "Understanding Responses API events" section into a clear description of the flow of relevant events we get back from OpenAI, then have a table showing how we'll map those relevant messages to Ably events.

rainbowFi · 2025-12-16T17:16:49Z

src/pages/docs/guides/ai-transport/openai-sdk.mdx

+
+**Key points**:
+
+- **Multiple concurrent responses are handled correctly**: The subscriber receives interleaved tokens for three concurrent AI responses, and correctly pieces together the three separate messages:


I never saw interleaved responses when I ran this script, any idea why? Luck, location, something else? It's not particularly important but I just want to make sure there isn't a change in the example code that is causing the behaviour

I was only seeing them intermittently when I first wrote it, and now I'm unable to get any at all, too! I think it would be good if users could observe this behaviour. One option would be to add small random delays before processing each event, what do you think?

(It's not ideal and distracts from the content of the guide)

Alternatively we could spin up two publisher instances at the same time: node publisher.mjs & node publisher.mjs — on testing this locally it seems to more reliably give interleaved events. But again it complicates the guide.

src/pages/docs/guides/ai-transport/openai-sdk.mdx

lawrence-forooghian · 2025-12-16T20:05:35Z

@rainbowFi I've updated the publisher to add an additional prompt ("Write a one-line poem about carrot cake"); missed this out of the original code but it was used when generating the responses shown here

This guide provides a concrete example of how to implement the message-per-token pattern that Mike documented (the one linked to in this guide). I initially got Claude to generate this but replaced a fair chunk of its output. I trusted that its prose is consistent with our tone of voice and AI Transport marketing position (whether mine is, I have no idea) and in general trusted its judgements about how to structure the document. I would definitely welcome opinions on all of the above, especially from those familiar with how we usually write docs. I have tried to avoid repeating too much content from the message-per-token page and have in particular not tried to give an example of hydration since it seems like a provider-agnostic concept.

GregHolmes · 2025-12-17T17:12:04Z

I haven't yet fully gone through this PR. But I have noticed that AI really enjoys boldening things. It's not something our docs really do though.

src/pages/docs/guides/ai-transport/openai-sdk.mdx

GregHolmes

I've added a few comments.
But also, just one other thing, AI seems to like making text bold where we don't tend to do that within the docs. I'd suggest removing the bold part,such as:

1. **Stream start**: First event arrives with response ID
2. **Content deltas**: Multiple `response.output_text.delta` events with incremental text
3. **Stream completion**: Stream ends when all events have been received```

to

The stream lifecycle typically follows this pattern:

Stream start: First event arrives with response ID
Content deltas: Multiple response.output_text.delta events with incremental text
Stream completion: Stream ends when all events have been received

src/pages/docs/guides/ai-transport/openai-sdk.mdx

GregHolmes · 2025-12-18T11:07:57Z

src/pages/docs/guides/ai-transport/openai-sdk.mdx

+- An OpenAI API key
+- An Ably API key
+
+**Useful links:**


I think this would be better suited at the bottom of the page. A further reading, or within next steps.
Is the quickstart at openai relevant? or would it be better to link to specific feature pages that we cover too.

src/pages/docs/guides/ai-transport/openai-sdk.mdx

GregHolmes · 2025-12-18T11:18:13Z

src/pages/docs/guides/ai-transport/openai-sdk.mdx

+This is only a representative example for a simple "text in, text out" use case, and may not reflect the exact sequence of events that you'll observe from the OpenAI API. It also does not handle response generation errors or refusals.
+</Aside>
+
+1. `{ type: 'response.created', response: { id: 'resp_abc123', … }, … }`: This gives us the response ID, which we'll include in all messages that we publish to Ably for this response. When we receive this event, we'll publish an Ably message named `start`.


Also, 11 point list seems a bit overwhelming. Is it worth breaking this down into a table such as:

OpenAIEvent and Action columns.
Do we then need json or instead the first column would have the type. response.in_progress, second will be the action. Ignore. etc?

GregHolmes · 2025-12-18T11:18:57Z

src/pages/docs/guides/ai-transport/openai-sdk.mdx

+
+**Important implementation notes:**
+
+- **Don't await publish calls**: As shown in the code above, `channel.publish()` is called without `await`. This maximizes throughput by allowing Ably to batch acknowledgments. Messages are still published in order. For more details, see [publishing tokens](/docs/ai-transport/features/token-streaming/message-per-token#publishing) in the message-per-token guide.


Is this too late in the page to be pointing this out? for example we could do Aside's for each part in key parts of the page?

There isn't really a good point to put this earlier - we're showing people the code in a block and then providing the informative notes afterwards. I think that's okay because we've linked to the feature docs which discuss this in more detail, but happy for you to suggest an alternative location

src/pages/docs/guides/ai-transport/openai-sdk.mdx

Co-authored-by: Greg Holmes <iam@gregholmes.co.uk>

lawrence-forooghian force-pushed the AIT-token-streaming-OpenAI-SDK branch from deae500 to 3d12fdf Compare December 12, 2025 18:21

lawrence-forooghian added the review-app Create a Heroku review app label Dec 12, 2025

ably-ci temporarily deployed to ably-docs-ait-token-str-qkwgk2 December 12, 2025 18:22 Inactive

lawrence-forooghian commented Dec 12, 2025

View reviewed changes

lawrence-forooghian requested review from GregHolmes and mschristensen December 12, 2025 18:24

lawrence-forooghian commented Dec 12, 2025

View reviewed changes

lawrence-forooghian force-pushed the AIT-token-streaming-OpenAI-SDK branch from 3d12fdf to 9876218 Compare December 12, 2025 18:27

ably-ci temporarily deployed to ably-docs-ait-token-str-qkwgk2 December 12, 2025 18:27 Inactive

lawrence-forooghian force-pushed the AIT-token-streaming-OpenAI-SDK branch from 9876218 to 281713a Compare December 12, 2025 18:33

ably-ci temporarily deployed to ably-docs-ait-token-str-qkwgk2 December 12, 2025 18:33 Inactive

lawrence-forooghian marked this pull request as ready for review December 12, 2025 18:33

lawrence-forooghian force-pushed the AIT-token-streaming-OpenAI-SDK branch from 281713a to 4f188f3 Compare December 12, 2025 18:34

ably-ci temporarily deployed to ably-docs-ait-token-str-qkwgk2 December 12, 2025 18:35 Inactive

GregHolmes and others added 4 commits December 15, 2025 09:37

Add AI transport as a product within docs

2f792d4

chore: Add AI Transport examples filter

fb61e35

chore: Add AI Transport product tile to the homepage

1a7fb97

Link to the pending `/ai-transport` overview page.

Merge pull request #3029 from ably/ait-148-examples

176fa2b

[AIT-148] AI Transport example filter and product tile

mschristensen force-pushed the feature/AIT-51-token-streaming-granular-history branch 2 times, most recently from 0e663af to 52e32d8 Compare December 15, 2025 13:38

mschristensen added 5 commits December 16, 2025 12:23

ait/token-streaming: add message per token page

abf95d0

ait/message-per-token: add intro

c90532b

Add intro describing the pattern, its properties, and use cases.

ait/message-per-token: add token publishing

bd3bc78

Includes continuous token streams, correlating tokens for distinct responses, and explicit start/end events.

ait/message-per-token: token streaming patterns

5be4688

Splits each token streaming approach into distinct patterns and shows both the publish and subscribe side behaviour alongside one another.

ait/message-per-token: client hydration patterns

e5ea672

Includes hydration with rewind and hydration with persisted history + untilAttach. Describes the pattern for handling in-progress live responses with complete responses loaded from the database.

Base automatically changed from feature/AIT-51-token-streaming-granular-history to AIT-129-AIT-Docs-release-branch December 16, 2025 12:23

rainbowFi reviewed Dec 16, 2025

View reviewed changes

src/pages/docs/guides/ai-transport/openai-sdk.mdx Show resolved Hide resolved

lawrence-forooghian force-pushed the AIT-token-streaming-OpenAI-SDK branch from 4f188f3 to cabebd0 Compare December 16, 2025 20:02

ably-ci temporarily deployed to ably-docs-ait-token-str-qkwgk2 December 16, 2025 20:03 Inactive

lawrence-forooghian force-pushed the AIT-token-streaming-OpenAI-SDK branch from cabebd0 to f827802 Compare December 16, 2025 20:13

ably-ci temporarily deployed to ably-docs-ait-token-str-qkwgk2 December 16, 2025 20:14 Inactive

Fix ups from review comments

413dd0a

ably-ci temporarily deployed to ably-docs-ait-token-str-qkwgk2 December 17, 2025 21:33 Inactive

rainbowFi reviewed Dec 17, 2025

View reviewed changes

src/pages/docs/guides/ai-transport/openai-sdk.mdx Show resolved Hide resolved

GregHolmes requested changes Dec 18, 2025

View reviewed changes

rainbowFi and others added 2 commits December 19, 2025 15:48

Update src/pages/docs/guides/ai-transport/openai-sdk.mdx

6317372

Co-authored-by: Greg Holmes <iam@gregholmes.co.uk>

Update src/pages/docs/guides/ai-transport/openai-sdk.mdx

8a74dab

Co-authored-by: Greg Holmes <iam@gregholmes.co.uk>

ably-ci temporarily deployed to ably-docs-ait-token-str-qkwgk2 December 19, 2025 15:48 Inactive

Update src/pages/docs/guides/ai-transport/openai-sdk.mdx

91291ef

Co-authored-by: Greg Holmes <iam@gregholmes.co.uk>

ably-ci temporarily deployed to ably-docs-ait-token-str-qkwgk2 December 19, 2025 15:49 Inactive

Update src/pages/docs/guides/ai-transport/openai-sdk.mdx

3b00594

Co-authored-by: Greg Holmes <iam@gregholmes.co.uk>

ably-ci temporarily deployed to ably-docs-ait-token-str-qkwgk2 December 19, 2025 15:50 Inactive

Fixup based on style review comments

a6865d4

ably-ci temporarily deployed to ably-docs-ait-token-str-qkwgk2 December 19, 2025 16:23 Inactive

matt423 force-pushed the AIT-129-AIT-Docs-release-branch branch from 400eb09 to f8056cb Compare December 23, 2025 10:41


		You should see publisher output similar to the following:

		```text


		Key points:

		- Multiple concurrent responses are handled correctly: The subscriber receives interleaved tokens for three concurrent AI responses, and correctly pieces together the three separate messages:


		Important implementation notes:

		- Don't await publish calls: As shown in the code above, `channel.publish()` is called without `await`. This maximizes throughput by allowing Ably to batch acknowledgments. Messages are still published in order. For more details, see [publishing tokens](/docs/ai-transport/features/token-streaming/message-per-token#publishing) in the message-per-token guide.

AI Transport: Add a guide for token streaming using the OpenAI SDK #3024

Are you sure you want to change the base?

AI Transport: Add a guide for token streaming using the OpenAI SDK #3024

Uh oh!

Conversation

lawrence-forooghian commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

coderabbitai bot commented Dec 12, 2025

Review skipped

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lawrence-forooghian commented Dec 16, 2025

Uh oh!

GregHolmes commented Dec 17, 2025

Uh oh!

Uh oh!

GregHolmes left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

7 participants

lawrence-forooghian commented Dec 12, 2025 •

edited

Loading