Skip to content

Using Streaming

gen returns a Generation object, which is a wrapper around the response from the LM.

Enabling streaming in APPL is simple. Just set the stream parameter to True when calling gen. The return is a generator that yields the response in chunks, but you can still access the complete response.

Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
import appl
from appl import ppl, records
from appl.func import gen

appl.init()


@ppl
def func(stream=True):
    # adopted from https://cookbook.openai.com/examples/how_to_stream_completions
    "Count to 100, with a comma between each number and no newlines. E.g., 1, 2, 3, ..."
    return gen(stream=stream)


content = func(stream=False)
print(f"Content: {content}")

content = func(stream=True)
print(f"Content: {content}")