Skip to content

Using Streaming

gen returns a Generation object, which is a wrapper around the response from the LM.

Enabling streaming in APPL is simple. Just set the stream parameter to True when calling gen. The return is a generator that yields the response in chunks, but you can still access the complete response.

Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
from typing import Optional

from appl import gen, ppl


@ppl
def func(
    stream: bool = True, max_tokens: Optional[int] = None, max_relay_rounds: int = 10
):
    # adopted from https://cookbook.openai.com/examples/how_to_stream_completions
    "Count to 50, with a comma between each number and no newlines. E.g., 1, 2, 3, ..."
    return gen(
        stream=stream,
        temperature=0.0,
        max_tokens=max_tokens,
        max_relay_rounds=max_relay_rounds,
    )


# content = func(stream=False)
# print(f"Content: {content}")

content = func(stream=True)
print(f"Content: {content}")

content = func(stream=True, max_tokens=25, max_relay_rounds=10)
print(f"Content: {content}")