Skip to content

Servers

APPL manages various LM backends as servers, where each server contains a set of configurations, such as the model name, base url, and default parameters.

Generation Parameters

The parameters used in the gen function to interact with the LM servers are unified into the OpenAI format, which is supported by the litellm package.

See the documentation of litellm.completion for the full list of parameters, where the required parameters are managed by APPL:

  • model: configured in the server settings. You may directly use the model name in the gen function without configuring that, but you can create alias names and specify the default parameters by configuring the server settings.
  • messages: the full conversation (a list of messages) stored in the context when the gen function is called.

You can specify other parameters supported by litellm.completion function.

Server Configurations

The basic servers are configured as follows: 

servers:
  default: gpt4o-mini
  gpt4o-mini: # the name of the server
    model: gpt-4o-mini # the model name
  gpt4o:
    model: gpt-4o

You may specify another server of gpt-4o-mini with default temperature to be 0.0 in the appl.yaml file: 

servers:
  # default: gpt4o-mini-temp0  # (1)
  gpt4o-mini-temp0: # (2)
    model: gpt-4o-mini
    temperature: 0.0 # (3)

  1. you may set the default server here, so you don't need to specify the server name in the gen function.
  2. Then when you call gen("gpt4o-mini-temp0"), the default temperature will be 0.0.
  3. You could still override the temperature by specifying it in the gen function.

We provide examples of configurations for different servers in our setup guide. See also the list of available models in litellm.

Multiple Servers Example

In the following, we provide a complete example for using multiple servers in different gen calls. The used servers are configured in this example.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
from appl import gen, ppl


@ppl
def add1(a, b):
    f"what's {a} plus {b}?"
    return gen("azure-gpt35")


@ppl
def add2(a, b):
    f"what's {a} plus {b}?"
    return gen("claude-35-sonnet")


@ppl
def add3(a, b):
    f"what's {a} plus {b}?"
    return gen("moonshot-8k")


@ppl
def add4(a, b):
    f"what's {a} plus {b}?"
    return gen("srt-llama2")


# print(add1("one", "two"))
# print(add2("three", "four"))
# print(add3("five", "six"))
# print(add4("seven", "eight"))