Setup Guide

You need to set up API keys or your own LLM backends to interact with LLMs. Besides, APPL has many configurations that you can customize to fit your needs.

Setup Environment Variables

Using Dotenv (Recommended)

We recommended you to store all your environment variables, including API keys, in a .env file in the root directory of your project (or other directories, see Priority of Configurations for more details). Based on the python-dotenv package, APPL will automatically load the environment variables into the current environment.

Remember to setup your .gitignore file

Make sure to add .env to your .gitignore file to prevent it from being committed to your repository.

For example, you can create a .env file with the following content to specify your OpenAI API key:

.env

OPENAI_API_KEY=<your openai api key>

Export or Shell Configuration

Alterantively, you can export the environment variables directly in your terminal, or add them to your shell configuration file (e.g., .bashrc, .zshrc). For example:

export OPENAI_API_KEY=<your openai api key>

Setup APPL Configuration

Default Configs

default_configs.yaml contains the default configurations for APPL.

metadata: {}

settings:
  logging:
    format: >- # The format of the log message, change HH:mm:ss.SSS to HH:mm:ss for the default loguru format
      <green>{time:YYYY-MM-DD HH:mm:ss}</green> |
      <level>{level: <8}</level> |
      <cyan>{name}</cyan>:<cyan>{function}</cyan>:<cyan>{line}</cyan> |
      <level>{message}</level>
    log_level: "INFO" # The level of the log messages
    max_length: 800 # The maximum length of the log message in bash
    suffix_length: 200 # The length of the suffix (when truncated)
    log_file:
      enabled: false # default to not log to a file
      path_format: './logs/{basename}_{time:YYYY_MM_DD__HH_mm_ss}'
      # The path to the log file, ext will be added automatically
      log_level: null # default to use the same log level as stdout
    display:
      configs: false # Display the configurations
      configs_update: false # Display the updates of the configurations
      docstring_warning: true # Display the warning message when docstring are excluded
      llm_raw_call_args: false # Display the raw args for the llm calls
      llm_raw_response: false # Display the raw response of the llm calls
      llm_raw_usage: false # Display the raw usage of the llm calls
      llm_call_args: true # Display the args for the llm calls
      llm_usage: true # Display the usage of the llm calls
      llm_response: true # Display the response of the llm calls
      llm_cache: false # Display the cache info
      llm_cost: true # Display the cost of the calls
      tool_calls: true # Display the tool calls
      tool_results: true # Display the results of the tool calls
      stream_interval: 0.5 # The interval in second to log the stream info
      rich:
        language: "markdown" # The language of the rich output
        theme: "monokai" # The theme of the rich output
        line_numbers: false # Whether to display the line numbers
        word_wrap: true # Whether to wrap the words

  tracing:
    enabled: false # default to not trace the calls
    path_format: './dumps/traces/{basename}_{time:YYYY_MM_DD__HH_mm_ss}'
    # The path to the trace file, ext will be added automatically
    strict_match: true # when saving and loading cache, whether need to match the generation id
  messages:
    colors:
      system: red
      user: green
      assistant: cyan
      tool: magenta
  misc:
    suppress_litellm_debug_info: true

# When using APIs through litellm,
#   see the list of available models at https://docs.litellm.ai/docs/providers
# When using SRT server,
#   set model to "srt" and api_base to the address of the server.
servers:
  default: gpt4o-mini
  # create aliases for the models, you may use the model name as server name directly
  gpt35-turbo: # the name of the server, should avoid using '.' in the name
    model: gpt-3.5-turbo # the model name
  gpt4-turbo:
    model: gpt-4-turbo
  gpt4o:
    model: gpt-4o
  gpt4o-mini:
    model: gpt-4o-mini
  _dummy:
    model: _dummy

You should setup your own default server.

The default server (currently gpt-4o-mini) set in APPL could be outdated and changed in the future. We recommend you to specify your own default model in the appl.yaml file.

Override Configs

You can override these configurations by creating a appl.yaml file in the root directory of your project (or other directories, see Priority of Configurations for more details). A typical usage is to override the servers configuration to specify the LLM servers you want to use, as shown in the following example appl.yaml file.

settings:
  logging:
    # log_file:
    #   enabled: true
    display:
      configs: false
      llm_call_args: false # true
      llm_response: false # true
      llm_cache: false
      llm_cost: true # true
  tracing:
    enabled: true

# example for setting up servers
servers:
  # default: azure-gpt35 # override the default server according to your needs
  azure-gpt35: # the name of the server
    model: azure/gpt35t # the model name
    # temperature: 1.0 # set the default temperature for the calls to this server
  gpt4-preview:
    model: gpt-4-0125-preview
  claude-35-sonnet:
    model: claude-3-5-sonnet-20240620
  claude-3-opus:
    model: claude-3-opus-20240229
  moonshot-8k:
    model: moonshot-v1-8k
    provider: custom # https://docs.litellm.ai/docs/providers/custom
    api_key: os.environ/MOONSHOT_API_KEY # setup your API key in the environment variable
    base_url: "https://api.moonshot.cn/v1"
    # add cost, see also https://docs.litellm.ai/docs/proxy/custom_pricing
    input_cost_per_token: 1.2e-5
    output_cost_per_token: 1.2e-5
    cost_currency: "CNY"
  # litellm has supported deepseek now, you can simply set up the deepseek server like this:
  # deepseek-chat:
  #   model: deepseek/deepseek-chat
  # deepseek-coder:
  #   model: deepseek/deepseek-coder
  # Below use deepseek apis to illustrate how to set up a group of apis that not (yet) supported by litellm
  deepseek-chat:
    model: deepseek-chat
    provider: custom
    api_key: os.environ/DEEPSEEK_API_KEY
    base_url: "https://api.deepseek.com/v1"
    input_cost_per_token: 1.0e-6
    output_cost_per_token: 2.0e-6
    cost_currency: "CNY"
  deepseek-coder:
    template: deepseek-chat
    model: deepseek-coder
  srt-llama2: # llama2 served using SRT
    model: custom/default # the default is just a irrelevant name, could be changed to your preference
    base_url: "http://127.0.0.1:30000/v1" # the example address of the SRT server

How configurations are updated?

The configurations are implemented as a nested dictionary in Python using addict.Dict. The update is recursively applied according to this link.

Other file formats like JSON or TOML are also supported.

You can also use other file formats, such as JSON (appl.json) or TOML (appl.toml), to specify the configurations. We recommend using YAML for better readability.

Setup LLMs

You can configure the LLM servers in the appl.yaml file by overriding the servers configuration as shown in the example above.

LLM APIs

APPL uses litellm to support various LLM APIs using the OpenAI format. Please refer to the list of supported providers.

You need to setup the corresponding API keys for the LLM backend you want to use in environment variables and specify corresponding configurations in appl.yaml.

An example of .env file that matches the appl.yaml example above to support using APIs from OpenAI, Anthropic, Azure, Moonshot, and DeepSeek is as follows:

.env

OPENAI_API_KEY=<your openai api key>
# Anthropic environment variables
ANTHROPIC_API_KEY=<your anthropic api key>
# Azure environment variables
AZURE_API_KEY=<your azure api key>
AZURE_API_BASE=<the base url of the API>
AZURE_API_VERSION=<the version of the API>
# Moonshot environment variables
MOONSHOT_API_KEY=<your moonshot api key>
# DeepSeek environment variables
DEEPSEEK_API_KEY=<your deepseek api key>

Local LLMs

We recommend using SGlang Runtime (SRT) to serve the local LLMs, which is fast and supports the regex constraints. You can install it following the official guide.

To serve local LLMs, please following SGLang's official guide.

Using Llama-2-7b-chat-hf as an example:

python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000

You may also use vLLM to host a local server, and the usage in APPL is similar to SRT.

Setup Tracing

APPL Tracing

You can enable APPL tracing by overriding the tracing configuration to true in appl.yaml.

appl.yaml

settings:
  tracing:
    enabled: true

To resume from a previous trace, you can specify the APPL_RESUME_TRACE environment variable with the path to the trace file. See more details in the tutorial.

LangSmith

To enable LangSmith tracing, you need to to obtain your API key from LangSmith and add the following environment variables to your .env file:

.env

LANGCHAIN_TRACING_V2=true
LANGCHAIN_API_KEY=<your api key>
# [Optional] specify the project name
# LANGCHAIN_PROJECT=<your project name>

Priority of Configurations

Let's say you are running a script (main.py) in the project directory which calls appl.init, and you have the following directory structure:

.env
appl.yaml
project/
├── .env
├── appl.yaml
└── main.py

Starting from the directory containing the file calling appl.init (in this case, main.py), APPL will walk up the directory tree to find custom configurations. The configurations in the files closer to the file calling appl.init will have higher priority.

In this case, the configurations in appl.yaml will be loaded first, followed by the ones in project/appl.yaml with overriding the previous ones. The environment variables in .env will be loaded first, followed by the ones in project/.env with overriding the previous ones.

Difference between .env and appl.yaml

• .env is used to store environment variables, including API keys. The contents should not be committed to the repository as they may contain sensitive information.
• appl.yaml is used to store APPL configurations, such as LLM servers and tracing settings. When used in a project, it should not contain sensitive information. The settings in general should be committed to the repository so that others can reproduce the results.