Using Tracing

APPL supports tracing APPL functions and LM calls to facilitate users to understand and debug the program executions. The trace is useful for reproducing (potentially partial) execution results by loading cached responses of the LM calls, which enables failure recovery and avoids the extra costs of resending these calls. This also unlocks the possibility of debugging one LM call conveniently.

Enabling APPL Tracing

To enable tracing in APPL, you can set the tracing configuration to true in appl.yaml:

appl.yaml

settings:
  tracing:
    enabled: true
    path_format: <Your custom path for trace files>
    # The default path format is "./dumps/traces/{basename}_{time:YYYY_MM_DD__HH_mm_ss}"
    strict_match: true # For loading the trace files, explain later.

Visualizing the Trace

Then we run the QA example with tracing enabled:

import appl
from appl import AIRole, gen, ppl
from appl.const import NEWLINE

appl.init()


@ppl(ctx="copy")  # copy the context from caller
def get_answer(question: str):
    question  # append to the prompt
    return gen()  # return as a future object


@ppl  # marks APPL function
def answer_questions(quotation: str, questions: list[str]):
    "Extract the name of the author from the quotation below and answer questions."
    quotation  # append to the prompt
    with AIRole():  # assistant message
        f"The name of the author is {gen(stop=NEWLINE)}"  # specify the prefix
    return [get_answer(q) for q in questions]  # parallelize calls


quotation = '"Simplicity is the ultimate sophistication." -- Leonardo da Vinci'
questions = [
    "In what era did the author live?",
    "What is the most famous painting of the author?",
]
for ans in answer_questions(quotation, questions):
    print(ans)

$ python answer_questions.py

You can find the result trace file in the specified path. The default location is ./dumps/traces/answer_questions_<the timestamp>.pkl. You can then visualize the traces using the script:

$ python -m appl.cli.vis_trace <path to the trace file> -o <output file>

The default output file is a HTML file, which can be viewed in a browser. We provide a sample trace file here.

If you specify the output file to be a .json file, the script will generate a JSON file that is loadable by Chrome's tracing viewer (with address: chrome://tracing/). The loaded trace will look like this:

Resuming from a Previous Trace

You can reproduce the results from a previous trace by specifying the APPL_RESUME_TRACE environment variable with the path to the trace file:

$ APPL_RESUME_TRACE=<path to the trace file> python answer_questions.py

Then each LM call will be loaded from the trace file if it exists. Such loading can be useful for:

• Debugging a specific LM call: the LM calls before that can be loaded from the trace file, therefore no need to resend them with extra costs.
• Reproducible results: the trace file can be shared with others to reproduce the same results.
• Recovery from failures: if the program fails, you can resume from the trace file to avoid resending the LM calls.

strict_match for calls with same prompts

When strict_match is False, the LM calls with the same prompt will load the same response from the trace file. To load the response for each LM call correspondingly, you can set strict_match to True (which is the default setting), then the gen_id of the LM call will also be used for matching.

LangSmith Tracing

Optionally, you can use LangSmith to inspect the LM calls and responses in the trace files. You need to obtain your API key from LangSmith and add the following environment variables to your .env file:

.env

LANGCHAIN_TRACING_V2=true
LANGCHAIN_API_KEY=<your api key>
# [Optional] specify the project name
# LANGCHAIN_PROJECT=<your project name>

LangSmith may contain inaccurate statistics for asynchronous LM calls

When running the example, the time statistics for the get_answer function calls are not consistent.

Nonetheless, it is sometimes useful to record and inspect the LM calls and responses using LangSmith.