Tutorial: App versioning

Learn how to use Weave Model to track and version your application and its parameters

Tracking the inputs, outputs, metadata as well as data flowing through your app is critical to understanding the performance of your system. However versioning your app over time is also critical to understand how modifications to your code or application parameters change your outputs. Weave’s Model class is how these changes can be tracked in Weave.

In this tutorial you’ll learn:

• How to use Weave Model to track and version your application and its parameters.
• How to export, modify and re-use a Weave Model already logged.

Using weave.Model

The weave.Model class is currently only supported in Python.

Using Weave Models means that parameters such as model vendor ids, prompts, temperature, and more are stored and versioned when they change.

To create a Model in Weave, you need the following:

• a class that inherits from weave.Model
• type definitions on all class fields
• a typed invoke function with the @weave.op() decorator

When you change the class fields or the code that defines your model, these changes will be logged and the version will be updated. This ensures that you can compare the generations across different versions of your app.

In the example below, the model name, temperature and system prompt will be tracked and versioned:

Python

import json
from openai import OpenAI

import weave

@weave.op()
def extract_dinos(wmodel: weave.Model, sentence: str) -> dict:
    response = wmodel.client.chat.completions.create(
        model=wmodel.model_name,
        temperature=wmodel.temperature,
        messages=[
            {
                "role": "system",
                "content": wmodel.system_prompt
            },
            {
                "role": "user",
                "content": sentence
            }
            ],
            response_format={ "type": "json_object" }
        )
    return response.choices[0].message.content

# Sub-class with a weave.Model
class ExtractDinos(weave.Model):
    client: OpenAI = None
    model_name: str
    temperature: float
    system_prompt: str

    # Ensure your function is called `invoke` or `predict`
    @weave.op()
    def invoke(self, sentence: str) -> dict:
        dino_data  = extract_dinos(self, sentence)
        return json.loads(dino_data)

TypeScript

This feature is not available in TypeScript yet.  Stay tuned!

Now you can instantiate and call the model with invoke:

Python

weave.init('jurassic-park')
client = OpenAI()

system_prompt = """Extract any dinosaur `name`, their `common_name`, \
names and whether its `diet` is a herbivore or carnivore, in JSON format."""

dinos = ExtractDinos(
    client=client,
    model_name='gpt-4o',
    temperature=0.4,
    system_prompt=system_prompt
)

sentence = """I watched as a Tyrannosaurus rex (T. rex) chased after a Triceratops (Trike), \
both carnivore and herbivore locked in an ancient dance. Meanwhile, a gentle giant \
Brachiosaurus (Brachi) calmly munched on treetops, blissfully unaware of the chaos below."""

result = dinos.invoke(sentence)
print(result)

TypeScript

This feature is not available in TypeScript yet.  Stay tuned!

Now after calling .invoke you can see the trace in Weave now tracks the model parameters as well as the code for the model functions that have been decorated with weave.op(). You can see the model is also versioned, “v21” in this case, and if you click on the model you can see all of the calls that have used that version of the model

A note on using weave.Model:

• You can use predict instead of invoke for the name of the function in your Weave Model if you prefer.
• If you want other class methods to be tracked by weave they need to be wrapped in weave.op()
• Parameters starting with an underscore are ignored by weave and won’t be logged

Exporting and re-using a logged weave.Model

Because Weave stores and versions Models that have been invoked, it is possible to export and re-use these models.

Get the Model ref In the Weave UI you can get the Model ref for a particular version

Using the Model Once you have the URI of the Model object, you can export and re-use it. Note that the exported model is already initialised and ready to use:

Python

# the exported weave model is already initialised and ready to be called
new_dinos = weave.ref("weave://morgan/jurassic-park/object/ExtractDinos:ey4udBU2MU23heQFJenkVxLBX4bmDsFk7vsGcOWPjY4").get()

# set the client to the openai client again
new_dinos.client = client

new_sentence = """I also saw an Ankylosaurus grazing on giant ferns"""
new_result = new_dinos.invoke(new_sentence)
print(new_result)

TypeScript

This feature is not available in TypeScript yet.  Stay tuned!

Here you can now see the name Model version (v21) was used with the new input:

What’s next?

• Follow the Build an Evaluation pipeline tutorial to start iteratively improving your applications.