Automatic Tool Execution

18 min

Introduction

Functionary provides a model that can make intelligent decisions regarding which functions/tools to use. However, it does not actually execute the function/tool. To bring this up another level, you can even automatically execute functions/tools once it is decided by Functionary! In this guide, you will learn how to do that with chatlab. The code used in this tutorial is provided in this Github repository.

Prerequisite

An understanding of how to run Functionary vLLM server and make API requests to the running server
Basic skills in Python programming and interacting with APIs

What you'll learn

How to use chatlab
How to configure and integrate chatlab directly with Functionary
How to get a model response grounded in function outputs end-to-end with Functionary and chatlab

What you'll need

A machine capable of running inference with a local Functionary-v1.4 model (at least 24GB GPU VRAM)
A machine with Functionary's dependencies installed

Setup and Requirements

Functionary

Start a Functionary vLLM server with functionary-v1.4 model

python3 server_vllm.py --model meetkai/functionary-7b-v1.4 --max-model-len 4096

The functionary-v1.4 model is trained on context-window of 4K so pass in --max-model-len of 4096.

Chatlab

Install the chatlab python binary package. In this tutorial, we will use version 1.3.0.

pip3 install chatlab==1.3.0

Note on Requirements

Please note that Chatlab's Chat class currently doesn't support Parallel Function calling. Thus, this tutorial is compatible with Functionary Version 1.4 only and may not work correctly with Functionary Version 2.* models.

Define Python Function

Let's assume that you are one of the car dealers at Functionary car dealership. You would like to create a chatbot that can assist you or your customers in quickly getting the prices of certain car models available in the dealership. You will create this Python function:

def get_car_price(car_name: str):
    """this function is used to get the price of the car given the name
    :param car_name: name of the car to get the price
    """
    car_price = {
        "rhino": {"price": "$20000"},
        "elephant": {"price": "$25000"} 
    }
    for key in car_price:
        if key in car_name.lower():
            return {"price": car_price[key]}
    return {"price": "unknown"}

This function queries a dictionary mapping car model names to its respective prices. It returns an "unknown" value if the car model name is not found.

Before you begin

Before you begin, let's imagine packages like chatlab are not around.

Manually create LLM Function

Now, a customer approaches your car dealer chatbot asking about the price of the car model "Rhino". Firstly, you would need to manually convert the Python function into the tool dictionary required in OpenAI API:

functions = [
    {
        "name": "get_car_price",
        "description": "this function is used to get the price of the car given the name",
        "parameters": {
            "type": "object",
            "properties": {
                "car_name": {
                    "type": "string",
                    "description": "name of the car to get the price"
                }
            },
            "required": ["car_name"]
        }
    }
]

Thereafter, you would perform inference on Functionary.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="functionary")

messages = [
    {
        "role": "user",
        "content": "What is the price of the car named 'Rhino'?"
    }
]

assistant_msg = client.chat.completions.create(
    model="meetkai/functionary-7b-v1.4",
    messages=messages,
    functions=functions,
    temperature=0.0,
).choices[0].message

This yields the following response:

ChatCompletionMessage(
    content=None,
    role='assistant',
    function_call=FunctionCall(
        arguments='{"car_name": "Rhino"}',
        name='get_car_price'
    ),
    tool_calls=None,
    tool_call_id=None,
    name=None
)

Manually execute function

As we can see above, Functionary makes the correct decision to call the `get_car_price` function with `Rhino` as input. However, we need to manually execute this function and append its output to the conversation for Functionary to generate appropriate model responses back to the customer.

import json

def execute_function_call(message):
    if message.function_call.name == "get_car_price":
        car_name = json.loads(message.function_call.arguments)["car_name"]
        results = get_car_price(car_name)
    else:
        results = f"Error: function {message.function_call.name} does not exist"
    return results

if assistant_msg.function_call is not None:
    results = execute_function_call(assistant_msg)
    messages.append({"role": "assistant", "name": assistant_msg.function_call.name, "content": assistant_msg.function_call.arguments})
    messages.append({"role": "function", "name": assistant_msg.function_call.name, "content": str(results)})
    
    output_msg = client.chat.completions.create(
        model="meetkai/functionary-7b-v1.4",
        messages=messages,
        functions=functions,
        temperature=0.0,
    ).choices[0].message
    
    print(output_msg)

This yields the final response:

ChatCompletionMessage(
    content="The price of the car named 'Rhino' is $20000.",
    role='assistant',
    function_call=None,
    tool_calls=None,
    tool_call_id=None,
    name=None,
)

This simple example shows that Functionary can:

Intelligently decide on the correct function to use given the conversation
Analyze the function output and generate response grounded in the output

However, as you can see, this requires manually creating the function configuration and executing the functions called until a model response is generated. This is where automatic tool execution will be helpful.

Call real python functions automatically

Now, we show that the Functionary can be further enhanced with automatic execution of Python functions. To call the real Python function, get the result and extract the result to respond, you can use chatlab. The following example uses chatlab==1.3.0:

import chatlab
import asyncio

chat = chatlab.Chat(model="meetkai/functionary-7b-v1.4", base_url="http://localhost:8000/v1", api_key="functionary")
chat.register(get_car_price)
asyncio.run(chat.submit("What is the price of the car named 'Rhino'?", stream=False))

for message in chat.messages:
    role = message["role"].upper()
    if "function_call" in message:
        func_name = message["function_call"]["name"]
        func_param = message["function_call"]["arguments"]
        print(f"{role}: call function: {func_name}, arguments:{func_param}")
    else:
        content = message["content"]
        print(f"{role}: {content}")

The output will look like this:

USER: What is the price of the car named 'Rhino'?
ASSISTANT: call function: get_car_price, arguments:{
    "car_name": "Rhino"
}
FUNCTION: {'price': {'price': '$20000'}}
ASSISTANT: The price of the car named 'Rhino' is $20000.

Now, Functionary will be called iteratively and chatlab will automatically execute any function called by Functionary until no more functions are to be called and a model response is generated for the customer. This is all done with the ease of a single command:

chat.submit("What is the price of the car named 'Rhino'?")

Congratulations

Congratulations on completing this tutorial. We hope you have learnt about how to further harness the power of Functionary by combining with automatic tool execution libraries like chatlab. Feel free to try out the example notebooks in the Github repository and explore Functionary's function calling capabilities with your own functions.

Summary

Performing inference using Functionary
Experiencing how Functionary calls functions and generates model responses
Learning how to execute functions called by Functionary automatically with chatlab