LLM Chat

New in v14.21.0

BETA

This feature is currently in BETA, which means:

We reserve the right to make breaking changes without prior notice.
You might lose the chat's history when resetting to an old entity revision. If this happens, you can solve the associated error by clearing the chat's history.

Reference

For a more technical API reference, please see the following pages:

Chat

A Chat enables the user to have a conversation with an LLM from within the app editor.

import viktor as vkt


class Parametrization(vkt.Parametrization):
    chat = vkt.Chat("Chatbot", method="call_llm")

The conversation (including the latest user prompt) can be obtained through the params argument and is of type ChatConversation.

get_messages can be used to get the full history of the conversation, which returns a list of messages with format:

{
    'role': str,        # user | assistant
    'content': str,     # the actual message
}

The messages can be directly fed to the LLM (some reformatting might be necessary, depending on which LLM you are using).

The streamed text should then, together with the conversation, be returned from the Chat method within a ChatResult. Since the LLM stream might contain more than only text responses, some processing is likely necessary (depending on which LLM you are using). The maximum response size is 75000 characters.

An LLM implementation using the Anthropic package, for example, could look like:

class Controller(vkt.Controller):
    ...

    def call_llm(self, params, **kwargs):
        conversation = params.chat
        
        client = Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
        llm_response = client.messages.create(
            max_tokens=1024, 
            messages=conversation.get_messages(),
            model="claude-3-5-sonnet-latest",
        )

        return vkt.ChatResult(conversation, llm_response.content[0].text)

Or, when a streamed response is desired:

class Controller(vkt.Controller):
    ...

    def call_llm(self, params, **kwargs):
        conversation = params.chat
        
        client = Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
        llm_response_stream = client.messages.create(
            max_tokens=1024, 
            messages=conversation.get_messages(),
            model="claude-3-5-sonnet-latest",
            stream=True
        )
        
        # filter only the actual text addition (delta) events from the streamed response
        text_stream = map(
            lambda delta_chunk: delta_chunk.delta.text,
            filter(lambda chunk: chunk.type == 'content_block_delta', llm_response_stream)
        )
        
        return vkt.ChatResult(conversation, text_stream)

Expand to see all available arguments

In alphabetical order:

first_message (optional): the first message that will appear on top of the empty conversation
```
chat = vkt.Chat(..., first_message="How can I help you?")
```
flex (optional): the width of the field between 0 and 100 (default=33)
```
chat = vkt.Chat(..., flex=50)
```
method (required): the controller method that will be triggered when the user clicks the "send" button
```
chat = vkt.Chat(..., method='call_llm')
```
placeholder (optional): placeholder message in the user prompt
```
chat = vkt.Chat(..., placeholder="Ask anything")
```
visible (optional): can be used when the visibility depends on other input fields
```
chat = vkt.Chat(..., visible=vkt.Lookup("another_field"))
```
See 'Hide a field' for elaborate examples.