ExaDev Public Notebook

Search

SearchSearch

Using a local LLM with Breadboard via LM Studio

18 Jun 20243 min read

Introduction

Here we will demonstrate the integration between Breadboard and a language model running locally on LM Studio.

Objective

The goal of this project is to showcase how easy it is to integrate Breadboard with a language model and to provide a step-by-step example using the LM Studio software.

Prerequisites

  • LM Studio software installed on your local machine, see Source section for more detailed steps.
  • A basic understanding of language models and Breadboard.

Inputs and Parameters

1. Max Tokens

  • Description: Specifies the maximum length of the response generated by the language model.
  • Example Value: -1 (indicating no limit on response length for this example).

2. Stream

  • Description: A flag that indicates whether the language model should return the response as it is being generated or wait for completion.
  • Example Value: true or false.

3. System Context

  • Description: Defines the context in which the language model should reply.
  • Example Value: "Professional Chef" (in this example, the language model responds as a professional chef).

4. Temperature

  • Description: Adjusts the creativity of the language model’s responses. Lower values make the model more deterministic.
  • Example Value: 1 (to ensure a balanced response).

5. User Context

  • Description: The question or command provided to the language model.
  • Example Value: "List the ingredients needed to make a cake."

Step-by-Step Guide

Step 1: Setting Up Inputs

  1. Max Tokens: Set to -1 for this example.
  2. Stream: Set to true or false based on your requirement.
  3. System Context: Set to "Professional Chef".
  4. Temperature: Set to 1 to maintain response consistency.
  5. User Context: Enter the question "List the ingredients needed to make a cake."

Step 2: Running the Board

  • Execute the board with the provided inputs.
  • Open LM Studio to observe the response generation in real time.

Step 3: Viewing the Response

  • The response will be generated word by word, if the stream flag is set to True LM Studio will return a response for each word. If set to False LM studio will wait until the whole response is completed and return it all in its entirety.
  • Observe the final response in LM Studio or on your board.

Example Output

Request

Max Tokens: -1
Stream: true
System Context: "Professional Chef"
Temperature: 1
User Context: "List the ingredients needed to make a cake."

Response

To make a cake, you will need the following ingredients:
 
- Flour
- Sugar
- Eggs
- Butter
- Baking powder
- Milk
- Vanilla extract

Conclusion

This example illustrates the simplicity and potential of integrating breadboards with language models using LM Studio. By following the steps outlined above, you can create responsive and contextually aware applications.

Source

README

Setup

Graph

Board

Graph View

Backlinks