Running local LLMs on laptop (CPU) — A tutorial on LM Studio

The future of AI is open-source and local models.

This is my first post on local LLMs tutorial series . Read it till end.

image — shutterstock.com

As soon as local 7 billion models like Mistral , Vicuna , Zephyr ,Openchat , Openhermes, Llama-2 ,etc was released , I was curious to run it on my laptop , but as I was not having dedicated Graphics card , so , I faced problems .

While I was searching , I found out something very amazing which many developers are unaware of .

Why local Models ?

Privacy Assurance

One of the primary motivations for using local LLMs is to uphold data privacy. By processing information on your local machine, you eliminate concerns related to data exposure and unauthorized access. This becomes particularly significant when dealing with sensitive data or scenarios where privacy compliance is paramount.

Offline Accessibility

Local models provide a solution for scenarios with limited or no internet connectivity. This ensures that AI applications remain functional even in environments with restricted or intermittent network access. It’s a game-changer for applications that require consistent performance regardless of the network status.

Experimentation Freedom

Local models empower developers to experiment freely. You can iterate rapidly, fine-tune models, and explore creative use cases without being bound by external API constraints. This flexibility fosters innovation and allows developers to push the boundaries of what’s possible.

Running LLMs on CPU — A Practical Guide

Format Conversion

Before unleashing the power of local models, it’s crucial to convert LLMs into compatible formats like GGML or GGUF from safetensors . These formats, conveniently available on platforms like Hugging Face, serve as the bridge between the model and the local environment (thanks to **TheBloke** ) .

Using Ctransformers and GPT4All

• Then you can use ctransformers or GPT4All for running it locally . Let’s highlight their significance, emphasizing their role in creating a seamless chat interface. This interface not only simplifies interaction with the model but also opens doors for innovative applications beyond conventional use cases.

#installing
pip install ctransformers
pip install gpt4all

# chatting after installing
from ctransformers import AutoModelForCausalLM
llm = AutoModelForCausalLM.from_pretrained(
                                    'models/llama-2–7b-chat.ggmlv3.q4_0.bin',
                                    model_type='llama')py

from gpt4all import GPT4All
llm = GPT4All("models/llama-2–7b-chat.ggmlv3.q4_0.bin")

• The best among all is to download and run **LM Studio** ,which does not require any above mentioned steps to do. It is a standalone system which does all for you.

Why LM Studio ?

User-Friendly Interface

LM Studio stands out for its user-friendly interface, making it accessible to a broader audience. The absence of coding requirements lowers the entry barrier, allowing individuals with diverse backgrounds to harness the power of sophisticated models.

Simplified Workflow

The simplified workflow of LM Studio, from download to launch, streamlines the process of interacting with different models. This efficiency is crucial for developers and enthusiasts who seek a hassle-free experience in working with powerful language models.

Local Server Support

The integration of a local HTTP server, mirroring OpenAI’s API, adds a layer of convenience for developers. It facilitates seamless integration into projects, providing a local environment that emulates the functionality of cloud-based APIs.

4. Uncensored chat — since it supports uncensored models also , so you can directly chat with it for blocked contents . for example -

As you can see in the bottom , it took only 2.27 seconds to generate text with only cpu and 16 GB RAM windows laptop. It highlights the efficiency of local processing.

Multimodal Support

Beyond language models, LM Studio’s support for vision models extends its utility. Developers can leverage this capability for tasks such as image summarization, expanding the horizons of what can be achieved with a single platform and also use it with it as an API.

How to use LM Studio

1. Download and Install:

• Visit the LM Studio website (https://lmstudio.ai/) and download the installer for your operating system (Windows, macOS, or Linux).
• Run the installer and follow the on-screen instructions.

2. Launch LM Studio:

• Once installed, launch the LM Studio application.

3. Find a Model:

• Browse Featured Models: Explore the models suggested on the home screen like zephyr -7b , code-llama-7b , vicuna-7b , mistral-7b-instruct , etc .
• Search for a Model: Use the search bar to find specific models by name or type.

4. Download a Model:

• Select the model you want to use and click the “Download” button.
• Ensure you have sufficient storage space as LLMs can be large , and try to use 4–5 bit quantization models which take only about 5 GB storage and are really fast .

5. Configure Settings (Optional):

• GPU Selection: If you have a compatible GPU, you can enable GPU acceleration for faster performance.
• Temperature: Adjust the temperature to control creativity and randomness in model responses.
• Response Length: Set the desired length for model responses.
• Other Settings: Explore additional settings for fine-tuning and experimentation.

6. Interact with the Model:

• chat — you can directly chat with it by visiting AI chat section, selecting model and giving prompt.
• Local Server — you can visit local server section and start server . This will start server **http://localhost:1234/v1/models **which will act in similar way as OpenAI API . Then you can copy the code given on right panel and integrate it in your projects .

from openai import OpenAI

# Point to the local server
client = OpenAI(base_url="http://localhost:1234/v1", api_key="not-needed")

history = [
    {"role": "system", "content": "You are an intelligent assistant. You always provide well-reasoned answers that are both correct and helpful."},
    {"role": "user", "content": "Hello, introduce yourself to someone opening this program for the first time. Be concise."},
]

while True:
    completion = client.chat.completions.create(
        model="local-model", # this field is currently unused
        messages=history,
        temperature=0.7,
        stream=True,
    )

    new_message = {"role": "assistant", "content": ""}

    for chunk in completion:
        if chunk.choices[0].delta.content:
            print(chunk.choices[0].delta.content, end="", flush=True)
            new_message["content"] += chunk.choices[0].delta.content

    history.append(new_message)

    # Uncomment to see chat history
    # import json
    # gray_color = "\033[90m"
    # reset_color = "\033[0m"
    # print(f"{gray_color}\n{'-'*20} History dump {'-'*20}\n")
    # print(json.dumps(history, indent=2))
    # print(f"\n{'-'*55}\n{reset_color}")

    print()
    history.append({"role": "user", "content": input("\n>>> ")})

After starting local server ,

integrating local model in project- using openchat 7b

• vision — you can download vision models like NousHermes vision and start with it in AI chat section

7. Experiment and Explore:

• Try different prompts to see the model’s capabilities.
• Use model-specific features like text generation, image summarization ,translation, code generation, or question answering with different models and different parameters.

In conclusion, the combination of local LLMs and tools like LM Studio marks a significant stride towards democratizing AI. From privacy and offline accessibility to a simplified workflow and uncensored chat capabilities, these developments pave the way for a more inclusive and innovative AI landscape.