Running llama 2 on CPU

Photo by Mateusz D on Unsplash

Setup

Signup with meta here to get permission here or OR find instruction here

Signup with Hugging Face here with the same email_id you used to signup with meta.

Download 3B ggml model here llama-2–13b-chat.ggmlv3.q4_0.bin

Note: Download takes a while due to the size, which is 6.8G.

Setup python and virtual environment

Install langchain library which is a wrapper around llama.cpp

from langchain.llms import LlamaCpp
from langchain import PromptTemplate, LLMChain
from langchain.callbacks.manager import CallbackManager
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler

callback_manager = CallbackManager([StreamingStdOutCallbackHandler()])

llm = LlamaCpp(
    model_path="./models/llama-2-13b-chat.ggmlv3.q4_0.bin",
    input={"temperature": 0.75, "max_length": 2000, "top_p": 1},
    callback_manager=callback_manager,
    verbose=True,
)

prompt = """
 Who is mahatma gandhi?
"""
prompt = prompt.format(text)
llm(prompt)

When you run this program you should see output from the trained llama model.

Congratulations if you are able to run this successfully.

My setup is Mac Pro (2.6 GHz 6-Core Intel Core i7, Intel Radeon Pro 560X 4 GB). Although I do have a small gpu that came with mac but you should be able to run without this.

There is a comprehensive guide from replicate which you can refer for more details — Replicate

Leave a comment if you have a problem.