How can you deploy llama 3.2 1b as an API endpoint on hugging face space ?

Table of contents :

1. Introduction
2. Install LLama.cpp python lib on your machine
3. Setup Code for llama cpp in python and Fastapi
4. Containerise the above application using docker
5. Create a Docker space on hugging face spaces
6. Deploy the container on hugging face spaces

Introduction :

In this blog we are going to cover , how we can deploy an LLM , Llama 3.2 1b q4 , and expose it as an endpoint on hugging face spaces on a docker space . We are also going to containerise this application .

Step 1 : Install LLama.cpp python lib on your machine

pip install llama-cpp-python

This will take some time , but you just have to run this command .

Step 2 : Setup Code for llama cpp in python and Fastapi

Now lets setup the code for building an endpoint in FastAPI .

Initially let’s just setup the code to download and Initialise the LLM.

from llama_cpp import Llama

# Initialize the LLM 
llm = Llama.from_pretrained(
    repo_id="hugging-quants/Llama-3.2-1B-Instruct-Q4_K_M-GGUF", 
    filename="llama-3.2-1b-instruct-q4_k_m.gguf"
)

Let’s setup the FastAPI endpoint code .

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from llama_cpp import Llama

# Initialize the LLM once when the application starts
llm = Llama.from_pretrained(
    repo_id="hugging-quants/Llama-3.2-1B-Instruct-Q4_K_M-GGUF", 
    filename="llama-3.2-1b-instruct-q4_k_m.gguf"
)

app = FastAPI()

class ChatRequest(BaseModel):
    message: str

@app.post("/chat")
async def chat_completion(request: ChatRequest):
    try:
        response = llm.create_chat_completion(
            messages=[
                {"role": "user", "content": request.message}
            ]
        )
        return {
            "response": response['choices'][0]['message']['content']
        }
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

Above is the entire code that will download and setup an LLM endpoint .

Step 3 : Containerise the above application using docker

Now lets setup the Dockerfile.

# Use the official Python image with a specific version
FROM python:3.12.5-slim

RUN useradd -m -u 1000 user
USER user
ENV PATH="/home/user/.local/bin:$PATH"

# Set the working directory in the container
WORKDIR /app

# Install system dependencies as root
USER root
RUN apt-get update && apt-get install -y --no-install-recommends \
    gcc \
    g++ \
    cmake \
    git \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

# Switch back to the non-root user
USER user

# Copy the requirements file into the container
COPY --chown=user ./requirements.txt requirements.txt

# Install dependencies
RUN pip install --no-cache-dir -r requirements.txt

# Copy the application code into the container
COPY --chown=user ./llm.py /app/llm.py
COPY --chown=user ./llama-3.2-1b-instruct-q4_k_m.gguf /app/llama-3.2-1b-instruct-q4_k_m.gguf

# Expose the application port
EXPOSE 7860

# Define the command to run the application
CMD ["uvicorn", "llm:app", "--host", "0.0.0.0", "--port", "7860"]

Above covers installing all the c compilers you will require for llama cpp .

Note : Change your your_file_name in the above code , to your file name where you have saved the FastAPI code.

Let’s also setup the requirements.txt file . I have directly provided the file here and you have to copy and save it as requirements.txt .

annotated-types==0.7.0
anyio==4.6.2.post1
certifi==2024.8.30
charset-normalizer==3.4.0
click==8.1.7
diskcache==5.6.3
fastapi==0.115.5
filelock==3.16.1
fsspec==2024.10.0
h11==0.14.0
huggingface-hub==0.26.2
idna==3.10
Jinja2==3.1.4
llama_cpp_python==0.3.2
MarkupSafe==3.0.2
numpy==2.1.3
packaging==24.2
pydantic==2.9.2
pydantic_core==2.23.4
PyYAML==6.0.2
requests==2.32.3
sniffio==1.3.1
starlette==0.41.2
tqdm==4.67.0
typing_extensions==4.12.2
urllib3==2.2.3
uvicorn==0.32.0

Now to install these libraries , just run the command below .

pip install -r requirements.txt

Step 4 : Create a Docker space on hugging face spaces

Login to your huggingface account and go to the Spaces section .

hugging spaces

Click on Create new Space and you would be directed to this screen .

hugging face space creation

Now as shown in the below image , enter your desired space name and Select Space SDK as Docker and select Blank docker .

Now let the space hardware be the same and also select Public so that you can test your endpoint on postman , now click on create space .

Step 5 : Deploy the container on hugging face spaces

Once you click on create , you can see the steps in the commit your code to your hugging face space , as given in the image below .

Note : Upload/commit all of the code .

Conclusion :

Above blog tells you how can you deploy an LLM on CPU on a free hugging face spaces . Stay tuned for more such blogs on latest technology that is out there !!