Starting with Qwen2.5-Coder-7B-Instruct Locally and using HuggingFace Inference Endpoint

Introduction

Qwen2.5-Coder-7B-Instruct is a versatile language model developed by Alibaba Cloud, optimized for code generation and technical instruction. This guide will help you set up and run the model locally for various coding tasks and queries, leveraging the power of your CUDA-compatible GPU.

Table Of Contents

1. Running Inference Locally.
2. Using Hugging Face Inference Endpoint

1. Running locally

Prerequisites

Before diving into the setup, make sure you have the following prerequisites in place:

1. Hardware: A CUDA-compatible GPU.
2. Software:

• Python 3.10 or higher.
• Installed versions of torch, transformers, accelerate, and other necessary dependencies.

Installation

First, install the required Python libraries. Open your terminal and run the following command:

pip install torch transformers accelerate

Ensure your CUDA and GPU drivers are properly installed and configured to leverage the GPU for faster computations.

Local Inference Setup

We will use the transformers library to load and run the model locally. Here's the basic setup for loading the Qwen2.5-Coder-7B-Instruct model and tokenizer.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Define the model name
model_name = "Qwen/Qwen2.5-Coder-7B-Instruct"

# Load the model with automatic device mapping and caching
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",  # Automatically select the best data type (FP16/FP32)
    device_map="auto",   # Automatically allocate model across available devices
    cache_dir='cache'    # Local cache directory for storing the model
)

# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained(
    model_name,
    cache_dir='cache'
)

This setup will automatically detect your GPU and utilize it for running the model efficiently.

Running a Code Generation Prompt

Let’s start by generating a piece of code using a sample prompt. We will use a simple instruction for generating a quicksort algorithm.

# Define the prompt
prompt = "write a quick sort algorithm."

# Format the messages for the model
messages = [
    {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."},
    {"role": "user", "content": prompt}
]

# Prepare the input text using the tokenizer's chat template
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

# Tokenize the input
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

# Generate a response
generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=512  # Maximum number of tokens to generate
)

# Remove the input tokens from the output
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

# Decode and print the response
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

print(response)

Result

The model responds with a Python implementation of the QuickSort algorithm:

def quick_sort(arr):
    if len(arr) <= 1:
        return arr
    else:
        pivot = arr[len(arr) // 2]
        left = [x for x in arr if x < pivot]
        middle = [x for x in arr if x == pivot]
        right = [x for x in arr if x > pivot]
        return quick_sort(left) + middle + quick_sort(right)

# Example usage:
arr = [3, 6, 8, 10, 1, 2, 1]
sorted_arr = quick_sort(arr)
print("Sorted array:", sorted_arr)

Testing Another Prompt

Let’s test the model with a non-coding query to see its general capabilities.

prompt = "How many 'r's are in word 'strawberry'?"
messages = [
    {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."},
    {"role": "user", "content": prompt}
]

# Prepare the input text
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

# Tokenize and generate a response
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=512
)

# Decode and print the response
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)

Result

The model responds accurately:

The word "strawberry" contains 3 'r's.

Troubleshooting Tips

1. Memory Issues: If you encounter memory issues, try reducing the batch size or offloading part of the model to the CPU.
2. CUDA Compatibility: Ensure that your CUDA and PyTorch versions are compatible.
3. Cache Management: Clear the cache directory periodically to free up disk space.

2. Running on Hugging Face Inference Endpoint

First you have to create dedicated endpoint on HuggingFace (defaults will do)

Navigate to https://ui.endpoints.huggingface.co/<your user name>/new

After it’s procured you can access it using HTTPS or OpenAI Python wrapper

HTTPS

import requests

url = "https://<your_ip>.us-east-1.aws.endpoints.huggingface.cloud/v1/chat/completions"
headers = {
    "Authorization": "Bearer your_key",
    "Content-Type": "application/json"
}
payload = {
    "model": "tgi",
    "messages": [
        {
            "role": "user",
            "content": "How many 'r's are in word 'strawberry'?"
        }
    ],
    "stream": True,
    'temperature': 2.0,
    "max_tokens": 500
}
import json
response = requests.post(url, headers=headers, json=payload, stream=True)
for chunk in response.iter_content(chunk_size=None):
    if chunk:
        # Decode the chunk from bytes to a string
        data_chunk = chunk.decode('utf-8').strip()

        # Check if the chunk starts with 'data: ' and remove it
        if data_chunk.startswith("data: "):
            json_data = data_chunk[len("data: "):]  # Remove 'data: ' prefix

            try:
                # Parse the remaining string as JSON
                parsed_data = json.loads(json_data)

                # Extract the 'content' field from the 'choices' array
                if 'choices' in parsed_data:
                    content = parsed_data['choices'][0]['delta'].get('content', '')
                    # Append the content to the full message
                    print(content, end="")
                    # Print the content incrementally

            except json.JSONDecodeError:
                print("Error parsing JSON:", data_chunk)

Output

Results are wonky — they differ with every test. I think you need to set temperature and other params suppressing hallucination.

There are two 'r's in the word 'strawberry'
...
The word "strawberry" contains three 'r's
...
The word 'strawberry' contains 3 'r's.
...
There are 2 'r's in the word 'strawberry'

OpenAI Python API

# If necessary, install the openai Python library by running 
# pip install openai

from openai import OpenAI

client = OpenAI(
 base_url="https://your_ip.us-east-1.aws.endpoints.huggingface.cloud/v1/", 
 api_key="your_key" 
)

chat_completion = client.chat.completions.create(
 model="tgi",
 messages=[
    {
        "role": "user",
        "content": "How many 'r's are in word 'strawberry'?"
    }
],
 stream=True,
 max_tokens=50
)

for message in chat_completion:
 print(message.choices[0].delta.content, end="")

Output

Results are much better (because of OpenAI the client code defaults)

The word "strawberry" contains 3 'r's.
...
There are 3 'r's in the word 'strawberry'.
...
To determine the number of 'r's in the word "strawberry", let's go through it step by step:

1. The word is "strawberry".
2. Let's count the occurrences of 'r':
   - S (

Troubleshooting Tips

Because of remote execution this section is limited to making sure you have HUGGING_FACE_TOKEN set in your environment.

Conclusion

This guide provided a brief introduction to running the Qwen2.5-Coder-7B-Instruct model locally. With this setup, you can generate code snippets, provide explanations, and much more, leveraging the model’s capabilities. Experiment with different prompts and explore the vast potential of this powerful language model.