Exploring AI: Running Large Language Models on Laptop Using IPEX-LLM (Part 2: Initial Results and Analysis)

In the previous part of this series, we discussed the basics of running large language models (LLMs) on GPUs and iGPUs, specifically using the IPEX-LLM. Now, we move on to the practical aspects and my initial results running these models locally.

Running Models with Ollama and IPEX-LLM

For this experiment, I used Ollama in conjunction with IPEX-LLM. Ollama provides a framework for building and running language models on local machines, and its integration with IPEX-LLM enhances performance on Intel GPUs, making it feasible to execute complex AI models locally.

My Journey with Mistral 7B on an iGPU

I tested the capabilities of IPEX-LLM by running two instruct versions of Mistral 7B, Q4_0 and Q4_K_M (both with 4-bit quantization, with K_M being more accurate), on an Intel Core Ultra 9 185GH machine equipped with an iGPU. The process was surprisingly smooth, thanks to the clear instructions provided on the IPEX-LLM website. While I encountered minor roadblocks, like navigating the HuggingFace website to find the correct model versions, the overall setup was straightforward.

Optimizing for Efficiency: Success with iGPU

Despite running on iGPU, the response time was quicker than the language models operating on cloud servers that I have tried. I recorded the average ms/token, and the results were impressive:

ms/token (milliseconds per token): Simply put, lower ms/token values indicate faster model performance. For example, if one model processes tokens at 5 ms/token, it will generate 200 tokens per second, while another model with 10 ms/token would only generate half as much.

Mistral Q4_0: Achieved 62 ms/token

Mistral Q4_K_M: Achieved 82 ms/token

GPU Usage

There was no significant difference in terms of performance for both models, as the average GPU utilization hovered around 95%, showcasing remarkable efficiency on the iGPU. This is significant compared to traditional reliance on powerful, dedicated GPUs or cloud resources.

GPU usage for Mistral Q4_0

GPU usage for Mistral Q4_K_M

Why Local AI with iGPUs Matters

The ability to run complex AI models on iGPUs holds tremendous potential. It democratizes AI by making powerful tools accessible to a wider range of users, including:

• Educational institutions: Equip students with the power of AI for learning and exploration, allowing hands-on experience without needing expensive hardware.
• Small startups and businesses: Access AI capabilities without large initial costs, minimizing the need for new purchases, and lowering operational costs. This opens up the ability to develop and test AI models locally before scaling up.
• Hobbyists and individual developers: Easier accessibility to AI tools, enabling innovation and experimentation in AI.

Next Step

Stay tuned for the final part of this series, where I’ll share my overall experience working with IPEX-LLM and provide helpful tips along the way.