不用find-tune也能保證Llama 2不亂說話！善用“Grammar”客製化輸出內容

本篇接續上篇開源的Llama 2有哪些玩法？本篇文章介紹開發者與Llama 2的互動方式，說明如何在Llama 2運用GBNF。

LLM（Large Language Module）最讓人詬病的問題之一是「亂說話」，得到不是預期的回應。若問題發生在日常對話影響不大，若開發者串接後續商業邏輯，那麼問題就多了，輕則拋出錯誤處理，重則導致一連串錯誤邏輯。

本篇文章介紹如何利用文法（Grammar）規則限縮Llama 2的生成內容，讓輸出維持在可控範圍同時保留LLM發揮創意、生成內容的優勢。

GBNF

llama.cpp用GBNF (GGML BNF)格式來定義形式文法（formal grammars），例如你可以用來限制模型生成JSON格式或只用Emojis做出回應，詳細說明可參考原文GBNF Guide。

以下GBNF文法範例會示範套用過後的效果，讓你能更容易理解它帶來的好處。

GBNF範例

List

List是llama.cpp內的範例grammar，旨在讓Llama 2輸出條列式內容。文法內容如下：

root ::= item+
item ::= "- " [^\r\n\x0b\x0c\x85\u2028\u2029]+ "\n"

輸入執行指令：

./main -m ./models/llama-2-13b-chat/ggml-model-Q4_K_M.gguf -n 256 --grammar-file grammars/list.gbnf -ins

Example: List

Car

List結構非常單純，再來測試複雜一點的例子，文法內容如下：

root ::= CarAndOwner
PerformanceFeature ::= "{"   ws   "\"engine\":"   ws   string   ","   ws   "\"horsepower\":"   ws   number   ","   ws   "\"topSpeed\":"   ws   number   "}"
PerformanceFeaturelist ::= "[]" | "["   ws   PerformanceFeature   (","   ws   PerformanceFeature)*   "]"
SafetyFeature ::= "{"   ws   "\"airbags\":"   ws   number   ","   ws   "\"parkingSensors\":"   ws   number   ","   ws   "\"laneAssist\":"   ws   number   "}"
SafetyFeaturelist ::= "[]" | "["   ws   SafetyFeature   (","   ws   SafetyFeature)*   "]"
AudioFeature ::= "{"   ws   "\"brand\":"   ws   string   ","   ws   "\"speakers\":"   ws   number   ","   ws   "\"hasBluetooth\":"   ws   boolean   "}"
AudioFeaturelist ::= "[]" | "["   ws   AudioFeature   (","   ws   AudioFeature)*   "]"
Features ::= "{"   ws   "\"audio\":"   ws   AudioFeature   ","   ws   "\"safety\":"   ws   SafetyFeature   ","   ws   "\"performance\":"   ws   PerformanceFeature   "}"
Featureslist ::= "[]" | "["   ws   Features   (","   ws   Features)*   "]"
Owner ::= "{"   ws   "\"firstName\":"   ws   string   ","   ws   "\"lastName\":"   ws   string   ","   ws   "\"age\":"   ws   number   "}"
Ownerlist ::= "[]" | "["   ws   Owner   (","   ws   Owner)*   "]"
Car ::= "{"   ws   "\"make\":"   ws   string   ","   ws   "\"model\":"   ws   string   ","   ws   "\"year\":"   ws   number   ","   ws   "\"colors\":"   ws   stringlist   ","   ws   "\"features\":"   ws   Features   "}"
Carlist ::= "[]" | "["   ws   Car   (","   ws   Car)*   "]"
CarAndOwner ::= "{"   ws   "\"car\":"   ws   Car   ","   ws   "\"owner\":"   ws   Owner   "}"
CarAndOwnerlist ::= "[]" | "["   ws   CarAndOwner   (","   ws   CarAndOwner)*   "]"
string ::= "\""   ([^"]*)   "\""
boolean ::= "true" | "false"
ws ::= [ \t\n]*
number ::= [0-9]+   "."?   [0-9]*
stringlist ::= "["   ws   "]" | "["   ws   string   (","   ws   string)*   ws   "]"
numberlist ::= "["   ws   "]" | "["   ws   string   (","   ws   number)*   ws   "]"

這段GBNF旨在結構化呈現車子的訊息，訊息包含車子的「品牌」、「型號」、「年份」、「顏色」、「性能」、「安全性」、「音響」、所屬人等資訊。

輸入執行指令：

./main -m ./models/llama-2-13b-chat/ggml-model-Q4_K_M.gguf -n 256 --grammar-file grammars/car.gbnf -ins --repeat_penalty 1.17647 --ctx_size 2048 --temp 0.2

Example: Car

將這段輸出整理一下：

{
    "car": {
        "make": "Tesla",
        "model": "Model S",
        "year": 2015,
        "colors": [
            "black"
        ],
        "features": {
            "audio": {
                "brand": "Bowers & Wilkins",
                "speakers": 17,
                "hasBluetooth": true
            },
            "safety": {
                "airbags": 8,
                "parkingSensors": 4,
                "laneAssist": 2015
            },
            "performance": {
                "engine": "electric",
                "horsepower": 762,
                "topSpeed": 155
            }
        }
    },
    "owner": {
        "firstName": "Elon",
        "lastName": "Musk",
        "age": 49
    }
}

測試結果很好，如此多層的巢狀結構也能夠輕易輸出，這讓Llama 2在商業邏輯的應用上變得更容易掌控與串接。

Weather

再來一個範例，來限制Llama 2只能回答特定的字詞，文法內容如下：

root ::= answer
answer ::= (weather | complaint | yesno)
weather ::= ("Sunny." | "Cloudy." | "Rainy.")
complaint ::= "I don't like talking about the weather."
yesno ::= ("Yes." | "No.")

輸入執行指令：

./main -m ./models/llama-2-13b-chat/ggml-model-Q4_K_M.gguf -n 256 --grammar-file grammars/weather.gbnf -ins --repeat_penalty 1.17647 --ctx_size 256 --temp 0.2

Example: Weather

Example: Yes or No

Example: Complaint

結果很棒！完全限制了Llama 2不會亂說多餘的話。

Book

再來看最後一個書本資訊的範例，文法內容如下：

root ::= Book
Auther ::= "{"   ws   "\"firstName\":"   ws   string   ","   ws   "\"lastName\":"   ws   string   ","   ws   "\"age\":"   ws   number   "}"
Autherlist ::= "[]" | "["   ws   Auther   (","   ws   Auther)*   "]"
Book ::= "{"   ws   "\"title\":"   ws   string   ","   ws   "\"subtitle\":"   ws   string   ","   ws   "\"authers\":"   ws   Autherlist   ","   ws   "\"genres\":"   ws   stringlist   ","   ws   "\"pages\":"   ws   number   ","   ws   "\"description\":"   ws   string   ","   ws   "\"isbn\":"   ws   string   "}"
Booklist ::= "[]" | "["   ws   Book   (","   ws   Book)*   "]"
string ::= "\""   ([^"]*)   "\""
boolean ::= "true" | "false"
ws ::= [ \t\n]*
number ::= [0-9]+   "."?   [0-9]*
stringlist ::= "["   ws   "]" | "["   ws   string   (","   ws   string)*   ws   "]"
numberlist ::= "["   ws   "]" | "["   ws   string   (","   ws   number)*   ws   "]"

輸入執行指令：

./main -m ./models/llama-2-13b-chat/ggml-model-Q4_K_M.gguf -n 256 --grammar-file grammars/book.gbnf -ins

Example: Book 1

整理一下結構：

{
    "title": "The Alchemist",
    "subtitle": "A Fable About Following Your Dreams",
    "authers": [
        {
            "firstName": "Paulo",
            "lastName": "Coelho",
            "age": 71
        }
    ],
    "genres": [
        "Fantasy",
        "Adventure",
        "Self-Help"
    ],
    "pages": 224,
    "description": "A magical tale about a shepherd boy named Santiago who embarks on a journey to fulfill his personal legend and find his treasure. A story of spiritual and self-discovery that will inspire you to follow your dreams and listen to your heart.",
    "isbn": "978-0-14-303533-6"
}

Example: Book 2

整理一下結構：

{
    "title": "The Lean Startup",
    "subtitle": "How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses",
    "authers": [
        {
            "firstName": "Eric",
            "lastName": "Ries",
            "age": 57
        }
    ],
    "genres": [
        "Business",
        "Entrepreneurship",
        "Management"
    ],
    "pages": 336,
    "description": "A comprehensive guide to launching and growing a successful startup, with practical advice on how to use innovation, experimentation, and customer feedback to build a sustainable business.",
    "isbn": "978-0-385-34011-7"
}

得到的結果也符合我的預期，唯一美中不足的是部分內容是虛構的，例如ISBN無法查到這本書，得到的內容結構完全符合我要的格式，對於會後商業邏輯的串接完全沒有問題。

GBNF生成工具

GBNF文法規則要怎麼產生呢？從llama.cpp的GBNF Guide，瞭解規則後就可以開始撰寫屬於自己的規則，這是其中一種做法。

對於剛入門的新手推薦使用現成的工具幫助快速上手：

json-schema-to-grammar

如果你熟悉JSON Schema可以用llama.cpp提供的python腳本json_schema_to_grammar.py轉換GBNF。

Grammer Builder

如果你熟悉Typescript可以用別人已經做好的網頁應用Grammar Builder轉換GBNF。

在試用Llama 2的過程中，有發現它的輸出結構很不穩定，有時候長篇大論，有時候以清單方式呈現，想到對話類型的簡單應用之外，中間一度很苦惱Llama 2要如何串接到更複雜的應用，例如用特定格式整理汽車、書本資訊。

仔細瞭解後發現GBNF完美解決了這個問題，並且不需要fine-tune模型就能得到結果，效果非常好，唯一美中不足的部分是無法避免Llama 2的幻想，為了讓Llama 2理解業務內容，最終還是需要fine-tune模型才能做到，下一篇文章來介紹如何fine-tune Llama 2。

本文同步刊登於Softaverse - 不用find-tune也能保證Llama 2不亂說話！善用“Grammar”客製化輸出內容