本地推理,單機執行,MacM1晶片系統基於大語言模型C++版本LLaMA部署“本地版”的ChatGPT

劉悅的技術部落格發表於2023-03-24
MacACM晶片模型C++ChatGPT

OpenAI公司基於GPT模型的ChatGPT風光無兩，眼看它起朱樓，眼看它宴賓客，FaceBook終於坐不住了，釋出了同樣基於LLM的人工智慧大語言模型LLaMA，號稱包含70億、130億、330億和650億這4種引數規模的模型，引數是指神經網路中的權重和偏置等可調整的變數，用於訓練和最佳化神經網路的效能，70億意味著神經網路中有70億個引數，由此類推。

在一些大型神經網路中，每個引數需要使用32位或64位浮點數進行儲存，這意味著每個引數需要佔用4位元組或8位元組的儲存空間。因此，對於包含70億個引數的神經網路，其儲存空間將分別為8 GB或12GB。

此外，神經網路的大小不僅取決於引數的數量，還取決於神經元的數目，層數和其他結構引數等。因此，70億的神經網路可能會佔用更多的儲存空間，具體取決於網路的結構和實現細節。

因此這種體量的模型單機跑絕對夠我們喝一壺，所以本次使用最小的LLaMA 7B模型進行測試。

LLaMA專案安裝和模型配置

和Stable-Diffusion專案如出一轍，FaceBook開源的LLaMA專案預設寫死使用cuda模式，這也就意味著必須有 NVIDIA 的 GPU來訓練和執行，不過好在大神GeorgiGerganov 用 C++ 基於 LLaMA 專案重寫了一個跑在 CPU 上的移植版本 llama.cpp應用。

llama.cpp首先適配的就是蘋果的M系列晶片，這對於果粉來說無疑是一個重大利好，首先透過命令拉取C++版本的LLaMA專案：

git clone https://github.com/ggerganov/llama.cpp

隨後進入專案目錄：

llama.cpp

在專案中，需要單獨建立一個模型資料夾models:

mkdir models

隨後去huggingface官網下載LLaMA的7B模型檔案：https://huggingface.co/nyanko7/LLaMA-7B/tree/main

是的，主模型檔案已經達到了13.5gb之巨，如果本地硬碟空間告急，請謹慎下載。

隨後在models目錄建立模型子目錄7B:

mkdir 7B

將tokenizer.model和tokenizer_checklist.chk放入和7B平行的目錄中：

➜  models git:(master) ✗ ls  
7B                      tokenizer.model         tokenizer_checklist.chk

隨後將checklist.chk consolidated.00.pth和params.json放入7B目錄中：

➜  7B git:(master) ✗ ls  
checklist.chk       consolidated.00.pth  params.json

至此，模型就配置好了。

LLaMA模型轉換

由於我們沒有使用FaceBook的原版專案，所以它的模型還需要進行轉換，也就是轉換為當前C++版本的LLaMA可以執行的模型。

這裡透過Python指令碼進行轉換操作：

python3 convert-pth-to-ggml.py models/7B/ 1

第一個引數是模型所在目錄，第二個引數為轉換時使用的浮點型別，使用 float32，轉換的結果檔案會大一倍，當該引數值為 1時，則使用 float16 這個預設值，這裡我們使用預設資料型別。

程式輸出：

➜  llama.cpp git:(master) ✗ python convert-pth-to-ggml.py models/7B/ 1  
{'dim': 4096, 'multiple_of': 256, 'n_heads': 32, 'n_layers': 32, 'norm_eps': 1e-06, 'vocab_size': -1}  
n_parts = 1  
  
Processing part 0  
  
Processing variable: tok_embeddings.weight with shape: torch.Size([32000, 4096]) and type: torch.float16  
Processing variable: norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: output.weight with shape: torch.Size([32000, 4096]) and type: torch.float16  
Processing variable: layers.0.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.0.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.0.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.0.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.0.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.0.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.0.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.0.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.0.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.1.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.1.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.1.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.1.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.1.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.1.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.1.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.1.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.1.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.2.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.2.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.2.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.2.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.2.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.2.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.2.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.2.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.2.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.3.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.3.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.3.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.3.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.3.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.3.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.3.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.3.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.3.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.4.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.4.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.4.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.4.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.4.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.4.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.4.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.4.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.4.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.5.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.5.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.5.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.5.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.5.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.5.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.5.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.5.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.5.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.6.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.6.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.6.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.6.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.6.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.6.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.6.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.6.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.6.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.7.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.7.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.7.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.7.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.7.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.7.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.7.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.7.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.7.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.8.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.8.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.8.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.8.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.8.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.8.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.8.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.8.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.8.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.9.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.9.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.9.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.9.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.9.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.9.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.9.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.9.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.9.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.10.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.10.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.10.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.10.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.10.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.10.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.10.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.10.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.10.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.11.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.11.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.11.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.11.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.11.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.11.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.11.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.11.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.11.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.12.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.12.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.12.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.12.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.12.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.12.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.12.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.12.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.12.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.13.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.13.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.13.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.13.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.13.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.13.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.13.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.13.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.13.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.14.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.14.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.14.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.14.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.14.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.14.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.14.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.14.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.14.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.15.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.15.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.15.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.15.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.15.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.15.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.15.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.15.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.15.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.16.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.16.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.16.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.16.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.16.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.16.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.16.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.16.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.16.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.17.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.17.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.17.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.17.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.17.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.17.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.17.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.17.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.17.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.18.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.18.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.18.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.18.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.18.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.18.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.18.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.18.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.18.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.19.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.19.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.19.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.19.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.19.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.19.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.19.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.19.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.19.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.20.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.20.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.20.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.20.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.20.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.20.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.20.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.20.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.20.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.21.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.21.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.21.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.21.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.21.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.21.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.21.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.21.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.21.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.22.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.22.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.22.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.22.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.22.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.22.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.22.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.22.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.22.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.23.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.23.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.23.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.23.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.23.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.23.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.23.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.23.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.23.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.24.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.24.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.24.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.24.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.24.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.24.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.24.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.24.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.24.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.25.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.25.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.25.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.25.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.25.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.25.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.25.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.25.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.25.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.26.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.26.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.26.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.26.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.26.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.26.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.26.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.26.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.26.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.27.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.27.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.27.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.27.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.27.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.27.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.27.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.27.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.27.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.28.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.28.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.28.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.28.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.28.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.28.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.28.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.28.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.28.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.29.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.29.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.29.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.29.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.29.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.29.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.29.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.29.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.29.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.30.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.30.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.30.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.30.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.30.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.30.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.30.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.30.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.30.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.31.attention.wq.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.31.attention.wk.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.31.attention.wv.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.31.attention.wo.weight with shape: torch.Size([4096, 4096]) and type: torch.float16  
Processing variable: layers.31.feed_forward.w1.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.31.feed_forward.w2.weight with shape: torch.Size([4096, 11008]) and type: torch.float16  
Processing variable: layers.31.feed_forward.w3.weight with shape: torch.Size([11008, 4096]) and type: torch.float16  
Processing variable: layers.31.attention_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Processing variable: layers.31.ffn_norm.weight with shape: torch.Size([4096]) and type: torch.float16  
  Converting to float32  
Done. Output file: models/7B//ggml-model-f16.bin, (part 0)

可以看到，如果轉換成功，會在models/7B/目錄生成一個C++可以呼叫的ggml-model-f16.bin模型檔案。

LLaMA模型呼叫

接下來就可以呼叫轉換後的模型了，首先在編譯C++專案：

make

程式返回：

➜  llama.cpp git:(master) ✗ make  
I llama.cpp build info:   
I UNAME_S:  Darwin  
I UNAME_P:  arm  
I UNAME_M:  arm64  
I CFLAGS:   -I.              -O3 -DNDEBUG -std=c11   -fPIC -pthread -DGGML_USE_ACCELERATE  
I CXXFLAGS: -I. -I./examples -O3 -DNDEBUG -std=c++17 -fPIC -pthread  
I LDFLAGS:   -framework Accelerate  
I CC:       Apple clang version 14.0.0 (clang-1400.0.29.202)  
I CXX:      Apple clang version 14.0.0 (clang-1400.0.29.202)  
  
cc  -I.              -O3 -DNDEBUG -std=c11   -fPIC -pthread -DGGML_USE_ACCELERATE   -c ggml.c -o ggml.o  
c++ -I. -I./examples -O3 -DNDEBUG -std=c++17 -fPIC -pthread -c utils.cpp -o utils.o  
c++ -I. -I./examples -O3 -DNDEBUG -std=c++17 -fPIC -pthread main.cpp ggml.o utils.o -o main  -framework Accelerate  
./main -h  
usage: ./main [options]  
  
options:  
  -h, --help            show this help message and exit  
  -i, --interactive     run in interactive mode  
  -ins, --instruct      run in instruction mode (use with Alpaca models)  
  -r PROMPT, --reverse-prompt PROMPT  
                        in interactive mode, poll user input upon seeing PROMPT (can be  
                        specified more than once for multiple prompts).  
  --color               colorise output to distinguish prompt and user input from generations  
  -s SEED, --seed SEED  RNG seed (default: -1)  
  -t N, --threads N     number of threads to use during computation (default: 4)  
  -p PROMPT, --prompt PROMPT  
                        prompt to start generation with (default: empty)  
  --random-prompt       start with a randomized prompt.  
  -f FNAME, --file FNAME  
                        prompt file to start generation.  
  -n N, --n_predict N   number of tokens to predict (default: 128)  
  --top_k N             top-k sampling (default: 40)  
  --top_p N             top-p sampling (default: 0.9)  
  --repeat_last_n N     last n tokens to consider for penalize (default: 64)  
  --repeat_penalty N    penalize repeat sequence of tokens (default: 1.3)  
  -c N, --ctx_size N    size of the prompt context (default: 512)  
  --ignore-eos          ignore end of stream token and continue generating  
  --memory_f16          use f16 instead of f32 for memory key+value  
  --temp N              temperature (default: 0.8)  
  -b N, --batch_size N  batch size for prompt processing (default: 8)  
  -m FNAME, --model FNAME  
                        model path (default: models/llama-7B/ggml-model.bin)  
  
c++ -I. -I./examples -O3 -DNDEBUG -std=c++17 -fPIC -pthread quantize.cpp ggml.o utils.o -o quantize  -framework Accelerate

編譯成功後，本地會生成一個main.cpp檔案。

隨後根據編譯後輸出的說明文件直接呼叫模型即可：

./main -m ./models/7B/ggml-model-f16.bin -p 'Hi i am '

程式輸出：

➜  llama.cpp git:(master) ✗ ./main -m ./models/7B/ggml-model-f16.bin -p 'hi i am'  
main: seed = 1679400707  
llama_model_load: loading model from './models/7B/ggml-model-f16.bin' - please wait ...  
llama_model_load: n_vocab = 32000  
llama_model_load: n_ctx   = 512  
llama_model_load: n_embd  = 4096  
llama_model_load: n_mult  = 256  
llama_model_load: n_head  = 32  
llama_model_load: n_layer = 32  
llama_model_load: n_rot   = 128  
llama_model_load: f16     = 1  
llama_model_load: n_ff    = 11008  
llama_model_load: n_parts = 1  
llama_model_load: ggml ctx size = 13365.09 MB  
llama_model_load: memory_size =   512.00 MB, n_mem = 16384  
llama_model_load: loading model part 1/1 from './models/7B/ggml-model-f16.bin'  
llama_model_load: .................................... done  
llama_model_load: model size = 12853.02 MB / num tensors = 291  
  
system_info: n_threads = 4 / 10 | AVX = 0 | AVX2 = 0 | AVX512 = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | VSX = 0 |   
  
main: prompt: ' hi i am'  
main: number of tokens in prompt = 6  
     1 -> ''  
 13450 -> ' hi'  
   423 -> 'i'  
 25523 -> ' am'  
  
sampling parameters: temp = 0.800000, top_k = 40, top_p = 0.950000, repeat_last_n = 64, repeat_penalty = 1.300000  
  
  
 hi i am a pythoner, but sunk to become a ruby

說實話，推理速度實在不敢恭維，也可能是因為筆者的電腦配置太渣導致。

結語

LLaMA 7B模型總體上需要純英文的提示詞（prompt），對中文的理解能力還不夠，優勢是確實可以單機跑起來，當然本地跑的話，減少了網路傳輸資料的環節，推理效率自然也就更高，對於普通的AI愛好者來說，足矣。

相關文章