README.md
1# Summary
2For Llama enablement, please see the [Llama README page](../llama/README.md) for complete details.
3
4This page contains Llama2 specific instructions and information.
5
6
7## Enablement
8
9We have verified running Llama 2 7B [mobile applications](#step-6-build-mobile-apps) efficiently on select devices including the iPhone 15 Pro, iPhone 15 Pro Max, Samsung Galaxy S22 and S24, and OnePlus 12.
10
11Since Llama 2 7B needs at least 4-bit quantization to fit even within some of the highend phones, results presented here correspond to 4-bit groupwise post-training quantized model.
12
13## Results
14
15### Llama2 7B
16Llama 2 7B performance was measured on the Samsung Galaxy S22, S24, and OnePlus 12 devices. The performance measurement is expressed in terms of tokens per second using an [adb binary-based approach](#step-5-run-benchmark-on).
17
18|Device | Groupwise 4-bit (128) | Groupwise 4-bit (256)
19|--------| ---------------------- | ---------------
20|Galaxy S22 | 8.15 tokens/second | 8.3 tokens/second |
21|Galaxy S24 | 10.66 tokens/second | 11.26 tokens/second |
22|OnePlus 12 | 11.55 tokens/second | 11.6 tokens/second |
23
24Below are the results for two different groupsizes, with max_seq_length 2048, and limit 1000, based on WikiText perplexity using [LM Eval](https://github.com/EleutherAI/lm-evaluation-harness).
25
26|Model | Baseline (FP32) | Groupwise 4-bit (128) | Groupwise 4-bit (256)
27|--------|-----------------| ---------------------- | ---------------
28|Llama 2 7B | 9.2 | 10.2 | 10.7
29
30## Prepare model
31
32You can export and run the original Llama 2 7B model.
33
341. Llama 2 pretrained parameters can be downloaded from [Meta's official website](https://ai.meta.com/resources/models-and-libraries/llama-downloads/) or from [Hugging Face](https://huggingface.co/meta-llama/Llama-2-7b).
35
362. Edit `params.json` file. Replace `"vocab_size": -1` with `"vocab_size": 32000`. This is a short-term workaround.
37
383. Export model and generate `.pte` file:
39 ```
40 python -m examples.models.llama.export_llama --checkpoint <checkpoint.pth> --params <params.json> -kv --use_sdpa_with_kv_cache -X -qmode 8da4w --group_size 128 -d fp32
41 ```
424. Create tokenizer.bin.
43 ```
44 python -m extension.llm.tokenizer.tokenizer -t <tokenizer.model> -o tokenizer.bin
45 ```
46
47 Pass the converted `tokenizer.bin` file instead of `tokenizer.model` for subsequent steps.
48
49
50# Run
51
52Running will be the same [by following this step](../llama/README.md#step-4-run-on-your-computer-to-validate).
53