Llama-2 proj weight

[shanhx2000]

In greedyopt.py, why k, v, weight ratio is set to 1/8? For Llama2-7B, shouldn't it be 1? (4096x4096)

"Llama-2-7B": {
'q': 1, 'k': 1/8, 'v': 1/8, 'o': 1,
'gate': 2.6875, 'up': 2.6875, 'down': 2.6875
},

[chromecast56]

8 queries per kv head in GQA

[shanhx2000]

Thanks for the prompt response. Based on the models on HF (https://huggingface.co/meta-llama/Llama-2-70b-hf/), it seems only 70B uses GQA and "num_key_value_heads": 8 / "num_attention_heads": 64 gives the 8 you provided. For 7B model, I found "num_attention_heads": 32 and "num_key_value_heads": 32, without using GQA. Shouldn't it be 1 in that case?