opencompass评测InternLM1.8B

1 配置 opencompass环境

git clone -b 0.2.4 https://github.com/open-compass/opencompass
pip install -e . -i https://mirrors.163.com/pypi/simple/
pip install -r requirements.txt -i https://mirrors.163.com/pypi/simple/
pip install protobuf -i https://mirrors.163.com/pypi/simple/
pip install modelscope -i https://mirrors.163.com/pypi/simple/

2 下载模型

from modelscope import snapshot_download
model_dir = snapshot_download('Shanghai_AI_Laboratory/internlm2_5-1_8b-chat',cache_dir='./model')

3 查看可用配置项

python tools/list_configs.py internlm ceval

4 修改测试文件

4.1 代码形式

修改`configs/models/hf_internlm/hf_internlm2_chat_1_8b.py文件：

from opencompass.models import HuggingFaceCausalLM

models = [
    dict(
        type=HuggingFaceCausalLM,
        abbr='internlm2-1.8b-hf',
        path="/data/coding/model/Shanghai_AI_Laboratory/internlm2_5-1_8b-chat",
        tokenizer_path='/data/coding/model/Shanghai_AI_Laboratory/internlm2_5-1_8b-chat',
        model_kwargs=dict(
            trust_remote_code=True,
            device_map='auto',
        ),
        tokenizer_kwargs=dict(
            padding_side='left',
            truncation_side='left',
            use_fast=False,
            trust_remote_code=True,
        ),
        max_out_len=100,
        min_out_len=1,
        max_seq_len=2048,
        batch_size=8,
        run_cfg=dict(num_gpus=1, num_procs=1),
    )
]

执行run.py：

python run.py --datasets ceval_gen --models hf_internlm2_chat_1_8b --debug

4.2 命令行形式

python run.py --datasets ceval_gen --hf-path /data/coding/model/Shanghai_AI_Laboratory/internlm2_5-1_8b-chat --tokenizer-path /data/coding/model/Shanghai_AI_Laboratory/internlm2_5-1_8b-chat --tokenizer-kwargs padding_side='left' truncation='left' trust_remote_code=True --model-kwargs trust_remote_code=True device_map='auto' --max-seq-len 2048 --max-out-len 16 --batch-size 4 --num-gpus 1 --debug

5 执行结果