Qwen2模型量化时关于bitsandbytes安装的问题
问题描述：
from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig,BitsAndBytesConfig
CUDA_DEVICE = "cuda:0"
model_name_or_path = '/qwen2-1.5b-instruct'
Tokenizer = AutoTokenizer.from_pretrained(model_name_or_path,use_fast=False)
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16
)
Model = AutoModelForCausalLM.from_pretrained(model_name_or_path,
device_map="auto",#CUDA_DEVICE,
#load_in_8bit=True,
quantization_config=bnb_config,
#torch_dtype="auto"
)
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": prompt}
]
# "genarate chat"
input_ids = Tokenizer.apply_chat_template(messages,tokenize=False,add_generation_prompt=True)
model_inputs = Tokenizer([input_ids], return_tensors="pt").to(CUDA_DEVICE)
generated_ids = Model.generate(model_inputs.input_ids,top_p=0.2,max_new_tokens=512)
generated_ids = [
output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]
response = self.Tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
在模型量化时遇到如下问题：
RuntimeError:
CUDA Setup failed despite GPU being available. Please run the following command to get mor e information:
python -m bitsandbytes
Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues
解决方法1：
可以卸载重新安装bitsandbytes，
step1：pip uninstall bitsandbytes
step2：pip install bitsandbytes
python3 -m bitsandbytes
解决方法2：
Step1:确认系统查找动态库libcudart.so的路径,将查找到的路径添加进去便解决这个问题
find / -name “libcudart.so*” #我的libcudart.so在/usr/local/cuda-11.4中
export LD_LIBRARY_PATH=“/usr/local/cuda-11.4/bin:$LD_LIBRARY_PATH”
python3 -m bitsandbytes
参考资料：
《Ubuntu18.04+CUDA11安装bitsandbytes出现的问题》
https://blog.csdn.net/steptoward/article/details/135507131
《大模型训练时关于bitsandbytes安装的问题》
https://gitcode.csdn.net/662f78a69ab37021bfb31bf0.html
《bitsandbytes 报错》
https://blog.csdn.net/weixin_43967256/article/details/134006312