MindSpore Transformers即时MindFormers套件的目标是构建一个大模型训练、微调、评估、推理、部署的全流程开发套件，提供业内主流的Transformer类预训练模型和SOTA下游任务应用，涵盖丰富的并行特性。期望帮助用户轻松的实现大模型训练和创新研发。
安装MindFormers
直接使用pip安装
pip install mindformers有时候训练或推理还需要源代码，那就git clone下载：
git clone https://portrait.gitee.com/huanglei_Sorry/mindformers支持的模型如下
当前MindFormers支持的模型列表如下：
| 模型 | 参数 | 序列 | 预训练 | 微调 | 推理 | LoRA | 对话 | 评估 |
|---|---|---|---|---|---|---|---|---|
| LLaMA2 | 7B | 4K | ✓ | ✓ | generate | ✓ | ✓ | PPL |
| 13B | 4K | ✓ | ✓ | generate | ✓ | ✓ | PPL | |
| 70B | 4K | ✓ | ✓ | generate | ✓ | ✓ | PPL | |
| LLaMA3 | 8B | 8K | - | ✓ | generate | - | ✓ | - |
| 70B | 8K | ✓ | ✓ | generate | - | ✓ | - | |
| Baichuan2 | 7B | 4K | - | ✓ | generate | ✓ | ✓ | PPL |
| 13B | 4K | - | ✓ | generate | ✓ | ✓ | PPL | |
| GLM2 | 6B | 2K | - | ✓ | generate | ✓ | ✓ | PPL / Rouge |
| GLM3 | 6B | 2K | - | ✓ | generate | - | ✓ | - |
| GLM3-32K | 6B | 32K | - | ✓ | generate | - | ✓ | - |
| Qwen | 7B | 8K | - | ✓ | docs | ✓ | ✓ | C-Eval |
| 14B | 8K | - | ✓ | docs | ✓ | ✓ | C-Eval | |
| Qwen1.5 | 7B | 32K | ✓ | ✓ | docs | - | ✓ | - |
| 14B | 32K | ✓ | ✓ | docs | - | ✓ | - | |
| 72B | 32K | ✓ | ✓ | docs | - | ✓ | - | |
| QwenVL | 9.6B | 2K | - | ✓ | generate | - | ✓ | - |
| InternLM | 7B | 2K | - | ✓ | generate | ✓ | ✓ | PPL |
| 20B | 2K | - | ✓ | generate | ✓ | ✓ | PPL | |
| InternLM2 | 7B | 2K | - | ✓ | generate | - | ✓ | - |
| 20B | 4K | - | - | generate | - | ✓ | - | |
| Yi | 6B | 2K | - | ✓ | generate | ✓ | ✓ | - |
| 34B | 4K | ✓ | ✓ | generate | ✓ | ✓ | - | |
| Mixtral | 8x7B | 32K | ✓ | ✓ | docs | ✓ | ✓ | - |
| DeepSeek Coder | 33B | 4K | - | ✓ | docs | ✓ | ✓ | - |
| CodeLlama | 34B | 4K | ✓ | ✓ | generate | ✓ | ✓ | HumanEval |
| GPT2 | 13B | 2K | ✓ | ✓ | generate | ✓ | ✓ | PPL |
使用示例
# 1. 单机多卡快速启动方式，默认8卡启动
bash scripts/msrun_launcher.sh "run_mindformer.py \
--config {CONFIG_PATH} \
--run_mode {train/finetune/eval/predict}"
# 2. 单机多卡快速启动方式，仅设置使用卡数即可
bash scripts/msrun_launcher.sh "run_mindformer.py \
--config {CONFIG_PATH} \
--run_mode {train/finetune/eval/predict}" WORKER_NUM
# 3. 单机多卡自定义启动方式
bash scripts/msrun_launcher.sh "run_mindformer.py \
--config {CONFIG_PATH} \
--run_mode {train/finetune/eval/predict}" \
WORKER_NUM MASTER_PORT LOG_DIR JOIN CLUSTER_TIME_OUT