使用OpenAI API进行Model Fine-tuning

• 1 基本信息
• 2 操作示例
  • 2.1 准备数据
  • 2.2 模型训练
  • 2.3 模型推理

1 基本信息

参考资料：

• 官方指南：https://platform.openai.com/docs/guides/fine-tuning
• 微调接口：https://platform.openai.com/docs/api-reference/fine-tunes
• 数据接口：https://platform.openai.com/docs/api-reference/files
• 价格：https://openai.com/pricing#language-models

基本步骤：

• 制作并上传训练数据
• 训练一个新的微调模型
• 使用训练好的微调模型

2 操作示例

2.1 准备数据

最佳实践：https://platform.openai.com/docs/guides/fine-tuning/preparing-your-dataset

• prompt建议以固定的分隔符结尾，比如: :, -> 或 \n\n###\n\n，该分隔符只用于标识prompt的结束，不在prompt的其他任何地方出现
• completion建议以空格开头
• completion同样建议以固定的分隔符结尾，比如: \n 或 ###，该分隔符只用于标识completion的结束，不在completion的其他任何地方出现
• 建议准备至少 数百条 数据用于微调训练
• 在使用微调模型进行推理时，输入的prompt需要遵循与训练时一致的句式格式，并用微调数据中使用的completion结束符对模型返回的completion进行截断

训练数据需要整理成以下 JSONL 格式：

{"prompt": "<prompt text>", "completion": "<ideal generated text>"}
{"prompt": "<prompt text>", "completion": "<ideal generated text>"}
{"prompt": "<prompt text>", "completion": "<ideal generated text>"}
...

以情感分析任务为例，官方给出的示例数据为：

{"prompt":"Overjoyed with the new iPhone! ->", "completion":" positive"}
{"prompt":"@lakers disappoint for a third straight night https://t.co/38EFe43 ->", "completion":" negative"}

我们让ChatGPT参考该样例，再生成20条类似格式的数据：

{"prompt":"I just got a promotion at work! ->", "completion":" positive"}
{"prompt":"The traffic today was absolutely terrible. ->", "completion":" negative"}
{"prompt":"I'm so grateful for my supportive friends and family. ->", "completion":" positive"}
{"prompt":"I can't believe how rude the customer service was at that store. ->", "completion":" negative"}
{"prompt":"I'm so excited to go on vacation next week! ->", "completion":" positive"}
{"prompt":"I'm feeling really down today. ->", "completion":" negative"}
{"prompt":"I just finished reading a great book. ->", "completion":" positive"}
{"prompt":"The weather has been awful lately. ->", "completion":" negative"}
{"prompt":"I had a wonderful time at the concert last night. ->", "completion":" positive"}
{"prompt":"I'm so frustrated with my job right now. ->", "completion":" negative"}
{"prompt":"I love spending time with my family. ->", "completion":" positive"}
{"prompt":"I'm so disappointed with the election results. ->", "completion":" negative"}
{"prompt":"I'm so grateful for my health. ->", "completion":" positive"}
{"prompt":"That movie was terrible. ->", "completion":" negative"}
{"prompt":"I'm so proud of my child for doing well in school. ->", "completion":" positive"}
{"prompt":"I had a terrible experience at the restaurant last night. ->", "completion":" negative"}
{"prompt":"I'm so happy with my new car. ->", "completion":" positive"}
{"prompt":"I'm feeling really anxious about the upcoming deadline. ->", "completion":" negative"}
{"prompt":"I had a fantastic time at the party last night. ->", "completion":" positive"}
{"prompt":"I'm so upset about the recent news. ->", "completion":" negative"}

保存数据：

import json

data = [
    {"prompt":"Overjoyed with the new iPhone! ->", "completion":" positive"},
    {"prompt":"@lakers disappoint for a third straight night https://t.co/38EFe43 ->", "completion":" negative"},
    {"prompt":"I just got a promotion at work! ->", "completion":" positive"},
    {"prompt":"The traffic today was absolutely terrible. ->", "completion":" negative"},
    {"prompt":"I'm so grateful for my supportive friends and family. ->", "completion":" positive"},
    {"prompt":"I can't believe how rude the customer service was at that store. ->", "completion":" negative"},
    {"prompt":"I'm so excited to go on vacation next week! ->", "completion":" positive"},
    {"prompt":"I'm feeling really down today. ->", "completion":" negative"},
    {"prompt":"I just finished reading a great book. ->", "completion":" positive"},
    {"prompt":"The weather has been awful lately. ->", "completion":" negative"},
    {"prompt":"I had a wonderful time at the concert last night. ->", "completion":" positive"},
    {"prompt":"I'm so frustrated with my job right now. ->", "completion":" negative"},
    {"prompt":"I love spending time with my family. ->", "completion":" positive"},
    {"prompt":"I'm so disappointed with the election results. ->", "completion":" negative"},
    {"prompt":"I'm so grateful for my health. ->", "completion":" positive"},
    {"prompt":"That movie was terrible. ->", "completion":" negative"},
    {"prompt":"I'm so proud of my child for doing well in school. ->", "completion":" positive"},
    {"prompt":"I had a terrible experience at the restaurant last night. ->", "completion":" negative"},
    {"prompt":"I'm so happy with my new car. ->", "completion":" positive"},
    {"prompt":"I'm feeling really anxious about the upcoming deadline. ->", "completion":" negative"},
    {"prompt":"I had a fantastic time at the party last night. ->", "completion":" positive"},
    {"prompt":"I'm so upset about the recent news. ->", "completion":" negative"},
]

with open("fine-tuning.jsonl", "w") as f:
    for item in data:
        json.dump(item, f)
        f.write("\n")

上传数据到OpenAI：

import os

openai.File.create(
  file=open("fine-tuning.jsonl", "rb"),
  purpose='fine-tune'
)

记录返回的文件id：

<File file id=xxx at 0x7fe7d80b4890> JSON: {
  "bytes": 1918,
  "created_at": 1685516851,
  "filename": "file",
  "id": "【文件id】",
  "object": "file",
  "purpose": "fine-tune",
  "status": "uploaded",
  "status_details": null
}

历史文件的id可通过 openai.File.list() 来查看

2.2 模型训练

创建Fine-tuning任务：

openai.FineTune.create(training_file="【文件id】")

查看组织下所有的Fine-tuning任务：

openai.FineTune.list()

查看某个Fine-tuning任务的详细信息：

openai.FineTune.retrieve(id="【任务id】")

• 查看详细信息时，其中包含的message会显示本次微调的计费信息：Fine-tune costs $x.xx，以及微调任务对应的结果模型id：Uploaded model: curie:ft-personal-2023-05-31-07-17-56

注：Fine-tuning阶段的计费取决于训练数据的token数和微调时的轮次参数“epochs”（来源：Pricing页面最下方的FAQ）：

There are two components to fine-tuning pricing: training and usage.

When training a fine-tuned model, the total tokens used will be billed according to our training rates. Note that the number of training tokens depends on the number of tokens in your training dataset and your chosen number of training epochs. The default number of epochs is 4.

(Tokens in your training file * Number of training epochs) = Total training tokens

Once you fine-tune a model, you’ll be billed only for the tokens you use. Requests sent to fine-tuned models are billed at our usage rates.

2.3 模型推理

使用模型训练时得到的模型ID进行推理：

ft_model = 'curie:ft-personal-2023-05-31-07-17-56'
res = openai.Completion.create(model=ft_model, prompt='I just finished a challenging workout and feel great! ->', max_tokens=1, temperature=0)
res['choices'][0]['text']  # 结果：' positive'

推理阶段的费用可以通过 res['usage'] 中的token数换算得到，具体数值可参考下表：