Rethinking with Retrieval Faithful Large Language Model Inference

• 概
• Rethinking with retrieval (RR)
• 代码

He H., Zhang H. and Roth D. Rethinking with retrieval: faithful large language model inference. arXiv preprint arXiv:2301.00303, 2023.

概

LLM (Large Language Model) + 检索.

Rethinking with retrieval (RR)

• CoT (Chain of thought) 已经被证明是一种挖掘 LLM 能力的有效方式, 但是, 这种方式还是容易发生 LLM 胡编乱造的情况. RR 就是希望 LLM 在思考的时候能够有一个理性的参照以获得更好的更准确的结果 (上图 (c)).

• 让我们以 "Did Aristotle use a laptop?" 来分析这个流程.

• 首先, CoT 的做法是希望 LLM 将这个问题分解然后一步步的解决, 理想的步骤是:

  1. "Aristotle died in 322 BC.";
  2. "The first laptop was invented in 1980.";
  3. "So the answer is no.".

• 但是, 其中涉及到很多的 '常识', 倘若 LLM 本身并没有很好学习这些知识, 即使整体的步骤是对的, 也有可能回答错误. 在实际中可能会产生如下的几种回答 (加粗的是错误的):

  1. [R1] "Aristotle died in 2000. The first laptop was invented in 1980. Thus, Aristotle used a laptop. So the answer is yes."
  2. [R2] "Aristotle died in 322BC. The first laptop was invented in 2000. Thus, Aristotle did not user a laptop. So the answer is no."
  3. [R2] "Aristotle died in 322BC. The first laptop was invented in 1980. Thus, Aristotle did not user a laptop. So the answer is no."

• 甚至, 有些时候, 回答是对的, 但是所依据的常识是错误的 (R2).

• 所以, 我们需要借助外部的数据库, 比如 Wikipedia. 对于上面的问题, 涉及到两个知识点:

  1. [K1] Aristotle (384–322 BC) was a Greek philoso￾pher and polymath during the Classical pe￾riod in Ancient Greece. ...
  2. [K2] The Epson HX-20, the first laptop computer, was invented in 1980. ...

• RR 所依赖的 Faithful inference 可以这样定义. 对于某个问题 \(Q\), 产生了一组思维链 \(R_1, R_2, \ldots, R_N\) 以及它们所对应的答案 \(P_1, P_2, \ldots, P_N\), faithful inference 定义为:

  \[R^* := \mathop{\arg\max} \limits_{R_i, i=1,2,\ldots, N} \mathbb{I}[P_i = P] f_{\mathcal{KB}}(R_i). \]

  其中 \(f_{\mathcal{KB}}: R \rightarrow \mathbb{R}^+\) 评估思维链 \(R\) 和外部知识库的吻合程度.

代码

[official]