首页 > 其他分享 >昇思25天学习打卡营第19天|LSTM+CRF序列标注

昇思25天学习打卡营第19天|LSTM+CRF序列标注

时间:2024-07-07 10:26:28浏览次数:8  
标签:25 seq tags 19 batch length score 打卡 size

概述

序列标注指给定输入序列,给序列中每个Token进行标注标签的过程。序列标注问题通常用于从文本中进行信息抽取,包括分词(Word Segmentation)、词性标注(Position Tagging)、命名实体识别(Named Entity Recognition, NER)等。

条件随机场(CRF)

对序列进行标注,实际上是对序列中每个Token进行标签预测,可以直接视作简单的多分类问题。但是序列标注不仅仅需要对单个Token进行分类预测,同时相邻Token直接有关联关系。

x=\begin{Bmatrix} x_0,...,x_n \end{Bmatrix}为输入序列,y=\begin{Bmatrix} y_0,...,y_n \end{Bmatrix}为输出的标注序列,输出序列y的概率为:

P(y|x)=\frac{exp(Score(x,y))}{\sum_{y'\in Y}exp(Score(x,y'))}

定义两个概率函数

1. 发射概率函数\psi _{EMIT}:表示x_i\rightarrow y_i的概率

2. 转移概率函数\psi _{TRANS}:表示y_{i-1}\rightarrow y_i的概率

于是可以得到Score的计算公式:

Score(x,y)=\sum_ilog\psi _{EMIT}(x_i\rightarrow y_i)+log\psi_{TRANS}(y_{i-1}\rightarrow y_i)

设标签集合为T,构造大小为\left | T \right |\times \left | T \right |的矩阵P,用于存储标签间的转移概率。

实现CRF层的前向训练部分,将CRF和损失函数做合并,选择分类问题常用的负对数似然函数,则有:

Loss=-log(P(y|x))

Loss=-log(\frac{exp(Score(x,y))}{\sum_{y'\in Y}exp(Score(x,y'))}) \newline=log(\sum_{y'\in Y}exp(Score(x,y'))-Score(x,y))

Score计算

def compute_score(emissions, tags, seq_ends, mask, trans, start_trans, end_trans):
    # emissions: (seq_length, batch_size, num_tags)
    # tags: (seq_length, batch_size)
    # mask: (seq_length, batch_size)

    seq_length, batch_size = tags.shape
    mask = mask.astype(emissions.dtype)

    # 将score设置为初始转移概率
    # shape: (batch_size,)
    score = start_trans[tags[0]]
    # score += 第一次发射概率
    # shape: (batch_size,)
    score += emissions[0, mnp.arange(batch_size), tags[0]]

    for i in range(1, seq_length):
        # 标签由i-1转移至i的转移概率(当mask == 1时有效)
        # shape: (batch_size,)
        score += trans[tags[i - 1], tags[i]] * mask[i]

        # 预测tags[i]的发射概率(当mask == 1时有效)
        # shape: (batch_size,)
        score += emissions[i, mnp.arange(batch_size), tags[i]] * mask[i]

    # 结束转移
    # shape: (batch_size,)
    last_tags = tags[seq_ends, mnp.arange(batch_size)]
    # score += 结束转移概率
    # shape: (batch_size,)
    score += end_trans[last_tags]

    return score

Normalizer计算

Normalizer可以改写为以下形式:

log(\sum_{y'_{0,i}\in Y}exp(Score_i))=log(\sum_{y'_{0,i-1}\in Y}exp(Score_{i-1}+h_i+P))

Normalizer代码实现如下:

def compute_normalizer(emissions, mask, trans, start_trans, end_trans):
    # emissions: (seq_length, batch_size, num_tags)
    # mask: (seq_length, batch_size)

    seq_length = emissions.shape[0]

    # 将score设置为初始转移概率,并加上第一次发射概率
    # shape: (batch_size, num_tags)
    score = start_trans + emissions[0]

    for i in range(1, seq_length):
        # 扩展score的维度用于总score的计算
        # shape: (batch_size, num_tags, 1)
        broadcast_score = score.expand_dims(2)

        # 扩展emission的维度用于总score的计算
        # shape: (batch_size, 1, num_tags)
        broadcast_emissions = emissions[i].expand_dims(1)

        # 根据公式(7),计算score_i
        # 此时broadcast_score是由第0个到当前Token所有可能路径
        # 对应score的log_sum_exp
        # shape: (batch_size, num_tags, num_tags)
        next_score = broadcast_score + trans + broadcast_emissions

        # 对score_i做log_sum_exp运算,用于下一个Token的score计算
        # shape: (batch_size, num_tags)
        next_score = ops.logsumexp(next_score, axis=1)

        # 当mask == 1时,score才会变化
        # shape: (batch_size, num_tags)
        score = mnp.where(mask[i].expand_dims(1), next_score, score)

    # 最后加结束转移概率
    # shape: (batch_size, num_tags)
    score += end_trans
    # 对所有可能的路径得分求log_sum_exp
    # shape: (batch_size,)
    return ops.logsumexp(score, axis=1)

Viterbi算法

在完成前向训练部分后,需要实现解码部分。Viterbi算法与计算Normalizer类似,使用动态规划求解所有可能的预测序列得分。不同的是在解码时同时需要将第i个Token对应的score取值最大的标签保存,供后续使用Viterbi算法求解最优预测序列使用。

取得最大概率得分ScoreScore,以及每个Token对应的标签历史HistoryHistory后,根据Viterbi算法可以得到公式:

P_{0,i}=max(P_{0,i-1})+P_{i-1,i}

代码实现:

def viterbi_decode(emissions, mask, trans, start_trans, end_trans):
    # emissions: (seq_length, batch_size, num_tags)
    # mask: (seq_length, batch_size)

    seq_length = mask.shape[0]

    score = start_trans + emissions[0]
    history = ()

    for i in range(1, seq_length):
        broadcast_score = score.expand_dims(2)
        broadcast_emission = emissions[i].expand_dims(1)
        next_score = broadcast_score + trans + broadcast_emission

        # 求当前Token对应score取值最大的标签,并保存
        indices = next_score.argmax(axis=1)
        history += (indices,)

        next_score = next_score.max(axis=1)
        score = mnp.where(mask[i].expand_dims(1), next_score, score)

    score += end_trans

    return score, history

def post_decode(score, history, seq_length):
    # 使用Score和History计算最佳预测序列
    batch_size = seq_length.shape[0]
    seq_ends = seq_length - 1
    # shape: (batch_size,)
    best_tags_list = []

    # 依次对一个Batch中每个样例进行解码
    for idx in range(batch_size):
        # 查找使最后一个Token对应的预测概率最大的标签,
        # 并将其添加至最佳预测序列存储的列表中
        best_last_tag = score[idx].argmax(axis=0)
        best_tags = [int(best_last_tag.asnumpy())]

        # 重复查找每个Token对应的预测概率最大的标签,加入列表
        for hist in reversed(history[:seq_ends[idx]]):
            best_last_tag = hist[idx][best_tags[-1]]
            best_tags.append(int(best_last_tag.asnumpy()))

        # 将逆序求解的序列标签重置为正序
        best_tags.reverse()
        best_tags_list.append(best_tags)

    return best_tags_list

CRF层

CRF的输入需要考虑输入序列的真实长度,因此除发射矩阵和标签外,加入 seq_length 参数传入序列Padding前的长度,并实现生成mask矩阵的 sequence_mask 方法。

代码实现:

import mindspore as ms
import mindspore.nn as nn
import mindspore.ops as ops
import mindspore.numpy as mnp
from mindspore.common.initializer import initializer, Uniform

def sequence_mask(seq_length, max_length, batch_first=False):
    """根据序列实际长度和最大长度生成mask矩阵"""
    range_vector = mnp.arange(0, max_length, 1, seq_length.dtype)
    result = range_vector < seq_length.view(seq_length.shape + (1,))
    if batch_first:
        return result.astype(ms.int64)
    return result.astype(ms.int64).swapaxes(0, 1)

class CRF(nn.Cell):
    def __init__(self, num_tags: int, batch_first: bool = False, reduction: str = 'sum') -> None:
        if num_tags <= 0:
            raise ValueError(f'invalid number of tags: {num_tags}')
        super().__init__()
        if reduction not in ('none', 'sum', 'mean', 'token_mean'):
            raise ValueError(f'invalid reduction: {reduction}')
        self.num_tags = num_tags
        self.batch_first = batch_first
        self.reduction = reduction
        self.start_transitions = ms.Parameter(initializer(Uniform(0.1), (num_tags,)), name='start_transitions')
        self.end_transitions = ms.Parameter(initializer(Uniform(0.1), (num_tags,)), name='end_transitions')
        self.transitions = ms.Parameter(initializer(Uniform(0.1), (num_tags, num_tags)), name='transitions')

    def construct(self, emissions, tags=None, seq_length=None):
        if tags is None:
            return self._decode(emissions, seq_length)
        return self._forward(emissions, tags, seq_length)

    def _forward(self, emissions, tags=None, seq_length=None):
        if self.batch_first:
            batch_size, max_length = tags.shape
            emissions = emissions.swapaxes(0, 1)
            tags = tags.swapaxes(0, 1)
        else:
            max_length, batch_size = tags.shape

        if seq_length is None:
            seq_length = mnp.full((batch_size,), max_length, ms.int64)

        mask = sequence_mask(seq_length, max_length)

        # shape: (batch_size,)
        numerator = compute_score(emissions, tags, seq_length-1, mask, self.transitions, self.start_transitions, self.end_transitions)
        # shape: (batch_size,)
        denominator = compute_normalizer(emissions, mask, self.transitions, self.start_transitions, self.end_transitions)
        # shape: (batch_size,)
        llh = denominator - numerator

        if self.reduction == 'none':
            return llh
        if self.reduction == 'sum':
            return llh.sum()
        if self.reduction == 'mean':
            return llh.mean()
        return llh.sum() / mask.astype(emissions.dtype).sum()

    def _decode(self, emissions, seq_length=None):
        if self.batch_first:
            batch_size, max_length = emissions.shape[:2]
            emissions = emissions.swapaxes(0, 1)
        else:
            batch_size, max_length = emissions.shape[:2]

        if seq_length is None:
            seq_length = mnp.full((batch_size,), max_length, ms.int64)

        mask = sequence_mask(seq_length, max_length)

        return viterbi_decode(emissions, mask, self.transitions, self.start_transitions, self.end_transitions)

BiLSTM+CRF模型

其中LSTM提取序列特征,经过Dense层变换获得发射概率矩阵,最后送入CRF层。具体实现如下:

class BiLSTM_CRF(nn.Cell):
    def __init__(self, vocab_size, embedding_dim, hidden_dim, num_tags, padding_idx=0):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=padding_idx)
        self.lstm = nn.LSTM(embedding_dim, hidden_dim // 2, bidirectional=True, batch_first=True)
        self.hidden2tag = nn.Dense(hidden_dim, num_tags, 'he_uniform')
        self.crf = CRF(num_tags, batch_first=True)

    def construct(self, inputs, seq_length, tags=None):
        embeds = self.embedding(inputs)
        outputs, _ = self.lstm(embeds, seq_length=seq_length)
        feats = self.hidden2tag(outputs)

        crf_outs = self.crf(feats, tags, seq_length)
        return crf_outs

完成模型设计后,我们生成两句例子和对应的标签,并构造词表和标签表。

embedding_dim = 16
hidden_dim = 32

training_data = [(
    "清 华 大 学 坐 落 于 首 都 北 京".split(),
    "B I I I O O O O O B I".split()
), (
    "重 庆 是 一 个 魔 幻 城 市".split(),
    "B I O O O O O O O".split()
)]

word_to_idx = {}
word_to_idx['<pad>'] = 0
for sentence, tags in training_data:
    for word in sentence:
        if word not in word_to_idx:
            word_to_idx[word] = len(word_to_idx)

tag_to_idx = {"B": 0, "I": 1, "O": 2}

接下来实例化模型,选择优化器并将模型和优化器送入Wrapper。

model = BiLSTM_CRF(len(word_to_idx), embedding_dim, hidden_dim, len(tag_to_idx))
optimizer = nn.SGD(model.trainable_params(), learning_rate=0.01, weight_decay=1e-4)

grad_fn = ms.value_and_grad(model, None, optimizer.parameters)

def train_step(data, seq_length, label):
    loss, grads = grad_fn(data, seq_length, label)
    optimizer(grads)
    return loss

将生成的数据打包成Batch,按照序列最大长度,对长度不足的序列进行填充,分别返回输入序列、输出标签和序列长度构成的Tensor。

def prepare_sequence(seqs, word_to_idx, tag_to_idx):
    seq_outputs, label_outputs, seq_length = [], [], []
    max_len = max([len(i[0]) for i in seqs])

    for seq, tag in seqs:
        seq_length.append(len(seq))
        idxs = [word_to_idx[w] for w in seq]
        labels = [tag_to_idx[t] for t in tag]
        idxs.extend([word_to_idx['<pad>'] for i in range(max_len - len(seq))])
        labels.extend([tag_to_idx['O'] for i in range(max_len - len(seq))])
        seq_outputs.append(idxs)
        label_outputs.append(labels)

    return ms.Tensor(seq_outputs, ms.int64), \
            ms.Tensor(label_outputs, ms.int64), \
            ms.Tensor(seq_length, ms.int64)

对模型进行预编译后,训练500个step。

from tqdm import tqdm

steps = 500
with tqdm(total=steps) as t:
    for i in range(steps):
        loss = train_step(data, seq_length, label)
        t.set_postfix(loss=loss)
        t.update(1)

最后将预测的index序列转换为标签序列,打印输出结果,查看效果。

idx_to_tag = {idx: tag for tag, idx in tag_to_idx.items()}

def sequence_to_tag(sequences, idx_to_tag):
    outputs = []
    for seq in sequences:
        outputs.append([idx_to_tag[i] for i in seq])
    return outputs

sequence_to_tag(predict, idx_to_tag)

得到输出标签

[['B', 'I', 'I', 'I', 'O', 'O', 'O', 'O', 'O', 'B', 'I'],
 ['B', 'I', 'O', 'O', 'O', 'O', 'O', 'O', 'O']]

总结

LSTM用于提取序列特征,CRF用于序列标注,从而实现语义的切分。

标签:25,seq,tags,19,batch,length,score,打卡,size
From: https://blog.csdn.net/m0_62894627/article/details/140241748

相关文章

  • 《Programming from the Ground Up》阅读笔记:p19-p48
    《ProgrammingfromtheGroundUp》学习第2天,p19-p48总结,总计30页。一、技术总结1.objectfilep20,Anobjectfileiscodethatisinthemachine'slanguage,buthasnotbeencompletelyputtogether。之前在很多地方都看到objectfile这个概念,但都没有看到起定义,这次......
  • springboot巡更系统 毕业设计-附源码10192
    摘 要目前,在日常生活中随处可见社区巡更人员对特定的区域进行定期或者不定期的安全巡查管理。包括勤前训示、必到点签到、巡更路线等,各项勤务工作均由巡更员本人在执勤日志本中手工填写,且工作点分散,不利于统一监管,存在信息化手段不足,勤务信息获取、输入复杂,监管信息不能......
  • 练19.4 骑车与走路
    题目链接:https://bzoj.org/d/rumen/p/P111Description在清华校园里,没有自行车,上课办事会很不方便。但实际上。并非去办任何事情都是骑车快,因为骑车总要找车、开锁、停车、锁车等,这要耽误一些时间。假设找到自行车,开锁并骑上自行车的时间为27秒;停车锁车的时间为23秒;步行每秒......
  • 代码随想录算法训练营第四天 | 24. 两两交换链表中的节点 、 19.删除链表的倒数第N个
    24.两两交换链表中的节点 题目:.-力扣(LeetCode)思路:这题关键是要每次进行两个结点的操作,并且每次都要保存其前结点,做题思路比较清晰,但是总是处理不好边界问题,总是越界。代码:/***Definitionforsingly-linkedlist.*structListNode{*intval;*List......
  • 代码随想录刷题day 4 | 两两交换链表中的节点 19.删除链表的倒数第N个节点 面试题
    24.两两交换链表中的节点迭代的版本:最重要的就是要知道循环变量怎么取,对于这道题,我们只需要存储需要交换的两个节点的前一个节点即可,只有当这个节点后面有两个节点时才进入循环,其实把握住这一点之后这题就非常容易了。递归的版本:这道题用递归做简直不要太简单,首先明白递归结束......
  • Solution - Atcoder ARC125E Snack
    观察到这种都是数量上限的限制,且求\(\max\)。所以可以考虑网络流建模,而流量就对应着给的糖果数。令\(S\)为源点,\(T\)为汇点,编号为\(1\simn\)的点对应的糖果的种类,编号为\(n+1\simn+m\)的点对应的小孩。连边\((S,i,a_i)\),表示第\(i\)种糖果数不超过\(a_i\)......
  • leetcode 257. 二叉树的所有路径
    给你一个二叉树的根节点 root ,按 任意顺序 ,返回所有从根节点到叶子节点的路径。叶子节点 是指没有子节点的节点。 示例1:输入:root=[1,2,3,null,5]输出:["1->2->5","1->3"]示例2:输入:root=[1]输出:["1"]java解题思路及代码实现递归法packagecom.java......
  • 昇思25天学习打卡营第11天|ResNet50图像分类
    文章目录昇思MindSpore应用实践基于MindSpore的ResNet50图像分类1、ResNet50简介2、数据集预处理及可视化3、构建网络构建BuildingBlock构建BottleneckBlock构建ResNet50网络4、模型训练5、图像分类模型推理Reference昇思MindSpore应用实践本系列文章主......
  • 如何完美解决 “error pulling image configuration: download failed after attempts
    如何完美解决"errorpullingimageconfiguration:downloadfailedafterattempts=6:dialtcp59.188.250.54"......
  • Python以中心裁剪图片,一个中国大陆身份证的尺寸是88mm*55mm,通常使用的身份证像素分辨
    1#以中心裁剪图片2#一个中国大陆身份证的尺寸是88mm*55mm,通常使用的身份证像素分辨率是336*2563defcrop_image_by_center(input_image_path,width_mm=88,height_mm=55):4#打开图片5image=Image.open(input_image_path)6image=ImageOps.ex......