patchTST代码复现心得

patchTST代码复现心得

Nie Y, Nguyen N H, Sinthong P, et al. A time series is worth 64 words: Long-term forecasting with transformers[J]. arXiv preprint arXiv:2211.14730, 2022.

代码

• 先来预测模块

def forecast(self, x_enc, x_mark_enc, x_dec, x_mark_dec):

 x_mark_enc, x_dec, x_mark_dec这三都没用着啊

# Normalization from Non-stationary Transformer
means = x_enc.mean(1, keepdim=True).detach()
x_enc = x_enc - means
stdev = torch.sqrt(
    torch.var(x_enc, dim=1, keepdim=True, unbiased=False) + 1e-5)
x_enc /= stdev

老三样，从NStrans学来的减去均值除以标准差

# do patching and embedding
x_enc = x_enc.permute(0, 2, 1)
# u: [bs * nvars x patch_num x d_model]
enc_out, n_vars = self.patch_embedding(x_enc)#B*N,N.dmodel

• 来到patch_embedding，这是新设计的

class PatchEmbedding(nn.Module):

self.padding_patch_layer = nn.ReplicationPad1d((0, padding))

 # do patching
        n_vars = x.shape[1]#B，N，seq
        x = self.padding_patch_layer(x)#进行填充以免过界
        x = x.unfold(dimension=-1, size=self.patch_len, step=self.stride)
        #它提取滑动窗口。这里在最后一个维度（dimension=-1，即 seq 维度）上提取大小为 patch_len 的滑动窗口，并以 stride 为步长进行滑动。
#结果张量的形状将变为 (B, N, new_seq, patch_len)，
        #居然是在这里变成四维的，B，12，8，12

填充完数据之后，用滑动窗口的方法把数据转成2D

x = torch.reshape(x, (x.shape[0] * x.shape[1], x.shape[2], x.shape[3]))
#重新形状化为三维张量(B * N, head, patch_len)
# Input encoding
x = self.value_embedding(x) + self.position_embedding(x)
#(B * N, head, dmodel)
return self.dropout(x), n_vars

reshape完了之后分别进行两种嵌入方法，pos位置嵌入和value_embedding

self.value_embedding = nn.Linear(patch_len, d_model, bias=False)

• 嵌入完了进到encoder

enc_out, attns = self.encoder(enc_out)#B*N,N,dmodel

这时候enc_ou的第一个维度和transformer很不一样

class EncoderLayer

new_x, attn = self.attention(
    x, x, x,#自注意力qkv
    attn_mask=attn_mask,
    tau=tau, delta=delta
)
x = x + self.dropout(new_x)#残差连接

进到注意力部分

 B, L, _ = queries.shape#B,seq,dmodel
        _, S, _ = keys.shape
        H = self.n_heads
#其实L和S是一个数
        queries = self.query_projection(queries).view(B, L, H, -1)#B, L, H, dmodel/h
        keys = self.key_projection(keys).view(B, S, H, -1)#一样的计算方法
        values = self.value_projection(values).view(B, S, H, -1)#H 表示头的数量-1 表示自动计算该维度

• 此时B的数字是batch*N数字很大

class FullAttention(nn.Module):

用的是FullAttention

B, L, H, E = queries.shape#B,seq,head ,64
_, S, _, D = values.shape
scale = self.scale or 1. / sqrt(E)#注意力权重的缩放因子

scores = torch.einsum("blhe,bshe->bhls", queries, keys)#张量乘法
#blhe 和 bshe 分别代表 queries 和 keys 张量的维度标签

A = self.dropout(torch.softmax(scale * scores, dim=-1))#softmax 操作，将它们转换为注意力权重。dropout 处理，以增强模型的鲁棒性。
V = torch.einsum("bhls,bshd->blhd", A, values)#张量乘法和求和操作
#得到注意力权重，然后将这些权重应用到值上，以获得加权后的值张量
if self.output_attention:
    return V.contiguous(), A
else:
    return V.contiguous(), None

• 之后回到encoderlayer输出

y = x = self.norm1(x)#层归一化
y = self.dropout(self.activation(self.conv1(y.transpose(-1, 1))))
y = self.dropout(self.conv2(y).transpose(-1, 1))

return self.norm2(x + y), attn

• 之后跳回到主预测部分。encout维度是B*N,N,dmodel

enc_out = torch.reshape(##B，N,N,dmodel
    enc_out, (-1, n_vars, enc_out.shape[-2], enc_out.shape[-1]))
# z: [bs x nvars x d_model x patch_num]
enc_out = enc_out.permute(0, 1, 3, 2)#B，N，dmodel ，N

进行维度重塑

self.head = FlattenHead(configs.enc_in, self.head_nf, configs.pred_len,
                        head_dropout=configs.dropout)

# Decoder
dec_out = self.head(enc_out)  # z: [bs x nvars x target_window]
dec_out = dec_out.permute(0, 2, 1)

• 这就是decode部分了

这个head是他定义的，此时x是四维 # x: [bs x nvars x d_model x patch_num]

def forward(self, x):  # x: [bs x nvars x d_model x patch_num]
    x = self.flatten(x)
    x = self.linear(x)
    x = self.dropout(x)
    return x

self.flatten = nn.Flatten(start_dim=-2)
self.linear = nn.Linear(nf, target_window)
self.dropout = nn.Dropout(head_dropout)

操作也很稀松

# De-Normalization from Non-stationary Transformer
dec_out = dec_out * \
          (stdev[:, 0, :].unsqueeze(1).repeat(1, self.pred_len, 1))
dec_out = dec_out + \
          (means[:, 0, :].unsqueeze(1).repeat(1, self.pred_len, 1))
return dec_out

最后再来一个反归一化

总结

x_mark_enc, x_dec, x_mark_dec这三都没用着啊

来到patch_embedding，这是新设计的

decode部分