Transformers - Path to Deep Learning 1

In this chapter, we will introduce the endpoint of traditional ML, the cornerstone of DL (Deep Learning), and the foundation of LLMs—Transformers.

I believe that after finishing this chapter, we will not be far from LLMs.

Compared to the native Transformers, its advanced wrapper, Keras, is much easier to use and more suitable for beginners, especially for those who do not have a good foundation in mathematics and theory. We will also use Keras in this large chapter.

The content of this chapter is indeed too much, so it is divided into 4 small chapters, which means 4 articles to introduce them separately.

By the way, I personally think that the original paper on Transformers, 'Attention Is All You Need', is a very worthwhile paper to study. However, before reading that paper, allow me to explain it so that it won't feel so unfamiliar when you read it.

Next, we have two branches, one leaning towards the language output type of LLM direction, and the other is the Computer Vision direction, which is what I am currently researching. If you think you want to understand the LLM direction more, you can read the later categorized specialized articles, starting from Labels. If you think you prefer CV, we will start from PyTorch.

Let's start from the beginning. This chapter is the most basic and also the most important Attention mechanism.

First, we need to answer a few questions:

Why do we use theAttention mechanism?

Here I will directly quote a passage from AI:

Before Attention appeared, RNN/LSTM had slow serial training speeds and struggled to remember information that was far apart in long texts. Attention allows each token to directly associate with words at any position in the entire text, capturing long-distance dependencies and supporting parallel computation, fully utilizing GPU computing power, making it possible to train large language models.

What is the Attention mechanism used for?

We know that it is difficult to understand the meaning of a word in context by itself. For example, in the sentence 'Apple Company Invented Phones', if we only see 'Apple', you might think of the fruit, but considering the context, we know that 'apple' in this sentence refers to the tech company that invented phones.

Let's start with vectors.

Before discussing the index Q, let's clarify what vectors are in the context of ML.

----------------------------------------------------------------------------------------------------------------------------

First, when we classify each word, we will give it a (generally) three-dimensional vector.

For example: King = { 14, 56, 38 }. male = { 8, 34, 25 } (this is fake, I don't know what it is)

So why do we need to give each word an index? The reason is that we need a quantity that can relate all words together. Vectors solve this problem well.

Just like the distance of vectors, the closer the distance between the labeled vectors of two words, the higher the relationship between the two words in reality.

Generally speaking, for most cases, there is a very interesting fact:

When we subtract the vector of Male from the vector of King (which I just made up), we will get the vector of Queen. Think about it, in terms of real logic, when we subtract all male kings from all kings, don't we get the set of queens? Hey, this is the role of vectors.

Now that we understand the role of vectors in word embeddings, we can conveniently discuss Q, K, and V.

Let's start with Q. What is Q? Q here stands for 'Query', which is an index.

Q represents where we want to look.

What about K? K stands for Key, which is a feature label.

What does that mean? For example, if our input token is 'apple', then its Key is fruit/company.

And V? V stands for Value, which is the core information carried by this token.

This refers to the deeper information carried by this token, such as social attributes, functions, etc.Advanceddata.

Having understood what these three letters are, we need to process this token.

Let's assume the word vector that this word carries is X_a.

Here, each W represents a weight, which is the weight information learned by the model during training.

Let's assume we input '苹果公司发明了手机'.

In English, that is 'Apple Company invented phones'.

Thus, we have obtained the values of the Q, K, and V vectors.

Alright, next, let's process these three values.

We take the Q value of the word 'apple' and perform a dot product with all the K values of this sentence to get the correlation between the two tokens (all values below are fictional).

Q_A · Ka = 3.2 (itself, high correlation)

Q_A · Kc = 1.4 (apple is a company, relatively high correlation)    

Q_A · Ki = 9.6 (there is a subject-verb relationship between the two words, very high correlation)

Q_A · Kp = 2.2 (the two words have a subject-object relationship, high correlation)

Next, we divide these obtained values by the square root of Dk (Dk is the vector dimension) (Softmax normalization).

If Dk is very large, then the resulting values will be very large, and the model will not be able to train.

The purpose of dividing like this is to make the model trainable. To cool down the data.

After cooling down the values, we do one thing:Normalization 

What does this do? It encodes the values after Softmax into a range between 0 and 1, where all values add up to 1.

At this point, we have obtained what is calledWeights.

Apple - Apple = 0.3

Apple - Company = 0.05

Apple - Invented = 0.5

Apple - phone = 0.15

At this time, V comes into play. We sum up all the V values of this sentence, and we get the information carried by the word Apple.

To summarize into a formula, it looks like this:

By now, you have understood the Attention mechanism.

Don't rush, we are not done yet. Here, we might wonder, the weight of 'Invented' is as high as 0.5, does that mean the importance of other features has been diluted? Indeed it has. So in 'Attention Is All You Need', they inventedMulti-Head Attention mechanism.

The principle is as follows: we prepare H groups (usually 8-16 groups) of Wa, Wi, Wc, Wp, and perform parallel computations. For example, one group of Wp specifically observes the relationship between the subject and the object, while one group of Wi specifically observes the subject-verb relationship.

In this way, we combine the results of all groups, called(Concat),The result obtained gives us a new and more comprehensive understanding of this word.

On another note, previously we mentioned that we put all this stuff together for parallel computation, but there is a problem: how does the machine distinguish the order? For example, the machine might confuse 'Zhang San hits Li Si' with 'Li Si hits Zhang San'. Wouldn't that be chaotic? Thus, we introducePositional Encoding.

So, before we throw that chunk into parallel computation, we add a positional encoding to each word (usually generated by sine/cosine), so it won't get mixed up.

Alright, that's all about Attention. In the next issue, we will introduce Encoder and Decoder.

Thank you so much for reading this far~ Goodbye~

这章，我们来介绍传统ML的终点，DL （Deep Learning深度学习）的基石，也是LLM的基座——Transformers。

相信这一章的内容结束后，我们离LLM也不远了。

相比于原生的Transformers， 它的高级Wrapper， Kiras就更好用，也更适合新手，尤其是数学，理论基础不好的人去学习。这里我们这一大章节也会利用Kiras。

这一章的内容实在是过多，所以分4小章节，也就是4篇文章来分别介绍。

顺带一提，个人认为Transformers的发源文章，“Attention Is All You Need”，是一非常值得研读的论文。不过，在读那篇论文之前，容许我讲解一下，这样读起来就不会那么生分。

接下来，我们有两个分支，一个是偏向于语言输出类的LLM方向，一个是Computer Vision方向，也就是我现在正在研究的方向。如果你认为自己更加想要了解LLM方向，可以读后面的有分类的专门的文章，我们从Label讲起。如果你认为你更偏好CV，我们从PyTorch讲起。

我们接下来从头开启。这一章节是最基础的也是最重要的Attention机制。

我们首先需要回答几个问题：

为什么我们要用Attention机制？

这里我直接引一段AI的原话：

在Attention出现前，RNN/LSTM串行训练速度慢，还难以记住长文本里距离较远的信息。Attention让每个token能直接关联全文任意位置的词，既能捕捉长距离依赖，又支持并行运算，充分利用GPU算力，让训练大语言模型成为可能。

Attention机制用来干嘛的呢？

我们知道，单凭一个词你很难理解它在上下文里面什么意思，比如Apple在“Apple Company Invented Phones”这个句子里，单只一个Apple，你可能理解为苹果这个水果，但是综合上下文，我们知道，apple在这句话里面指的是这个发明了手机的科技公司。

我们先从向量开始讲。

在讲索引Q之前，我们先来理清楚在ML的语境下，向量是什么。

----------------------------------------------------------------------------------------------------------------------------

首先，我们在给每一个词做分类的时候，会给他做一个（一般是）三维的向量。

如： King = { 14, 56, 38 }.   male = { 8, 34, 25 } （假的，我也不知道是多少）

那么这时候我们为什么要给每一个词做一个索引呢？原因就是我们需要一个能够把所有词语联系起来的量。向量很好地解决了这一点。

就像向量的距离，两个词语被label的向量的距离越近，那么在现实中，两个词语的联系就越高。

一般来说，对于大部分情况，有一个很有意思的事实：

当我们把 King的向量，（也就是我刚才乱编的），减去 Male的向量，我们会得到Queen的向量。想一想，对于现实逻辑，当我们在所有的国王里面减去所有男性国王，是不是我们就得到了女王这个集合。欸，就这就是向量的作用。

既然我们理解了在词向量里，向量的作用，我们就方便讲 Q，K，V了。

首先从Q开始讲。Q是什么？Q这里代表“Query”，也就是索引。

Q代表我们要去哪里找。

K呢？ K代表Key，也就是特征标签。

什么意思？比如我们输入Token是“苹果”，那么它的Key，就是 水果/公司

V呢？V代表Value也就是我们这个Token身上带的核心信息。

也就是这个Token身上带的深层信息，比如社会属性，功能等高级的数据。

得知了这三个字母是啥，我们就要对这个Token处理了。

我们假定这个词语自带的词语向量是 X_a

这里每一个W代表权重，是我们在训练中模型学会的权重信息。

假定我们输入“苹果公司发明了手机”

用英文也就是“ Apple Company invented phones"

这样，我们就得到了Q，K，V三个向量的值。

好了，接下来，我们来处理这三个值。

我们把apple这个词的Q值与这段话的所有K值做点积。得到两个Token的关联度（下面所有的值都是虚构的）

Q_A · Ka = 3.2 （自己就是自己，关联度高）

Q_A · Kc = 1.4    （apple是一个公司，关联度偏高）    

Q_A · Ki = 9.6   （两个词之间是主谓关系，关联度非常高）

Q_A · Kp = 2.2   （两个词是主语和宾语关系，关联高）

接下来，我们把这些得到的值，除以根号下Dk（Dk是向量维度）（Softmax化）

如果Dk很大，那么得出来的值就很大，模型训练不动。

我们这样子除的目的就是让模型训练的动。给数据降温。

在经过降温的值之后，我们做一个事情：归一化 

这个事情在做什么？把Softmax后的值encode成0-1之间的值，所有的值加起来等于1。

这时候，我们就得到了所谓的权重。

Apple - Apple = 0.3

Apple - Company = 0.05

Apple - Invented = 0.5

Apple - phone = 0.15

这时，V派上用场。我们把这个句子所有的V值加起来，就得到了了Apple这个词所带有的信息。

总结一下成一个公式，长这样：

\(\text{Attention}(Q, K, V) = \text{Softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V\)

到这里，你已经理解了Attention机制了。

别急，我们还没结束。这里，我们是否会想，刚才的“Invented”竟然有0.5这么多的权重，那其他的特征的重要性是不是被稀释了？还真是。于是在“Attention Is All You Need”中，他们发明了多头Attention机制。（Multi-Head Attention）

原理是这样的：我们准备H组（通常是8-16组）的Wa，Wi，Wc，Wp，并行运算，比如，有一组Wp专门观察主语和宾语的关系，有一组Wi专门观察主谓关系。

这样，我们把所有组的结果联合起来，叫做（Concat），得到的结果，于是对这个词有了全新，更加全方位的认知。

另一个方面，在之前，我们说了我们把这一堆东西放在一起并行运算，那么有个问题，机器怎么分得清语序呢？举个例子，机器会把“张三打李四”和“李四打张三”混为一谈。这不乱套了嘛，于是引入了位置编码。（Positional Encoding）

于是，我们在把那一坨丢去并行运算前，给每一个词加上一个位置编码，（通常由正弦/余弦生成），这样就不会乱套了。

好了，这就是Attention的所有内容了。我们下期介绍Encoder和Decoder。

万分感谢看到这里的你～再见～