Let's get the core idea of the Seq2Seq model

The Seq2Seq model is mainly used to achieve the conversion from one sequence to another, such as French-English translation. The Seq2Seq model consists of two deep neural networks.The deep neural network can be other neural networks such as RNN(Recurrent neural network) or LSTM(Long short-term memory). The Seq2Seq model uses a neural network to map the input sequence to a fixed-dimensional vector, which is an encoding process; then another neural network maps this vector to the target sequence, which is a decoding process.The model structure of Seq2Seq is shown in Fig1. The model inputs the sentence "ABC" and then generates "WXYZ" as the output sentence.

Fig1 Model structure of Seq2Seq

Encoding and decoding

Encoding and decoding is the core part of the Seq2Seq model.The deep learning network uses RNN as an example to explain the principle of encoding and decoding. The structure of encoding and decoding is shown in Fig 2.

Fig 2 Structure of encoding and decoding

Encoding: The RNN reads each symbol of the input sequence 𝑋 sequentially. When each symbol is read, the hidden state ℎ<𝑡 > of the RNN changes, the formula is as follows. After reading the end of the sequence, the hidden state of the RNN is the summary 𝐶 of the entire input sequence.

Decoding: After another RNN is trained, the output sequence is generated by predicting the next symbol 𝑦𝑡 . At this time, the hidden state ℎ <𝑡> of the decoder at time 𝑡 is calculated as follows.𝑦𝑡 and ℎ<𝑡> are 𝑦𝑡−1 and 𝑐 for condition.

The conditional distribution for the next symbol is

Train the encoder and decoder of the RNN together, and find the maximum conditional log-likelihood function as follows:

Where θ is a set of model parameters, and each (𝑥𝑛 , 𝑦𝑛 ) is a combination of an input sequence and an output sequence from the training set.

The specific structure of the hidden unit of the RNN is shown in FIG. 3, which includes an update gate z and a reset gate r. The role of the update gate is to select whether the hidden state is updated by the new hidden state. The role of the reset gate is to determine whether the previous hidden state was ignored.

Fig 3 Structure of hidden unit

The specific process of calculating the activation function of the 𝑗𝑡ℎ hidden unit in the RNN is as follows:

The calculation formula of the reset gate 𝑟𝑗 is:

Where 𝜎 is the logical sigmoid function, and 𝐽 represents the 𝑗𝑡ℎ element of the vector. 𝑥 and ℎ <𝑡−1> represent the input sequence and the previous hidden state, respectively. 𝑊𝑟 and 𝑈𝑟 are learning weight matrices.

Then calculate the update gate 𝑧𝑗 in the same way:

The activation function of the last hidden unit ℎ𝑗 is:

In this formula, when the reset gate approaches 0, the hidden state will force the previous state to be ignored and reset with the current input. This method can make hidden state delete unimportant information in the future.

The Attention Mechanism

When the original Seq2Seq model is translated, the source sentence is compressed into a fixed-length vector, which makes it difficult for the neural network to process long sentences. After the Attention mechanism was proposed, this problem was effectively solved. The core idea of the Attention mechanism: each time the model generates a word in translation, it searches for the most relevant set of positions in the source sentence. Then, the model will base on the context vector associated with these source positions and all the target words previously generated To predict new target words.

Fig 4 Given the source sequence (𝑥1 , 𝑥2 , . . . , 𝑥𝑇 ), try to generate the 𝑡𝑡ℎ target word 𝑦𝑡

In [ ]: