Introduction

Recurrent neural networks, (RNNs), are a class of neural networks that stores relevant parts of the input variables and use this information to predict output in the future. RNNs repetitively perform the same computational operation on every element of a sequence and each output is calculated based on the previous computations (Figure 2-63). This computational process captures the interdependency of the sequential time series data to estimate the outputs, that is why RNNs process sequential data very well [76].

Figure 2-64 Complete Structure of RNN Cell

As shown in Figure 2-64, an RNN can be considered as a chain of repeating modules. In standard RNNs (Figure 2-63), this repeating module will have a very simple structure, such as a single tanh layer. Hidden state or memory cell of this structure preserves information from the previous input variables. At time step t, the memory cell’s current state (ht) is a function of input state vectors at that current time step (Xt) and hidden state at the previous time step (ht-1), so h(t)=f(ht-1,Xt). Its output at time step t, denoted by (yt), is also a function of the previous state and the current input (Figure 2-63). For standard RNN cells, the output (yt) and the hidden state (ht) at a given time step are same.

Figure 2-65 A Recurrent Neuron Network Unrolled through Time [77]

Although RNNs can better capture nonlinearity in time series problems, they are weak on learning long-term dependencies due to vanishing of gradient during the backpropagation process ([78, 79]). Moreover, traditional RNNs learn a time series sequence based on a predetermined time lag, but it is difficult to find an optimal time window size in an automatic way ([78] [80]).