RECURRENT NEURAL NETWORKS

Recurrent neural networks(RNNs) are another popular deep learning neural network,along with convolutional neural networks.Convolutional neural networks specialize in processig images,while RNNs specialize in processing sequences.RNNs are derived from feedforward neural networks and can use their internal state(memory) to process variable-length sequences of inputs.RNNs have been aplied in sequence prediction,text analysis,and speech recognition.

Figure 1 shows the schematic diagram of a recurrent neural network and its unfolded structure.X is the sequencial input data ,O is the sequential output data,h is the recurrent unit (also called cell),and v is the feedback loop. As show in Figure 1 the recurrent structure occurs repeatedly in the neural network. The h recurrent unit can be implemented as long short-terem memory k (LSTM) network unit and a gated recurrent unit(GRU).

Figure 1: The schematic diagram of recurrent neural network and its unfolded structure

Figure 2 shows an interesting article that explains the differences of RNN,LSTM,and GRU.

Figure 2: The schematic diagram of RNN,LSTM,and GRU

Figure 3 shows the four types of recurrent neural networks.

• One to one: One input and one output
• One to many: One input and multiple output
• Many to one: Multiple inputs and one output
• Many to many: Multiple inputs and multiple output

Figure 3: The different types of vanilla RNN

For more details about recursive neural networks,see the following:

[embed]Power of Recurrent Neural Networks (RNN): Revolutionizing AI The article explains what is a recurrent neural network, LSTM & types of RNN, why do we need a recurrent neural…www.simplilearn.com

1.VANILLA RNNs

A vanilla RNN is the simplest recurrent neural network that has an input vector, an output vector,and a recurrent unit or cell, as shown in Figure 1.1. Within the recurrent unit ,there is an activation function,called tanh.You can also use a different activation function.

Figure 1.1: The vanilla RNN structure

The easiest waty to implement vanilla RNN is to use the Keras functioncalled SimpleRNNS;see the following links for more details.

[embed]tf.keras.layers.SimpleRNN | TensorFlow v2.16.1 Fully-connected RNN where the output is to be fed back as the new input.www.tensorflow.org

[embed]Keras documentation: SimpleRNN layer Keras documentationkeras.io

The following is another example of vanilla RNNs using Keras SimpleRNN.It comes from an impresssive website called Easy-deep-learning-with-Keras,which contains many illustrative example codes on deep learning neural networks;see Figure 1.2.

buomsoo-kim/Easy-deep-learning-with-Keras: Keras tutorial for beginners (using TF backend) (github.com)

2.Long-Short Term Memory

Long short term memory (LSTM) is the most popular recurrent neural network architecture.Unlike conventional feedforward neural networks, LSTM has feedback connections. It can be process both single input data (e.g.,an image ) and a sequence of data (e.g.,sequences of image in a video).LSTM networks were developed to deal with the vanishing gradient and are well-suited for classification and prediction based on time-series data.

Figure 2.1 shows an ordinary LSTM unit,consisting of a cell,an input gate,an output gate,and a forgetting gate.The cell remembers values over arbitrary time intervals,and the three gates regulate the flow of information in and out of the cell.

Figure 2.1: The long short memory unit structure

LSTM has been commonly used in the following ;

• Time-series prediction
• Time-series anomaly detection
• Speech recognition
• Language translation
• Music composition
• Text generation
• Hardwriting recognition
• Human action recognition
• Robot control

There are 5 types of LSTM networks:

• LSTM classic
• Peephole connections
• Gated recurrent unit
• Multiplicative LSTM (2017)
• LSTM’s with attention

See following website about the five types of LSTM network:

[embed]404: Not Found | Exxact Exxactblog.exxactcorp.com

Now let’s look at some RNN ptyhon example code.

This is the output of the example,showing te inputs and outputs of the SimpleRNN:

This code shows how to define a SimpleRNN depth and pass input data through this layer. As a result, the throughput at the last time step of the RNN cannot be achieved for each sequence. This simple example is useful for capturing the basic working of RNNs. In more complex scenarios, such RNN layers can be used for tasks such as time series predictions, language components, and sequence analysis.

This is another example of Keras SimpleRNN,showing how to create an RNN model and show the model summary.

This is the output of the previous example,showing the inputs and the outputs of the SimpleRNN:

The following is another Keras SimpleRNN example,predicting the airline passenger numbers,which is modified based on the example code at numbers ,which is modified based on the examplecode at https://www.datatechnotes.com/2018/12/rnn-example-with-keras-simplernn-in.html .

First the airline passenger numbers are retrieved from a website:

df=pd.read_csv(“https://raw.githubusercontent.com/jbrownlee/Datasets/master/airline-passengers.csv”,usecols=[1],engine=”python”)

Then put the first 80 percent of the data as training data and use the remaining 20 percent for prediction.

Tp=int(df.shape[0]*0.8)

Then build s simple RNN model,train the model wtih the training data(first 80 percent of the totatl data),predict the data,and finally plot the resuls.

Output:

The plot output of the last example.This image showing the original data, the predictions,and the vertical red line representing the 80 percent prediction point.The x-axis consist of data points at different time and the y-axis consists of the number of air passengers.The predicted result after the 80 percent prediction point agrees well with the original data,even though the model has never seen the data before.

This another example shows how to create a simple Keras LSTM network.It was modified based on the Tensorflow RNN website.

Output:

3.Natural Language Processing and Python Natural Language Toolkit

Natural language processing(NLP) is a form of arficial intelligence that deals with human (natural) languages,in particular with the processing and analysis of large amounts of natural language data.The main challenges in NPL are speech recognition,natural language understanding and natural language generation.

The Python Natural Language Toolkit(NLTK) is a leading open source libraray for natural language processing.It has easy-to-use interfaces with more than 50 corpora lexical resources(such as WordNet).In linguistics, a corpus is a language resource consisting of a large and structured set of texts.NLKT has a number of libraries for classification,tokenization,stemming,tagging,parsing,semantic reasoning,and more.

For more details about NLTK,see following:

[embed]NLTK NLTK is a leading platform for building Python programs to work with human language data. It provides easy-to-use…www.nltk.org

This examples shows a Python example code for using the NLTK library for text analysisThe text in the code is from the Wikipedia page “Atrificial Intelligence”.

• maxent_ne_chunker: Model used for named entity recognition (NER).
• words: Word database used for NER.
• punkt: Model used for sentence and word tokenization.
• averaged_perceptron_tagger: Model used for part-of-speech (POS) tagging.

Outputs:

• print(tokens)

In this section, we define a long text string named sentence. We then use NLTK's word_tokenize function to split this text into individual words, which are stored in the tokens list.

• print(tagged)

In this step, we assign part-of-speech tags to each token (word). The nltk.pos_tag function analyzes each token and determines its type (noun, verb, adjective, etc.). The tagged list contains tuples, each consisting of a token and its corresponding POS tag

• print(entities)

Finally, we perform named entity recognition (NER) using the nltk.chunk.ne_chunk function, which takes the POS-tagged tokens and identifies named entities (e.g., person names, locations, organization names). The entities variable is a tree structure that represents these named entities.

There is a continuation of this output. However, I am sharing this piece because it is too long. If you want to see the completed code:

RNN/NLTK.py at main · jaguuai/RNN (github.com)