EXPERIMENTS WITH CHARACTER GENERATING RECURSIVE NEURAL NETWORKS (2016)

I should start by thanking Andrej Karpathy for without his code this project would have been impossible.

My work on recursive neural networks, began by osmosis. A few different ideas I had been working on all merged handily into one whole. My previous project had investigated deep dream, and the order and chaos therein. I started to look at large corpora of text, such as the entire works of Shakespeare, and the bible. These large volumes of text normally followed a system of subdivision. The idea intrigued me that these large works of text could be broken down and a specific verse or line found with a sequence of a few numbers. I found it exciting that these short sections were often about 140 characters, the same as a tweet. It struck me that a tweet is usually a memory, condensed into 140 characters, and indeed this is the reason for their length. I pondered that if a tweet could represent a memory, with a few numbers to specify the time and date on which it occurred, then it must form part of a larger whole, in the same way that one phrase in the bible is part of a larger book.

While I researched these volumes of text, I found that many of them were linked to the Char-RNN project, as it takes these sort of works as input. I found that the works of Shakespeare has already been used to make new Shakespeare to great effect. Somebody had even made the same links in their mind that I had, and had started an automatic twitter account that every day generated a new bible quote using Char-RNN.

I set up the software and began by reproducing the results of others with works like Shakespeare and the Bible as they are easily accessible. From there I moved onto my Facebook chat history. I realised that I had the entirety of it as a .JSON file so I wrote a program to parse it and make a plane text file consisting simply of line after line of :

SENDER : TIME : DATE : MESSAGE CONTENT

The Char-RNN model produced was fascinating because it picked up on little things, such as the frequency with which each person spoke, as well as the general formatting of the document. It even picked up on messages that had been sent repeatedly, creating large swathes of output containing only one message repeated over and over. It did not, as I had hoped, manage to mimic the people it was learning from, It managed to understand that many conversations happened on the same topics, but the messages were rarely specific to the 'sender'.

This got me thinking, about whether this computer program is close to sentience, I could not make up new text based on old text in anywhere near the same manner. As it is just a statistical model, is our brain not only a statistical model, if so a wildly more complex one.

In the end I wound up using the Dictionary as an input for the software, as I thought it would be interesting to investigate how the computer reacts when I task it with defining certain words. This is achieved by giving the computer a starting series of letters (in this case the word to be defined) to start from when it begins writing. Interestingly the computer would often 'refuse' to define a certain word, in that it would begin a new line immediately after the input phrase and continue to ignore it. It was fascinating to me which words it would or would not refuse, and the process of changing settings until it would define the word, albeit with an appendage such as an input of 'art' would end up as 'artpolynomial' or 'artistic'. When it would define a word, the definitions were usually wildly inaccurate, drawing from all the other irrelevant definitions it had read and writing commonly used words or phrases. Once in a while however, it would say something that made sense in one way or another, never an accurate definition, but one that at least made you think the computer was really trying to get it right.

This set of Ideas formed the basis for continuing work visualising the similarities of virtual neural networks to the brain.

Extended Examples of Computer Writing.

Bible, 2016 : https://docs.google.com/document/d/1vjUphqvE33Nvm1jib1W1I6LC2eOugj0ZRvHwf0p47MY/edit?usp=sharing

Generated Dictionary : https://docs.google.com/document/d/1pofn9lQOpWCGz272R2KMq2c1rSEtj8p0VKBkw_ub1K0/edit?usp=sharing

Pride and Prejudice, 2016 : https://docs.google.com/document/d/1Kf2Q9K9FJC0y3ZUWQLBrTKI5viJaVE-1sT3zFTPeX18/edit?usp=sharing