31st July 2015

Neural network is 10 times bigger than the previous world record

Digital Reasoning, a developer of cognitive computing, recently announced that it has trained the largest neural network in the world to date with a stunning 160 billion parameters. Google’s previous record was 11.2 billion, while the Lawrence Livermore National Laboratory trained a neural network with 15 billion parameters.

The results of Digital Reasoning’s research with deep learning and neural networks were published in the Journal of Machine Learning and Arxiv alongside other notable companies like Google, Facebook, and Microsoft. They were presented at the prestigious 32nd International Conference on Machine Learning in Lille, France, earlier this month.

Neural Networks are computer systems that are modelled after the human brain. Like the human brain, these networks can gather new data, process it, and react to it. Digital Reasoning’s paper, titled “Modelling Order in Neural Word Embeddings at Scale,” details both the impressive scope of their neural network as well as the exponential improvement in quality.

In their research, Matthew Russell, Digital Reasoning’s Chief Technology Officer, and his team evaluated neural word embeddings on “word analogy” accuracy. Neural networks generate a vector of numbers for each word in a vocabulary. This allowed the research team to do “word math.” For instance, “king” minus “man” plus “woman” would yield a result of “queen.” There is an industry standard dataset of around 20,000 word analogies. Google's previous accuracy on this metric was a 76.2% accuracy rate. In other words, Google was able to get 76.2% of the word analogies "correct" in their system. Stanford's best score is a 75.0% accuracy. Digital Reasoning’s model achieves a score of 85.8% accuracy, which is a near 40% reduction in error over both Google and Stanford, a massive advancement in the state of the art.

“We are extremely proud of the results we have achieved, and the contribution we are making daily to the field of deep learning,” said Russell. “This is a tremendous accomplishment for the company and marks an important milestone in putting a defensible stake in the ground towards our position as not just a thought leader in the space, but as an organisation that is truly advancing the state of the art in a rigorous, peer reviewed way.”

Comments »