Artificial Intelligence
Neural Networks
The AI of the past used brute-force computing to analyze data and present them in a way that seemed human. The programmer supplied the intelligence in the form of decision trees and algorithms. Imagine that you were trying to build a machine that could play tic-tac-toe. You would give it specific rules on what move to make, and it would follow them. That is essentially how IBM’s Big Blue computer beat chess Grandmaster Garry Kasparov in 1997, by using a supercomputer to calculate every possible move faster than he could.
Today’s AI uses machine learning in which you give it examples of previous games and let it learn from those examples. The computer is taught what to learn and how to learn and makes its own decisions. What’s more, the new AIs are modeling the human mind itself using techniques similar to our learning processes. Before, it could take millions of lines of computer code to perform tasks such as handwriting recognition. Now it can be done in hundreds of lines. What is required is a large number of examples so that the computer can teach itself.
The new programming techniques use neural networks — which are modeled on the human brain, in which information is processed in layers and the connections between these layers are strengthened based on what is learned. This is called deep learning because of the increasing numbers of layers of information that are processed by increasingly faster computers. These are enabling computers to recognize images, voice, and text — and to do human-like things.
AI has applications in every area in which data are processed and decisions required. Wired founding editor Kevin Kelly likened AI to electricity: a cheap, reliable, industrial-grade digital smartness running behind everything. He said that it “will enliven inert objects, much as electricity did more than a century ago. Everything that we formerly electrified we will now ‘cognitize.’ This new utilitarian AI will also augment us individually as people (deepening our memory, speeding our recognition) and collectively as a species. There is almost nothing we can think of that cannot be made new, different, or interesting by infusing it with some extra IQ. In fact, the business plans of the next 10,000 start-ups are easy to forecast: Take X and add AI This is a big deal, and now it’s here.”
-
YANN LECUN IS among those bringing a new level of artificial intelligence to popular internet services from the likes of Facebook, Google, and Microsoft.
As the head of AI research at Facebook, LeCun oversees the creation of vast “neural networks” that can recognize photos and respond to everyday human language. And similar work is driving speech recognition on Google’s Android phones, instant language translation on Microsoft’s Skype service, and so many other online tools that can “learn” over time. Using vast networks of computer processors, these systems approximate the networks of neurons inside the human brain, and in some ways, they can outperform humans themselves.
This week in the scientific journal Nature, LeCun—also a professor of computer science at New York University—details the current state of this “deep learning” technology in a paper penned alongside the two other academics most responsible for this movement: University of Toronto professor Geoff Hinton, who’s now at Google, and the University of Montreal’s Yoshua Bengio. The paper details the widespread progress of deep learning in recent years, showing the wider scientific community how this technology is reshaping our internet services—and how it will continue to reshape them in the years to come.
But as LeCun tells WIRED, deep learning will also extend beyond the internet, pushing into devices that can operate here in the physical world—things like robots and self-driving cars. Just last week, researchers at the University of California at Berkeley revealed a robotic system that uses deep learning tech to teach itself how to screw a cap onto a bottle. Early this year, big-name chip maker Nvidia and an Israeli company called Mobileye revealed that they were developing deep learning systems that can help power self-driving cars.
LeCun has been exploring similar types of “robotic perception” for over a decade, publishing his first paper on the subject in 2003. The idea was to use deep learning algorithms as a way for robots to identify and avoid obstacles as they moved through the world—something not unlike what’s needed with self-driving cars. “It’s now a very hot topic,” he says.
Yes, Google and some many others have already demonstrated self-driving cars. But according to researchers, including LeCun, deep learning can advance the state of the art—just as it has vastly improved technologies such as image recognition and speech recognition. Deep learning algorithms date back to the 1980s, but now that they can tap the enormously powerful network of machines available to today’s companies and researchers, they provide a viable way for systems to teach themselves tasks by analyzing enormous amounts of data.
“This is a chance for us to change the model of learning from very shallow, very confined statistics to something extremely open-ended,” Sebastian Thrun, who helped launched the Google self-driving car project, said of deep learning in an interview this past fall.
Thrun has left Google, but odds are, the company is already exploring the use of deep learning techniques with its autonomous cars (the first of which are set to hit the road this summer). According to Google research fellow Jeff Dean, the company is now using these techniques across dozens of services, and self-driving cars, which depend so heavily on image recognition, are one of the more obvious applications.
Trevor Darrell, one of the researchers working on deep learning robots at Berkeley, says his team is also exploring the use of the technology in autonomous automobiles. “From a researchers perspective, their are many commonalities in what it takes to move an arm to insert a peg into a hole and what it takes to navigate a car or a flying vehicle through an obstacle course,” he says.
Deep learning is particularly interesting, he says, because it has transformed so many different areas of research. In the past, he says, researchers used very separate techniques for speech recognition, image recognition, translation, and robotics. But now one this one set of techniques—though a rather broad set—can serve all these fields.
The result: all of these fields are suddenly evolving at a much faster rate. Face recognition has hit the mainstream. So has speech recognition. And the sort of autonomous machines his team is working on, Darrell says, could reach the commercial market within the next five years. AI is here. But it will soon arrive in a much bigger way.
Deep Learning
A machine-learning technique that has already given computers an eerie ability to recognize speech and categorize images is now creeping into industries ranging from computer security to stock trading. If the technique works in those areas, it could create new opportunities but also displace some workers.
Deep learning, as the technique is known, involves applying layers of calculations to data, such as sound or images, to recognize key features and similarities. It offers a powerful way for machines to recognize similarities that would normally be abstruse to a computer: the same face seen from different angles, for instance, or a word spoken in different accents (see “10 Breakthrough Technologies 2013: Deep Learning”). The mathematical principles that underlie deep learning are relatively simple, but when combined with huge quantities of training data and computer systems capable of powerful parallel computations, the technique has resulted in dramatic progress in recent years, especially in voice and image recognition.
For example, Google uses deep learning for voice recognition on Android phones, while Facebook uses the technology to identify friends in users’ photographs (see “Facebook Creates Software That Matches Faces Almost as Well as You Do”).
Other tech companies are following. At an event in Boston last week, two researchers from eBay described how the company is using deep learning to categorize products in images posted by sellers. By studying images that have already been tagged, the system can tell the difference, for example, between a pair of flip-flops and a pair of flats. This is helping to improve eBay’s search engine, especially for products that haven’t been tagged very well.
People in other fields and industries are starting to show an interest in deep learning. At the Boston event, researchers, engineers, and entrepreneurs discussed progress in the field and its potential application in advertising, finance, and medicine. One attendee who had previously applied machine learning techniques to hedge funds had founded a startup to use deep learning to predict market shifts like a sudden plunge in a currency’s value. Another attendee, from a major U.S. insurance company, was looking into using deep learning to identify fraudulent claims.
Andrew Ng, a leading figure in the field, and both an associate professor at Stanford and chief scientist at the Chinese company Baidu, said at the conference that deep learning has already proven useful. “One of the things Baidu did well early on was to create an internal deep learning platform,” Ng said. “An engineer in our systems group decided to apply it to decide a day in advance when a hard disk is about fail. We use deep learning to detect when there might’ve been an intrusion. Many people are now learning about deep learning and trying to apply it to so many problems.”
Deep learning is being tested by researchers to glean insights from medical imagery. Emmanuel Rios Velazquez, a postdoctoral researcher at the Dana-Farber Cancer Institute in Boston, is exploring whether deep learning could help to more accurately predict a patient’s outcome from images of his or her cancer.
Drug discovery is another promising area. Olexandr Isayev, a research scientist from the University of North Carolina at Chapel Hill, has shown that deep learning algorithms can help train computers to pick out potentially useful drug molecules from hundreds of millions of candidates. Isayev fed data from hundreds of thousands of experiments into his computer systems, and then had his system predict how a molecule might bind to a particular group of proteins. “A typical machine-learning algorithm does one objective function,” he said. “[With deep learning] you can do multiple optimizations. For example, you might want to maximize binding with this protein but minimize binding with some other protein.”
Deep learning doesn’t work best for everything, as Isayev’s work demonstrates. He says the improvements it offered for computerized drug discovery were modest compared to what it could do for computerized image recognition.
Even so, the potential for deep learning to be applied more broadly can be seen with the emergence of some well-funded startups. Palo Alto-based MetaMind, which has developed a deep learning platform, was founded by Richard Socher, who studied under Ng at Stanford. “We have people classifying fashion, cars, houses, satellite images, and each of these are already gigantic industries,” Socher says. “The beauty of deep learning is that, from the raw input to the final output, it’s all learned.”
-
Self Teaching Machines
At first, the video clip doesn’t look like a landmark in science. Given the task of teaching itself how to beat 49 classic Atari video games, the computer clearly has no idea what’s going on — but it doesn’t last. Within two hours, it plays those games like a fortysomething who hasn’t seen daylight since the early 1980s, announcing itself in the process as the first artificial intelligence system to teach itself disparate tasks from scratch.
There was a time when AI was the preserve of science fiction, of Isaac Asimov and I, Robot. Now it is big business. DeepMind Technologies, the British company that developed the system, was snapped up by Google for £300 million last year.
In October DeepMind said, in turn, that it was acquiring Vision Factory, a UK start-up working on image recognition, and Dark Blue Labs, another homegrown fledgeling doing similar work in speech recognition. It also launched a tie-up with the University of Oxford computer science department. And with these deals, Google has put itself squarely at the head of the field of “deep learning”, the AI process that many hope will lead to the industry’s holy grail — achieving human-like intelligence in machines.
Given that Google’s ambitions in the sector are obvious, smaller companies with any intelligence of their own might be expected to give it a wide berth. Not so. With the big groups focused on the bigger and more distant goal of cracking the nut known as general AI, entrepreneurs are focusing on filling in the gaps, using the giant strides made already to tackle more immediate and niche challenges.
Magic Pony Technology is using this more narrow AI to improve video processing. The London-based start-up’s platform can improve the resolution of any image, or compress it to a smaller file size with no loss of quality, because it has seen enough similar images to guess intelligently the details that should be there — in the same way that a person could draw the rest of a chair if handed a picture that shows only the middle part. The company is aiming the technology at mobile brands serving emerging markets such as India and Africa, where bandwidth is not strong to handle the video-streaming demands of smartphone users.
“We’re a team of seven at the moment, largely part-time,” Rob Bishop, Magic Pony’s co-founder, says, “but we can do powerful things. With developments like Amazon web services, we can run our own supercomputer in the cloud from just a few thin laptops.” Magic Pony is about to close a seven-figure round of investment, which may sound fanciful to naysayers. After all, AI has been touted as the next big thing since the days when Atari represented the pinnacle of video games. However, the processing power was not there then: programmers had to teach computers rigid rules over how things worked, so AI stumbled.
“It’s like the old joke about the Daleks,” says Mike Lynch, the cofounder of Autonomy and former machine learning academic at Cambridge, who is investing in several young AI companies as the chief executive of Invoke Capital. “Their plans for world domination were fine until they hit a step. But while AI was at the level of a sea urchin a few years ago, it’s now around the level of a two-year-old.”
The unique fact about this particular two-year-old is that it can handle certain tasks — such as pattern-spotting, fact-checking or cross-referencing — far better than any adult. A computer also can work much more quickly and take on a far greater workload, all without a break.
Celaton, based in Milton Keynes, brands its inSTREAM platform as “the best knowledge worker you’ve ever hired”, able to make sense of the reams of unstructured information that bombard large organisations on a daily basis.
Celaton says that inSTREAM can recognise the nature of a customer complaint, for example, understand why it has happened and craft a personalised response with minimum human input. The company says that turnover will rise to £4.5 million in the next financial year.
Perhaps the most promising role for narrow AI may lie in the “internet of things”, the near-future network where devices from your watch to your fridge will communicate with each other on a continual basis. Invoke has invested in Neurence, an AI company developing a “big brain in the cloud”.
Not that the machines will be talking solely to each other. “The internet of things won’t be possible without a simple way to interact with all of these devices,” Vishal Chatrath, of VocalIQ, says. The Cambridgebased start-up has developed an alternative to Apple’s Siri that engages the user in conversation. The company is releasing a trial app next month.
The ambition does not end there. “One of our key projects is to develop a car that can talk to you, like in Knight Rider,” Mr Chatrath says. “That’s the level we’re targeting.”
If all this has been achieved by the equivalent of a two-year-old, imagine what AI could be doing by the time it starts school.
-
Fact Checking
Computers can now do fact-checking for any body of knowledge, according to Indiana University network scientists, writing in an open-access paper published June 17 in PLoS ONE.
Using factual information from summary infoboxes from Wikipedia* as a source, they built a “knowledge graph” with 3 million concepts and 23 million links between them. A link between two concepts in the graph can be read as a simple factual statement, such as “Socrates is a person” or “Paris is the capital of France.”
In the first use of this method, IU scientists created a simple computational fact-checker that assigns “truth scores” to statements concerning history, geography and entertainment, as well as random statements drawn from the text of Wikipedia. In multiple experiments, the automated system consistently matched the assessment of human fact-checkers in terms of the humans’ certitude about the accuracy of these statements.
Dealing with misinformation and disinformation
In what the IU scientists describe as an “automatic game of trivia,” the team applied their algorithm to answer simple questions related to geography, history, and entertainment, including statements that matched states or nations with their capitals, presidents with their spouses, and Oscar-winning film directors with the movie for which they won the Best Picture awards. The majority of tests returned highly accurate truth scores.
Lastly, the scientists used the algorithm to fact-check excerpts from the main text of Wikipedia, which were previously labeled by human fact-checkers as true or false, and found a positive correlation between the truth scores produced by the algorithm and the answers provided by the fact-checkers.
Significantly, the IU team found their computational method could even assess the truthfulness of statements about information not directly contained in the infoboxes. For example, the fact that Steve Tesich — the Serbian-American screenwriter of the classic Hoosier film “Breaking Away” — graduated from IU, despite the information not being specifically addressed in the infobox about him.
Using multiple sources to improve accuracy and richness of data
“The measurement of the truthfulness of statements appears to rely strongly on indirect connections, or ‘paths,’ between concepts,” said Giovanni Luca Ciampaglia, a postdoctoral fellow at the Center for Complex Networks and Systems Research in the IU Bloomington School of Informatics and Computing, who led the study.
“If we prevented our fact-checker from traversing multiple nodes on the graph, it performed poorly since it could not discover relevant indirect connections,” said Ciampaglia. “But because it’s free to explore beyond the information provided in one infobox, our method leverages the power of the full knowledge graph.
“These results are encouraging and exciting. We live in an age of information overload, including abundant misinformation, unsubstantiated rumors and conspiracy theories whose volume threatens to overwhelm journalists and the public. Our experiments point to methods to abstract the vital and complex human task of fact-checking into a network analysis problem, which is easy to solve computationally.”
Expanding the knowledge base
Although the experiments were conducted using Wikipedia, the IU team’s method does not assume any particular source of knowledge. The scientists aim to conduct additional experiments using knowledge graphs built from other sources of human knowledge, such as Freebase, the open-knowledge base built by Google, and note that multiple information sources could be used together to account for different belief systems.
The team added a significant amount of natural language processing research, but they note that additional work remains before these methods could be made available to the public as a software tool.
* The team selected Wikipedia as the information source for their experiment due to its breadth and open nature. Although Wikipedia is not 100 percent accurate, previous studies estimate the online encyclopedia is nearly as reliable as traditional encyclopedias, but also covers many more subjects, the researchers note.
IQ Tests
Computers have never been good at answering the type of verbal reasoning questions found in IQ tests. Now a deep learning machine unveiled in China is changing that.
Just over 100 years ago, the German psychologist William Stern introduced the intelligence quotient test as a way of evaluating human intelligence. Since then, IQ tests have become a standard feature of modern life and are used to determine children’s suitability for schools and adults’ ability to perform jobs.
These tests usually contain three categories of questions: logic questions such as patterns in sequences of images, mathematical questions such as finding patterns in sequences of numbers and verbal reasoning questions, which are based around analogies, classifications, as well as synonyms and antonyms.
It is this last category that has interested Huazheng Wang and pals at the University of Science and Technology of China and Bin Gao and buddies at Microsoft Research in Beijing. Computers have never been good at these. Pose a verbal reasoning question to a natural language processing machine and its performance will be poor, much worse than the average human ability.
Today, that changes thanks to Huazheng and pals who have built a deep learning machine that outperforms the average human ability to answer verbal reasoning questions for the first time.
In recent, years, computer scientists have used data mining techniques to analyze huge corpuses of texts to find the links between words they contain. In particular, this gives them a handle on the statistics of word patterns, such as how often a particular word appears near other words. From this it is possible to work out how words relate to each other, albeit in a huge parameter space.
The end result is that words can be thought of as vectors in this high-dimensional parameter space. the advantage is that they can then be treated mathematically: compared, added, subtracted like other vectors. This leads to vector relations like this one: king – man + woman = queen.
This approach has been hugely successful. Google uses it for automatic language translation by assuming that word sequences in different language represented by similar vectors are equivalent in meaning. So they are translations of each other.
But this approach has a well-known shortcoming: it assumes that each word has a single meaning represented by a single vector. Not only is that often not the case, verbal tests tend to focus on words with more than one meaning as a way of making questions harder.
Huazheng and pals tackle this by taking each word and looking for other words that often appear nearby in a large corpus of text. They then use an algorithm to see how these words are clustered. The final step is to look up the different meanings of a word in a dictionary and then to match the clusters to each meaning.
This can be done automatically because the dictionary definition includes sample sentences in which the word is used in each different way. So by calculating the vector representation of these sentences and comparing them to the vector representation in each cluster, it is possible to match them.
The overall result is a way of recognizing the multiple different senses that some words can have.
Huazheng and pals have another trick up their sleeve to make it easier for a computer to answer verbal reasoning questions. This comes about because these questions fall into several categories that require slightly different approaches to solve.
So their idea is to start by identifying the category of each question so that the computer then knows which answering strategy it should employ. This is straightforward since the questions in each category have similar structures.
So questions that involve analogies are like these:
Isotherm is to temperature as isobar is to? (i) atmosphere, (ii) wind, (iii) pressure, (iv) latitude, (v) current.
and
Identify two words (one from each set of brackets) that form a connection (analogy) when paired with the words in capitals: CHAPTER (book, verse, read), ACT (stage, audience, play).
Word classification questions are like this:
Which is the odd one out? (i) calm, (ii) quiet, (iii) relaxed, (iv) serene, (v) unruffled.
And questions looking for synonyms and antonyms are like these:
Which word is closest to IRRATIONAL? (i) intransigent, (ii) irredeemable, (iii) unsafe, (iv) lost, (v) nonsensical.
And
Which word is most opposite to MUSICAL? (i) discordant, (ii) loud, (iii) lyrical, (iv) verbal, (v) euphonious.
Spotting each type of question is relatively straightforward for a machine learning algorithm, given enough to examples to learn from. And this is exactly how Huazheng and co do it.
Having identified the type of question, Huazheng and buddies then devise an algorithm for solving each one using the standard vector methods but also the multi-sense upgrade they’ve developed.
They compare this deep learning technique with other algorithmic approaches to verbal reasoning tests and also with the ability of humans to do it. For this, they posed the questions to 200 humans gathered via Amazon’s Mechanical Turk crowdsourcing facility along with basic information about their ages and educational background.
And the results are impressive. “To our surprise, the average performance of human beings is a little lower than that of our proposed method,” they say.
Human performance on these tests tends to correlate with educational background. So people with a high school education tend to do least well, while those with a bachelor’s degree do better and those with a doctorate perform best. “Our model can reach the intelligence level between the people with the bachelor degrees and those with the master degrees,” say Huazheng and co.
That’s fascinating work that reveals the potential of deep learning techniques. Huazheng and co are clearly optimistic about future developments. “With appropriate uses of the deep learning technologies, we could be a further step closer to the true human intelligence.”
Deep learning techniques are currently sweeping through computer science like wildfire and the revolution they are creating is still in its early stages. There’s no telling where this revolution will take us but one thing is for sure: William Stern would be amazed.
Algorthms Taking Over
Digital technology is advancing so fast that computers could soon solve every problem, writes James Dean
By the time we celebrate New Year’s Eve 2025, your smartphone will understand your slurred requests for a midnight taxi, the car that collects you will be driverless and, when it drops you off, your sudden craving for a pre-bed snack will be satisfied by drone delivery.
“Machine learning” is developing so rapidly that experts have told The Times that the next decade will bring a revolution in computers and robotics, leading to intelligent machines and huge technological and social changes.
The Royal Society, Britain’s leading scientific institution, has set up a working party to investigate machine learning, whereby a computer takes in the data produced every day by the planet’s people and internet-connected objects to deduce patterns and, in effect, becomes intelligent.
Experts said that by 2025 electronic personal assistants such as Apple’s Siri will be able to communicate with us as if they were humans, and to anticipate our behaviour and daily priorities.
Internet companies such as Facebook already use algorithms to predict what we will like, for instance in deciding what will appear in our newsfeed or as adverts. As these pick up more data, the distinction between what humans consider intelligent and what computers are capable of will narrow.
“They’re constantly learning and, increasingly, they have perceptive capabilities,” Zoubin Ghahramani, a professor of information engineering at Cambridge University, said. “Machinelearning systems aren’t programmed to be fixed, but to improve and grow themselves in the same way that biological systems learn to improve.”
The consequences will be profound. Robots will be able to watch surgeons and learn how to copy their movements. Nursing-home staff will be assisted by robots that also can keep people company. Drones will take on delivery roles. Peter Donnelly, professor of statistical science at Oxford University, said: “Almost everything on our smartphones has machine learning at its core. In ten years there will be a whole lot of complex apps, central to our lives, that we can’t even conceive of now.” The secrets of human life are quietly being transferred to the digital brains of machines. Every second, they learn more about our foibles by digesting the vast amounts of personal information we put online.
Social networks, smartphone apps and countless internet-connected services are helping machines to learn about us so quickly that by 2025 we will be able to talk to artificially intelligent personal assistants as if they were humans.
The evolution is being catalysed by a process known as “machine learning”, whereby a computer is fed huge amounts of data from which it is able to draw its own meanings. Machine learning will be central to spurring advances in robotics for hospitals, home care and transport over the next decade, according to experts polled by The Times.
Such is the perceived importance of the field that last month the Royal Society created a machine-learning working group, comprising several leading thinkers, to explore its likely benefits and pitfalls.
“Twenty years ago, a computer programmer would have to work out a problem and then type out the code that allowed the computer to solve it,” said Peter Donnelly, chairman of the working group and professor of statistical science at the University of Oxford. “With machine learning, they programme the steps that allow the computer to learn the solution to the problem. You can’t programme a computer in a car and tell it what to do in every circumstance, because you can’t possibly conceive of them all.”
Professor Donnelly said that the expansion of the internet and the increased power of computers had revolutionised the field since the first forays were made about 20 years ago, and that ten years from now we would be using and relying on complex apps beyond our current imagination.
The power of machine learning is expected to accelerate rapidly in the coming years as computers digest more information about us from the internet, and the algorithms that power them become more advanced. This will help to create powerful virtual assistants that are able to pre-empt the needs of their human masters. Zoubin Ghahramani, a professor of information engineering at the University of Cambridge, said: “A lot of these algorithms are already affecting our lives, but most people don’t think about the fact that they are all forms of robot intelligence. “Every aspect of your activity on Facebook is controlled by this intelligence — what appears in your news feed, what adverts it shows you.”
Professor Ghahramani predicted that smartphone assistants would be “superintelligent” in ten years’ time. “We can already talk to them in a primitive way but we can’t have conversations with them,” he said. “By 2025 we will, and they’ll be much more natural. If you were to allow it, the assistant would know things about you — your priorities, what it should remind you about, clever recommendations it could make.”
There will be limits to the progress, however. Paul Newman, principal investigator at the University of Oxford’s mobile robotics group, said: “We’re not going to have something that helps us around the house. I don’t see the generalist robot coming for a long time. “Intelligent robots will first appear in places where they have a clearly defined role — that’s why there are already so many robots on car production lines. We have evolved extraordinary capabilities as humans, but look at how long it took us to evolve. I think the idea of a humanoid robot is jaundiced by science fiction, and at the moment, it is just that.”
Change Index page
Neural Networks
The AI of the past used brute-force computing to analyze data and present them in a way that seemed human. The programmer supplied the intelligence in the form of decision trees and algorithms. Imagine that you were trying to build a machine that could play tic-tac-toe. You would give it specific rules on what move to make, and it would follow them. That is essentially how IBM’s Big Blue computer beat chess Grandmaster Garry Kasparov in 1997, by using a supercomputer to calculate every possible move faster than he could.
Today’s AI uses machine learning in which you give it examples of previous games and let it learn from those examples. The computer is taught what to learn and how to learn and makes its own decisions. What’s more, the new AIs are modeling the human mind itself using techniques similar to our learning processes. Before, it could take millions of lines of computer code to perform tasks such as handwriting recognition. Now it can be done in hundreds of lines. What is required is a large number of examples so that the computer can teach itself.
The new programming techniques use neural networks — which are modeled on the human brain, in which information is processed in layers and the connections between these layers are strengthened based on what is learned. This is called deep learning because of the increasing numbers of layers of information that are processed by increasingly faster computers. These are enabling computers to recognize images, voice, and text — and to do human-like things.
AI has applications in every area in which data are processed and decisions required. Wired founding editor Kevin Kelly likened AI to electricity: a cheap, reliable, industrial-grade digital smartness running behind everything. He said that it “will enliven inert objects, much as electricity did more than a century ago. Everything that we formerly electrified we will now ‘cognitize.’ This new utilitarian AI will also augment us individually as people (deepening our memory, speeding our recognition) and collectively as a species. There is almost nothing we can think of that cannot be made new, different, or interesting by infusing it with some extra IQ. In fact, the business plans of the next 10,000 start-ups are easy to forecast: Take X and add AI This is a big deal, and now it’s here.”
-
YANN LECUN IS among those bringing a new level of artificial intelligence to popular internet services from the likes of Facebook, Google, and Microsoft.
As the head of AI research at Facebook, LeCun oversees the creation of vast “neural networks” that can recognize photos and respond to everyday human language. And similar work is driving speech recognition on Google’s Android phones, instant language translation on Microsoft’s Skype service, and so many other online tools that can “learn” over time. Using vast networks of computer processors, these systems approximate the networks of neurons inside the human brain, and in some ways, they can outperform humans themselves.
This week in the scientific journal Nature, LeCun—also a professor of computer science at New York University—details the current state of this “deep learning” technology in a paper penned alongside the two other academics most responsible for this movement: University of Toronto professor Geoff Hinton, who’s now at Google, and the University of Montreal’s Yoshua Bengio. The paper details the widespread progress of deep learning in recent years, showing the wider scientific community how this technology is reshaping our internet services—and how it will continue to reshape them in the years to come.
But as LeCun tells WIRED, deep learning will also extend beyond the internet, pushing into devices that can operate here in the physical world—things like robots and self-driving cars. Just last week, researchers at the University of California at Berkeley revealed a robotic system that uses deep learning tech to teach itself how to screw a cap onto a bottle. Early this year, big-name chip maker Nvidia and an Israeli company called Mobileye revealed that they were developing deep learning systems that can help power self-driving cars.
LeCun has been exploring similar types of “robotic perception” for over a decade, publishing his first paper on the subject in 2003. The idea was to use deep learning algorithms as a way for robots to identify and avoid obstacles as they moved through the world—something not unlike what’s needed with self-driving cars. “It’s now a very hot topic,” he says.
Yes, Google and some many others have already demonstrated self-driving cars. But according to researchers, including LeCun, deep learning can advance the state of the art—just as it has vastly improved technologies such as image recognition and speech recognition. Deep learning algorithms date back to the 1980s, but now that they can tap the enormously powerful network of machines available to today’s companies and researchers, they provide a viable way for systems to teach themselves tasks by analyzing enormous amounts of data.
“This is a chance for us to change the model of learning from very shallow, very confined statistics to something extremely open-ended,” Sebastian Thrun, who helped launched the Google self-driving car project, said of deep learning in an interview this past fall.
Thrun has left Google, but odds are, the company is already exploring the use of deep learning techniques with its autonomous cars (the first of which are set to hit the road this summer). According to Google research fellow Jeff Dean, the company is now using these techniques across dozens of services, and self-driving cars, which depend so heavily on image recognition, are one of the more obvious applications.
Trevor Darrell, one of the researchers working on deep learning robots at Berkeley, says his team is also exploring the use of the technology in autonomous automobiles. “From a researchers perspective, their are many commonalities in what it takes to move an arm to insert a peg into a hole and what it takes to navigate a car or a flying vehicle through an obstacle course,” he says.
Deep learning is particularly interesting, he says, because it has transformed so many different areas of research. In the past, he says, researchers used very separate techniques for speech recognition, image recognition, translation, and robotics. But now one this one set of techniques—though a rather broad set—can serve all these fields.
The result: all of these fields are suddenly evolving at a much faster rate. Face recognition has hit the mainstream. So has speech recognition. And the sort of autonomous machines his team is working on, Darrell says, could reach the commercial market within the next five years. AI is here. But it will soon arrive in a much bigger way.
Deep Learning
A machine-learning technique that has already given computers an eerie ability to recognize speech and categorize images is now creeping into industries ranging from computer security to stock trading. If the technique works in those areas, it could create new opportunities but also displace some workers.
Deep learning, as the technique is known, involves applying layers of calculations to data, such as sound or images, to recognize key features and similarities. It offers a powerful way for machines to recognize similarities that would normally be abstruse to a computer: the same face seen from different angles, for instance, or a word spoken in different accents (see “10 Breakthrough Technologies 2013: Deep Learning”). The mathematical principles that underlie deep learning are relatively simple, but when combined with huge quantities of training data and computer systems capable of powerful parallel computations, the technique has resulted in dramatic progress in recent years, especially in voice and image recognition.
For example, Google uses deep learning for voice recognition on Android phones, while Facebook uses the technology to identify friends in users’ photographs (see “Facebook Creates Software That Matches Faces Almost as Well as You Do”).
Other tech companies are following. At an event in Boston last week, two researchers from eBay described how the company is using deep learning to categorize products in images posted by sellers. By studying images that have already been tagged, the system can tell the difference, for example, between a pair of flip-flops and a pair of flats. This is helping to improve eBay’s search engine, especially for products that haven’t been tagged very well.
People in other fields and industries are starting to show an interest in deep learning. At the Boston event, researchers, engineers, and entrepreneurs discussed progress in the field and its potential application in advertising, finance, and medicine. One attendee who had previously applied machine learning techniques to hedge funds had founded a startup to use deep learning to predict market shifts like a sudden plunge in a currency’s value. Another attendee, from a major U.S. insurance company, was looking into using deep learning to identify fraudulent claims.
Andrew Ng, a leading figure in the field, and both an associate professor at Stanford and chief scientist at the Chinese company Baidu, said at the conference that deep learning has already proven useful. “One of the things Baidu did well early on was to create an internal deep learning platform,” Ng said. “An engineer in our systems group decided to apply it to decide a day in advance when a hard disk is about fail. We use deep learning to detect when there might’ve been an intrusion. Many people are now learning about deep learning and trying to apply it to so many problems.”
Deep learning is being tested by researchers to glean insights from medical imagery. Emmanuel Rios Velazquez, a postdoctoral researcher at the Dana-Farber Cancer Institute in Boston, is exploring whether deep learning could help to more accurately predict a patient’s outcome from images of his or her cancer.
Drug discovery is another promising area. Olexandr Isayev, a research scientist from the University of North Carolina at Chapel Hill, has shown that deep learning algorithms can help train computers to pick out potentially useful drug molecules from hundreds of millions of candidates. Isayev fed data from hundreds of thousands of experiments into his computer systems, and then had his system predict how a molecule might bind to a particular group of proteins. “A typical machine-learning algorithm does one objective function,” he said. “[With deep learning] you can do multiple optimizations. For example, you might want to maximize binding with this protein but minimize binding with some other protein.”
Deep learning doesn’t work best for everything, as Isayev’s work demonstrates. He says the improvements it offered for computerized drug discovery were modest compared to what it could do for computerized image recognition.
Even so, the potential for deep learning to be applied more broadly can be seen with the emergence of some well-funded startups. Palo Alto-based MetaMind, which has developed a deep learning platform, was founded by Richard Socher, who studied under Ng at Stanford. “We have people classifying fashion, cars, houses, satellite images, and each of these are already gigantic industries,” Socher says. “The beauty of deep learning is that, from the raw input to the final output, it’s all learned.”
-
Self Teaching Machines
At first, the video clip doesn’t look like a landmark in science. Given the task of teaching itself how to beat 49 classic Atari video games, the computer clearly has no idea what’s going on — but it doesn’t last. Within two hours, it plays those games like a fortysomething who hasn’t seen daylight since the early 1980s, announcing itself in the process as the first artificial intelligence system to teach itself disparate tasks from scratch.
There was a time when AI was the preserve of science fiction, of Isaac Asimov and I, Robot. Now it is big business. DeepMind Technologies, the British company that developed the system, was snapped up by Google for £300 million last year.
In October DeepMind said, in turn, that it was acquiring Vision Factory, a UK start-up working on image recognition, and Dark Blue Labs, another homegrown fledgeling doing similar work in speech recognition. It also launched a tie-up with the University of Oxford computer science department. And with these deals, Google has put itself squarely at the head of the field of “deep learning”, the AI process that many hope will lead to the industry’s holy grail — achieving human-like intelligence in machines. Given that Google’s ambitions in the sector are obvious, smaller companies with any intelligence of their own might be expected to give it a wide berth. Not so. With the big groups focused on the bigger and more distant goal of cracking the nut known as general AI, entrepreneurs are focusing on filling in the gaps, using the giant strides made already to tackle more immediate and niche challenges.
Magic Pony Technology is using this more narrow AI to improve video processing. The London-based start-up’s platform can improve the resolution of any image, or compress it to a smaller file size with no loss of quality, because it has seen enough similar images to guess intelligently the details that should be there — in the same way that a person could draw the rest of a chair if handed a picture that shows only the middle part. The company is aiming the technology at mobile brands serving emerging markets such as India and Africa, where bandwidth is not strong to handle the video-streaming demands of smartphone users.
“We’re a team of seven at the moment, largely part-time,” Rob Bishop, Magic Pony’s co-founder, says, “but we can do powerful things. With developments like Amazon web services, we can run our own supercomputer in the cloud from just a few thin laptops.” Magic Pony is about to close a seven-figure round of investment, which may sound fanciful to naysayers. After all, AI has been touted as the next big thing since the days when Atari represented the pinnacle of video games. However, the processing power was not there then: programmers had to teach computers rigid rules over how things worked, so AI stumbled.
“It’s like the old joke about the Daleks,” says Mike Lynch, the cofounder of Autonomy and former machine learning academic at Cambridge, who is investing in several young AI companies as the chief executive of Invoke Capital. “Their plans for world domination were fine until they hit a step. But while AI was at the level of a sea urchin a few years ago, it’s now around the level of a two-year-old.”
The unique fact about this particular two-year-old is that it can handle certain tasks — such as pattern-spotting, fact-checking or cross-referencing — far better than any adult. A computer also can work much more quickly and take on a far greater workload, all without a break.
Celaton, based in Milton Keynes, brands its inSTREAM platform as “the best knowledge worker you’ve ever hired”, able to make sense of the reams of unstructured information that bombard large organisations on a daily basis.
Celaton says that inSTREAM can recognise the nature of a customer complaint, for example, understand why it has happened and craft a personalised response with minimum human input. The company says that turnover will rise to £4.5 million in the next financial year.
Perhaps the most promising role for narrow AI may lie in the “internet of things”, the near-future network where devices from your watch to your fridge will communicate with each other on a continual basis. Invoke has invested in Neurence, an AI company developing a “big brain in the cloud”.
Not that the machines will be talking solely to each other. “The internet of things won’t be possible without a simple way to interact with all of these devices,” Vishal Chatrath, of VocalIQ, says. The Cambridgebased start-up has developed an alternative to Apple’s Siri that engages the user in conversation. The company is releasing a trial app next month.
The ambition does not end there. “One of our key projects is to develop a car that can talk to you, like in Knight Rider,” Mr Chatrath says. “That’s the level we’re targeting.”
If all this has been achieved by the equivalent of a two-year-old, imagine what AI could be doing by the time it starts school.
-
Fact Checking
Computers can now do fact-checking for any body of knowledge, according to Indiana University network scientists, writing in an open-access paper published June 17 in PLoS ONE.
Using factual information from summary infoboxes from Wikipedia* as a source, they built a “knowledge graph” with 3 million concepts and 23 million links between them. A link between two concepts in the graph can be read as a simple factual statement, such as “Socrates is a person” or “Paris is the capital of France.”
In the first use of this method, IU scientists created a simple computational fact-checker that assigns “truth scores” to statements concerning history, geography and entertainment, as well as random statements drawn from the text of Wikipedia. In multiple experiments, the automated system consistently matched the assessment of human fact-checkers in terms of the humans’ certitude about the accuracy of these statements.
Dealing with misinformation and disinformation
In what the IU scientists describe as an “automatic game of trivia,” the team applied their algorithm to answer simple questions related to geography, history, and entertainment, including statements that matched states or nations with their capitals, presidents with their spouses, and Oscar-winning film directors with the movie for which they won the Best Picture awards. The majority of tests returned highly accurate truth scores.
Lastly, the scientists used the algorithm to fact-check excerpts from the main text of Wikipedia, which were previously labeled by human fact-checkers as true or false, and found a positive correlation between the truth scores produced by the algorithm and the answers provided by the fact-checkers.
Significantly, the IU team found their computational method could even assess the truthfulness of statements about information not directly contained in the infoboxes. For example, the fact that Steve Tesich — the Serbian-American screenwriter of the classic Hoosier film “Breaking Away” — graduated from IU, despite the information not being specifically addressed in the infobox about him.
Using multiple sources to improve accuracy and richness of data
“The measurement of the truthfulness of statements appears to rely strongly on indirect connections, or ‘paths,’ between concepts,” said Giovanni Luca Ciampaglia, a postdoctoral fellow at the Center for Complex Networks and Systems Research in the IU Bloomington School of Informatics and Computing, who led the study.
“If we prevented our fact-checker from traversing multiple nodes on the graph, it performed poorly since it could not discover relevant indirect connections,” said Ciampaglia. “But because it’s free to explore beyond the information provided in one infobox, our method leverages the power of the full knowledge graph.
“These results are encouraging and exciting. We live in an age of information overload, including abundant misinformation, unsubstantiated rumors and conspiracy theories whose volume threatens to overwhelm journalists and the public. Our experiments point to methods to abstract the vital and complex human task of fact-checking into a network analysis problem, which is easy to solve computationally.”
Expanding the knowledge base
Although the experiments were conducted using Wikipedia, the IU team’s method does not assume any particular source of knowledge. The scientists aim to conduct additional experiments using knowledge graphs built from other sources of human knowledge, such as Freebase, the open-knowledge base built by Google, and note that multiple information sources could be used together to account for different belief systems.
The team added a significant amount of natural language processing research, but they note that additional work remains before these methods could be made available to the public as a software tool.
* The team selected Wikipedia as the information source for their experiment due to its breadth and open nature. Although Wikipedia is not 100 percent accurate, previous studies estimate the online encyclopedia is nearly as reliable as traditional encyclopedias, but also covers many more subjects, the researchers note.
IQ Tests
Computers have never been good at answering the type of verbal reasoning questions found in IQ tests. Now a deep learning machine unveiled in China is changing that.
Just over 100 years ago, the German psychologist William Stern introduced the intelligence quotient test as a way of evaluating human intelligence. Since then, IQ tests have become a standard feature of modern life and are used to determine children’s suitability for schools and adults’ ability to perform jobs.
These tests usually contain three categories of questions: logic questions such as patterns in sequences of images, mathematical questions such as finding patterns in sequences of numbers and verbal reasoning questions, which are based around analogies, classifications, as well as synonyms and antonyms.
It is this last category that has interested Huazheng Wang and pals at the University of Science and Technology of China and Bin Gao and buddies at Microsoft Research in Beijing. Computers have never been good at these. Pose a verbal reasoning question to a natural language processing machine and its performance will be poor, much worse than the average human ability.
Today, that changes thanks to Huazheng and pals who have built a deep learning machine that outperforms the average human ability to answer verbal reasoning questions for the first time.
In recent, years, computer scientists have used data mining techniques to analyze huge corpuses of texts to find the links between words they contain. In particular, this gives them a handle on the statistics of word patterns, such as how often a particular word appears near other words. From this it is possible to work out how words relate to each other, albeit in a huge parameter space.
The end result is that words can be thought of as vectors in this high-dimensional parameter space. the advantage is that they can then be treated mathematically: compared, added, subtracted like other vectors. This leads to vector relations like this one: king – man + woman = queen.
This approach has been hugely successful. Google uses it for automatic language translation by assuming that word sequences in different language represented by similar vectors are equivalent in meaning. So they are translations of each other.
But this approach has a well-known shortcoming: it assumes that each word has a single meaning represented by a single vector. Not only is that often not the case, verbal tests tend to focus on words with more than one meaning as a way of making questions harder.
Huazheng and pals tackle this by taking each word and looking for other words that often appear nearby in a large corpus of text. They then use an algorithm to see how these words are clustered. The final step is to look up the different meanings of a word in a dictionary and then to match the clusters to each meaning.
This can be done automatically because the dictionary definition includes sample sentences in which the word is used in each different way. So by calculating the vector representation of these sentences and comparing them to the vector representation in each cluster, it is possible to match them.
The overall result is a way of recognizing the multiple different senses that some words can have.
Huazheng and pals have another trick up their sleeve to make it easier for a computer to answer verbal reasoning questions. This comes about because these questions fall into several categories that require slightly different approaches to solve.
So their idea is to start by identifying the category of each question so that the computer then knows which answering strategy it should employ. This is straightforward since the questions in each category have similar structures.
So questions that involve analogies are like these:
Isotherm is to temperature as isobar is to? (i) atmosphere, (ii) wind, (iii) pressure, (iv) latitude, (v) current.
and
Identify two words (one from each set of brackets) that form a connection (analogy) when paired with the words in capitals: CHAPTER (book, verse, read), ACT (stage, audience, play).
Word classification questions are like this:
Which is the odd one out? (i) calm, (ii) quiet, (iii) relaxed, (iv) serene, (v) unruffled.
And questions looking for synonyms and antonyms are like these:
Which word is closest to IRRATIONAL? (i) intransigent, (ii) irredeemable, (iii) unsafe, (iv) lost, (v) nonsensical.
And
Which word is most opposite to MUSICAL? (i) discordant, (ii) loud, (iii) lyrical, (iv) verbal, (v) euphonious.
Spotting each type of question is relatively straightforward for a machine learning algorithm, given enough to examples to learn from. And this is exactly how Huazheng and co do it.
Having identified the type of question, Huazheng and buddies then devise an algorithm for solving each one using the standard vector methods but also the multi-sense upgrade they’ve developed.
They compare this deep learning technique with other algorithmic approaches to verbal reasoning tests and also with the ability of humans to do it. For this, they posed the questions to 200 humans gathered via Amazon’s Mechanical Turk crowdsourcing facility along with basic information about their ages and educational background.
And the results are impressive. “To our surprise, the average performance of human beings is a little lower than that of our proposed method,” they say.
Human performance on these tests tends to correlate with educational background. So people with a high school education tend to do least well, while those with a bachelor’s degree do better and those with a doctorate perform best. “Our model can reach the intelligence level between the people with the bachelor degrees and those with the master degrees,” say Huazheng and co.
That’s fascinating work that reveals the potential of deep learning techniques. Huazheng and co are clearly optimistic about future developments. “With appropriate uses of the deep learning technologies, we could be a further step closer to the true human intelligence.”
Deep learning techniques are currently sweeping through computer science like wildfire and the revolution they are creating is still in its early stages. There’s no telling where this revolution will take us but one thing is for sure: William Stern would be amazed.
Algorthms Taking Over
Digital technology is advancing so fast that computers could soon solve every problem, writes James Dean
By the time we celebrate New Year’s Eve 2025, your smartphone will understand your slurred requests for a midnight taxi, the car that collects you will be driverless and, when it drops you off, your sudden craving for a pre-bed snack will be satisfied by drone delivery.
“Machine learning” is developing so rapidly that experts have told The Times that the next decade will bring a revolution in computers and robotics, leading to intelligent machines and huge technological and social changes.
The Royal Society, Britain’s leading scientific institution, has set up a working party to investigate machine learning, whereby a computer takes in the data produced every day by the planet’s people and internet-connected objects to deduce patterns and, in effect, becomes intelligent.
Experts said that by 2025 electronic personal assistants such as Apple’s Siri will be able to communicate with us as if they were humans, and to anticipate our behaviour and daily priorities.
Internet companies such as Facebook already use algorithms to predict what we will like, for instance in deciding what will appear in our newsfeed or as adverts. As these pick up more data, the distinction between what humans consider intelligent and what computers are capable of will narrow.
“They’re constantly learning and, increasingly, they have perceptive capabilities,” Zoubin Ghahramani, a professor of information engineering at Cambridge University, said. “Machinelearning systems aren’t programmed to be fixed, but to improve and grow themselves in the same way that biological systems learn to improve.”
The consequences will be profound. Robots will be able to watch surgeons and learn how to copy their movements. Nursing-home staff will be assisted by robots that also can keep people company. Drones will take on delivery roles. Peter Donnelly, professor of statistical science at Oxford University, said: “Almost everything on our smartphones has machine learning at its core. In ten years there will be a whole lot of complex apps, central to our lives, that we can’t even conceive of now.” The secrets of human life are quietly being transferred to the digital brains of machines. Every second, they learn more about our foibles by digesting the vast amounts of personal information we put online.
Social networks, smartphone apps and countless internet-connected services are helping machines to learn about us so quickly that by 2025 we will be able to talk to artificially intelligent personal assistants as if they were humans.
The evolution is being catalysed by a process known as “machine learning”, whereby a computer is fed huge amounts of data from which it is able to draw its own meanings. Machine learning will be central to spurring advances in robotics for hospitals, home care and transport over the next decade, according to experts polled by The Times.
Such is the perceived importance of the field that last month the Royal Society created a machine-learning working group, comprising several leading thinkers, to explore its likely benefits and pitfalls.
“Twenty years ago, a computer programmer would have to work out a problem and then type out the code that allowed the computer to solve it,” said Peter Donnelly, chairman of the working group and professor of statistical science at the University of Oxford. “With machine learning, they programme the steps that allow the computer to learn the solution to the problem. You can’t programme a computer in a car and tell it what to do in every circumstance, because you can’t possibly conceive of them all.”
Professor Donnelly said that the expansion of the internet and the increased power of computers had revolutionised the field since the first forays were made about 20 years ago, and that ten years from now we would be using and relying on complex apps beyond our current imagination.
The power of machine learning is expected to accelerate rapidly in the coming years as computers digest more information about us from the internet, and the algorithms that power them become more advanced. This will help to create powerful virtual assistants that are able to pre-empt the needs of their human masters. Zoubin Ghahramani, a professor of information engineering at the University of Cambridge, said: “A lot of these algorithms are already affecting our lives, but most people don’t think about the fact that they are all forms of robot intelligence. “Every aspect of your activity on Facebook is controlled by this intelligence — what appears in your news feed, what adverts it shows you.”
Professor Ghahramani predicted that smartphone assistants would be “superintelligent” in ten years’ time. “We can already talk to them in a primitive way but we can’t have conversations with them,” he said. “By 2025 we will, and they’ll be much more natural. If you were to allow it, the assistant would know things about you — your priorities, what it should remind you about, clever recommendations it could make.”
There will be limits to the progress, however. Paul Newman, principal investigator at the University of Oxford’s mobile robotics group, said: “We’re not going to have something that helps us around the house. I don’t see the generalist robot coming for a long time. “Intelligent robots will first appear in places where they have a clearly defined role — that’s why there are already so many robots on car production lines. We have evolved extraordinary capabilities as humans, but look at how long it took us to evolve. I think the idea of a humanoid robot is jaundiced by science fiction, and at the moment, it is just that.”
Change Index page
-
Fact Checking
William Flew Auckland