How can learning agents 'learn to use the web'?

One of the most compelling individual examples of progress with machine learning (ML) and artificial intelligence (AI) involves digital “learning agents” that work on ML algorithms to actually navigate the web, and to use specific in-page functionality in much the same ways that humans do.

Through the power of more sophisticated machine learning processes, computers have become able to “see” images and decipher what they mean. Engineers have been able to program AI technologies with an astonishing level of granularity – in the sense that computers can now “read” text off of a visual page with a high degree of literacy. This requires a remarkable amount of resources – to use raw pixel inputs to let the technology perceive shapes of letters, numbers and text characters – and then to use natural language processing to string those characters together, and come up with commands and responses.

However, another of the primary pathways to learning agent improvement is iteration. The programs are essentially “trained” to do the right thing from a human perspective, and refine their capabilities according to training sets.

An excellent example of all of this progress can be found at OpenAI's “Mini World of Bits” page that talks about reinforcement learning agents that perceive sets of raw pixels in a small webpage and can “produce keyboard and mouse actions.”

Web users can see the technologies outputting those keyboard and mouse events with user-like movements on small webpages: to operate drop-down list boxes, check boxes with logic, respond to text inputs, choose colors, and much more. OpenAI states that “one may utilize an unrestricted amount of pretraining on the training environments.”

All of this shows that artificial intelligence and machine learning are progressing rapidly, and that this will require human responses to keep up. The types of rote technology built into webpages to prove that a user is “not a robot” may have to be significantly upgraded in order to be effective as artificial intelligence essentially escapes some of the pens we've created for it. At the same time, there is an exciting set of applications for AI agents being able to use the web in a meaningful way – for a while now, people have talked about using artificial intelligence to improve recommendation engines, or go surf the web for results. Now, these same artificial intelligence agents could be used to work with controls on the web as well.