With competitors such as Google evolving their AI models, Anthropic is adding a key feature to its Claude Sonnet model that allows it to interact with computers. The model is now capable of following instructions it’s provided with to “move a cursor around their computer’s screen, click on relevant locations, and input information via a virtual keyboard,” just as a human user would.
Anthropic believes having Claude use a computer is an important step for the future of AI because it opens the door to general computer skills that translate to a variety of applications. “Enabling AIs to interact directly with computer software in the same way people do will unlock a huge range of applications that simply aren’t possible for the current generation of AI assistants,” said the company.
However, this new feature is very much rough around the edges and currently performs like a beta release. Several actions that “people perform effortlessly—scrolling, dragging, zooming—currently present challenges for Claude and we encourage developers to begin exploration with low-risk tasks.” The company says it expects to rapidly improve performance as it receives feedback from users.
This will be a welcomed feature for users who have tasks that can be easily automated, with the bonus of being able to do so without needing to do any kind of specific model training. For example, software developers will be able to use Claude to improve the quality assurance process by having the AI repetitively test applications to try and find any bugs.
While Anthropic isn’t the first to implement this feature, having it jump into the fray will undoubtedly push other companies in the AI space to continue to improve their own offerings. Time will tell how effective and useful these general computer skills will be, and how much employment displacement humans will see.