常時起動型Ray-Ban MetaグラスがOpenClawで日常業務を高速化する新研究

A research team developed an OpenClaw agent for smart glasses to find out how continuously perceiving AI changes the way people use agentic AI systems.

Researchers from the University of Colorado, the Gwangju Institute of Science and Technology, and Google have introduced VisionClaw, an always-on agentic AI that pairs continuous first-person perception with the autonomous execution of digital tasks.

The team set out to bridge the gap between digital and real life: AI agents can run software and handle tasks on the web, but they have no window into the physical world. Smart glasses, on the other hand, capture their surroundings through cameras and microphones but can barely act on their own.

With VisionClaw, the researchers wanted to find out whether an always-on AI holds up in everyday life and how real-world interactions shift when perception and action live inside a single system.

Image: Xiaoan Liu et al.

How VisionClaw works

VisionClaw connects a displayless Ray-Ban Meta to Gemini Live and OpenClaw through a custom smartphone app. The glasses continuously stream audio and individual frames from the user's surroundings to Gemini, which processes the multimodal input and either replies directly by voice or kicks off tasks through OpenClaw. The agent taps into tools like a browser, email, calendar, or web search, then feeds the results back to the language model. The setup ties continuous first-person perception to agentic execution of digital tasks.

Image: Xiaoan Liu et al.

The researchers ran two studies to see how well VisionClaw holds up in practice and how people actually use a system like this.

In the first study, they compared VisionClaw against two stripped-down systems with 12 participants: an always-on AI running on the Ray-Ban Meta that perceives the environment but can't perform general agent actions, and a smartphone version of OpenClaw that handles agentic tasks but has no continuous awareness of the surroundings. Participants worked through four tasks involving real objects or physical documents, such as taking notes from paperwork, composing emails, researching products, or controlling devices.

Faster results with less effort

According to the paper, VisionClaw completed tasks 13 to 37 percent faster depending on the task, and users rated it 7 to 46 percent less demanding. Mental effort, time pressure, and frustration all dropped. Success rates were statistically similar overall, but VisionClaw fell to around 58 percent on the note-taking task because the glasses' camera couldn't reliably capture small or visually challenging objects like receipts.

"Results show that integrating perception and execution enables faster task completion and reduces interaction overhead compared to non-always-on and non-agent baselines", the researchers write.

Image: Xiaoan Liu et al.

In a second, autobiographical field study, the researchers looked at how VisionClaw performs in daily use. Four of the paper's authors used the system themselves over an extended period, logging 55 active participant days. During that time, they generated 555 voice-initiated interactions totaling 25.8 hours of use. The researchers analyzed what people actually used VisionClaw for and identified six usage categories: information retrieval (30 percent), shopping (19 percent), saving content (16 percent), communication (14 percent), remembering (12 percent), and control (9 percent).

Beyond those categories, the field study surfaced four emergent interaction patterns: open-ended, multi-step conversations with the AI agent; spontaneous capture and later recall of information; more unobtrusive but sometimes less reliable screenless AI use; and growing usefulness over time as the system accumulated personal data. Taken together, the paper argues, this points to a shift from isolated voice commands toward continuous, context-driven use.

Image: Xiaoan Liu et al.

"Beyond performance gains, deployment findings reveal a shift in interaction: tasks are initiated opportunistically during ongoing activities, and execution is increasingly delegated rather than manually controlled. These results suggest a new paradigm for wearable AI agents, where perception and action are continuously coupled to support situated, hands-free interaction", the researchers write.

VisionClaw: Open source on GitHub

The authors argue that VisionClaw points beyond individual use cases toward a new kind of human-AI interaction. Rather than responding to one-off commands like a traditional voice assistant, an always-on system acts more like a continuous, context-aware companion, with perception, memory, and action all working in concert. They also flag open challenges: privacy risks from constant recording, the handling of large volumes of personal data, and the need to design systems that stay unobtrusive in the background.

On the technical side, it's worth noting that the researchers used a Ray-Ban Meta without a display, even though Meta already sells a version with a built-in display in the US. A display could meaningfully expand and simplify AI use by surfacing results directly in the user's field of view, making them easier to verify at a glance.

Methodologically, the small sample sizes limit what we can take away: the first study included only 12 participants, and the second just four. The bigger problem is that the field study was conducted entirely by four of the paper's authors: people who built the system and know exactly how it works. Google researchers were also involved, and Google has said it plans to launch AI glasses based on Android XR and Gemini later this year. With that in mind, the study shouldn't be read as a fully unbiased evaluation.

The paper "VisionClaw: Always-On AI Agents Through Smart Glasses" is freely available online, and VisionClaw itself is open source on GitHub.