実践におけるOpenEnv：現実世界環境でのツール利用エージェントの評価

OpenEnv in Practice: Evaluating Tool-Using Agents in Real-World Environments Back to Articles OpenEnv in Practice: Evaluating Tool-Using Agents in Real-World Environments

AI agents often perform impressively in controlled research settings, yet struggle when deployed in real-world systems where they must reason across multiple steps, interact with real tools and APIs, operate under partial information, and recover from errors in stateful, permissioned environments—highlighting a persistent gap between research success and production reliability.

OpenEnv is an open-source framework from Meta and Hugging Face designed to address this challenge by standardizing how agents interact with real environments. As part of this collaboration, Turing contributed a production-grade calendar management environment to study tool-using agents under realistic constraints such as access control, temporal reasoning, and multi-agent coordination.

In this post, we explore how OpenEnv works in practice, why calendars serve as a powerful benchmark for real-world agent evaluation, and what our findings reveal about the current limitations of tool-using agents.

OpenEnv is a framework for evaluating AI agents against real systems rather than simulations. It provides a standardized way to connect agents to real tools and workflows while preserving the structure needed for consistent and reliable evaluation.

OpenEnv uses a gym-oriented API (reset, step, action, observations) like OpenAI's Gymnasium. Also, OpenEnv uses a standard MCP tool call interface to connect to envs which provides a consistent interface across domains and simulation to production environments.

The environments maintain state across multiple actions—enabling long-horizon reasoning—and can connect directly to real APIs and tools such as browsers, code repositories, or calendars. This shifts evaluation from "Can this work in a controlled demo?" to "Can this operate reliably in the real world?"

The Calendar Gym: A Production-Grade Benchmark

Calendar systems are deceptively complex. While scheduling a meeting seems simple, real-world calendar management requires agents to reason over time, permissions, multiple users, and incomplete information—often across several dependent steps. These properties make calendars a powerful testbed for evaluating tool-using agents outside controlled simulations.

To ground OpenEnv in this kind of realistic, demanding use case, Turing built a production-grade calendar management environment referred to as the Calendar Gym. Rather than simulating scheduling in the abstract, it exposes agents to the same constraints they would face in real calendar systems: Access Control Lists across users and calendars, limited visibility into other users' state, and multi-step workflows where actions must be chained in the correct order. Agents interact with a rich set of calendar operations—from listing calendars to modifying events and permissions—and must handle failed actions, incorrect assumptions, and missing permissions. Each session runs in an isolated environment, enabling reliable comparisons across runs.

Below is a code example of how to use the Calendar Gym. We explore the environment, discover available tools, list calendars, create an event, and print the result.

from openenv_wrapper.client import MCPEnvClient from openenv_wrapper.data_models import MCPAction with MCPEnvClient.from_hub(base_url="TuringEnterprises/calendar-gym") as client: # Connect and reset the environment result = client.reset() print("Reset successful:", result.observation.success) # Discover available tools result = client.step(MCPAction(action_type="ListToolsAction")) print("Available tools:", len(result.observation.tools_list)) # List calendars result = client.step(MCPAction( action_type="ToolCallAction", tool_name="calendars_list", arguments={} )) calendars = result.observation.tool_result["items"] print("Calendars:", calendars) # Create an event result = client.step(MCPAction( action_type="ToolCallAction", tool_name="events_insert", arguments={ "calendarId": "primary", "summary": "Team Sync", "start": {"dateTime": "2026-01-15T14:00:00Z"}, "end": {"dateTime": "2026-01-15T15:00:00Z"} } )) print("Event created:", result.observation.success) Below is an excerpt of what the Calendar Gym returns when you call ListToolsAction. Each entry includes the tool name plus an input schema (what arguments the tool accepts).

{ "tools_list": [ { "name": "calendars_list", "description": "List calendars visible to the current user.", "input_schema": { "type": "object", "properties": {}, "additionalProperties": false } }, { "name": "events_insert", "description": "Create an event in a calendar.", "input_schema": { "type": "object", "properties": { "calendarId": { "type": "string" }, "summary": { "type": "string" }, "start": { "type": "object", "properties": { "dateTime": { "type": "string" } }, "required": ["dateTime"] }, "end": { "type": "object", "properties": { "dateTim