Netflix、因果推論の自動化ワークフローをオープンソース化

Netflix open-sourced an agentic workflow for Observational Causal Inference (OCI) that reduces toil in causal analysis. Given observational data and the human user's analysis plan, the agent uses an actor-critic loop to estimate causality, write a report, and suggest next steps.

The agent workflow builds on existing OCI tools at Netflix. The goal is to automate error-prone or repetitive tasks such as sensitivity analysis or tracking multiple iterations, leaving higher-level tasks such as framing questions and evaluating results to the human user. OCI analysis is framed as target trial emulation; that is, as finding the optimal A/B test for answering the question. The Netflix team evaluated their workflow on the Atlantic Causal Inference Conference (ACIC) competition dataset and found that it was "competitive" against benchmark systems. According to Netflix,

Using agents for causal inference poses a challenge: how do we evaluate agents’ performance on tasks without ground truth? To meet this challenge, our workflow combines process audits with human oversight. To enable others to learn from and critique our workflow, we have open-sourced a lightweight, standalone version. We hope this work stimulates more research and development on agentic evaluation in the absence of ground truth.

The human analyst sets up the process by creating a templated Jupyter notebook and an analysis plan. The actor agent uses the plan to produce a spec, fill in the parameters of the notebook, and execute it. The critic reviews the output of the notebook; gives it a rating: not_satisfactory, satisfactory_with_caveats, or fully_satisfactory; and recommends spec changes.

OCI Agent Workflow - Image Source: Netflix Blog

To show the system in action, Netflix published a case study of using the workflow to estimate the impact on retention of new entertainment types (such as games). The analysis plan specified the "treatment" as the number of days engaging with the new type and the "outcome" as 2-month retention. For a baseline, they simply fed this plan to a Claude model, which decided to do a simple linear regression to produce an estimated effect.

When the research team used the oci-agent workflow, they produced an estimated effect that was "just 25% of the baseline." The critic agent flagged several issues, including potential early adopter bias and a failed placebo test. The workflow provides agents with playbooks for these and other scenarios, and it automates conducting multiple analyses using "tweaked" parameters.

Fabio Piazza, a senior product manager at Owkin, wrote on LinkedIn:

Netflix continues to quietly set the bar on applied AI. Rather than only checking the agent's output, they make every step transparent. Agents publish plans, specs, plots, and notebooks that humans can inspect and re-execute. They pair these "process audits" with human oversight, and split the work across two agents, one to run the analysis, one to critique it and surface gaps. A great reminder that the frontier isn't just better models, it's better workflows around them.

Taikai Takeda, a director of software engineering at Indeed.com, posted on X:

I'm not very knowledgeable about causal inference, so if I just casually throw it at an LLM, it might do some half-baked regression analysis and I'd end up satisfied with that....[OCI is] designed not so much for the agent to spit out answers, but to guide through the correct steps while leaving artifacts that experts can verify later. It includes experiments on public datasets and case studies, making the article easy to understand. AI agents are really good at lowering the entry barrier for specialized tasks, aren't they.

The oci-agent source code is available on GitHub.

About the Author

Anthony Alford

Anthony is a Senior Director, Development at Genesys where he is working on several AI and ML projects related to customer experience. He has over 20 years experience in designing and building scalable software. Anthony holds a Ph.D. degree in Electrical Engineering with specialization in Intelligent Robotics Software and has worked on various problems in the areas of human-AI interaction and predictive analytics for SaaS business optimization.

Show moreShow less