Credit GenieがInsights Agentを活用し、AI財務アシスタントを改善した方法

*This is a guest post by David Li, AI Engineer @ Credit Genie, Jeffrey Ngai, Staff AI Engineer @ Credit Genie, Goyo Lozano Palacio, Applied AI Engineer @ Credit Genie, and Charles Yuan, AI Engineer @ Credit Genie.*

Adding an AI financial assistant to the Credit Genie mobile app

At Credit Genie, we strive to help our customers develop a deeper understanding of their finances. To accomplish this, we used LangGraph to build AskGenie - an AI financial assistant integrated with our mobile app.

Building and launching AskGenie was only half of the challenge. After its launch, we had to find a way to understand how users engage with it by analyzing the corresponding LangSmith traces.

In this post, we discuss how LangSmith Insights Agent helped us answer this question and how these learnings led us to extend AskGenie’s capabilities.

Setting up Insights Agent to analyze customer interactions at scale

We configured Insights Agent to run under two configurations:

• a V1 configuration that uses a custom prompt and a set of four predefined categories
• a V2 configuration that uses the same prompt of V1 while leaving the categories unspecified.

Insights Agent has the ability to automatically generate categories from the analysis of the LangSmith traces, and we were interested to observe whether different insights would emerge from an unconstrained analysis.

Both configurations use Anthropic’s Sonnet 4.5 as the thinking model and Anthropic’s Haiku 4.5 as the summarization model.

Detecting gaps in functionality from Insight Agent’s report

Interestingly, the unconstrained V2 configuration of Insights Agent also produced four categories. These autogenerated categories gave us a more nuanced understanding of the user interactions with AskGenie.

Predefined Categories (V1)

%

Autogenerated Categories (V2)

%

Customer Support

36%

Repayment Management

33%

Financial Insights

30%

Financial Insights

30%

Product & Features

25%

Advance Acquisition

27%

Policy & Compliance

9%

Access Barriers

10%

While we expected that customers would interact with AskGenie to explore their finances (as indicated by the “Financial Insights” cluster), the results also highlighted a high volume of customer support questions. This was not something we anticipated, especially considering that the Credit Genie mobile app already offers a separate customer support chatbot.

A deeper dive into these clusters indicated that some customers reach out to AskGenie to change the repayment date of the cash advances that they received from Credit Genie. In fact, the results from the V1 configuration surfaced “Repayment Date Modifications” as a relevant subcluster and the results from the V2 configuration included a similar “Repayment Date Modification Requests” subcluster.

Additionally, we learned that users regularly ask to check the status of their cash advances and their balance (as suggested for example by the “Advance Status and Tracking” and “Balance and Fee Questions” subclusters).

These insights into real user behavior highlighted some gaps in AskGenie’s capabilities: the agent should 1) provide our customers information about their cash advances and 2) give them the ability to update repayment dates.

Closing the gaps

Based on what we learned from Insights Agent, we improved AskGenie by adding a tool to retrieve information about the approved cash advance amount for the user, the expected delivery date of the cash advance, the scheduled repayment date, and the user’s cash advance history.

We also extended AskGenie in a way that allows our customers to modify the repayment date or apply for a new cash advance directly from the chat interface.

Finally, instead of re-routing users to a separate support bot,we introduced an in-chat pop-up window that allows customers to submit support requests without leaving AskGenie.

Making Insights Agent part of Credit Genie’s AI development process

At Credit Genie, Insights Agent has become an integral part of how we interpret feedback from our customers at scale, communicate this feedback to our Product team, and iterate on our AI products.

For AskGenie, we now run Insights Agent on a weekly schedule to observe shifts or emerging trends in user behavior and react accordingly. This helps us with faster cycles of product ideation, experimentation, and refinement.

About Credit Genie

Credit Genie is a mobile-first financial wellness platform designed to help individuals take control of their financial future. We leverage AI to provide personalized financial insights, and we are building an ecosystem of tools and services that provides instant access to cash and helps our customers build their credit. Our goal is to empower every customer to achieve long-term financial stability.

David Li is an AI Engineer at Credit Genie developing cutting-edge agentic AI systems that deliver real-time financial intelligence and seamless customer experiences. In his free time, you can find him exploring new restaurants, editing videos, or on the basketball court.

Jeffrey Ngai is a Staff AI Engineer at Credit Genie, building and scaling AI-driven systems that improve financial access and user experience while delivering high-impact, production-ready solutions that bridge product and infrastructure. In his free time, Jeffrey enjoys traveling, exploring new cocktail spots with friends, and diving into new skills and hobbies.

Goyo Lozano Palacio is an Applied AI Engineer at Credit Genie, designing evaluation systems for consumer-facing financial AI agents and fostering AI adoption inside the company. Outside of work, Goyo is exploring new cities, tracking down great food, and experimenting with new cooking recipes at home. He’s most engaged when he’s building something or trying activities he’s never done before, like salsa dancing.

‍Charles Yuan is an AI Engineer at Credit Genie, where he builds and scales consumer-facing AI systems, including the company’s flagship AskGenie product. Charles is particularly interested in turning early-stage AI prototypes into robust, user-facing systems that operate at scale. Outside of work, Charles enjoys reading, watching movies, boxing, and playing Pokemon.