# Don't Approve the AI Model. Approve the App - Doneyli De Jesus, Langfuse **URL:** \ **Category:** Content Creators\ **Tags:** machine-learning, ai-engineer, safety, benchmarking\ **Created:** [4 November 2026 10:00 UTC](https://www.artofsm.art/t/dont-approve-the-ai-model-approve-the-app-doneyli-de-jesus-langfuse/27435 "2026-11-04T10:00:35Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![artesia](https://www.artofsm.art/user_avatar/www.artofsm.art/artesia/32/36_2.png) [@artesia](https://www.artofsm.art/u/artesia)\ **Post date:** [4 November 2026 10:00 UTC](https://www.artofsm.art/t/dont-approve-the-ai-model-approve-the-app-doneyli-de-jesus-langfuse/27435/1 "2026-11-04T10:00:35Z")
[![](https://www.artofsm.art/uploads/default/original/3X/0/c/0cb3c212d5485074d126f507dc7db10d8a305410.jpeg "Don't Approve the AI Model. Approve the App - Doneyli De Jesus, Langfuse") ](https://www.youtube.com/watch?v=-uBABM2AuFg) Doneyli De Jesus from Langfuse explains that in regulated industries like banking, AI deployment focuses on certifying applications rather than models by using tailored evaluation frameworks with proprietary data, strict compliance measures, and human expert involvement to ensure performance and safety. This approach enables banks to maintain on-premises control, meet regulatory requirements, and accelerate AI evaluation, with Langfuse providing open-source tools to support these processes. ---
**Author:** ![artesia](https://www.artofsm.art/user_avatar/www.artofsm.art/artesia/32/36_2.png) [@artesia](https://www.artofsm.art/u/artesia)\ **Post date:** [4 November 2026 10:24 UTC](https://www.artofsm.art/t/dont-approve-the-ai-model-approve-the-app-doneyli-de-jesus-langfuse/27435/3 "2026-11-04T10:24:42Z")
Doneyli De Jesus from Langfuse by ClickHouse shares insights from working with a top-five bank to implement safety guardrails for deploying AI applications in regulated industries, particularly financial services. She emphasizes that these industries prioritize application performance over public model benchmarks, as internal data and use cases are highly sensitive and proprietary. The bank’s focus was on certifying AI applications, such as a 10K analyst tool for investment bankers, rather than the underlying models themselves. This approach ensures that AI solutions align with the firm’s unique context and operational requirements. In regulated sectors like banking, three critical requirements shape AI deployment: maintaining on-premises infrastructure, ensuring data and model access are strictly gated without internet exposure, and deploying solutions in air-gapped environments to protect sensitive information. Many banks are revisiting their own data centers and investing in local GPU infrastructure to meet these demands. Additionally, some banks have internal research teams that develop and fine-tune their own models, underscoring the need for strict control and compliance in AI workflows. To address these challenges, Langfuse helped the bank develop a robust evaluation framework centered around a “golden data set” that serves as a benchmark for testing AI applications. This framework operationalizes test cases similar to software QA processes, generating traces and scores that provide evidence of model performance across multiple dimensions such as accuracy, regulatory compliance, and completeness. By integrating tools to handle specific weaknesses—like using deterministic Python code to improve numerical accuracy—the bank was able to raise model performance from 60% to over 90%, ensuring the application met strict quality gates before production deployment. The evaluation process combines deterministic scoring, path validation, and LLM-based qualitative judgments to comprehensively assess AI agent outputs. This multi-dimensional scoring system allows the bank to set customized thresholds for different use cases, with some dimensions like regulatory compliance acting as binary gates that can block deployment if not met. The framework also incorporates human domain experts who review failure cases through an annotation queue, providing feedback that continuously refines the golden data set and improves the AI application iteratively. In conclusion, Doneyli highlights three key takeaways for deploying AI in regulated industries: build your own benchmarks tailored to your data and use cases, approve AI applications rather than models to maintain flexibility, and involve domain experts in setting quality thresholds. This approach enables banks to keep sensitive data on-premises, maintain strict compliance, and accelerate evaluation cycles from weeks to minutes. Langfuse provides open-source tools and detailed documentation to help organizations reproduce this methodology and safely operationalize AI solutions in complex, regulated environments. ## Useful Links - [Langfuse GitHub Repository and Blog Post](https://langfuse.com/) — Contains source code and detailed methodology to reproduce the evaluation framework described in the video.