From Benchmarks to Reality: Lessons from Deployed AI Systems

Putting AI models into production is rarely straightforward. Even when a model performs well in controlled settings, real-world deployment introduces a different set of challenges: distribution shift, unpredictable user inputs, latency and cost constraints, and the need to meet user expectations while enforcing safety and moderation.

In this talk, we’ll walk through two case studies: a feature in the Remini app that transforms portraits into a user’s “best self”, and the system powering Evernote’s AI Assistant. In both cases, the core modeling approach is relatively simple to describe — but turning these ideas into real-world systems took months of iteration and exposed a range of issues that are often underexplored in research.

We’ll focus on what actually breaks when models meet real users, the trade-offs between model quality, robustness, and system constraints, and the techniques we used to make these systems hold up in practice.

ANDREA FERRETTI

Andrea is an AI Research Engineer at Bending Spoons, with a background in mathematics and a long-standing passion for programming. He works on language models for Evernote and previously on computer vision systems for the Remini app.

He was previously an early member of UniCredit’s R&D department in Milan, working on applied machine learning in production.

Earlier in his career, he worked in pure mathematics at the Max Planck Institute in Bonn and the University of Lille. He holds a PhD in Algebraic Geometry from the University of Rome.

Joining us in person offers the opportunity for direct networking and interaction with both speakers and participants.
Please complete the form below to reserve your seat.