AI Nutrition Coach

The AI Nutrition Coach was an early attempt at creating an agentic personal nutrition coach to help me lose weight and improve my nutrition

Problem

I've been a long-time quantified selfer, and nutrition tracking and improvement has always been really difficult. Keeping tracking of what you've eaten is time-consuming and error-prone, and even when you have a lot of the data it's not clear what to do with that. If you knew already, you probably wouldn't need a coach.

Solution

LLM-powered agents offer the ability to not only facilitate tracking using natural language inputs (or photos), but also the ability to give high quality, personalized behavioral recommendations to help people achieve their nutrition goals.

Technical Architecture

• Next.js and React for client application
• LlamaIndex for memory and conversation management
• Python-based state-machine framework for agentic behior
• Postgres for data persistence
• Telegram for chat support
• GPT-4 for AI capabilities

Implementation Details

The most intersting part of the implementation of the was the hand-rolled agent framework. "Framework" is maybe strong, but it worked by defining a statemachine that would encode ensure that the model took the correct action based on what kind of work it was doing. So if the user was logging a meal but gave incomplete information, it wouldn't call the tool to look up nutrition data or give feedback. The key workflows within the app were onboarding, meal logging, and daily / weekly check-ins. Each workflow had a set of prompts and tools associated with it, and coordiation was managed by the models and the state machine.

Challenges & Leanrings

• Wrangling the agentic behavior before frameworks like LangGraph was a challenge. As the app grew in complexity it became more and more difficult to get reliable behavior, which prompted the addition of the state machine.
• Nutrition is a great use-case for coaching applications because eating is a behavior that people engage in regularly. This gives you a lot of "at bats" and a quick feedback cycle to test up prompt and feature updates
• People's behaviors are dependant on other people. I was living with my partner at the time and the quality of the recommendations was a function not just of my preferences but hers too. Good coaching products need to take that into account.
• Behavior change happens when you avoid the long tail of failure modes. It was surprising to see when I would and wouldn't follow the coach's advice. It often came down to small things like whether I remembered the coach's suggested ingredients when I was grocery shopping or whether I bought some unhealthy food item that I shouldn't have that would mock me until I'd eaten it all.

Results & Impact

We never released the application for wider use, but I personally lost 10 lbs in about four months and learned a lot about how to improve my nutrition intake.