Apps Building Process
Build Sprint Context
These notes were captured during a main build sprint when we were hitting model rate limits. At that time we settled on Kimi K2 as the production pick, and the final entry in this set lands in mid-October 2024.
Project Context
This case study documents the work done as a Learning Engineer at Playlab to help build system prompts and optimize AI-powered educational applications deployed within high schools in Ghana. The project focused on creating curriculum-aligned assessment and lesson planning tools for Ghana's Senior High School (SHS) system.
Optimization
Migrating from expensive models to cost-effective alternatives while maintaining quality.
Engineering
Refining system prompts with categorization and explicit formatting rules.
Evaluation
Comprehensive testing of multiple AI models for production fitness.
Timeline Summary
Identified that apps are too pricey. Intervention Courses - Teacher Planning app is the most used, currently using Claude Sonnet 4.5. Decision made to keep Sonnet 4.5 for this critical app given importance, or potentially move to cheaper model.
Sonnet models hitting rate limits and high costs. Goal: get as many apps as possible to Kimi K2 given best cheap tool calling ability. Started reviewing Sonnet conversations and testing same conversations with Kimi.
Costs became unbearable. Doubling down on moving all apps to Kimi K2 and improving tool calling. Also addressing issue with Claude Sonnet constantly providing assessments with question B being the main answer.