kwindla on why evals are central to LLM engineering, with notes on Jev
The author argues evals are now key to scaling LLM systems and shares notes on how Typesafe/Jev could fit into existing toolkits and eval workflows.
Almost everyone I know is spending more time thinking about evals, these days, than we did a year ago. For the past two years, we've been speed-running computer science/software engineering history, trying to figure out how to build useful, robust, real-world systems that leverage LLMs. (Re-inventing lots of wheels, as always happens during a big computing transition.) Last year at this time, it seemed like the big, general questions we were asking mostly revolved around how to think about the loops, tool calls, and context management triad of problems. Now, a lot of my conversations are about evals. Evals unlock scaling, maintainability, modularity, cost management, legal/compliance, and research into new architecture directions. Probably a bunch of other stuff, too. But building good evals is not a competence that most software engineers have (yet). Some really helpful notes on how Typesafe/Jev might fit into our toolkits, below, including some thoughts on evals ...
This post quotes another post: x.com/fleetingbits/status/2100292587621863590