Tutorial: turning an LLM into a decision model for ticket routing
Using a support ticket example, it explains freezing the backbone, adding LoRA, replacing the language model head with a decision head, and applying softmax per question to output calibrated probabilities.
How to turn an LLM into a decision model? Say we have a support ticket that says, “I was charged twice for invoice 4411. Please refund.” Our application needs to make two decisions. Which team should handle the ticket, and whether to issue a refund. An LM would generate text token by token while a decision model would score the available choices directly. Here’s how to turn an LLM into one: → Define the questions and choices. Give the model the ticket alongside the decisions to make. For the team, the choices are billing, technical, and sales. For the refund, they are yes and no. → Keep the backbone and freeze its weights. The backbone processes the input and builds internal representations of it. Add LoRA, small trainable components that adapt those representations while the original weights stay frozen. → Replace the language model head. The language model head predicts the next token. Replace it with a decision head that gathers evidence from the ticket for each option, lets the questions and options interact, and produces one score per choice. → Convert scores into probabilities. Apply softmax, which converts scores into probabilities, separately for each question. You get one probability distribution over the teams and another over the refund choices. → Train on the correct answers. For this training example, the labels are billing and yes. Cross-entropy loss measures how well the predictions match those labels, and training updates only the decision head and LoRA components. → Calibrate on held-out examples. After training, use examples excluded from training to adjust the model’s confidence. Its predicted probabilities should reflect how often those predictions are correct. At inference time, one forward pass scores all the choices. Your application gets probabilities it can use to route the ticket and make the refund decision. I trained a JEV-style decision model locally using Qwen3.5 0.8B. Accuracy went from 37% to 65% in just 60 steps, with 10 minutes of training on 4GB of VRAM. Check out my YouTube video for the full walkthrough, from setup and training to evaluation and inference. YouTube video: https://youtu.be/yKEesJaCwEc (subscribe for more content like this)
This post quotes another post: x.com/703601972/status/2108658332739485746