Pipecat runs Jev in parallel with the LLM to trigger tool calls early
Traditional tool calls must wait for the full token stream before executing; the author runs Jev in parallel with the LLM so a structured event triggers the tool call while the LLM is still generating text.
Perfect demo. 10/10. Nothing to add. ... just kidding, I always have something to add (sorry). One of the things we do fairly often in Pipecat pipelines is "greedy inference." If you have partial transcriptions you can run the LLM on the partials and just throw away the output if it turns out the transcription wasn't finished or wasn't accurate. Same thing for streaming output from the LLM. You can run guardrails/safety checks sentence-by-sentence, prompt the LLM so you can use the first token for some kind of decision, or whatever. But we haven't been able to do this kind of thing with traditional LLM tool calls, because with an LLM you have to wait for the entire tool stanza to be emitted, one token at a time, before you can execute the tool call. Jev changes that. You can put Jev in parallel with an LLM, ask Jev to output a structured even that triggers a tool call, and run the tool call while the LLM is outputting the text the user will hear/read.
This post quotes another post: “Replacing the LLM's tool-calling decisions with Jev, tested”