Jev goes multimodal: image-based gesture and object detection decisions
The author shows putting camera frames into Jev's state and using Choice, Noul, Score and Box in parallel to judge gestures, whether something is present, and where it is.
Jev is now multimodal. It can see. Here’s how it works and what you can build with it: Jev is a decision model. You give it information, ask questions, and get answers your code can use directly. Previously, that information was text or JSON. Now it can include images. Start with how a Jev request works. You send a “state,” which is simply the information you want Jev to examine. Then you ask questions about it. Each question specifies the kind of answer you need. → Choice picks from options you define, such as “thumbs up” or “fist.” → Noul returns the probability that something is true, such as whether a hand is visible. → Score rates something on a scale you define, such as how open the hand is. With multimodal Jev, images can be part of both the state and the questions. Put a camera frame in the state, and your questions can ask about what’s visible in it. Put a reference photo in a question’s instructions, and you can ask whether an object in the camera frame matches the one in that photo. Jev can now also identify an object and locate it for you. The new Box type brings a YOLO-like capability to Jev. Describe what you want to find, such as “the hand in this image,” and it returns a bounding box around it. Your app can use those coordinates to highlight the object or crop it. One detail matters here. Box returns coordinates even when the target is absent, so pair it with a Noul presence check before using the result. My demo asks about presence, gesture, and location together. Jev answers these independent questions in parallel, and the app combines the results. These capabilities open up prototypes for gesture controls, locating buttons in screenshots, and checking product photos against visual references. Jev’s focus on fast decisions also makes it promising for interactive applications where response time matters, with image latency worth testing in your own workflow. Images and Box are still experimental and require the images “danger zone” header. I cover how it works, the API, and the demo app in the detailed video below. Chapters 0:00 Intro 0:36 Live demo 1:41 What's covered 2:22 How Jev works 3:06 Question types 3:57 Playground demo 6:59 Demo app walkthrough 10:30 Using Multiple images 11:25 Parallel questions 12:40 Danger zone headers 13:07 Outro If you’re new to Jev and want to understand how decision models work, I’ve written a detailed article about it. It’s quoted below.
This post quotes another post: x.com/akshay_pachaar/status/2101037514945597645