Feeding ASCII art into Jev lets a text-only model make robot-arm decisions
The author fed ASCII-rendered images to Jev, a text-only decision model, to judge what a robot arm should do, and it worked surprisingly well.
Do dumber things. I wired vision into JEV in a very hacky way and it worked great 😂 Last week’s rabbit hole was getting caught up on robot arms. Hardware, training pipelines, MuJoCo, sim2real, all of it. I wanted to understand how the pieces fit together and get enough of a working understanding to start building things. The last time I touched this stuff, my world was ABB RobotStudio, KUKA, and very deterministic machines doing exactly what you told them to do. It was all Faro laser trackers and immaculate localization with no vision in the loop. Coming back to it through vision language action models and learned policies has been wild. There are about six projects I want to build now, but somewhere in the middle of this, JEV FOMO hit. Anyone can make a 10 second latency vision+text API call to Astra. Training VLA models is also getting super easy, I had the arm running on TurboVLA and SmolVLA within 24 hours. That’s sorta table stakes. Can we possibly do it in a dumber way? When I searched around for JEV vision repos, I found a lot of people using other models as vision sidecars, or doing binned action outputs on existing 7B open source models. I wanted to see if I could wrangle extra capability out of the existing JEV model purely through harnessing. Side note, I’m bearish on fine-tuning and bullish on harnessing. Not enough people are single-model-to-rule-them-all-bitter-lesson pilled. So how do you force a low latency text only classifier to understand something useful about a visual scene? Not everything in the image. Just enough to answer a specific question about what’s happening. What information would you have to give it, and how ridiculous could the representation be before it stopped working? Turns out, the old ASCII text trick worked awesome. The goal was photons in, actions out. I ran a Python process to do simple color thresholding (no OpenCV cheating) and convert the camera frames into a 100 × 100 character array. I piped that into JEV and set up a training dojo in MuJoCo to iterate on all the tuning coefficients. A few million test cases later, covering all the annoying combinations (white rook on a white square, surrounded by eight pieces, etc.), I had a fairly robust system. The whole point was to provide fine alignment when pieces were off nominal in their positioning, and detect missed grabs. I should note that JEV is not running the entire pipeline. I’ve got Opus 5.5 as a high level orchestration machine, a stockfish chess engine, a classical kinematics trajectory planner, and a few other housekeeping tools running. Because this is a fairly bounded problem, we can utilize classical programming where it makes sense. The JEV engine gets turned on when we’re roughly positioned in open loop mode. This hybrid open-than-closed loop approach saves a ton of compute and tokens. It’s played through about 10 full games now. I’ve been pitting Opus 5.5 against Astra 6, because StockFish gets a bit boring. The models are worse, which adds a bit of variety to the games. Every night I start a game and let them duel it out. It feels very Westworld, having it whirring in the background while I’m doing research. Highly recommend. Anyways, full 60 second video below. It’s obviously the worst possible solution to the problem, but that was kind of the point. Anyone in SF want to talk VLAs next week?
This post quotes another post: x.com/andrewmccalip/status/2106011735501295626