ThePrimeagen uses Cloudflare's Jev to speed up Omarchy QA tests
He is trying to cut the desktop QA agent's login flow from 2 minutes toward 30 seconds using Jev, which has image capabilities, for judgment, and is asking for help on the second step of deciding what counts as correct.
Omarchy QA update There are two things that are very hard about building a QA agent harness. 1. How to go fast in the day and age of agents (which are very very slow) 2. What is correct? What do we even test? The first one is less exciting but the second one has been this last weeks challenge. When I originally started this, to have a basic agent crawl the desktop in QEMU was about ~15 minutes to open the lock screen and then type in password and confirm that we are back on the desktop Then a few changes I was down to ~5 for median. That sucks. With more changes and now a custom harness I am down to ~2 minutes. That still sucks. So for the last week I have been experimenting with using Clef (cloudflare's jev, it has image capabilities, which is what i needed) and I am hoping to get this number down from 2 minutes to 30 seconds for * logging in * system menu * lock screen * password * back to desktop * confirm i am done. If I can get this then I can officially move to step 2, which... well I am not stoked about step 2 and may need some help. So if you are interested in helping, let me know.