The video examines the use of Jev and the open-source Liar model for skill routing in agent operations, showing that while Jev performs strongly out of the box, Liar can achieve comparable accuracy after fine-tuning on relevant data. Despite some limitations, Liar’s local, lightweight, and trainable nature offers a promising, cost-effective alternative for skill prediction tasks, encouraging further experimentation and development in AI agent workflows.
The video explores the use of Jev and Jev-style models, specifically focusing on skill routing in agent operations. The presenter discusses the challenge of selecting from a large number of skills—122 in their case—and how Jev models can efficiently shortlist relevant skills for agents. Using real session data, Jev was able to include the correct skill in its top three picks 69% of the time. The presenter then introduces Liar, an open-weight Jev-style model, which initially performed less well with only 28% accuracy but showed promise after fine-tuning.
Liar is an open-source model developed by Convai Innovations and available on GitHub and Hugging Face. It operates by scoring multiple options based on a question and state input, using transformer layers to process the data. Unlike Jev, Liar is limited to handling about 20 options due to token constraints but is lightweight and can run locally on most machines. The model requires fine-tuning to perform well on specific tasks, as it is not a zero-shot decision engine like Jev.
The presenter tested Liar on a skills prediction task using a long Telegram session with 364 messages and 122 skills. Jev performed strongly, with 73% accuracy in the top three picks and 84% in the top ten. Untrained Liar lagged behind but still performed better than random guessing. After fine-tuning on a subset of the data, Liar’s accuracy improved significantly, nearly doubling its top pick accuracy and approaching Jev’s performance in the top ten skills.
Further fine-tuning Liar on a much larger dataset of over 112,000 training questions improved its top three pick accuracy to 71%, very close to Jev’s 79%. While Liar’s top pick accuracy also improved, it still trailed Jev slightly. Interestingly, the top ten accuracy decreased somewhat, possibly due to longer training sequences. Despite being slower than Jev, Liar’s local, free, and trainable nature makes it a compelling alternative for skill routing in agents, especially when cost and privacy are concerns.
In conclusion, the experiment demonstrated that while Jev remains superior out of the box, fine-tuned Liar models can achieve comparable performance in skill routing tasks. The presenter encourages viewers to share their experiences and ideas for further experiments with Jev and Liar, highlighting the potential of smaller decision models for practical applications in AI agent workflows. The video ends with an invitation to explore the presenter’s Agent Wiki project for additional resources and tools.