From Supplier Emails to Delivery Alerts: A Practical Test of Jev

Assume a situation where a manufacturer has to work with many different suppliers. This company may receive dozens or hundreds of emails a day regarding shipments that have been ordered. In the past, a team of supply chain managers would go over these emails, and decide for each one if there is some escalation or additional treatment needed.

Obviously, today’s generative AI and LLMs should allow us to save a lot of time for planners in this scenario. Let’s say we want to automatically alert planners when we receive a signal of a shipment being delayed, and safely ignore all other communications from suppliers. How does it look to actually build this automation system?

Building the automation

With a classical LLM

1. The system sends an email to the LLM, and asks if the email indicates a shipment delay.
2. The system parses the result - requiring some heuristics and error handling, since the LLM output is not always what you expect in format.
3. Make a decision whether or not to inform a planner.

With Jev

1. The system sends an email to Jev, and asks if the email indicates a shipment delay.
2. Jev can be configured to directly return the probability, in a type safe manner, that requires no parsing or content error handling.
3. Decide what your cutoff is - for example, if the chance of the email indicating a delay is over 40%, warn a planner.

The relative simplicity of the implementation is apparent. But that is not the only advantage. I will break down a quick demo of using each tool for this purpose, to show the cost advantage:

Classical LLM (Claude Opus, medium effort)

I started by writing out three emails. One clearly indicating a delay, one update not indicating a delay, and one ambiguous email signalling a potential problem upcoming. I then provide context for the model, and ask for each email - what is the chance the relevant email indicates a shipment will be late?

I asked to receive the response in a format that is computer friendly. The result came back with a fairly accurate estimation, and seemed to be quick and inexpensive:

For 3 emails, the test took 3 seconds, and cost somewhere between $0.0002 to $0.3403 (the total includes overhead from the environment and infrastructure).

Jev

Let’s look at the same test in Jev. Here, I defined the context as a “background” parameter in the state, and sent each email as a separate question:

The results ran quickly and came back with their structured answers. The probabilities are very similar to the ones that the LLM assigned, which in our simple example, would translate to the same results:

The break comes when discussing performance. The request, including all three email questions, took about 360 milliseconds to return (most of which was just the networking time on my specific computer setup). The estimated API cost came in at about ~$0.000025.

Bottom line

This post is meant for illustrative purposes only. I did not perform heavy benchmarking or throttling tests, and there are obviously optimizations that you can build into a system that is purpose-engineered for answering a question like this, using either type of model. We used LLMs in the classic “chatbot” version that an existing supply chain director might easily adopt, without talking about different structuring options the API has.

These quick results show that Jev gave similar performance as the best-in-class reasoning LLM, while running about 10x faster and being orders of magnitude cheaper. In a system built to process thousands of emails a day, that can make a real impact for usability. This is why I feel that probabilistic decision models must at least be part of the architecture discussion for any modern agentic decision-making system.

Here at Hexight, our system has been relying on probabilistic models for concrete decision making for years. These new frontier models are exciting additions that we are already implementing into our arsenal.