Dear readers,

I hope that you are as curious as I am and join me on this learning journey. So, get your curiosity ready and let’s get started. 🙂

As my learning journey on Evaluation Testing progresses, I want to share a post on the two major types of evaluation testing - objective and subjective. In my first post Evaluation Testing LLM I have focused on the general definition of what are Evals, why we need them and then had a deeper dive into LLM-as-a-judge concept.

Objective vs Subjective Evals

We do not have to always use LLM-as-a-judge. There are Evals which we can do perfectly in the way we have learned before Gen AI, namely with objective or also known as exact evaluations. In order to clearly see the difference between subjective and objective evaluations, let’s explore an example use case.

Example: Family Weekly Meal Plan Agentic Workflow

We have the following agentic workflow:

Step #1 - The LLM task is to create a family weekly meal plan for a dinner with the following criteria: 1 fish meal 1 meat meal 5 vegetarian meals All meals must be healthy, meaning - no added sugar, no fats, moderate usage of carbohydrates coming from potatoes and white rice.

Step #2 - The LLM task is to reflect on its output and to apply the revised feedback providing a final meal plan version.

Additional criteria - the workflow runs automatically every week on Monday morning.

Testing Plan for Family Weekly Meal Plan Agentic Workflow

Objective Evals

are everything that can be measured. It can be with and without ground truth (or also seen as a reference example).

Scenarios:

  • Total meals count = 7
  • Exactly 1 fish meal
  • Exactly 1 meat meal
  • Exactly 5 vegetarian meals
  • The process runs on Monday morning, once per week.

Analysis:

Scenarios 1 to 5 are candidates for objective evals. The total meals count can be achieved by asking the LLM to provide the response in a JSON structured output. Then you can relatively easily test it objectively by counting the output meals.

Scenarios 2-4 are more interesting for objective tests. Since they fall into the same category, I will deep dive into only one of them, namely - Scenario 2 (only 1 fish meal). The question here is how can we test if a recipe is a fish meal objectively? The current requirement is too generic and thus more difficult to build objective evals if we cannot predict what are the possible valid responses. What I believe is clever to do is to check if it is possible to change the requirement to make it more testable. In our example, I think that it is perfectly Okay to work with a list of concrete fishes that could be generated by the LLM. The steps below describe the flow of adjustment and test preparation.

  • Step #1 Limit fish options by adjusting the LLM prompt. In our case: salmon, tuna and herring fish.
  • Step #2 Build ground truth with few examples to compare against, in our case it will be providing meal names:
    • Baked salmon with basmati rice
    • Tuna pizza
    • Mediterranean salad with herring filet
  • Step #3 For the tests itself - use lexical search and compare the results against the ground truth. If we have Tuna Pasta and compare against the ground truth, we should get a positive match. If we have Cod fish with potatoes we should have a negative match, even if this is a fish meal.

For these scenarios, where objective evals are still not sufficient and we do see them as important business functionality, we can also use hybrid approach and add subjective evals with LLM-as-a-judge, for example.

Subjective Evals

are everything that is non-deterministic and cannot be measured precisely.

Scenarios

  • Evaluate whether the meal is healthy, meaning that:
    • Meal recipes that contain no added sugar and no fats.
    • Meal recipes that do not contain more than 150g potatoes or 100g rice per portion.
  • Evaluate whether the meal distribution of vegetarian, meat and fish meals in the weekly plan is correct (as mentioned in the objective scenarious)

The most important thing when designing evals and deciding upon the types is to have a clear distribution of what can be tested objectively. This is the simpler and also long term cost effective solution. Everything else should be tested subjectively. Here we have more options - please check my previous post on Evaluation Testing LLM. An important consideration when using LLM-as-a-judge is that, even though we can automate more subjective evals, we should also be aware of the potential higher costs - in the end, tokens are used for the evaluation itself as well.

Final Thoughts

In this blog we looked with an example on the 2 major types of evaluations - objective and subjective. This is a topic that I find crucial for building software with LLMs since Evals can give us fast feedback on each iteration and also serve us as a guard during our AI Engineering process.

Happy learning! 🙂