LLM-based Application Evaluation

Brain John Aboze

| December 16, 2024 | 8 mins |

This blog post was written by Brain John Aboze as part of the Deepchecks Community Blog. If you would like to contribute your own blog post, feel free to reach out to us via blog@deepchecks.com. We typically pay a symbolic fee for content that's accepted by our reviewers.

Introduction

The world stands witness to exponential advancements being made in probably the most interesting yet controversial trend of the 21st century: artificial intelligence. The global AI market is valued at $196.63 billion in 2025 and is expected to grow at a compound annual growth rate (CAGR) of 28.46% between 2024 and 2030. Estimates show that by 2030, AI will have contributed about $15.7 trillion, which is over 5x Africa’s current GDP. At both the personal and enterprise levels, generative AI is reshaping our interaction with AI; hence, it remains at an all-time high in demand. A Mckinsey survey reveals that 65% of organizations use Gen AI, with 75% of respondents anticipating disruptive changes from the technology. The survey highlights that the most common areas where organizations are implementing generative AI include sales and marketing, service development, and IT functions. It is also supported by a report from Gartner, which underlines that 38% of executive investments in Gen AI are for improving customer experience and retention, increasing revenue, and reducing costs.

Photo by Google DeepMind

The natural language processing (NLP) market is projected to reach $ 31.76 billion by the end of 2024, with a CAGR of 23.97% between 2024 and 2029. Large Language Models (LLMs) which have revolutionized how we build various applications, with interaction with users through natural language, market is projected to grow from $1,590 million in 2023 to $259,8 million in 2030 with a CAGR of 79.80% during this period. By 2025, an estimated number of 750 million applications will run with LLMs. Further, half of all digital work will be automated using these LLM-powered apps. As the demands for LLM-based applications gain momentum, a pressing need has emerged regarding how to assess these system capabilities against a multidimensional approach that guarantees accuracy and reliability. LLM-based applications are mainly common in the retail, e-commerce, marketing, education, finance and healthcare domains. This article aims to cover the evaluation of LLM-based applications across the model and product (application) evaluation.

LLM application across different domains, Author

Challenges in Evaluating LLM-based application

Evaluation of LLM-based application is an inherently complex task due to the intrinsic nature of LLMs and human language itself. The challenges inherent in LLM-based applications differ from those built with traditional machine learning and deep learning models. Let’s look at a few of them:

application evaluation, complexity reasons

Natural language responses and the absence of clear ground truth

Language is fluid and always contextual, cultural, and dependent on the intentions of its users. After all, what might be an appropriate response in one situation may turn into an inappropriate response in another. There are multiple ways to say one thing. LLMs generate responses in natural language; defining therefore what constitutes a “correct” or “optimal” answer is highly difficult and subjective, as the tone, context, or even cultural nuances can affect natural language. Unlike tasks with definite answers or structured data outputs, natural language doesn’t always have a single “correct” answer as multiple responses can be appropriate for a given input. This absence of clear ground truth complicates the evaluation process, as it is difficult to establish standard evaluation metrics effectively to measure the model’s outputs.

Stochastic nature of LLMs and irreproducibility of results

LLMs operate probabilistically, which introduces randomness into their outputs, such that identical inputs may produce different outputs. This unpredictability complicates reproducibility, a cornerstone of traditional evaluation processes. Traditional evaluation methods rely on reproducibility of outputs to assess performance changes over time or after iterations of modifications, but the variability in LLM responses undermine this approach.

Source: Dr Soumya Ranjan Mishra

Limitations of Current Evaluation Methods and the Need for Standardization

Quantitative evaluations using metrics and benchmarks can be misleading. Model builders may tune their models to excel in these specific evaluations-a phenomenon known as “benchmark overfitting”-but the models may fail to perform effectively in diverse, real-world applications. Academic benchmarks tend to be very limited and do not reflect the full richness of using language in real settings. The consequence of optimization for benchmarks is a model that performs very well during evaluations but does not serve practical purposes. In addition, qualitative evaluation methods are still evolving, with different criteria that depend on the application or use case. Due to the lack of standardized assessments, evaluation could be inconsistent and subjective over time. There’s a pressing need for the development of standard, robust evaluation frameworks that can accurately reflect a model’s performance in real-world scenarios.

Model Evaluation vs. Product Evaluation

The first thing to understand is that two kinds of LLM assessments exist: model evaluation and product evaluation. Understanding this distinction is key to ensuring that the right evaluation approach is applied depending on your goal.

Source: Author

Model evaluation focuses on LLM performance within the application; these evaluations are not tied to any specific application use case but solely on the LLM itself. Product evaluation, on the other hand, focuses on the overall system’s performance, including all the moving components of the LLM-based application and the effect on user experience.

Model Evaluation

The development and refinement of foundational LLMs primarily rests on the shoulders of the LLM providers such as OpenAI, Google, Meta and Anthropic. These foundational models serve various applications and evaluation in this scope cuts across robust language understanding and generation and safety and ethical standards. In the cases of fine-tuned models which represent customization of these foundation models to address specialized requirements, evaluating fine-tuned models scope ensures that it meets the specific performance criteria as well as comply with the respective domain-specifications. The quality of the LLM (either foundational or fine-tuned) directly impacts the effectiveness of applications built upon them.

Source: Author

Model Evaluation methodologies includes the following:

Eyeballing: This is simply the manual assessment which provides immediate feedbacks. This evaluation methodology is prone to bias, inconsistency, inefficiency and is difficult to scale.
Benchmarks: These are standardized tests or datasets used to assess LLM performance on various tasks. These benchmarks allow objective comparison of LLMs on different tasks. Some benchmarks include HELM, GLUE and BIG-bench. As previously mentioned, models can be optimized to perform well on benchmark at the expense of practical generalization.

HELM leaderboard

Human-as-a-Judge: This involves evaluating the performance of LLMs by engaging humans (both at the expert and non-expert level) to compare the text generated by the LLMs to that written by humans. Human judgements are still regarded as the gold standard for assessment of LLM quality, however it is time-consuming, expensive, prone to bias and difficult to scale.

Photo by RDNE Stock project

LLM-as-a-Judge: This simply utilizes another LLM to evaluate the outputs of an LLM (which serves as the primary model). Read more about LLM-as-a-Judge.
ML-as-a-Judge: This evaluation methodology involves utilizing specialized machine learning models to assess specific aspects of the LLM outputs such as toxicity, sentiment analysis or topic relevance. This is efficient for target evaluations with narrow focus.
Embedding-as-a-source: This utilizes the embeddings (numerical representation of text) to evaluate the LLM performance on semantic similarity or relevance. Looking at evaluation such as prompt-response relevance, similarity or consistency as well as distances between various themes or topics within the given prompt/response. This is also additionally dependent on the embedding models used.
NLP metrics computation: Utilizing NLP quantitative measures such as text statistics and quality. These are often focused on surface-level characteristics and may not reflect deeper understanding or appropriateness.
Pattern Recognition: This is vital for security and compliance, by detecting predefined patterns cutting across Personal identifiable information (PII), sensitive information, inappropriate/prohibited content or compliance exposures utilizing regex or pattern recognition algorithms. This evaluation can aid flagging issues for review or block responses.

Product Evaluation

When deploying an LLM within an application, evaluating the system holistically is imperative rather than focusing solely on the model. This holistic evaluation considers the LLM as one component of a larger ecosystem, including databases, APIs, user interfaces, and other backend services and integrations. This approach ensures that all components of the applications work to deliver the predefined objectives and goals.

Source: Author

LLM-based applications often rely on other system components such as databases, APIs, integration and user interface. Understanding the various interactions with the moving parts associated with is essential to guide system optimization efforts, fix integration challenges or bottlenecks to improve user experiences. This evaluation focuses on the direct impact on end-user satisfaction and business success, looking at the entire application/product pipeline from input to outputs. In order to run product evals, you need to have input test cases that reflect real-world scenarios and evaluation criteria.

Source: Author

In addition to these test cases, the evaluation metrics need to be defined that captures product effectiveness, performance and user satisfaction. It is worthy to note that both test cases and evaluation metrics for the product changes as the product evolves. Based on the dynamic nature of these systems, it would be a best practice to define global and specific evaluation metrics for the products. Some global evaluation criteria could include tone and style of response, compliance time, response correctness (which requires pre-defined ground truth answers) and product performance metrics such as response time, stability and resource utilization.

Product Evaluation methodologies includes the following:

End-user-in-the-loop: Feedback mechanisms are integrated directly into the applications/products, and the users become part of the process of evaluation. Thumbs-up and -down buttons, facilities for comments, or reporting mechanisms allow users to provide feedback regarding responses in real-time. This user-generated data will help to identify issues that might not have been foreseen at development time and support continuous improvement.
Human assessment: Human evaluators will provide insight into the user experience, usability, and appropriateness of the responses of an application within a real-world context as well as review interactions with application components. They are able to review how well the application serves the user needs, alignment with brand guidelines, and complex or subtle interactions.

Quality vs. Scalability Trade-Off in LLM Application Evaluation

Photo by Shiva Smyth

Evaluating LLM-based applications involves balancing the depth of insights with the practicality of scaling the evaluation process.

Below are the three primary evaluation methods:

Automated Scoring: These depend on quantitative metrics through automation and are thus lower-cost but highly scalable options. However, most of the automated scoring can offer surface-level analytics only, lacking depth compared to other methods of scoring.
Human Evaluation: Human evaluation provides high-value insights with a high degree of flexibility. However, it is less scalable and expensive, and biases or subjectivity might appear.
LLM-based: The use of LLM capability to drive assessments is faster than human judgments, yet deeper than automated evaluation; one has to be very careful with its implementation, and there are some questions regarding its reliability.

Final Notes

In conclusion, the evaluation of LLM-based applications requires a well-rounded approach that balances the distinct needs of both model evaluations and product evaluations. Among key considerations are time costs, workflow efficiency, and quality versus scalability trade-offs that will instruct how such evaluations are done. As the landscape of LLM-powered applications evolves and scales, continuous monitoring and integrating the evaluation processes into CI/CD pipelines will increasingly become important. This includes the representational nature of your test cases regarding real usage, integrating them with real-time user feedback, and continuous refinement of the evaluation metrics for the performance to remain robust in live environments. Finally, efficient evaluation ensures that LLMs are not just technically sound but also stand up to the expectations and needs of the end.