Integrating Deepchecks with AWS SageMaker AI: Step-by-Step Guide

Brain John Aboze

| May 01, 2025 | 4.5 mins

Introduction

Generative AI (GenAI) isn’t a “set it and forget it” technology; it has fuzzy inputs and outputs. Continuous evaluation is critical as systems built with this technology are inherently dynamic and rapidly evolving. To ensure that your GenAI application consistently delivers reliable, high-quality, and ethically sound outputs, it is critical to implement rigorous evaluation throughout the entire lifecycle. This also helps promptly detect issues like hallucination, bias, cost, and performance degradation.

Deepchecks is a leading end-to-end evaluation and observability solution. It was recently announced in the AWS re: Invent 2024 as a partner AI app within SageMaker, bringing its evaluation capabilities to the SageMaker environment. Deepchecks evaluation suites include automated scoring, version comparison, properties evaluation, golden set management, continuous monitoring, and debugging, enabling AI teams to assess model performance and quality systematically. Integrating Deepchecks with AWS SageMaker AI creates a seamless, end-to-end evaluation environment. SageMaker’s scalable, secure, and fully managed platform, combined with Deepchecks’ advanced evaluation capabilities, allows you to embed continuous monitoring and assessment directly into your development workflow. This integration not only extends SageMaker’s native capabilities but also enhances overall observability, ensuring that your GenAI applications are always performing at their best.

Setting Up Deepchecks as a Partner AI App

Amazon SageMaker Partner AI Apps enable coherent integration with AWS-curated third-party tools like Deepchecks. Partner AI Apps are pre-configured, fully managed applications that run within your SageMaker environment. AWS-curated AI apps, including Deepchecks, are deployed as complete application stacks. These application stacks leverage AWS services, ensuring operations run securely on service-managed AWS accounts. They also keep sensitive data within your trusted security boundaries and never share it with a third party.

The first step in setting up is having your organization’s administrator subscribe to Deepchecks via AWS marketplace. Deechecks subscriptions fall into three tiers:

Small: 200 tokens/sec (development/testing).
Medium: 500 tokens/sec (staging).
Large: 1300 tokens/sec (production).

The administrator can then use the SageMaker A I console to browse, discover, and provision the app. Once deployed, permissions are assigned to data scientists and ML developers, granting them access through Amazon SageMaker Studio or SageMaker Unified Studio without needing separate credentials. These permissions for operations include creating, updating, and managing Partner AI Apps and the ability to pass roles for secure operations.

Then, follow the AWS documentation to subscribe to Deepchecks via AWS Marketplace and configure the Partner AI App Permissions and execution role accordingly.

After your organization’s administrator configures Deepchecks, team members can log into the platform using a corporate single sign-on (SSO) login. Then, team members can access the SageMaker AI environment with the designated domain and user profile to use the Deepchecks app for LLM evaluation.

Note:A “ Deployed” status on Deepchecks for a Partner AI App indicates that the app is correctly configured in the SageMaker environment, is installed correctly, and is ready for functionality use. Look at the URL of the Amazon Resource Name (ARN) as well as the URL of the Deepchecks Software Development Kit (SDK), which are displayed when the Deepchecks Partner AI App is accessed.

Integrating Deepchecks with AWS SageMaker AI: Step-by-Step Guide

Version Comparison
AI-Assisted Annotations
CI/CD for LLMs
LLM Monitoring

Now, with Deepchecks deployed, your SageMaker environment is fully operational, and you’re all set to dive into GenAI development and harness the power of Deepchecks’ evaluation workflows.

To start, select “ Open Deepchecks LLM Evaluation” to launch a private instance of the Deepchecks LLM evaluation application within SageMaker. Next, generate and copy your Deepchecks API key and the ARN and SDK URL (AWS Partner App URL) needed to integrate Deepchecks LLM Eval for your GenAI application.

Note: As a best practice, store the API key in an environment variable named DEEPCHECKS_API_KEY.

Next, launch a Jupyter Notebook or Code Editor within the SageMaker Studio environment and install the SageMaker Python SDK and Deepchecks LLM Evaluation SDK to connect to the Deepcheck UI from our code. Install the required packages by running the following:

pip install sagemaker deepchecks-llm-client[sagemaker]

Next, set the necessary environment variables for seamless integration and authentication:

os.environ['AWS_PARTNER_APP_AUTH'] = 'true'
os.environ['AWS_PARTNER_APP_ARN'] = '<your-partner-app-ARN>'
os.environ['AWS_PARTNER_APP_URL'] = '<your-partner-app-URL>'
os.environ['DEEPCHECKS_API_KEY'] = '<your-Deepchecks-API-key>'

These variables ensure that Deepchecks connects securely with your SageMaker environment without exposing sensitive credentials.

Logging Interactions & Monitoring

DeepchecksLLMClient is the gateway for integrating Deepchecks’ evaluation capabilities into your generative AI workflows. With the client, you can log interaction, monitor performance, and gain detailed insights throughout the application lifecycle. The client can be initiated as follows:

from deepchecks_llm_client.client import DeepchecksLLMClient
from deepchecks_llm_client.data_types import ApplicationType, EnvType
import os

# Retrieve credentials from AWS Secrets Manager
dc_api_key = os.environ["DEEPCHECKS_API_KEY"]
dc_host = os.environ["AWS_PARTNER_APP_URL"]

# Initialize client
dc_client = DeepchecksLLMClient(
    api_token=dc_api_key,
    host=dc_host,
)

Additional parameters like log_level and silent_mode control logging verbosity and error handling, ensuring that you receive appropriate feedback without interrupting your workflow.

Next, you need to set up a new evaluation application and its scope as follows:

# Create a QA-focused application
dc_client.create_application(
    app_name="medical-chatbot",
    app_type=ApplicationType.QA,
    description="LLM for answering patient queries using medical guidelines",
    version_name="v1-claude-3",
)

The app_type can be QA, SUMMARIZATION, GENERATION, CLASSIFICATION, FEATURE_EXTRACTION and OTHER. Creating an application provides a structured context in which subsequent interactions can be logged and compared.

Next, capture interaction within our application and log key parameters such as input and output, expected output, timing information, environment types (which can be either EVAL, PROD, or PENTEST), and additional data such as retrieved context, history, and any custom properties that might be useful for debugging.

def evaluate_medical_response(question: str, model_response: str, context: list):
    """
    Log interactions between users and a medical chatbot.
    """  
    interaction_id = dc_client.log_interaction(
        app_name="medical-chatbot",
        version_name="v1-claude-3",
        env_type=EnvType.EVAL,
        input=question,
        output=model_response,
        information_retrieval=context,  # Retrieved documents from RAG
        annotation="good",  # Options: "good", "bad", "unknown"
        annotation_reason="Accurate citation of NIH guidelines",
        custom_props={"specialty": "cardiology"}
    )
    return interaction_id

# Example usage
question = "What's the first-line treatment for hypertension?"
response = "ACE inhibitors are recommended as initial therapy..."
context = ["NIH Guideline: ACE inhibitors reduce cardiovascular risk..."]
interaction_id = evaluate_medical_response(question, response, context)

By incorporating these processes, the DeepchecksLLMClient facilitates logging of complex interactions as well as the overall observability of your LLM apps much more easily. Once the interaction is logged, you can continue to the Deepchecks UI to view the results of your app’s checks and multiple versions with time. This robust evaluation framework is essential for maintaining high-quality generative AI solutions on AWS SageMaker.