How to Check the Accuracy of Your Machine Learning Model in 2025 | Deepchecks

How to Check the Accuracy of Your Machine Learning Model in 2025

Amos Rimon

| June 23, 2025 | 12 mins |

Introduction

Accuracy is among the most popular methods for validating ML models in classification problems and is a famous and widely used tool. Its widespread popularity is owed to its being straightforward-it is simple to understand and implement. For simple cases, it represents a valid measure of assessing model performance. Unfortunately, real-world scenarios are rarely simple. You often face cases where datasets are imbalanced, multiclass problems, or multilabel classification challenges. In these complex scenarios, high accuracy might not imply a good performance measure. As machine learning problems get more complex, calculating and interpreting accuracy becomes more difficult and requires special attention.

Photo by Khwanchai Phanthong

It is thus important to know what accuracy is measuring, how to derive it, and some of the caveats when it is used in different machine learning scenarios. This article gives an extended description of accuracy as a classification metric. More specifically, the article provides a definition of accuracy, shows its use in binary, multiclass, and multilabel settings, and touches on its main issues. To deepen your understanding, you will also find practical examples and the full code behind them here.

Accuracy

As one of the fundamental metrics for classification problems, accuracy refers to the measure of correct predictions made by the model. It is calculated as the number of correct predictions divided by all predictions. The accuracy formula in machine learning is as follows:

This is a very simple formula, giving rise to a very easily understandable definition of accuracy in those cases where the classification problem involves only two classes. Accuracy is an intuitive metric and easy to compute, but it assumes a binary classification context. Now, let’s see how to apply accuracy to multiclass and multilabel classification and discuss details concerning more complex cases.

Example implementation:

from sklearn.metrics import accuracy_score

# Example data
y_true = [0, 1, 0, 1, 0, 1, 1, 0]
y_pred = [1, 1, 0, 1, 0, 0, 1, 0]

# Calculate accuracy score
accuracy = accuracy_score(y_true, y_pred)

# Print the accuracy score
print("Accuracy Score:", accuracy)

The Accuracy Paradox

Default accuracy is an overall measure of performance for a model on the entire dataset. But this overall accuracy can be misleading, especially for cases in which the class distribution is imbalanced and correctly predicting the minority class is important. The model can achieve a high accuracy score in such cases by correctly predicting the majority class but consistently misclassifying the minority class, thus giving a wrong impression of good performance. Take, for instance, a cancer prediction model where it is essential to identify whether a patient’s sample is malignant. Misclassification of benign as malignant has grave effects, as it entails inflicting unnecessary treatments on perfectly healthy individuals and erodes trust in the diagnostic process. And cases are generally highly biased, with more benign than malignant.

Photo by Miguel Á. Padriñán

Let’s illustrate this with an example using the Wisconsin Breast Cancer dataset, which classifies breast tumor cases as benign or malignant.

By making the dataset imbalanced-removing most malignant cases so that only about 5.6% of cases are malignant-and using only a single feature, we challenge the model’s performance.

cancer_data_imbalanced = pd.concat(
[
    cancer_data[cancer_data["labels"] == "malignant"].sample(
        frac=0.1, random_state=random_seed
    ),
    cancer_data[cancer_data["labels"] == "benign"],
]
)
cancer_data_imbalanced = cancer_data_imbalanced.loc[:, ["mean texture", "labels"]]
cancer_data_imbalanced["labels"].value_counts(normalize=True)

Our model achieves an overall accuracy of approximately 94.64%, which initially seems impressive. However, a closer look at the class-level predictions using a confusion matrix reveals a different story: the model misdiagnoses almost all malignant cases. This result starkly contrasts with the high overall accuracy, demonstrating the accuracy paradox.

The high accuracy is an illusion in the case of the imbalanced data set, where it is very costly to misclassify the minority class. Similar cases occur when predicting rare but critical events, such as serious medical conditions, economic crises, terrorist attacks, or meteor impacts. In such instances, an accuracy score of 90% would be meaningless because not even one missed case should be allowed because it might lead to catastrophic results. In this way, depending only on accuracy is not enough and can be misleading.

To avoid this problem, ask yourself the following :

When accuracy doesn’t cut it as a good evaluation metric for your ML model, consider the following alternatives:

Metrics Description
Precision How many of the predicted positives are actually positive. Prioritize this when false positives are costly.
Recall (Sensitivity) How many of the actual positives are correctly identified. Prioritize this when missing positives is costly.
F1 Score A balanced metric combining precision and recall. Use when you need a single metric to summarize performance.
Confusion Matrix A detailed table showing true/false positives/negatives. Helps pinpoint where your model is making mistakes.
ROC Curve & AUC Visualizes the trade-off between true positive rate and false positive rate at various thresholds. Higher AUC means a better model overall.
PR-Curve Similar to ROC, but focuses on the trade-off between precision and recall. Helpful for imbalanced datasets.
Matthews Correlation Coefficient A comprehensive metric considering true/false positives/negatives, even with imbalanced classes. Ranges from -1 (worst) to +1 (best).

Accuracy in Multiclass Problems

In multiclass classification problems, accuracy is defined similarly to binary classification, but the calculation must account for multiple classes rather than just two. Here is the generalized formula for accuracy in multiclass problems:

Where

Let’s see an example. The following confusion matrix shows true values and predictions for a 3-class prediction problem.

We calculate accuracy by dividing the number of correct predictions (the corresponding diagonal in the matrix) by the total number of samples.

The result tells us that our model achieved a 44% accuracy on this multiclass problem.

However, calculating an overall accuracy metric also conceals class-level issues in the multiclass case, so it is important to examine class-level predictions.

For example, let’s make predictions on the Iris dataset by using the sepal columns.

iris_data_sepal = iris_data.loc[:, ["sepal width (cm)", "sepal length (cm)", "labels"]]
iris_data_sepal.sample(5, random_state=random_seed)

model = DecisionTreeClassifier(random_state=random_seed)
prediction_results = get_prediction_results(X_train, y_train, y_test, model)
prediction_results["Prediction success"].mean()

The overall accuracy is ~76.7%, which might not be that bad.

However, when we examine the results at the class level, the results are more diverse.

Accuracy is hard to interpret for individual classes in a multiclass problem, so we use the class-level recall values instead.

Accuracy in Multilabel Problems

Multilabel classification differs from multiclass classification because, in multilabel, an instance can belong to multiple classes simultaneously, while in multiclass, each instance belongs to only one class. These problems can be viewed as multiple binary classification problems, one for each class.

Let’s see an example based on the RCV1 data set. In this problem, we try to predict 103 classes represented as a big sparse matrix of output labels. To simplify our task, we use a 1000-row sample.

The model seems to be accurate when we compare predictions with test values.

rcv1 = datasets.fetch_rcv1()
rcv1_data, rcv1_target, sample_id, target_names = (
    rcv1["data"],
    rcv1["target"],
    rcv1["sample_id"],
    rcv1["target_names"],
)
samples = np.random.randint(0, rcv1_data.shape[0], 1000)
rcv1_data_sample = rcv1_data[samples]
rcv1_target_sample = rcv1_target[samples]
rcv1_data_sample.shape, rcv1_target_sample.shape

X_train, X_test, y_train, y_test = train_test_split(
    rcv1_data_sample.toarray(),
    rcv1_target_sample.toarray(),
    train_size=0.8,
    random_state=random_seed,
)
model = DecisionTreeClassifier(random_state=random_seed)
predictions = model.fit(X_train, y_train).predict(X_test)
predictions

However, this is not a meaningful result because it relies on the huge number of ‘Negative’ values in the class vectors. We have a problem similar to the imbalanced binary case. Only now, we have many imbalanced class vectors where the majority of classes are the ‘Negative’ values.

Therefore, to get a more meaningful understanding of the model’s performance, we need to compute the accuracy with regard to the various metrics.

Multilabel Accuracy or Hamming Score

Hamming Score is a metric used in multilabel settings that compares the total number of labels active in both reality and as predicted with the number of properly predicted labels.

Where

Plain text

def hamming_score(y_test, predictions):
    return (
        (y_test & predictions).sum(axis=1) / (y_test | predictions).sum(axis=1)
    ).sum() / predictions.shape[0]
hamming_score(y_test, predictions)

Hamming Loss

Hamming Loss measures the fraction of incorrect labels to the total number of labels, accounting for both false positives and false negatives. It ranges from 0 to 1, with 0 indicating no errors. The formula for Hamming Loss is:

Where

Plain text

def hamming_loss(y_test, predictions):
    return (y_test != predictions).sum().sum() / y_test.size
hamming_loss(y_test, predictions)

Beyond Hamming Score and Hamming Loss, you can use multilabel versions of standard classification metrics seen in binary and multiclass cases. For further exploration of multilabel metrics, see [Hyperlink to the article on Understanding Classification Metrics: Accuracy, Precision, Recall, F1 Score, ROC-AUC, and PR-AUC for Binary and MultiClass Models]. Such metrics and techniques could help you obtain a more realistic and accurate picture of the performance of the model in cases of multilabel classification problems and, therefore, overcome the limitation of using only a single accuracy metric.

Subset Accuracy or Exact Match Ratio

Subset Accuracy is also known as Exact Match Ratio or Labelset Accuracy. It is a more strict form of the Accuracy metric. In order for the prediction to be correct, all of the labels have to match exactly. Its formula is as follows:

Where

Plain text

def exact_match(y_test, predictions):
    return (y_test == predictions).all(axis=1).mean()
exact_match(y_test, predictions)

This strict criterion does not account for partial correctness. For example, for a model that predicted almost all of the labels correctly but missed only one, Subset Accuracy regards it as a failure. Therefore, the above-mentioned limitation makes Subset Accuracy less informative in practice, where partial correctness is valuable. The obvious alternative in this case is to use other metrics that allow for partial correctness, like the following:

Further Accuracy Types

The most important applications of accuracy in multiclass, multilabel, and binary problems have been analyzed. Naturally, according to your concrete problem, more accuracy adjustments could be useful. Here are some of the more common:

from sklearn.metrics import balanced_accuracy_score

y_true = [0, 1, 0, 1, 0, 1, 0, 1]
y_pred = [0, 0, 0, 1, 1, 1, 1, 0]

balanced_accuracy = balanced_accuracy_score(y_true, y_pred)
print(f"Balanced Accuracy: {balanced_accuracy}")

from sklearn.metrics import top_k_accuracy_score

y_true = [0, 1, 2, 2, 1]
y_pred_proba = [[0.2, 0.3, 0.5],[0.1, 0.6, 0.3],[0.3, 0.2, 0.5],[0.4, 0.4, 0.2],[0.1, 0.7, 0.2]]
k = 2
top_k_acc = top_k_accuracy_score(y_true, y_pred_proba, k=k)
print(f"Top-{k} Accuracy: {top_k_acc}")

from sklearn.metrics import log_loss, brier_score_loss

# Example data
y_true = [0, 1, 1, 0, 1]
y_pred_proba = [0.2, 0.6, 0.3, 0.4, 0.7]

log_loss_value = log_loss(y_true, y_pred_proba)
print(f"Log Loss: {log_loss_value}")
brier_score = brier_score_loss(y_true, y_pred_proba)
print(f"Brier Score: {brier_score}")

When to use Accuracy Score in ML

Accuracy score should be used when you want to know the skill of a model to classify data points correctly, irrespective of the prediction performance per class or label. It gives you an intuition for whether your data is suitable for your classification problem.

If you need to utilize the accuracy metric in your project, there are very simple-to-use packages like Deepchecks that give you in-depth reports on relevant metrics to evaluate your model. This makes it easier for you to understand your model’s performance better.

Be Sure How to Measure the Accuracy of Your ML

Whatever metric you choose, you should know what it is good for, its caveats, and what processes you can use to validate against its common pitfalls. The bigger the ML projects you have, the more complex the system of metrics you need to monitor. You have to learn about them, know how to implement them, and keep them in check continuously. These tasks can become hard to maintain and tend to introduce wrong metrics, measurements, and interpretations.

One way to make model evaluation, validation, and monitoring easier is to utilize ML solutions like deepchecks at the different stages of the ML lifecycle. It provides a broad range of already tried and tested metrics with worked-out implementation and detailed documentation. Using Deepchecks, you can choose from a wide range of verified and documented metrics to better understand the workings of your ML models and trust them more.

Final notes

As such, accuracy is not a universally applicable parameter for machine learning models, though it can be quite a useful one. Unbalanced datasets, multiclass issues, multilabel scenarios-it may severely restrict all of these. For these reasons, to do a good thorough performance assessment with your model, one should identify its limitations and attempt to investigate the use of alternative measures such as precision, recall, F1 score, and others, depending on the particular situation and their associated costs.

Additionally, Deepchecks can streamline the model evaluation, validation, and monitoring process. These tools offer a wide range of verified metrics and detailed documentation to ensure a deeper understanding and greater trust in your ML models.