Default Accuracy Metric¶

The default_accuracy metric evaluates the accuracy of predicted default probabilities by comparing them to observed default rates.

Metric Type: default_accuracy

Accuracy Calculation¶

The accuracy is calculated as: pd / odr (expected to observed ratio)

Where:

pd = Predicted Default probability (mean)
odr = Observed Default Rate (actual defaults / total volume)

Configuration Fields¶

Record-Level Data Format¶

For individual loan/account records:

metrics:
  accuracy_check:
    metric_type: "default_accuracy"
    config:
      name: ["portfolio_accuracy"]
      data_format: "record_level"
      prob_def: "predicted_probability" # Column with predicted probabilities (0.0-1.0)
      default: "default_flag" # Column with default indicators (0/1 or boolean)
      segment: [["product_type"]] # Optional: segmentation columns
      dataset: "loan_portfolio"

Summary-Level Data Format¶

For pre-aggregated data:

metrics:
  summary_accuracy:
    metric_type: "default_accuracy"
    config:
      name: ["aggregated_accuracy"]
      data_format: "summary_level"
      mean_pd: "avg_probability" # Column with mean probabilities (0.0-1.0)
      defaults: "default_count" # Column with default counts
      volume: "total_count" # Column with total observation counts
      segment: [["risk_grade"]] # Optional: segmentation columns
      dataset: "risk_summary"

Required Fields by Format¶

Record-Level Required¶

name: Metric name(s)
data_format: Must be "record_level"
prob_def: Probability column name
default: Default indicator column name
dataset: Dataset reference

Summary-Level Required¶

name: Metric name(s)
data_format: Must be "summary_level"
mean_pd: Mean probability column name
defaults: Default count column name
volume: Volume count column name
dataset: Dataset reference

Optional Fields¶

segment: List of column names for grouping

Output Columns¶

The metric produces the following output columns:

group_key: Segmentation group identifier (struct of segment values)
volume: Total number of observations
defaults: Total number of defaults
odr: Observed Default Rate (Defaults/Volume)
pd: Mean Predicted Default probability
accuracy: Calculated accuracy score

Fan-out Examples¶

Multiple Accuracy Metrics¶

metrics:
  portfolio_accuracy:
    metric_type: "default_accuracy"
    config:
      name: ["total_accuracy", "product_accuracy", "region_accuracy"]
      segment: [null, ["product_type"], ["region"]]
      data_format: "record_level"
      prob_def: "model_score"
      default: "default_indicator"
      dataset: "loan_data"

This creates three accuracy metrics:

Overall portfolio accuracy
Accuracy by product type
Accuracy by region

Mixed Data Formats¶

metrics:
  detailed_accuracy:
    metric_type: "default_accuracy"
    config:
      name: ["record_level_accuracy"]
      data_format: "record_level"
      prob_def: "probability"
      default: "default"
      dataset: "detailed_data"

  summary_accuracy:
    metric_type: "default_accuracy"
    config:
      name: ["summary_accuracy"]
      data_format: "summary_level"
      mean_pd: "mean_prob"
      defaults: "def_count"
      volume: "vol_count"
      dataset: "summary_data"