Feature Generation with Artemis and evoML

Project overview

Goal: generate domain-specific features for credit default risk classification, so that a simple, explainable model scores higher without changing the model itself.

Before: the baseline script with a template feature function (minimal feature engineering; logistic regression 0.817 ROC AUC on validation, 0.813 on test)

After: the same script with the generated features (20+ engineered features; logistic regression 0.857 on validation, 0.861 on test)

Target users: data scientists, machine learning engineers, credit risk analysts

Use cases:

• Automated feature generation for financial risk models
• Lifting an explainable model's performance through feature discovery

Why generate features with Artemis

Feature generation turns raw data into inputs a model can learn from. Sometimes that is an encoding or a rescaling. Often it means new features built from the interaction of existing ones, or from domain knowledge the dataset does not contain, while keeping an eye on noise, overfitting and leakage. Here a data scientist uses Artemis to generate such features for an existing model, as a post-hoc step, with the aim of lifting a linear model's performance.

The project predicts whether a customer will reach serious delinquency within two years, from their credit history. The end-to-end baseline analysis is here. The scripts use the evoML Python client to train and evaluate the models on our AutoML platform, but any script that trains a model and prints its score works the same way.

How it works

Artemis needs three things: a script it is allowed to change, a command that scores the result, and an Objective.

1. The script

credit_script.py loads the dataset, passes it through a feature engineering function, trains three models on evoML (XGBoost, LightGBM and logistic regression, with ROC AUC as the objective) and prints their scores. The feature engineering function starts as a template. It gives Artemis one place to add features, and the comments in it let you steer how far it goes:

# From credit_script.py (baseline)
def engineer_features(df: pd.DataFrame) -> pd.DataFrame:
    """
    Template function for feature engineering.

    Args:
        df (pd.DataFrame): Input dataframe
    Returns:
        pd.DataFrame: DataFrame with new features
    """
    # Create a copy to avoid modifying the original dataframe
    df_engineered = df.copy()
    feature_2 = df_engineered.columns[1]
    feature_3 = df_engineered.columns[2]

    # Example of creating a new feature by multiplying two existing features
    #df_engineered["feature_1"] = df_engineered[feature_3] + df_engineered[feature_2]

    return df_engineered

df = engineer_features(df)
print(df.columns)

A Markdown file next to the script, Features.md, describes the dataset: each column, its type and what the target means. The data is the Kaggle Give Me Some Credit set.

2. The benchmark

A Runner on your own machine executes the script as the benchmark command of the Project's Script. To make the score a measured Metric, the script writes it to a results file in the build directory after each run:

artemis_results.json

{ "logistic_roc_auc": 0.8167 }

The Runner reads that file after every benchmark run, and logistic_roc_auc becomes the Metric each new Version is scored on. Numbers printed to the console stay in the logs, so the script also prints all three models' scores for you to read:

See Custom Metrics for the file format. Run the Script once on the Baseline before you start, so there is a score to compare against.

3. The Objective

This is the Objective we gave Artemis, with Features.md attached as context, quoted as written:

This script executes a classification trial on an AutoML platform, where my best-performing model is LightGBM, with a ROC AUC score of 0.863 on validation and 0.866 on test. My logistic regression classifier achieves similar performance (validation 0.816, test 0.813). To me, this means my feature set is likely informative enough for an explainable model, but there could be some more domain-specific feature engineering I might have missed. Take a look at the commented-out section of my script for context, generate more technical features and attach them to the dataframe for upload. Don't change anything else.

Artemis then generates Versions of the script, runs each one on the Runner, reads logistic_roc_auc from the results file and carries the best Versions forward. A Version that fails to run is discarded. One that scores lower stays in the table and is ranked, but is not carried forward. In the current platform this is a Discovery Run with the script's ROC AUC as its measured Metric.

Results

Artemis lifted the logistic regression classifier by engineering new features into the dataset. The before and after, as evoML shows them:

Baseline:

Final Version:

ROC AUC for the logistic regression model went from 0.817 to 0.857 on validation and from 0.813 to 0.861 on test, an improvement of about 0.04 and 0.05.

The Feature Importance tab for the model shows the age-bucket features generated by Artemis among the strongest. Their contribution to the positive class (serious delinquency) is negative, which matches age being protective, and they sharpen the separation between risk strata, which is what raises the ROC AUC.

What Artemis built

Asked in the Version's Branch chat how many features it had engineered and how, the agent listed 14 new features, derived with sums, ratios, thresholds and interaction terms from domain knowledge about credit risk. evoML then encoded them in a type-aware, leakage-safe way (one-hot or target encoding for categorical buckets, scaling for continuous variables), handled missing values and applied the same transformations across the train, validation and test splits.

The final script has 20+ features across several categories:

# From credit_script_optimised.py (optimized features excerpt)

# Payment history features
df_engineered["total_delinquencies"] = (
    df_engineered["NumberOfTime30_59DaysPastDueNotWorse"]
    + df_engineered["NumberOfTime60_89DaysPastDueNotWorse"]
    + df_engineered["NumberOfTimes90DaysLate"]
)

df_engineered["delinquency_severity_score"] = (
    df_engineered["NumberOfTime30_59DaysPastDueNotWorse"] * 1
    + df_engineered["NumberOfTime60_89DaysPastDueNotWorse"] * 3
    + df_engineered["NumberOfTimes90DaysLate"] * 5
)

# Utilization features
df_engineered["very_high_utilization"] = (
    df_engineered["RevolvingUtilizationOfUnsecuredLines"] > 0.8
).astype(int)

# Income stability features
df_engineered["income_per_dependent"] = df_engineered["MonthlyIncome"] / (
    df_engineered["NumberOfDependents"] + 1
)

# Debt management features
df_engineered["high_debt_burden"] = (df_engineered["DebtRatio"] > 0.43).astype(int)

# Credit mix features
df_engineered["real_estate_to_total_credit_ratio"] = df_engineered[
    "NumberRealEstateLoansOrLines"
] / df_engineered["NumberOfOpenCreditLinesAndLoans"].replace(0, 1)

# Risk interaction features
df_engineered["high_debt_with_delinquency"] = (
    (df_engineered["DebtRatio"] > 0.5) & (df_engineered["total_delinquencies"] > 0)
).astype(int)

If you would rather handle some transformations yourself, say so in the Objective and start another run, or open the Version's Branch and ask the coding agent to change the script.