Documentation

# Examples

Realistic usage of the Platform API is demonstrated in the case studies shown below. Each example makes use of kortical.datasets to load the appropriate dataset, either as a dataframe (datasets.load) or by copying files into your current working directory (datasets.write_to_disk).

# Create and evaluate a model

This code snippet will train a series of model versions on some uploaded data, then deploy the best performing version to the model's development environment. Next, it will use a test set to create predictions, and evaluate its performance against the targets using the area under ROC metric:

from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

from kortical import datasets
from kortical.api.data import Data
from kortical.api.model import Model


# Prepare data
train_df, test_df = train_test_split(datasets.load('titanic'), train_size=0.8, random_state=0)
train_data = Data.upload_df(train_df, 'titanic_train', targets='Survived')

# Select model, train candidates (reduced training time for quick demonstration)
titanic_model = Model.create_or_select('titanic', delete_unpublished_versions=True, stop_train=True)
best_version = titanic_model.train_model(train_data, max_models_with_no_score_change=50)

# Deploy
development = titanic_model.get_environment()
model_instance = development.create_component_instance(best_version.id, wait_for_ready=True)

# Predict
predictions = model_instance.predict(test_df)

auc = roc_auc_score(predictions['Survived'], predictions['yhat_probs'])
print('Got an ROC AUC of', auc)

# Creating a model with custom model code

As an extension to the previous example, every detail of model creation and training is specified using model code (written in the Kortical language). In this example below, a model is trained with more cross-validation folds:

from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

from kortical import datasets
from kortical.api.data import Data
from kortical.api.model import Model

# Copy default model code from the platform/modify what you want to change
titanic_model_code = r"""
- ml_solution:
  - data_set:
    - target_column: Survived
    - problem_type: classification
    - evaluation_metric: area_under_roc_curve
    - fraction_of_data_set_to_use: 1.0
    - cross_validation_folds: 4                  
  - features:
    - numeric:
      - PassengerId
      - Pclass
      - Age
      - SibSp
      - Parch
      - Fare
    - categorical:
      - Sex
      - Embarked
    - text:
      - Name
      - Ticket
      - Cabin
    - date
  - models:
    - one_of:
      - xgboost
      - linear
      - random_forest
      - extra_trees
      - decision_tree
      - deep_neural_network
      - lightgbm
      """

# Prepare data
train_df, test_df = train_test_split(datasets.load('titanic'), train_size=0.8, random_state=0)
train_data = Data.upload_df(train_df, 'titanic_train', targets='Survived')

# Select model, train candidates (reduced training time for quick demonstration)
titanic_model = Model.create_or_select('titanic', delete_unpublished_versions=True, stop_train=True)
best_version = titanic_model.train_model(train_data, model_code=titanic_model_code, max_models_with_no_score_change=50)

# Deploy
development = titanic_model.get_environment()
model_instance = development.create_component_instance(best_version.id, wait_for_ready=True)

# Predict
predictions = model_instance.predict(test_df)

auc = roc_auc_score(predictions['Survived'], predictions['yhat_probs'])
print('Got an ROC AUC of', auc)

# Get predictions and explanations from an existing model

This code snippet will use a deployed model to carry out both predictions and explanations. A small number of predictions are made with fast explanations to reduce computing time. The most important features for predicting the positive class (Survived) are plotted in a bar chart.

from matplotlib import pyplot as plt

from kortical import datasets
from kortical.api.model import Model

# Prepare data
test_df = datasets.load('titanic')

# Get an instance of the model's default version
titanic_model = Model.get_model('titanic')
model_instance = titanic_model.get_development_model_instance()

# Predict and explain (only 3 rows are sampled for short run-time)
# For more accurate explanations, use a statistically significant sample size.
titanic_explanations = model_instance.predict(test_df.sample(3), explain_predictions=True, explain_profile='fast')

# Filter to get sum of those rows with explanation weights for the positive class ('[1]')
explains_summed = titanic_explanations.filter(like='[1]').sum()

# Filter these weights to only show those with a absolute value > 0.1 (or we will get a huge chart)
explains_summed[explains_summed.abs()>.01].sort_values().plot(kind='barh')

# make the plot look nice in matplotlib
plt.tight_layout()
plt.show()

# Creating and explaining a multi-class classification model

Like the previous example, this code snippet will use a deployed model to carry out both predictions and explanations; this is for a multi-class classification scenario.

from sklearn.metrics import f1_score
from sklearn.model_selection import train_test_split
from matplotlib import pyplot as plt

from kortical import datasets
from kortical.api.data import Data
from kortical.api.model import Model

# Prepare data
train_df, test_df = train_test_split(datasets.load('iris'), train_size=0.8, random_state=0)
train_data = Data.upload_df(train_df, 'iris_train', targets='species')

# Select model, train candidates (reduced training time for quick demonstration)
iris_model = Model.create_or_select('iris', delete_unpublished_versions=True, stop_train=True)
best_version = iris_model.train_model(train_data, max_models_with_no_score_change=50)

# Deploy
development = iris_model.get_environment()
model_instance = development.create_component_instance(best_version.id, wait_for_ready=True)

# Predict
predictions = model_instance.predict(test_df)
f1 = f1_score(predictions['species'], predictions[f'predicted_species'], average='weighted')
print('Got an F1 Score of', f1)

# Explain some predictions from this model
sample = test_df.sample(10, random_state=1)
some_iris_explanations = model_instance.predict(sample, explain_predictions=True, explain_profile='fast')

class_names = some_iris_explanations['species'].unique()
for class_name in class_names:
    # each class has its own set of explanation weights for all columns. The [<class_name>] suffix is used to filter
    explains_summed_this_class = some_iris_explanations.filter(like=f'[{class_name}]').sum()

    # filter these weights to only show those with a absolute value > 0.1 (or we will get a huge chart)
    explains_summed_this_class[explains_summed_this_class.abs() > .01].sort_values().plot(kind='barh', title=f'Top Drivers for {class_name}')
    plt.tight_layout()
    plt.show()

# Creating a multi-label classification model

This code snippet will use a deployed model to carry out predictions; this is for a multi-label problem (explanations are not currently supported for this scenario).

from sklearn.model_selection import train_test_split

from kortical import datasets
from kortical.api.data import Data
from kortical.api.model import Model

# Prepare data
train_df, test_df = train_test_split(datasets.load('enron_multi_label'), train_size=0.8, random_state=0)
enron_train = Data.upload_df(train_df, 'enron_train', targets=['cat0', 'cat2', 'cat3', 'cat4'])

# Select model, train candidates (reduced training time for quick demonstration)
enron_model = Model.create_or_select('enron', delete_unpublished_versions=True, stop_train=True)
best_version = enron_model.train_model(enron_train, max_models_with_no_score_change=50)

# Deploy
development = enron_model.get_environment()
model_instance = development.create_component_instance(best_version.id, wait_for_ready=True)

# Predict
predictions = model_instance.predict(test_df)

# Create and rank a series of experiments

This code snippet will run experiments which train on various subsets of titanic passenger data. After this, they are printed out in a list ranked according to model score.

from kortical import datasets
from kortical.api.data import Data
from kortical.api.model import Model
from kortical.api.experiment_framework import ExperimentFramework

# Prepare data
train_df = datasets.load('titanic')

# Here are two experiments, aiming to determine whether
# dropping the PassengerId column results in a better model.
experiment_dict = {
    'all_features': train_df,
    'dropped_passenger_id': train_df.drop(columns=['PassengerId'])
}

with ExperimentFramework('.titanic_experiments', is_minimising=False) as ef:
    for experiment_name, df in experiment_dict.items():
        # Prepare data
        train_data = Data.upload_df(df, f'{experiment_name}_df', targets='Survived')

        # Select model, train versions (reduced training time for quick demonstration)
        titanic_model = Model.create_or_select('titanic', delete_unpublished_versions=True, stop_train=True)
        best_version = titanic_model.train_model(train_data, max_models_with_no_score_change=50)

        ef.record_experiment(experiment_name, model_version=best_version)

    ef.list_experiments()

results = ef.get_experiment_results()

# Create and rank a series of experiments (with custom loss function)

The experiment framework may be used with a custom loss function. In this example, we train various subsets of house price data to predict sale price. Now, the root mean squared error (RMSE) is used as the new metric (replacing model score) to rank the list of experiments.

Care must be taken when writing a custom loss function; please refer to instance.train_model_custom_loss_on_top_n in the documentation.

import numpy as np
from sklearn.model_selection import train_test_split

from kortical import datasets
from kortical.api.data import Data
from kortical.api.model import Model
from kortical.api.experiment_framework import ExperimentFramework


# Define custom loss function
def custom_loss_function(df):
    # root mean squared error
    y = df['SalePrice']
    yhat = df['yhat']
    loss = np.sqrt(abs(y - yhat) ** 2)
    return loss.mean(), f'{loss} RMSE'

# Prepare data and experiments
train_df, test_df = train_test_split(datasets.load('house_price_regression'), train_size=0.8)
experiment_dict = {
    'feature_set1': train_df,
    'feature_set2': test_df.drop(columns=['Street', 'Alley', 'LotShape'])
}

with ExperimentFramework('.house_price_experiments') as ef:
    ef.clear_experiment_cache()

    for experiment_name, data in experiment_dict.items():
        # Prepare data
        train_data = Data.upload_df(data, 'train_data', targets='SalePrice')

        # Select instance, train_model (reduced training time for quick demonstration)
        house_model = Model.create_or_select('house_prices', delete_unpublished_versions=True, stop_train=True)
        results = house_model.train_model_custom_loss_on_top_n(
            train_data,
            test_df,
            custom_loss_function,
            max_models_with_no_score_change=50,
            n=10, is_minimising=True)
        
        ef.record_experiment(experiment_name, results)

    ef.list_experiments()

results = ef.get_experiment_results()

# Using Superhuman Calibration to improve model performance

In this example, a BBC model has been prepared; this classifies a block of text from an article as one of the following categories: tech, sport, politics, business, entertainment. In order to improve the model's performance, we use Superhuman Calibration to identify a subset of data where it can perform at a higher accuracy.

Please refer to the full documentation on Superhuman Calibration.

# Set up imports and model code

from sklearn.metrics import f1_score
from sklearn.model_selection import train_test_split

from kortical import api, datasets
from kortical.api import superhuman_calibration
from kortical.api.data import Data
from kortical.api.model import Model

# Fixing model code to reduce train time in example
model_code = r"""
- ml_solution:
  - data_set:
    - target_column: category
    - problem_type: classification
    - evaluation_metric: f1_score
    - data_set_random_seed: 1
    - modelling_random_seed: 786058994
  - features:
    - text:
      - text:
        - create_features:
          - tf-idf:
            - norm: l2
            - use_idf: False
            - smooth_idf: True
            - sublinear_tf: False
            - max_features: 50934
  - models:
    - linear:
      - C: 2.055
      - class_weight: null
      - max_iter: 1000000
      - tol: 0.0001
      - warm_start: False
      - multi_class: auto
      - solver: lbfgs
      - penalty: l2
      - dual: False
      - fit_intercept: False
"""

# Train and Superhuman Calibration workflow (continued...)


# Prepare a model
train_df, test_df = train_test_split(datasets.load('bbc'), train_size=0.8, random_state=0)
train_data = Data.upload_df(train_df, 'bbc_train', targets='category')
bbc_model = Model.create_or_select('bbc', delete_unpublished_versions=True, stop_train=True)
best_version = bbc_model.train_model(train_data, max_models_with_no_score_change=50, model_code=model_code)

# Deploy
development = bbc_model.get_environment()
model_instance = development.create_component_instance(best_version.id, wait_for_ready=True)

# Predict
predictions = model_instance.predict(test_df)
f1 = f1_score(predictions['category'], predictions[f'predicted_category'], average='weighted')
print('Got an F1 Score of', f1)

# Split test data into calibration/test set
fit_df, test_df = train_test_split(predictions, train_size=0.5, random_state=0)

# Get general thresholds (no defined strategy). Calculate the scores.
calibration_data = superhuman_calibration.calibrate(fit_df,
                                                    'category',
                                                    target_accuracy=0.97)
test_df = superhuman_calibration.apply(test_df, calibration_data)
superhuman_calibration.score(test_df, calibration_data)

# Save calibration data to be used at predict time, see the automation template for a working example...

# Predict workflow (continued...)

# Load calibration data for this model (we already have this in the variable calibration_data)

# Do predictions. In this example we reuse the test data
# but this should be any rows you would like to make predictions for
predictions = model_instance.predict(test_df)

# Apply superhuman calibration to the predictions
superhuman_calibration.apply(test_df, calibration_data, in_place=True)

# Return predictions

← kortical.api →