Documentation

# kortical.api.experiment_framework

# What is an Experiment?

An experiment is a variation of a training run, which aims to optimise a score. A 'score' simply refers to model score, although other metrics are possible (perhaps better suited to a business case). Examples of various experiments include:

  • Changing the dataset (e.g selecting a subset of features, column transformations)
  • Changing the training (e.g altering the model code, using a custom loss function)

This object represents an experiment framework, which is a place to store experiments. It allows for easy comparison and ranking of experiments, fast recall of data/training configurations, and efficient storage of the best performing models.

from kortical.api.experiment_framework import ExperimentFramework

# Initialise framework (with default parameters)
ef = ExperimentFramework(path=".experiments", is_minimising=True)  

print(ef.framework_path)         # Location of experiment framework
print(ef.is_minimising)          # Determines whether these experiments aim to minimise/maximise a metric

The experiment framework folder (called .experiments by default) is stored in the current working directory, unless otherwise specified; the complete folder structure is shown below. If a new experiment is recorded using an existing name, it is appended with a run number (i.e experiment, experiment_2, experiment_3...). Within an experiment's results folder, model_number is ordered by best result first.

.experiments
│   results_table.csv    
│
└───experiment_1
│   │   overview.yaml
│   │   best_model.yaml
│   │   dataset.csv
│   └───results
│       │   model_1.yaml
│       │   model_2.yaml
│       │   ...
│   
└───experiment_2
│   ...

# Experiments

# record_experiment

This function records the results of an experiment. The individual results are stored into the experiment framework folder and a summary of the result and its rank is printed to the console. Within the experiment folder, an overview file is created (storing the inputs described above) as well as a results folder containing model-specific details.

Often, a series of experiments will be recorded iteratively (refer to "Examples").

Inputs

  • experiment_name - the name of the experiment.
  • results (= None) - experimental results. Formatted as [(ModelVersion, num, str),...] (see output of model.train_model_custom_loss_on_top_n), or formatted as int/float.
  • model_version (= None) - a ModelVersion object, required only if results is None or number.
  • description (= None) - An optional longer description of the experiment, in string format.
  • custom_artifact_paths (= None) - Optional list of paths to files on the local disk that are copied to the experiment folder. Can be a path str or a list of paths [str,...].
  • tags (= None) - If you would like to append further experiment information to the high-level results table, pass in a dictionary here. The keys will create new columns, populated by the values.

Returns

from kortical.api.experiment_framework import ExperimentFramework

results_1 = [(modelversion_1, loss_1, format_result), (modelversion_2, loss_2, format_result)]
description_1 = "Experiment 1, house price regression. Columns 8 and 9 removed before training,  percentage error loss function."
data_paths = ['path-to-train-data', 'path-to-test-data', 'path-to resource-3']

# Initialise framework (with custom name) and record experiment
titanic_ef = ExperimentFramework('.titanic_experiments')
titanic_ef.record_experiment('experiment_1', results_1, description=description_1, custom_artifact_paths=data_paths)

# list_experiments

This function lists the name of every experiment that currently exists in the framework.

No Inputs

Returns

  • list - a list of experiment names currently recorded in the framework.
from kortical.api.experiment_framework import ExperimentFramework

ef = ExperimentFramework('.titanic_experiments')
experiment_names = ef.list_experiments()

# get_experiment_results

This function returns a list of all experimental results, ranked by model score (default). To order by custom loss, set order_by = 'custom_loss. For a chronological list of experiments, set order_by = 'time'.

Inputs

  • order_by (= 'model_score') - orders the list of experiments. Accepts one of the following arguments: custom_loss, model_score or time.

Returns

  • list - a list of experimental results, formatted as [(experiment_name, best_model[metric], metric, best_model, overview),...].
from kortical.api.experiment_framework import ExperimentFramework

# Initialise framework and get list of experiments (ordered by custom loss)
ef = ExperimentFramework('.titanic_experiments')
results = ef.get_experiment_results(order_by='custom_loss')

This function prints all experimental results to the console, ranked by model score (default). To order by custom loss, set order_by = 'custom_loss. For a chronological list of experiments, set order_by = 'time'.

Inputs

  • order_by (= 'model_score') - orders the list of experiments. Accepts one of the following arguments: custom_loss, model_score or time.
  • highlight_experiment (= None) - passing in the name of an existing experiment will highlight it in the ranked list, as well as print again at the top.

Returns

  • Prints to console.
from kortical.api.experiment_framework import ExperimentFramework

# Initialise framework and print experiments
ef = ExperimentFramework('.titanic_experiments')
results = ef.print_experiment_results()

# Print a list ordered by time (oldest to newest)
results = ef.print_experiment_results(order_by='time')

# Highlight an experiment
results = ef.print_experiment_results(highlight_experiment='drop_passenger_id')

#

Below is the format of the printout, shown with no highlighting (top) and highlighting a specific experiment (below). In this example, the experiments aim to maximise a model score by dropping 'Name' and/or 'PassengerId' columns from the titanic training set.


#

# ef_get_best_model_version

This function returns details about the best model version across the framework (or specified experiment), presented in a ModelVersion object.

Inputs

  • experiment_name (= None) - Specify this parameter to return the best model in a specific experiment, instead of searching through the entire framework.

Returns

  • ModelVersion - A ModelVersion object.
from kortical.api.experiment_framework import ExperimentFramework

ef = ExperimentFramework('.titanic_experiments')
best_model_version = ef.get_best_model_version()

# clear_experiment_cache

This function deletes all contents in the framework folder, resulting in an empty experiment framework; this includes any manually added files that are not experiment-related. If you wish to delete only specific experiments you can do this manually by deleting the folders for those experiments.

No Inputs

Returns

  • None
from kortical.api.experiment_framework import ExperimentFramework

ef = ExperimentFramework('.titanic_experiments')
ef.clear_experiment_cache()

← kortical.api.superhuman_calibration →