Documentation

# How To Build an Automation App

Note

To make it easier to flick back and forth, it's often useful to click the link to this doc and open in a new tab.

# Overview

This doc is a guide to creating a bespoke automation app, using the Instant App automation template.

This template should be used when you want to automate the parts of a workflow where machine learning is confident it can hit a target accuracy and leave the rest for manual processing. These will then be fed back in as training data and help to improve the automation going forward.

This app will have:

  • Preprocessing and transform code in python
  • The main task automation workflow
  • Automated Superhuman Calibration
  • API based model retraining
  • Automatic model publication based on the business case
  • Failover, redundancy, audit trail, easy scalability and other important production enterprise features
  • Continuous integration / Automated testing
  • An open stack based on python, uwsgi, nginx, docker and kubernetes

This doc will take you through:

  • Initial Setup
  • Creating An App
  • Customising The App
  • Superhuman Calibration for an Existing Model
  • Setting up the Git Repository / Continuous Integration
  • Testing the Deployed App
  • Integrating the Automation App

The tutorial assumes that you are using Github for source control and that you've got Pycharm and the GitHub CLI installed.

PRE-REQUISITIES

This tutorial assumes that you are using:

  • Kortical SDK. Refer to the Kortical SDK documentation setup page, which details a range of installation methods depending on your operating system and preferences.

# Creating An App

# 1. Create a codebase for your app.

Create the app, selecting the automation template:

> kortical app create <app_name>
Available app templates:

	[1] bbc_web_app_template
	[2] bigquery_app_template
	[3] excel_app_template
	[4] automation_template

Please select a template index:
> 4

This will create a folder with the specified app name; by default, this will be in your current directory.

# 2. Create a virtual environment for the app.

Although you may have already created a venv for running the Kortical package, the requirements for your app may clash with your currently installed libraries. It is strongly recommended to create a new venv for each app folder that is created, so running code locally is quick and easy.

Deactivate your current environment and enter the app directory. Then create a python 3.9 venv and install the requirements:

    > cd <your_chatbot_project_name>
> which python3.9                                    
> <path-to-python3.9>/python3.9 -m venv app-venv          
> source app-venv/bin/activate                  

    > cd <your_chatbot_project_name>
> where python3.9
> <path-to-python3.9>/python3.9 -m venv app-venv
> copy "app-venv\Scripts\python.exe" "app-venv\Scripts\python3.exe"         
> app-venv\Scripts\activate.bat            

    // Make sure to add code blocks to your code group

    This will create a venv called app-venv, to create a venv with a different name change the last parameter.

    Note

    All further steps assume that we’re in the app directory and have it's venv activated. To activate a venv: source <path-to-venv>/bin/activate, to exit a venv: deactivate.

    # 3. Prepare your train data.

    Copy your data into the folder <app_name>/data, and delete the file data/animal_shelter.csv.

    Run the script local/get_training_data.py to split the data into a train and test set; after this, upload the train data to Kortical.

    # 4. Build a model.

    In the platform, do exploratory data analysis, data cleaning, feature engineering, kick off a training run, Create Model. Wait until the training plateau, then publish a model from the Models page.

    # 5. Deploy the app.

    > cd <app_name>
> kortical app deploy

    (NB: This will take a few minutes)

    # Customising The App

    # Step 2

    Replace the config parameters for your project. Navigate to src -> <app_name> -> config -> config.yml and you'll see a list of parameters.

    api_key: changeme
model_name: "<app_name>"
target: 'OutcomeType'
target_accuracy: 0.8
not_automated_class: 'Not Automated'

    You'll need to replace these values with ones that make sense for you.

    api_key - This is the API key that users will need to pass in to access the apps endpoints. This should be something that's hard to guess. One way to create it is to go to https://www.guidgenerator.com/online-guid-generator.aspx, click Generate some GUIDS!, then click Copy to Clipboard and finally paste it into the yaml.

    model_name - This will automatically pick up the name of the app but if you have already created a model, you should change it here to whatever the model name is in the platform.

    target - This should be set to the name of the target column in the platform / dataset.

    target_accuracy - This should be set to whatever the human accuracy level is or perhaps a different level that is more optimal for the business. The Superhuman Calibration will then attempt to tune the model's performance up to this accuracy level.

    not_automated_class - This is the name of the class to use for entries we can't automate and keep the desired accuracy rate. This can either be an existing class or a new class.

    # Step 3

    Here we change the model code. Navigate to the Lab page for the Model in the platform and copy the model code.

    then in the app navigate to src -> <app_name> -> workflows -> common.py, find the variable model_code and copy in your code.

    # Step 4

    Next up is changing the code as appropriate, the functions you'll likely want to edit are mostly in src -> <app_name> -> workflows -> common.py:

    create_train_calibrate_and_test_datasets - Especially if you have time series data, you'll probably want to slice it differently here.

    preprocessing - Add any preprocessing code you like, if you'd like to remove all preprocessing, you can set it to:

    def preprocessing(df):
    pass

    postprocessing - Add any postprocessing code you like, if you'd like to remove all postprocessing, you can set it to:

    def postprocessing(df):
    return df

    # Step 5

    Changing the business case can be a little more involved, you'll find the code in src -> <app_name> -> workflows -> business_case.py:

    calculate - This function should return a single scalar value, ideally in currency value.

    should_publish- This function determines if the challenger model is good enough to unseat the champion should it exist and be published.

    If you do not yet know the enough details of the business case you could use automation rate as a proxy. eg: (Be sure to fix the import and keep in mind that all hyphens become underscores for python modules)

    from <app_name>.config import read_config

config = read_config("config.yml")
target = config['target']
target_accuracy = config['target_accuracy']
target_accuracy_tolerance = 0.03            # 3%
minimum_automation_rate_improvement = 0.01  # 1%


def calculate(calibration_results, print_results=False):
    automation_rate = calibration_results[target]['automation_overall']['automation']

    if print_results:
        print(f"Automation Rate {automation_rate*100:.2f}%")

    return automation_rate


def should_publish(challenger_calibration_results, champion_calibration_results):

    target = list(challenger_calibration_results.keys())[0]

    challenger_accuracy = challenger_calibration_results[target]['automation_overall']['accuracy']
    target_accuracy_delta = abs(challenger_accuracy - target_accuracy)

    # determine if the calibration_results accuracy is close enough to the target accuracy
    if challenger_accuracy < target_accuracy and \
        target_accuracy_delta > target_accuracy_tolerance:
        return False, f"The accuracy for the challenger model accuracy [{challenger_accuracy:.3f}] was not within the tolerance [{target_accuracy_tolerance}] of the target accuracy [{target_accuracy}]."

    challenger_automation_rate = calculate(challenger_calibration_results)

    # If there is no champion and the target accuracy is met publish the challenger
    if champion_calibration_results is None:
        return True, f"There is no current champion. Publishing the model is recommended. The challenger accuracy [{challenger_accuracy:.3f}], the difference to the target is [{target_accuracy_delta:.3f}]. The automation rate is [{challenger_automation_rate:.2f}]."

    champion_automation_rate = calculate(champion_calibration_results)

    automation_rate_change = challenger_automation_rate - champion_automation_rate

    # determine if the automation rate is a meaningful amount better if not return False
    if challenger_automation_rate < champion_automation_rate + minimum_automation_rate_improvement:
        return False, f"The challenger model does not meaningfully improve the automation rate [{automation_rate_change:.3f}]."

    return True, f"The challenger automation rate is [{challenger_automation_rate:.2f}], the target accuracy difference [{target_accuracy_delta:.3f}] is acceptable and the automation rate is improved by [{automation_rate_change:.2f}]."

    # Superhuman Calibration for an Existing Model

    # Step 1

    In Pycharm go to the file local -> adopt_trained_model.py in Pycharm.

    You should see a green arrow. Clicking this will bring up a menu where we can select Run 'adopt_trained_model'.

    You may be prompted for your login details and find that pressing enter doesn't submit your email, in this case you can press control + C to terminate the run. Having run the python script, a run config will have appeared in the drop down on the top right of Pycharm. You can click on this, click on Edit Configurations..., then you can scroll down and check the box labelled Emulate terminal in output console. The run adopt_trained_model.py again.

    Enter your credentials and save them.

    # Step 2

    There will be quite a lot of output but among that you should see

    Raw Model F1 Score: 0.908

    This is the raw model score and below that the calibration scores.

                      automation_rate  accuracy
Overall                  0.910380  0.946304
Class1                   0.836842  0.942122
Class2                   0.948910  0.968998
Class3                   0.895086  0.909353

                  precision    recall  f1_score   count
Overall            0.947228  0.946304  0.946465  3017.0
Class1             0.942122  0.921384  0.931638   318.0
Class2             0.968998  0.964824  0.966906  1393.0
Class3             0.909353  0.937685  0.923302   674.0

    Here we can see the the overall automation rate and accuracy on the first line and then further breakdown per class below.

    Below that again we'll find the business case reporting.

    Business Case:

Should Publish: True,
Reason: There is no current champion. Publishing the model is recommended. The challenger accuracy [0.946], the difference to the target is [0.004]. The automation rate is [0.91].

    If it did not recommend publishing you may need to dial back the target accuracy, train a better model or adjust the business case to make it pass.

    # Step 3

    The previous step published your model to the UAT environment.

    Go to your model in the platform, go to deployments, select Production and click publish on the ghost model and confirm. We could easily adjust the code to publish to production automatically but this gives us a chance to review model changes, read model reports and make an informed decision before publishing.

    # Step 4

    Deploy the app. In the command prompt run:

    > kortical app deploy --force

    (NB: This will take a few minutes)

    # Setting up the Git Repository / Continuous Integration

    TIP

    Refer to the page ML Ops - Deployment for further information.

    # Step 1

    While that's running, we can set up the git repo. We may need to open up a new terminal and navigate to our app, then run:

    (NB: This might require you install the GithubCLI, on Mac this can be done with brew install gh)

    > git init # Intialize repo
> gh repo create "<git hub organization>/<app_name>" --private --confirm # Create the repo
> git add . && git commit -m "initial commit" # Add and commit the code
> git push --set-upstream origin master # Set the upstream branch on GitHub

    # Step 2

    Test out the Continuous Integration. To do this we want to create a new branch. In the bottom right of the Pycharm window there is a branch icon and beside it, it says master. Clicking this opens a menu with an option New Branch. We can click this, enter a branch name such as branch1 and confirm.

    Then we can go to the config.yml and add some white space.

    Next we can go to the top menu Git -> Commit, we then have to enter any commit message. Then we want to click on the drop down beside the Commit button and select Commit & Push. This opens yet another dialog and we select push.

    # Step 3

    Then we can navigate to our new repo on Github, we should see our branch highlighted with a button to Compare & pull request, click this. Click Create pull request to create the pull request.

    After a few seconds it should pick up our workflow and start running it. We can see details to see how it's progressing.

    If we look at the details we should see that one of the tests failed due to our changes to the business case.

    # Step 4

    To fix the tests, if we find the assert in tests -> test_business_case_should_publish.py -> test_business_case_pass.

    assert reason.startswith("The challenger savings are ")

    We can change it to fix the test, as below.

    assert reason.startswith("The challenger automation rate is ")

    Then we can commit change, check Github and we should see the tests pass.

    We can also click the Files Changed tab on the pull request in Github and see what changes form part of this commit.

    We recommend adding branch protection rules for master to stop accidental commits Settings Tab -> Branches -> Branch Protection Rules -> Add Rule. We need to enter Branch name pattern, we can enter master. Then tick Require a pull request before merging and Require status checks to pass before merging (just top level ticks needed) and click the Create button. After that review and Save Changes and you're done.

    # Testing the Deployed App

    # Step 1

    To test the app's endpoints are doing what we want them to do, we can feed the test data back in and check the business case performance. This gives us confidence that predict is doing what we want it to.

    Adjust the assert in tests -> test_business_case.py -> test_business_case from assert savings > 3_000_000 to assert savings > 0.1 or whatever makes sense for your business.

    Go to the green tick beside tests -> test_business_case.py -> test_business_case and run it.

    This test requires user input, if pytest gives the error OSError: pytest: reading from stdin while output is captured! Consider using '-s'. then go to the dropdown on the top right of Pycharm, select Edit Configurations... and enter -s in Additional Parameters. Then run the test again.

    You should see the expected automation rate in the output of the test. eg:

    Automation Rate 93.14%

    # Integrating the Automation App

    The app is now deployed and working.

    You can let the dev team for the workflow system know that there are two endpoints for them to integrate to.

    # Predict.csv

    POST - https://<system_url>/app/<app_name>/predict.csv?api_key=<app_key_we_set_in_the_config_file>

    # Parameters

    file - A multipart-encoded file with data in the same format it was in for training in the platform, that we want predictions for. Rows with the not automated class should be sent for manual processing. Those with a different class should be accepted as automated.

    # Returns

    A CSV file with the same name as the endpoint predict.csv. This file includes the data sent to the endpoint but also a number of extra columns are added with the details of the predictions. The prediction will be in a column predicted_<target> where <target> is the name of the target you set.

    # Example

    An example of calling this endpoint from Python can be found in tests -> test_business_case.py -> test_business_case

    # Train

    POST - https://<system_url>/app/<app_name>/train?api_key=<app_key_we_set_in_the_config_file>

    # Parameters

    file - A multipart-encoded file with all the training data. This is a temporary solution and we will be shortly introducing Live Datasets, so no data is required to be passed in for training.

    # Returns

    Success! - If successful, otherwise it will contain an error.