Auditing Models

Introduction

This is the fifth and last tutorial in the Intro Series where we do a deeper dive into auditing models built using the Footings framework. Recall, one of the principles of the Footings framework is - models need to be audited using a second source such as excel. This is important in the actuarial world where models are contrived of many calculations which are used to value and price insurance. Actuaries need to be able to demonstrate how the calculations work to satisfy requirements whether they be internal, external, regulatory, audit, etc.

Auditing Models

We will continue to use the AddABC model with documentation for the prior tutorial.

We will start with the code from the prior tutorial.

from footings.model import (
    model,
    step,
    def_parameter,
    def_intermediate,
    def_return,
)

@model(steps=["_add_a_b", "_add_ab_c"])
class AddABC:
    """This model takes 3 parameters - a, b, and c and adds them together in two steps."""
    a = def_parameter(dtype=int, description="This is parameter a.")
    b = def_parameter(dtype=int, description="This is parameter b.")
    c = def_parameter(dtype=int, description="This is parameter c.")
    ab = def_intermediate(dtype=int, description="This holds a + b.")
    abc = def_return(dtype=int, description="The sum of ab and c.")

    @step(uses=["a", "b"], impacts=["ab"])
    def _add_a_b(self):
        """Add a and b together and assign to ab."""
        self.ab = self.a + self.b

    @step(uses=["ab", "c"], impacts=["abc"])
    def _add_ab_c(self):
        """Add ab and c together for final return."""
        self.abc = self.ab + self.c

To audit a model, one needs to instantiate it and call the audit method as shown below. We will first return the audit results as a native python object and in later sections, we will show how it can be exported to different file types.

AddABC(a=1, b=2, c=3).audit()

AuditContainer(name='AddABC', instantiation={'parameter.a': 1, 'parameter.b': 2, 'parameter.c': 3}, output={'abc': 6}, docstring='This model takes 3 parameters - a, b, and c and adds them together in two steps.\n\n.. rubric:: Parameters\n\n- **a (int)** - This is parameter a.\n- **b (int)** - This is parameter b.\n- **c (int)** - This is parameter c.\n\n.. rubric:: Intermediates\n\n- **ab (int)** - This holds a + b.\n\n.. rubric:: Returns\n\n- **abc (int)** - The sum of ab and c.\n\n.. rubric:: Steps\n\n1) **_add_a_b** - Add a and b together and assign to ab.\n2) **_add_ab_c** - Add ab and c together for final return.\n\n', signature='AddABC(*, a: int, b: int, c: int)', steps={'_add_a_b': AuditStepContainer(name='_add_a_b', method_name='_add_a_b', docstring='Add a and b together and assign to ab.', uses=('parameter.a', 'parameter.b'), impacts=('intermediate.ab',), output={'intermediate.ab': 3}, metadata={}, config=AuditStepConfig(show_method_name=True, show_docstring=True, show_uses=True, show_impacts=True, show_output=True, show_metadata=True)), '_add_ab_c': AuditStepContainer(name='_add_ab_c', method_name='_add_ab_c', docstring='Add ab and c together for final return.', uses=('intermediate.ab', 'parameter.c'), impacts=('return.abc',), output={'return.abc': 6}, metadata={}, config=AuditStepConfig(show_method_name=True, show_docstring=True, show_uses=True, show_impacts=True, show_output=True, show_metadata=True))}, metadata=None, config=AuditConfig(show_signature=True, show_docstring=True, show_steps=True, step_config=AuditStepConfig(show_method_name=True, show_docstring=True, show_uses=True, show_impacts=True, show_output=True, show_metadata=True)))

As the output shows, a dictionary is returned with information about the model including arguments passed on initialization, the steps, and the output. Do notice that each step shows output for each attribute that is impacted including the intermediate attributes in the model.

The goal of an audit is to provide the user with sufficient documentation describing what the model does. Using native python code might not be the best delivery format if someone is not a python user. Thus, the Footings framework provides the ability to output to a file format. As of right now, two file types are supported - .xlsx and .json.

The below code shows how to generate an audit file in excel.

AddABC(a=1, b=2, c=3).audit("Audit-AddABC.xlsx")

This produces an .xlsx file with five tabs. Below are screenshots of the the tabs. The file can also be downloaded here.

../../_images/main.png

Sheet: Main

../../_images/instantiation.png

Sheet: Instantiation

../../_images/_add_a_b.png

Sheet: _add_a_b

../../_images/_add_ab_c.png

Sheet: _add_ab_c

../../_images/output.png

Sheet: Output

The below code shows how to generate an audit file in excel. The audit file can be downloaded here.

AddABC(a=1, b=2, c=3).audit("Audit-AddABC.json")

There are options to configure audit output a certain way. To see this review the audit api.

Closing

This tutorial concludes the Intro Series of tutorials. With this tutorial we dug deeper into auditing models which is probably the most important feature of the Footings framework. The long-term goal is to continue to build out this feature so it will be able to generate an audit document sufficient to satisfy the needs of any auditor or regulator who needs an understanding of the model.

Note a more advanced topic of testing with the Footings framework builds off of the audit capabilities. At some point there will be a tutorial demonstrating this.

Running Models Pattern Series