treeinterpreter

To access the Pipeline's fitted model, just retrieve the ._final_estimator attribute from your pipeline

You are getting arrays of length 2 for bias and feature contributions for the very simple reason that you have a 2-class classification problem.

As explained clearly in this blog post by the package creators, in the 3-class case of the iris dataset you get arrays of length 3 (i.e. one array element for each class):

Lime: A short explanation of how it works, taken from their github page:

Intuitively, an explanation is a local linear approximation of the model's behaviour. While the model may be very complex globally, it is easier to approximate it around the vicinity of a particular instance. While treating the model as a black box, we perturb the instance we want to explain and learn a sparse linear model around it, as an explanation. The figure below illustrates the intuition for this procedure. The model's decision function is represented by the blue/pink background, and is clearly nonlinear. The bright red cross is the instance being explained (let's call it X). We sample instances around X, and weight them according to their proximity to X (weight here is indicated by size). We then learn a linear model (dashed line) that approximates the model well in the vicinity of X, but not necessarily globally.

There is much more detailed information in various links on the github page.

treeinterpreter: An explanation of how this one works is available on http://blog.datadive.net/interpreting-random-forests/ (this is for regression; an example for classification, which works very similarly, can be found here).

In short: suppose we have a node that compares feature F to some value and splits instances based on that. Suppose that 50% of all instances reaching that node belong to class C. Suppose we have a new instance, and it ends up getting assigned to the left child of this node, where now 80% of all instances belong to class C. Then, the contribution of feature F for this decision is computed as 0.8 - 0.5 = 0.3 (plus additional terms if there are more nodes along the path to leaf that also use feature F).

Comparison: The important thing to note is that Lime is a model-independent method (not specific to Decision Trees / RFs), which is based on local linear approximation. Treeinterpreter, on the other hand, specifically operates in a similar manner to the Decision Tree itself, and really looks at which features are actually used in comparisons by the algorithm. So they're really fundamentally doing quite different things. Lime says "a feature is important if we wiggle it a bit and this results in a different prediction". Treeinterpreter says "a feature is important if it was compared to a threshold in one of our nodes and this caused us to take a split that drastically changed our prediction".

This is difficult to answer definitively. They're probably both useful in their own way. Intuitively, you may be inclined to lean towards treeinterpreter at first glance, because it was specifically created for Decision Trees. However, consider the following example:

• Root Node: 50% of instances class 0, 50% class 1. IF F <= 50, go left, otherwise go right.
• Left Child: 48% of instances class 0, 52% class 1. Subtree below this.
• Right Child: 99% of instances class 0, 1% of instances class 1. Subtree below this.

This kind of setup is possible if the majority of instances go left, only some right. Now suppose we have an instance with F = 49 that got assigned to the left and ultimately assigned class 1. Treeinterpreter won't care that F was really close to ending up on the other side of the equation in the root node, and only assign a low contribution of 0.48 - 0.50 = -0.02. Lime will notice that changing F just a little bit would completely change the odds.

Which one is right? That's not really clear. You could say that F was really important because if it had been only a little bit different the prediction would be different (then lime wins). You could also argue that F did not contribute to our final prediction because we hardly got any closer to a decision after inspecting its value, and still had to investigate many other features afterwards. Then treeinterpreter wins.

To get a better idea here, it may help to also actually plot the learned Decision Tree itself. Then you can manually follow along its decision path and decide which features you think were important and/or see if you can understand why both Lime and treeinterpreter say what they say.

Yes, you can know the features and their contributions (weight is not the right term) that impacted the prediction of a specific observation. This actually constitutes the decision_path for how it made that decision of that particular observation. What you are looking for is TreeInterpreter.

The second question is: Why are there always two values for each variable and instance (such as [0.12 -0.12] for the first feature and first instance) that seem to be positive and negative, rather than one value of that feature?

So my answer is: Each of these lists (such as [0.12 -0.12]) just represents the contribution of a feature to the final probability of an instance being in class 1 and class 2. Remember, features never dictate what class an instance must be in, but instead, they increase or decrease the final class probabilities of an instance. So 0.12 means that Feature 1 added 0.12 to the probability of instance 0 being in class 1, and reduced its probability of being in class 2 by 0.12. These values are always symmetric, meaning that whatever makes an instance more likely to be in class 1 makes it less likely to be in class 2.

Similarly, Feature 2 reduced probability of instance 1 being class 1 by 0.05, and increased its probability of being class 2 by 0.05.

So each feature contributed (added or reduced) to instance 1 being in class 1 by: 0.12, -0.05, 0.22, 0.14, 0.07, 0.01. Now add these all up to the bias of being in class 1 (0.49854), you get 1, which is the final probability of that instance being in class 1, as shown by the model output.

Similarly, add all of the second values of each list, and add the bias of being in class 2 (0.50146), you get 0, which is the final probability of that instance being in class 2, as shown by the model output above.

Finally repeat the exact same exercise for instance 1, i.e. add -0.03, -0.11, 0.06, 0, -0.06, -0.04 to the bias of 0.49854, you get 0.32, which is the P{instance1 = class1}. And add 0.03, 0.11, -0.06, 0, 0.06, 0.04 to the bias of 0.50146, you get 0.68, which is the P{instance1 = class2}. Therefore these numbers constitute a full trajectory of contributions, from the initial bias to the final classification probability of an instance.

I had answered a very similar question at a conceptual level on datascience.stackexchange.com, please feel free to check it out by clicking here.

You should be using c.max() instead c.all() if you want to get the max element of the array. This section of code should give you what you want:

First of all, you need to cast your data in feasible format: contributions.shape is (1, 6, 2). Having contributions[0] makes it easy to iterate with zip: