Making columns that were not there

Halfway through Step 4 a tree found October on its own, out of twelve month columns nobody explained to it. Then the rest of that step was about how little you could safely conclude from it.

This step starts from the other side of that. There are things it can only reach by building a staircase, however deep you let it go and however many columns you hand it, and they are not exotic. The commonest one is the gap between two numbers.

The two halves of this step

In the first half you meet a small table where the answer is exactly that: a gap. You watch a tree fail on it, then hand it one column you built yourself, and watch a single question get every person right.

In the second half you go back to the bank file and try the same trick four times, properly measured, and it does not work once. Then you work out why, which is worth more than the trick.

The two halves together are the actual skill: knowing which missing column is worth building, and being willing to find out that the answer is none of them.

What you actually do, in order

1. Plot twelve shops on a grid, by hand, and try to separate them with a straight line.
2. Open a small table with two dates in it and watch a tree struggle.
3. Build one column, and watch one question get everybody right.
4. Take the lid off and see what the tree was doing instead.
5. Build four columns for the bank file that ought to help.
6. Measure them the way Step 4 taught you, and watch all four fail.
7. Build the one your own table needs, and measure it the same way.

Paper and pen. You want a page with room for a square about the size of your hand.

A chain of shops. For each shop you know the month it opened, counted from the month the company started, and the month a particular sale happened, counted the same way. Some sales were big and some were not.

Here are twelve real rows out of the table you will open in Part 2.

Draw it

Draw two lines for the edges of a square, one along the bottom and one up the left.

Along the bottom is opened. Mark 0 at the left, then 20, 40 and 60, evenly spaced. Up the left is sale. Mark 0 at the bottom, then 40, 80 and 120, evenly spaced. Squared paper makes this easier, and the back of an envelope is fine.

Now put a mark for each of the twelve shops: across to its opened month, up to its sale month. Use a small circle for the big sales and a cross for the rest. Twelve marks in all. Being a few millimetres out will not matter.

You cannot do it with one line. The best line across gets 10 of the 12, and so does the best line up and down. Never better, either way.

Now break the rule. Draw one line at any angle you like.

A line running diagonally, up and to the right, puts all six circles above it and all six crosses below. Every one of the twelve, and with room to spare: you can be quite sloppy about the angle and still get all of them.

Here is the same thing drawn properly, so you can check yours. The left panel is the best straight-across line there is, with the two shops it gets wrong. The right panel is one line at an angle.

Two straight lines, one across and one up and down, get eleven of the twelve. Three get all twelve, which is worth knowing, because that is exactly what a tree with three questions would do. It is not that a tree cannot separate these twelve. It is what it costs, and what happens to the thirteenth shop it has never seen.

What you just found out about trees

Every question a tree asks is one number against one cut. On a grid, that is a line straight up and down, or straight across. Never at an angle.

More questions means more lines, so a tree builds a staircase alongside a diagonal. With enough questions and enough shops the staircase gets close. It never becomes the diagonal, and every extra step costs a question and covers fewer shops.

Open band-05.ipynb. Every cell on this page is already in it, in this order. You run them; you do not type them. The typing is in Part 7, on your own table.

If it is a while since you started JupyterLab: open a terminal, move into the course folder with cd, then type jupyter lab. Parts 6 and 7 of the setup page walk through it slowly.

import pandas as pd
from sklearn.tree import DecisionTreeClassifier, export_text
from sklearn.model_selection import train_test_split

stores = pd.read_csv("data/stores.csv")

print(stores.shape)
print(stores["big_sale"].mean().round(4))

A plain read_csv with no sep=";" this time, because this file uses commas like most of them do. The bank file was the odd one.

print(stores.head(5))

   opened_month  sold_month  big_sale
0            34         101         1
1            48         109         1
2             6          45         1
3            15          83         1
4            46         110         1

X = stores[["opened_month", "sold_month"]]
y = stores["big_sale"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=0)

flat = DecisionTreeClassifier(max_depth=2, random_state=0)
flat.fit(X_train, y_train)

print(round(flat.score(X_test, y_test), 4))

Better than saying yes to everybody, which scores 0.5667 on this particular test half. That is the 62.67 percent from two cells ago, measured on the 150 shops that were held back rather than on all 600. Better than lazy, then, and not good.

deeper = DecisionTreeClassifier(max_depth=3, random_state=0)
deeper.fit(X_train, y_train)
print(round(deeper.score(X_test, y_test), 4))

loose = DecisionTreeClassifier(random_state=0)
loose.fit(X_train, y_train)
print(round(loose.score(X_train, y_train), 4))
print(round(loose.score(X_test, y_test), 4))
print(loose.get_n_leaves())

Read the last three together, because you have seen this shape before. Perfect on the shops it learned from, 0.9467 on the ones it has not, and twenty endings to do it. That is Step 3's memorising tree, in a new place, and this time you can see exactly what it is memorising: a staircase, climbing beside a diagonal it cannot draw.

Now the thing you worked out on paper.

stores["months_open"] = stores["sold_month"] - stores["opened_month"]

print(stores["months_open"].min(), stores["months_open"].max())
print(stores.head(3))

One line. Subtract one column from another and pandas does it for all 600 rows at once, the same way multiplying a column by a hundred did in Step 2.

X2 = stores[["opened_month", "sold_month", "months_open"]]

X2_train, X2_test, y_train, y_test = train_test_split(
    X2, y, test_size=0.25, random_state=0)

one_question = DecisionTreeClassifier(max_depth=1, random_state=0)
one_question.fit(X2_train, y_train)

print(round(one_question.score(X2_train, y_train), 4))
print(round(one_question.score(X2_test, y_test), 4))

Every shop it learned from, and every shop it never saw. One question.

In Step 3 a perfect score was a warning. Here it is not, and there are two separate reasons, and you should have both.

The second reason is that nothing leaked, and here is the test, which is narrower than you might guess.

print(export_text(one_question, feature_names=list(X2.columns)))

|--- months_open <= 24.50
|   |--- class: 0
|--- months_open >  24.50
|   |--- class: 1

Two years. A shop that has been open more than two years makes big sales, and one that has not does not.

The tree threw away both of the columns it was given and used only the one you built. It was never going to find that rule in one question, and it found it immediately once somebody wrote it down.

Look at what the unlimited tree was doing instead. This is the first four questions of the twenty-ending tree from Part 2.

print(export_text(loose, feature_names=list(X.columns), max_depth=2))

max_depth=2 here is a limit on the printing, not on the tree. The tree is unchanged; you are just asking to see the top of it.

|--- sold_month <= 51.50
|   |--- opened_month <= 17.50
|   |   |--- sold_month <= 36.00

Read the first three cuts as lines on your paper grid. One across at 51.5, one up and down at 17.5, then another across at 36. Across, up and down, across, and on it goes: twenty boxes in all, each a small rectangle with a single answer in it.

That is a staircase. It follows the diagonal, roughly, and every extra step costs a question and covers fewer shops. Sorted by size, the twenty boxes hold 1, 1, 2, 2, 4, 5 shops and upwards. Two of them hold a single shop each. Step 2 put it this way: a rule built around one person is not a rule, it is a note about that person.

The general shape of it, stated carefully

It is not that a tree cannot get there. Given enough questions and enough rows it will build a staircase against almost any boundary, and Part 2 watched it climb to 0.9467 doing exactly that.

What is true is narrower and more useful:

• One question can only be a line straight up or straight across. Anything at an angle costs many questions.
• Questions are the thing you are short of. Every one you spend on a staircase is one you are not spending on something else, and the deeper you go the fewer rows each ending holds, which is Step 3's whole subject.
• The staircase is fitted to your rows. The diagonal is a fact about the world. A staircase drawn round your 450 shops will be drawn slightly differently round the next 450.

So the column does not tell the model anything new. It makes something cheap that was expensive, and cheap is what you are buying.

Two kinds of new column, and one of them is nearly always a waste

• Something one question can already say. Age in five bands. Balance rounded to the nearest hundred. A column that is another column times two. A tree can cut age anywhere it likes, so these hand it nothing it did not have, and they usually do nothing at all. Usually, not always: if your model is short of questions, a band can save it one, and then it helps for that reason rather than because you added anything.
• Something one question cannot. A gap between two columns. A ratio. A count of how many of five things are true at once. These are the ones that can pay, and Parts 5 and 6 are about how often they do not.

Ask which kind you are about to build before you write any code. It will save you most of the time people waste on this.

Back to the real file, and to the 47-column table you built in Step 4.

Here are four columns a sensible person would build for it. Each one is a real idea, and one of them is yours.

• peak_month: one for March, September, October and December, zero otherwise. This is your Step 1 rule, the four months you found by hand, written as a column.
• contacts_total: this campaign's calls plus the previous campaign's. How much this person has been bothered altogether.
• balance_per_year: money in the bank divided by age. A rough measure of how well somebody is doing for their age, which a bank might care about.
• contacted_before: one if this person was ever called in an earlier campaign, zero otherwise. The flag from Step 4's Part 8.

One is a sum and one is a ratio, which Part 4 says can pay. Two are things one question could arguably already say. Before you run anything, write down which is which, and which single one you would bet on.

bank = pd.read_csv("data/bank.csv", sep=";")
y_bank = (bank["y"] == "yes").astype(int)

plain = ["age", "balance", "day", "campaign", "pdays", "previous"]
many = ["job", "marital", "education", "contact", "month", "poutcome"]

ready = bank[plain].copy()
for name in ["default", "housing", "loan"]:
    ready[name] = (bank[name] == "yes").astype(int)

ready = pd.concat([ready, pd.get_dummies(bank[many], dtype=int)], axis=1)

print(ready.shape)

Step 4's Part 4, in one cell. Every piece of it is from that step: .copy(), the loop turning three yes-or-no columns into ones and zeros, get_dummies making 38 boxes, and concat joining them on sideways.

extras = ready.copy()

extras["peak_month"] = bank["month"].isin(["mar", "sep", "oct", "dec"]).astype(int)
extras["contacts_total"] = bank["campaign"] + bank["previous"]
extras["balance_per_year"] = (bank["balance"] / bank["age"]).round(2)
extras["contacted_before"] = (bank["pdays"] != -1).astype(int)

print(extras.shape)
print(extras["peak_month"].sum())

.isin([...]) is from Step 1, where you used it to write the wider rule. The division and the addition work a column at a time, like the subtraction in Part 3.

base_scores = []
extra_scores = []
wins = 0

for seed in range(30):
    a, b, c, d = train_test_split(ready, y_bank, test_size=0.25, random_state=seed)
    plain_tree = DecisionTreeClassifier(max_depth=3, random_state=0)
    plain_tree.fit(a, c)
    base_score = plain_tree.score(b, d)

    a, b, c, d = train_test_split(extras, y_bank, test_size=0.25, random_state=seed)
    extra_tree = DecisionTreeClassifier(max_depth=3, random_state=0)
    extra_tree.fit(a, c)
    extra_score = extra_tree.score(b, d)

    base_scores.append(base_score)
    extra_scores.append(extra_score)

    if extra_score > base_score:
        wins = wins + 1

print(round(sum(base_scores) / 30, 4))
print(round(sum(extra_scores) / 30, 4))
print(wins)

The same loop as Step 4's Part 6, with one thing changed: both models get three questions, so the only difference between them is the four new columns.

The four short names come out of train_test_split in its fixed order, and it is worth saying them out loud once more: a is the training columns, b the test columns, c the training answers, d the test answers. Short names here because they are used twice and thrown away.

Four thought-out columns, built from a real finding and two honest ratios, and the average went down. They won 5 of the 30.

I also ran them one at a time, because four columns failing together could hide one that helps. The averages, against the 0.8902 to beat: peak_month 0.8882, contacts_total 0.8902, balance_per_year 0.8901, contacted_before 0.8902. Bronze at the bottom asks you to reproduce that.

One of the four is doing all the damage, and it is the one you were most sure of.

This is the part that is worth the step, and it starts with a question you can put to the model directly rather than guessing at: did it even use the column?

new_names = ["peak_month", "contacts_total",
             "balance_per_year", "contacted_before"]

used = pd.Series(0, index=new_names)

for seed in range(30):
    a, b, c, d = train_test_split(extras, y_bank, test_size=0.25, random_state=seed)
    tree = DecisionTreeClassifier(max_depth=3, random_state=0)
    tree.fit(a, c)
    weight = pd.Series(tree.feature_importances_, index=extras.columns)
    used = used + (weight[new_names] > 0)

print(used.to_string())

Three new things, all small.

• pd.Series(0, index=new_names) makes four counters with names on them, all starting at zero. You met pd.Series in Step 1, where you made a column of False the same way.
• tree.feature_importances_ is one number per column, saying how much work that column did. The trailing underscore means the tree worked it out, like statistics_ in Step 4. Step 6 is about what the size of these numbers means. Today you only care whether one is zero, because zero means the tree never asked about that column at all.
• weight[new_names] > 0 gives four True or False answers, and adding those to the counters turns each True into a 1, the way .astype(int) has all course.

peak_month          29
contacts_total       1
balance_per_year     5
contacted_before     0

Not one of them is what I expected when I wrote this page, and the first line is the opposite of it. Take them in turn.

peak_month did not get ignored. It got used, and it did harm.

This is the one I got wrong when I first wrote this page, and it is more interesting than what I expected.

I assumed the tree would leave it alone, because it already had twelve month columns and had picked October out of them. It does the opposite. It reaches for peak_month in 29 of the 30 splits, and it reaches high, because lumping four good months together makes a bigger, cleaner-looking split than any single month can.

Print the tree at random_state=0 and you can watch the damage. In Step 4 the second question was month_oct, and under it sat the leaf that caught six people in October after the sixteenth. With peak_month available, the tree asks that instead, and then both leaves underneath say no. The leaf that was catching people is gone, and the catches drop from 28 to 23.

So the coarse column won the argument and then had nothing useful to say. That is worth remembering: a tree takes the biggest split it can see right now, and the biggest split now is not always the best three questions later. Nobody warns you about this, and it is why Step 12 exists.

One correction to something you may believe from Step 1, while you are here. October is not special. In October 37 of 80 calls ended in a yes. In December it was 9 of 20 and in March 21 of 49, which are the same kind of rate. October wins because it holds 80 calls to December's 20, so the tree can build a bigger, steadier leaf out of it. Four months are genuinely good, and the tree picks the one it can lean on hardest.

contacts_total and balance_per_year were the right shape and the wrong idea

These two are the kind Part 4 said to look for: a sum and a ratio, neither of which one question can say.

The tree barely touched them: contacts_total once out of thirty and balance_per_year five times, and the average did not move.

The reason is simpler than the shape. Whether somebody opens a savings account has very little to do with their total number of phone calls, or with their money divided by their age. The arithmetic was legal. The idea was wrong.

A feature is a guess about how the world works, written as a column. Getting the shape right does not make the guess right, and no amount of care with the shape will rescue a guess that is not true.

contacted_before was already in the file three times over

Zero out of thirty. That bottom line is what a column with nothing whatever to add looks like, and Step 4 half told you already: it said contacted_before "never won a place in the top three questions". Now you have watched it lose thirty times.

It is the same 816 people, three times. pdays is -1 for exactly the other 3,705, so a cut anywhere between -1 and 1 makes the same two groups. poutcome_unknown, one of your 38 boxes, marks exactly the same 3,705, not almost. And previous is 0 for exactly those people too, so previous <= 0.5 is a third way to say it.

Three columns already said it in one question each. You built a fourth.

One column here was engineered, and somebody else did it

Look at pdays again. Days since this person was last contacted. That is not something the bank's computer recorded on the day. It is a subtraction between two dates that somebody did before they gave you the file, which makes it the one hand-built gap in the whole table.

Hold it to the same standard as your four.

without_pdays = ready.drop(columns=["pdays"])

scores = []
for seed in range(30):
    a, b, c, d = train_test_split(without_pdays, y_bank, test_size=0.25, random_state=seed)
    tree = DecisionTreeClassifier(max_depth=3, random_state=0)
    tree.fit(a, c)
    scores.append(tree.score(b, d))

print(round(sum(scores) / 30, 4))

.drop(columns=["pdays"]) hands back a copy of the table with that column missing. It leaves ready alone, the way .copy() does.

So somebody did the job you are learning, on the right column, for the right reason, and on this file it did not pay either. That is the honest shape of this work, and it is why the measuring matters more than the ideas.

No code here and no answer at the end. Your own table, the one you signed up in Step 0 and got into shape in Step 4.

The brief

Build the columns your table needs, decide which of them are worth anything, and prove it with thirty splits rather than one.

It is done when

• You have written down, before touching any code, three columns you think your table is missing, and one sentence each on why somebody who knows the subject would expect them to matter.
• For each of the three, you have said whether your model could already express it. If it could, you have said so and not built it.
• You have built the ones that survived that test, and measured them with the thirty-split loop from Part 5, one at a time and then all together.
• Your README holds one average and one win count for each column on its own, plus one pair for all of them together. Three new columns means four averages and four win counts, and the ones that lost are the point.
• You have written down one column your table would need and cannot have, and what somebody would have to start recording for you to get it. This one matters more than the rest of the part.

Working alone

If the table is about work you do, you are the person this step keeps telling you to go and find. Sit down with your three ideas before you build them and answer two questions in writing: what would I look at, and what do I wish somebody had recorded?

The second question is the one that pays, and it is the fifth item on the list above. Domain knowledge is not a thing other people have. It is what you know about this data that the table does not say.

If you know somebody else who works with this data

Ask them both questions before you build anything, and write their answer down word for word rather than in your own words. The wording is where the column hides.

Add to the README you started in Step 0.

• The shop table: what the tree scored on two columns, and what one question scored on three.
• Your sentence from Part 0, the one you wrote before you ran anything, and whether it was right.
• The four bank columns, the two averages, the win count, and one line on why they failed.
• Your own three columns, what each of them measured, and which survived.
• The column your table cannot have, and who would have to start recording what.

Six questions. Answer them in writing before you open the answers.

One counts more than the other five. You can get five right and still not pass this step.

Two of the answers below offer half marks. Half is not a pass on the marked one: if you got half there, read the answer, then come back tomorrow and write it out again from memory before you go on. Half on any of the other five is a pass.

1. A tree with no limit scored 0.9467 on the shops. One question on a column you built scored 1.0. What did the new column give the tree that it did not have before?
2. Your model is a tree. You replace age with five age bands and the score does not move at all. A colleague says the bands must be wrong. What do you say?
3. contacts_total is a sum of two columns, which is exactly the shape Part 4 said to look for, and it did nothing. Why not?

4. A colleague has a table of hospital visits, one row per visit. It holds admitted_on and discharged_on as dates, the ward, the patient's age, and twenty other columns. She is predicting whether a patient will be readmitted within a month. Her tree scores badly and she asks you which of three new columns to build.

   1. The length of stay in days, from the two dates.
   2. Age in five bands instead of the raw number.
   3. The share of visits to that ward that ended in a readmission, worked out across her whole table.

   For each of the three, say whether it can help her and why. One of them has almost nothing to offer a tree, and one of them is not safe to build the way she has described it.

5. The bank file has a day and a month and no year. Name one column you would build if it had a year, and say what a tree would do with it that it cannot do now.
6. You build a new column and your score improves by half a point on one split. Name the two things you do before you tell anybody.