Cheating, on purpose

By the end of this step you will never again trust a score without asking one question first: which people was this measured on?

That sounds small. It is the difference between a model that works and a model that embarrasses you in front of the people who paid for it.

The thing that happens in Part 3

In Step 2 your unlimited tree scored 0.9998. It found 520 of the 521 people who said yes and wasted no calls at all.

You cannot check that tree. It was shown all 4,521 people, so there is nobody left to test it on. That is the first thing to understand: the problem is not that the number is high, it is that no honest number exists for it.

So in this step you keep a quarter of the people back and train the same kind of tree on the other three quarters. It scores 0.9997 on the people it learned from and 0.7975 on the quarter it never met.

Saying no to every single person would have scored 0.878. Your near-perfect model does worse than a rule a child could write on a napkin.

Then you cheat three times, on purpose

The first cheat is loud. You leave the answer sitting in the table and score a flat 1.0 on people the tree has never seen. Nothing goes red. Nothing warns you.

The second is quiet, and it is the one that costs real people their jobs. You add a column that is allowed to be there, that is not the answer, that no check will ever complain about, and your score goes up. You will only catch it by asking a question about the world, not about the data.

The third attacks the split itself. You put every person in the file twice, change nothing else, and watch the same broken tree look respectable. That one is the marked question at the end.

What you actually do, in order

1. Memorise eight real people on paper, score full marks, then fail on four new ones.
2. Cut the file in two and check the two halves.
3. Watch the big tree fall.
4. Watch the small tree hold.
5. Leave the answer in the table, and see what a model does with it.
6. Add a column that is not the answer, and score higher than you have ever scored.
7. Enter every person twice, and score the same broken tree again.
8. Split your own table and get your first honest number from it.

Paper and pen. Laptop shut.

Here are eight real people from the file. Two things about each, and whether they said yes.

First, try to find a rule

Spend three minutes looking for one. Is it the young ones? The rich ones? Read the ages down the page: no, no, yes, no, yes, no, yes, yes. Now read the money in order of size: 480 yes, 574 no, 701 yes, 751 no, 2262 no, 3215 no, 7195 yes, 8556 yes.

There is no single cut of either column that gets more than six of these eight right. Check that yourself before you believe it.

Now stop looking and just memorise

Take two minutes and learn the eight answers by heart. No, no, yes, no, yes, no, yes, yes.

Cover the table with your hand. Write the eight answers from memory.

You will get eight out of eight. Full marks. A perfect score.

You have just done, with your hands, what your unlimited tree did in Step 2. It had 709 endings for 4,521 people. That is about six people per ending, and many endings held exactly one person.

An ending built around one person is not a rule. It is a note about that person.

Six words. You will hear all six in any job interview for this work.

Open band-03.ipynb. Every cell on this page is already in it, in this order. You run them; you do not type them. The only typing you do in this step is in Part 8, on your own table.

The first cell rebuilds everything you had at the end of Step 2: said_yes_before is 1 for the people who said yes to the last campaign and 0 for everybody else, X is the seven columns the model is allowed to look at, and y is the answer, 1 for yes and 0 for no.

Two things about the imports. The first line brings in two names at once, separated by a comma, instead of the two separate lines you wrote in Step 2. Same effect, less typing. The second line is new: scikit-learn keeps its tools in labelled drawers, and model_selection is the drawer for deciding which rows a model gets to see. That is where the splitter lives.

import pandas as pd
from sklearn.tree import DecisionTreeClassifier, export_text
from sklearn.model_selection import train_test_split

df = pd.read_csv("data/bank.csv", sep=";")
df["said_yes_before"] = (df["poutcome"] == "success").astype(int)

cols = ["age", "balance", "day", "campaign",
        "pdays", "previous", "said_yes_before"]

X = df[cols]
y = (df["y"] == "yes").astype(int)

print(X.shape, y.sum())

Now the cut.

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=0)

print(X_train.shape, X_test.shape)
print(y_train.sum(), y_test.sum())

Four new names on the left of one equals sign. train_test_split hands back four things at once, in that fixed order: the training columns, the test columns, the training answers, the test answers. Swap two of those names and everything below will run and be wrong, so copy the order exactly.

That line ends in an open bracket and carries on underneath. Python allows this: while a bracket is open, the line has not finished. The setup page warned you that a bracket you forget to close is an error, and it still is. A bracket you close on the next line is not. The indent on the second line is only there to make it easy to read.

test_size=0.25 means a quarter of the rows go into the test pile. random_state=0 means the rows are shuffled the same way every time you run it. That is why your numbers should match the ones on this page. If a future version of scikit-learn ever changes how it breaks a tie, they could drift by a digit, and the numbers in requirements.txt are the ones every figure here was produced with.

Before you measure anything, work out what the score to beat is. On the test pile only.

print(round((y_test == 0).mean(), 4))

This is the lazy rule from Step 1, aimed at the test pile: say the commonest answer to everybody and never think again. (y_test == 0) is True for everybody who said no. Taking the mean of a column of True and False gives you the share that are True, because True counts as 1 and False counts as 0.

The , 4 inside round is how many figures after the dot you want to keep. Every score on this page is rounded to four, so if you drop it you will see a number twenty digits long and think something has gone wrong. Nothing will have.

Now the thing this step exists for.

big = DecisionTreeClassifier(random_state=0)
big.fit(X_train, y_train)

print(round(big.score(X_train, y_train), 4))
print(round(big.score(X_test, y_test), 4))

Look at what changed from Step 2. You train on X_train, y_train. You score twice, and the second scoring hands it X_test, y_test, which the tree has never met.

Read those two numbers again.

On the people it learned from: 0.9997, which is as near perfect as makes no difference.

On the people it has never seen: 0.7975.

The bar was 0.878. Your model is a long way under it. On this measure, saying no to every single person, without a computer, without a course, without any of this, beats what you just built.

Now find out what that actually costs.

guess = big.predict(X_test)

print(((guess == 1) & (y_test == 1)).sum())
print(((guess == 1) & (y_test == 0)).sum())

Why it happened

The tree kept asking questions until every one of the 3,390 training people sat alone or nearly alone in their own ending. Print it if you like: the first five questions down the left-hand side are said_yes_before <= 0.50, then age <= 60.50, then pdays <= 401.00, then day <= 1.50, then balance <= 19.50. It goes 31 questions deep in places and finishes with 529 endings.

That wall is real. It is also about one person, not about people. The next person who arrives is on the wrong side of a wall that was built for somebody else.

This has a name and you now own it: overfitting. The tree learned the rows instead of the pattern.

Same split. Same data. Two questions instead of thirty-one.

small = DecisionTreeClassifier(max_depth=2, random_state=0)
small.fit(X_train, y_train)

print(round(small.score(X_train, y_train), 4))
print(round(small.score(X_test, y_test), 4))

The two numbers are almost the same. That tells you one specific thing: this tree is not memorising. It does about as well on strangers as it did on its own homework.

It does not tell you the tree is good, or honest, or worth shipping. Part 5 hands you a model with no gap at all between the two numbers that is completely worthless. A small gap rules out one fault. It is not a certificate.

It is also above the 0.878 bar, which the big tree was not.

guess_small = small.predict(X_test)

print(((guess_small == 1) & (y_test == 1)).sum())
print(((guess_small == 1) & (y_test == 0)).sum())

Put the two side by side

The big tree finds three more people. It burns 111 extra phone calls to do it.

If a call costs your bank five euros, the big tree spent 555 euros to find three extra customers. Somebody has to decide whether that was worth it, and that somebody is you, and you cannot decide it from a score alone.

One warning about those three, and it is the habit this whole step is for. Three is what this split gave you, and it happens to be the smallest gap of the thirty splits I ran: the usual gap is about thirteen customers for about a hundred wasted calls. The trade is the lesson. The exact price is not.

You have just seen a split catch a bad model. Now watch a split fail to catch a worse one.

You are going to put the answer itself into the table of columns, as if you had copied it in by mistake. People do this every week, and it is one of the two standard ways a real project produces a beautiful number that means nothing. Part 6 has the other one.

X_leak = X.copy()
X_leak["answer"] = y

L_train, L_test, y_train, y_test = train_test_split(
    X_leak, y, test_size=0.25, random_state=0)

leaky = DecisionTreeClassifier(max_depth=2, random_state=0)
leaky.fit(L_train, y_train)

print(round(leaky.score(L_train, y_train), 4))
print(round(leaky.score(L_test, y_test), 4))

X.copy() again, so that the table you have been using is left alone. Then one new column called answer, holding exactly the thing you are trying to guess.

In the printing line further down you will see cols + ["answer"]. A plus between two lists joins them end to end and hands back a new list of eight names. It does not change cols, which still holds seven and is used again in Part 7.

The split runs with the same random_state=0, so the same people go to the same side as before. y_train and y_test come out holding exactly what they held before. That is the whole point of random_state.

A flat, perfect score on 1,131 people it has never met.

Nothing turned red. No warning appeared. The split, the thing that caught the bad model in Part 3, did not make a sound here.

Now look at it

print(export_text(leaky, feature_names=cols + ["answer"]))

|--- answer <= 0.50
|   |--- class: 0
|--- answer >  0.50
|   |--- class: 1

You gave it seven columns of real information and one column of the answer, and it threw the seven away. Which is what it is built to do. It tries every cut of every column and keeps the best one, and nothing beats the answer.

How you catch it

Two habits, and you already have both of them.

1. Print the tree, the way you just did. If one column is doing all the work, look hard at that column.
2. Take that column out and run it again. Taking a column out means making a shorter list and passing that instead, the mirror of the plus you just met: seven_minus = ["age", "balance", "day", "campaign", "pdays", "previous"], then train on X[seven_minus].

That last one was easy to spot once you looked. This one is not, and this one is the reason this step is in the course.

The file has a column that has never been in your seven: duration. It is how many seconds the phone call lasted. If you did Step 1's Silver task you met it once and were asked to write down why the bank could never use it. This is that, with numbers.

It is not the answer. Nobody copied it in by mistake. It was collected honestly, it is in the file the bank shipped, and every check you have ever run would let it through.

cols_d = cols + ["duration"]

D_train, D_test, y_train, y_test = train_test_split(
    df[cols_d], y, test_size=0.25, random_state=0)

deep = DecisionTreeClassifier(max_depth=3, random_state=0)
deep.fit(D_train, y_train)

print(round(deep.score(D_test, y_test), 4))

guess_d = deep.predict(D_test)
print(((guess_d == 1) & (y_test == 1)).sum())
print(((guess_d == 1) & (y_test == 0)).sum())

cols + ["duration"] makes a new list of eight names without touching cols. max_depth=3 gives it one more question than the small tree had, which it needs to use the new column properly.

0.9072 on strangers. That is the best honest-looking number in this course so far. It beats the 0.878 bar properly, and it finds 64 of the 138 instead of 22.

Being fair to the thing you are about to disqualify: that is the friendliest of thirty splits for it. Over all thirty it averages 0.8933 and 41 catches, so the honest headline is "about twenty more customers", not forty-two.

It is not free. Wasted calls go from 5 to 31, so by the five-euro arithmetic in Part 4 you spent 130 euros to find 42 more customers on this split. That is a good trade, and you should be able to say so with the numbers in your hand rather than by pointing at the score. In a real job you would be tempted to send this in an email with an exclamation mark in it.

There isn't one. The call has not happened. Its length is not a fact about the customer, it is a fact about a call you have not made.

Long calls and yes answers do go together. Which one causes the other is a good argument to have over lunch and it does not matter here. What matters is the order in time: the length exists only after the call, and you have to choose who to ring before it.

The column is real. The number is real. The model is worthless, because on Monday morning the column is empty for every person you care about.

This is leakage, the fifth word in Part 1, and now it has a face.

Nobody hid it. Open data/bank-names.txt, the notes that came with the file, and just above the list of columns there is a heading: related with the last contact of the current campaign. Four columns sit under it. Whoever wrote that heading was telling you exactly which columns belong to the call itself. That is why Step 0 made you open the notes.

One more, and this one attacks the split itself.

First, check the file you have.

print(df.duplicated().sum())

.duplicated() marks every row that is an exact copy of a row further up. Summing it counts them.

twice = pd.concat([df, df], ignore_index=True)

print(twice.shape)
print(twice.duplicated().sum())

pd.concat stacks tables on top of each other. Handing it [df, df] stacks the file on itself, so every person now appears twice. ignore_index=True renumbers the rows from 0 so there are no repeated row numbers.

X2 = twice[cols]
y2 = (twice["y"] == "yes").astype(int)

X2_train, X2_test, y2_train, y2_test = train_test_split(
    X2, y2, test_size=0.25, random_state=0)

big2 = DecisionTreeClassifier(random_state=0)
big2.fit(X2_train, y2_train)

print(round(big2.score(X2_train, y2_train), 4))
print(round(big2.score(X2_test, y2_test), 4))

0.7975 became 0.9447. The tree is exactly as bad as it was. The data is exactly the same data. Nobody added a single new fact.

There is a tool for exact copies, and it is worth meeting now.

print(twice.drop_duplicates().shape)

.duplicated() marks the copies. .drop_duplicates() throws them away and hands back the table without them. It matches on the whole row: every value in every column has to be identical.

small2 = DecisionTreeClassifier(max_depth=2, random_state=0)
small2.fit(X2_train, y2_train)

print(round(small2.score(X2_test, y2_test), 4))

Do not read too much into that one number, and you already know why. Run the same comparison over thirty splits and the small tree gains nothing on average: it goes up on fifteen of them and down on the other fifteen. The big tree goes up on all thirty, by about fourteen points every time.

So the rule is not "duplicates raise your score". It is narrower and more useful than that. Copies pay off only for a model that can memorise, because memorising is the only way to profit from seeing the same person twice. The bigger the model, the more it gains. Yours gained fifteen points and went from useless to respectable-looking.

What happened

The split shuffles rows, not people. When a person appears twice, one copy can land in the training pile and the other in the test pile.

The tree memorises the copy it trained on. Then you test it on the twin, and it recognises it perfectly, because it has seen that exact person before.

About three quarters of the test pile has a twin sitting in the training pile. Your test set is not a test. It is most of the homework again, handed back in a different order.

No code here and no answer at the end. Use the table you signed up in Step 0 and modelled in Step 2.

The brief

Split your own table, train your depth 2 tree on the training half only, and get your first honest number.

It is done when

• You have four things with sensible names, from one train_test_split with random_state=0, and you have printed the shapes to check the split did what you think.
• You know the bar on your own test pile: what saying the commonest answer to everybody would score. Not Step 1's baseline, which was the whole file. The bar on your test half. You already have the tool for this from Step 0: run value_counts() on your test answers, take the biggest count, and divide it by the number of rows in the test half.
• You have your depth 2 tree scored twice, on the training half and on the test half, and both numbers are written in your README next to each other.
• You have an unlimited tree scored the same two ways, so you can see the gap open up on your own data.
• You have gone through your columns one at a time and asked the Part 6 question of each: would I know this, for a new person, at the moment I have to answer? Write the list. Mark every column you are not sure about.
• Anything you marked, you take out and run again, and you write down what the score did. Taking a column out means writing a shorter list of names and passing that one instead, the way Part 5 showed you.
• You have run .duplicated().sum() on your own table. If it is not zero, print a few of them with mine[mine.duplicated(keep=False)].head(10) and work out whether they are the same person twice or two people who happen to match. keep=False marks both copies rather than only the second, which is what you want when you are looking at them rather than deleting them.

Working alone

Somebody who works in the area your data comes from is the person to ask: a nurse for hospital data, a pharmacist for prescriptions, a shopkeeper for sales, a teacher for school data. If your table is about work you do, that person is you, and this is one of the few advantages you have over somebody who came to this from computing.

Go down your column list and ask one question of each: would I know this before the thing happens? Write the answers down in your own hand. You are looking for the one you hesitate over, and the hesitation is the finding.

If you know somebody else who works with this data

Ask them the same question, column by column, and write down what they say rather than what you expected them to say.

Add to the README you started in Step 0.

• Your table from Part 4: big tree and small tree, score, found, wasted, on the test pile.
• Which of the two you would hand over, and what the extra customers cost.
• What duration did, and the sentence you would say to the manager.
• Your own table's two scores, training and test, for both trees.
• Your column list from Part 8, with the ones you marked and what happened when you took them out.
• One line: what you wrote in Part 5 of Step 2, and whether this step agreed with you.

Six questions. Answer them in writing before you open the answers.

One counts more than the other five. You can get five right and still not pass this step.

1. Your tree scores 0.9997 on the training pile and 0.7975 on the test pile. Which of those two numbers do you put in an email, and why the other one is worthless.
2. The small tree found 22 of 138 and the big tree found 25. Why is the small tree still the better model?
3. You add a column and your test score jumps from 0.89 to 0.99. Name the first two things you do, in order.

4. A colleague splits her rows, trains on the training rows, and scores 0.91 on the test rows. She shows you the code and the split is done correctly.

   Then she finds that every customer was entered into the system twice, once with their name misspelled, so the two rows are not exact copies of each other.

   Is her 0.91 still honest? Say what is happening to it, and say what she should do. Then say why drop_duplicates() does not fix it.

5. duration is a real column, honestly collected, and it raises the score. In one sentence, what is wrong with using it?
6. You split by rows. Give one example, from any kind of work, where a person can end up on both sides of that split.