The table is not the truth

By the end of this step you will be able to do one thing, and it is a thing most people who work with data cannot do. You will be able to open a table you have never seen before and work out what is wrong with it, before you trust a single number in it.

That is worth saying plainly, because from the outside this step looks like it is not teaching you machine learning at all. There is no model here. You train nothing. You read code rather than write it.

Why the course starts here and not with a model

A model is a machine for finding patterns. It cannot tell whether a pattern is real. Feed it a broken table and it will find the broken pattern, print a good score, and nobody will notice for months.

In Part 3 a number is going to lie to you. It has two decimal places. It looks exactly like every number you have ever trusted. It means nothing at all. Once that has happened to you once, you check. For the rest of your working life, you check.

What you actually do, in order

1. Fill in a twelve-line form you cannot honestly answer, on paper, and make the fault this step is about with your own hand.
2. Open a real table of 4,521 bank customers and count what is in it.
3. Take an average, believe it, then find out it was nonsense.
4. Ask the standard tool for finding missing values what is missing, on a table you already know has a hole in it, and read its answer carefully.
5. Learn to tell a fault from a fact. This is harder than it sounds and it is mostly judgement.
6. Find the thing about this table that can never be fixed by anybody.
7. Sign up a table of your own, about something you care about, and carry it through the twenty-six steps after this one.

What you need before you start

Python and JupyterLab, on your own laptop. If you have never installed either, or do not know what they are, stop and do the setup page first. It assumes you have never installed anything in your life, and it takes about an hour.

You do not need to know how to code. Every line up to Part 7 is written for you and you only run it. In Part 7 you write four short lines of your own, about a table you choose.

Read this whole part first. Then do it with a pen. The laptop stays open only so you can read the questions; everything you write goes on paper.

Here is why. Later today a number on your screen is going to fool you. This is so that when it does, you have already made one of those numbers yourself, with your own hand, and know exactly how it happened. Ideas about data are slippery. Your own handwriting is not.

You need one sheet of paper, a pen, and your phone face down where you cannot read it. Write the numbers 1 to 12 down the left of the sheet.

Round one: the honest form

You are registering for something. A new job's paperwork, an account, a club. Twelve questions, and this round has one rule: answer only what you actually know, from memory. Nothing looked up, nothing worked out, no phone. If you do not know, leave the line blank and move on.

Give yourself about twenty seconds a line. Do not go back.

1. Your full name, spelled as it is on your passport or driving licence.

2. Your date of birth.

3. The full address where you live, including the postcode.

4. The day, month and year you moved in there.

5. Your height in centimetres, to the nearest centimetre.

6. Your shoe size.

7. The full name of somebody to contact in an emergency.

8. That person's phone number. From memory. The phone stays face down.

9. A second emergency contact, and their number.

10. The month and year you started your current job, course or main occupation.

11. The full registered name of that employer or school. Not what everybody calls it. The name on the paperwork.

12. Its main switchboard number.

Count your blanks. Write the number at the bottom of the sheet.

Round two: the real form

Now the rule every real form actually has. It will not be accepted with a blank on it. There is no box marked "I do not know" and there is nobody to ask.

Go back to every line you left empty and put something on it. Not a joke and not a refusal. Something that would get the form accepted: a plausible number, a round figure, a date that looks like a date, a name you can half remember. Whatever you would really write at half past four on a Friday to get the thing handed in.

Do it now, before you read on. It takes two minutes and the whole step turns on it.

Five words. Read them once. Then write each one in your own words in your notebook.

Open band-00.ipynb. It sits in the course folder, in the file list down the left of JupyterLab.

If none of that means anything yet, stop and do the setup page first. It takes about an hour and twenty minutes and it assumes you have never installed anything in your life.

Then run the cells from the top. Do not skip any.

1. Load the file.

   import pandas as pd
   
   df = pd.read_csv("data/bank.csv", sep=";")
   
   print(df.shape)

   Five small things are happening. Read them once now. You will see them in every step from here on.

   • import pandas as pd brings in pandas, the library that handles tables, and gives it the short name pd so you do not have to type the whole word. Every notebook in this course brings in pandas in its first cell, so you will type or run this line more than any other.
   • pd.read_csv means ask pandas to read a file of this kind. CSV is the plainest way to save a table.
   • sep=";" tells it that this file separates its columns with a semicolon. Most files use a comma, so most of the time you leave this off. Leave it off here and all 17 columns get squashed into one.
   • df is the name we are giving the table. It is short for data frame, which is what pandas calls a table. Everybody in this field calls it df, so you may as well start now.
   • Coming up in a moment you will see df["age"]. That means the column called age, out of the table called df. The square brackets mean reach inside. The quotes mean this is a name I am typing, not something for Python to go and look up. The name has to match the column exactly, capital letters and all.

   Nothing here is yours to invent yet. You are reading working code and running it, which is the whole of Step 0. You start writing in Step 1.

   You should see

   (4521, 17)

   That is 4,521 rows and 17 columns. If you see (4521, 1) you left out the sep.

   If you get ModuleNotFoundError: No module named 'pandas'

   Pandas is not installed where this notebook is running. Put %pip install pandas in a cell and run it. The % matters. Then restart the kernel and run from the top.

   If you get FileNotFoundError

   Python looked for data/bank.csv and it was not there. That means JupyterLab was started somewhere other than the course folder.

   Save your notebook. Close the browser tab. Go to the terminal window you started Jupyter from and press Control and C together to stop it. Then follow Part 6 of the setup page again, so your terminal is standing in the course folder, and start jupyter lab from there.

   Do not move the notebook to a different folder to make the error go away. It will come back in Step 1.

2. Name the columns.

   print(list(df.columns))

   You should see

   Seventeen names, starting age, job, marital, education.

   Read all seventeen out loud. You will meet pdays in Part 3 and it is the reason this step exists.

3. Look at three rows.

   df.head(3)

   You should see

   Three rows and 17 columns. Row 0 begins 30 unemployed married primary. Row 2 begins 35 management single tertiary.

   If your first row is not the 30 year old, you have opened a different file. Check step 1.

4. Take an average that is honest.

   print(round(df["age"].mean(), 2))
   print(df["age"].min(), df["age"].max())

   You should see

   41.17, then 19 87

   The youngest is 19 and the oldest is 87. An average age of 41.17 sits between them and makes sense. Hold on to that feeling. Part 3 takes it away.

There is a column called pdays. Going by the name, it holds how many days ago this person was last called. Hold that loosely: Part 3 is about to show you that a column name is a claim, not a fact.

print(round(df["pdays"].mean(), 2))

Now look at the column instead of the average.

print(df["pdays"].min(), df["pdays"].max())
print((df["pdays"] == -1).sum())

Do not guess what minus one means. Go and read.

notes = open("data/bank-names.txt").read()

for line in notes.split("\n"):
    if "pdays" in line:
        print(line)

That opens the notes file, cuts it into lines, and prints only the lines with the word pdays in them. You are not expected to be able to write that yet. You are expected to run it and read what comes back.

Now take the average again, this time of the people it actually applies to.

called_before = df["pdays"] != -1

print(called_before.sum())
print(round(df.loc[called_before, "pdays"].mean(), 2))

First, one new thing you will use constantly. To see what is actually inside a column of words, attach .value_counts() to it.

print(df["job"].value_counts())

Now, pandas has a tool for finding missing values. Run it.

print(df.isna().sum().sum())

Now count the word unknown in every column that has it. This next one is a loop. It does the same thing once for each column, and only prints when it finds something.

for name in df.columns:
    hits = (df[name] == "unknown").sum()
    if hits > 0:
        print(name, hits)

has_unknown = (df[["job", "education", "contact", "poutcome"]] == "unknown").any(axis=1)

print(has_unknown.sum())

You are now looking for problems, so you will start seeing them everywhere. Some of them are not problems. This part is how you tell.

print(df["balance"].min(), df["balance"].max())
print((df["balance"] < 0).sum())

Is that broken data? No. It is an overdraft. People really do owe the bank money. This one is fine.

print(df["campaign"].max())
print((df["campaign"] > 10).sum())

Is that broken data? Probably not. It is probably true, and that is worse. Stop and picture the person who answered the phone on the fiftieth call.

You will build things that decide who gets rung. Write that number down now, while it still bothers you.

There is a column for the day of the month and a column for the month.

print(df["day"].min(), df["day"].max())
print(sorted(df["month"].unique()))

Now look for a year.

print([name for name in df.columns if "year" in name])

The notes say the full file runs from May 2008 to November 2010. So two rows saying 5 May could be a year apart, or two years apart, and nothing in the table can tell you which came first.

So you cannot put these calls in order. You cannot ask whether the bank got better over time. You cannot split the old calls from the new ones.

This part has no code in it and no answer at the end. It is also the most important thing you will do in this step, because the table you pick here is the table you carry through every step after it.

Find a table about something you care about

It must be about a real thing near you. Your work, your town, your hobby, your family's business. Not a famous practice file.

Good places to look: a spreadsheet somebody at your work already keeps, an open data site run by your city or government, a sports league's published results, the export button on an app you use every day.

Two things you settle before you run anything

1. It is a CSV file, which means the file name ends in .csv. If yours is a spreadsheet ending in .xlsx, open it, choose File then Save As, and pick CSV from the list of formats. In Google Sheets it is File, then Download, then CSV. A CSV holds the same table with the colours and the formulas replaced by the values they worked out, which is all you need.
2. You can open it and scroll the whole thing. If you cannot, cut it down until you can. A table you have never looked at is not a table you can find three faults in.

Then the checker tests six more

These are the six it runs, in the order it prints them, and the page and the script agree because I went and read the script.

1. Between 200 and 50,000 rows. Fewer than 200 and you cannot learn anything. More than 50,000 and the later steps get slow on a plain laptop.
2. The column you named is really in the file. Spelling counts, and this is the one people fail first.
3. At least four other columns to guess from.
4. The column you want to guess holds at least two different answers, and no more than twenty. Twenty different answers is the point where you are predicting a number rather than choosing between answers, and the early steps need choosing.
5. At least 30 rows hold the less common answer. Fewer and your scores jump around and mean nothing.
6. You are allowed to have it. If it is about named people, somebody has to have said yes in writing. No script can test that, so you say it out loud by adding --i-may-use-this at the end.

python3 tools/check_my_table.py path/to/your.csv name_of_the_column_to_guess --i-may-use-this

Then write the one page

One page. Not two. It says:

1. What each row is. One row is one what?
2. Where it came from, who collected it, and why they bothered.
3. How many rows and how many columns.
4. What you want to guess, and how often each answer comes up.
5. Three things wrong with it. Named, with the number of rows affected for each.

Make a folder for your own work. Put three things in it.

1. Your table, as a file. The one that passed the checker.
2. Your one page, as a text file called README.md. A README is just a plain text file saying what is in a folder. The .md stands for Markdown, which is plain text with a few symbols in it for headings and lists. The easiest way to make one: in JupyterLab's file list, click the plus button, choose Text File, type your page, then right click the file and rename it to README.md.
3. Your notebook, with the cells you ran to find the three faults still in it.

Keep the notebook even though it is messy. It is proof that you looked, and every step after this one adds to it.

Six questions. Answer them in writing, in your notebook. Do not open the answers until all six are written down.

One of them counts more than the other five. You can get five right and still not pass this step.

1. What does df.shape tell you, and in what order?
2. df.isna().sum().sum() prints 0. Name two ways the table could still be full of holes.
3. A column of prices has a minimum of 0. Is that a fault or a fact? What would you do to find out?

4. Your file has 4,521 rows. You print the average of pdays and get 39.8.

   Then you notice 3,705 of those rows hold -1, and the file's notes say -1 means the client was never called before.

   What was your 39.8? Say what it is an average of.

5. Why is a negative bank balance fine and a negative number of days not?
6. The table has a day and a month but no year. Name one question you can no longer answer.