NLPMedium10 min25 pts

Spam Filter

Classify messages from six keyword counts. F1 is measured for the spam class, and predicting every message as ham scores 0.00.

Target

F1 ↑ 0.850

Baseline

0.000

Time

10 min

Points

Higher is better. Scored on 20 held-out rows.

Dataset

class 0
class 1

Each dot is one training row: feature 1 against feature 2, coloured by class.

Training split · 32 rows · 6 features. Scored by F1 on 20 held-out rows; target ≥ 0.85, baseline 0.

What this lab teaches

Turn keyword counts into a weighted sum and score the spam class with F1, where precision and recall both have to hold up.

Six counts, 32 rows

The training split is 32 rows of six keyword counts (each column ranges from 0 to 5) with 8 spam rows - every fourth training row is label 1. The hidden split is 20 rows with 5 spam. Counts are not centred, and their means differ across columns, so standardizing each column before fitting keeps one busy keyword from dominating every gradient step.

The F1 arithmetic you are judged on

Precision is TP / (TP + FP), recall is TP / (TP + FN), and F1 is their harmonic mean 2*TP / (2*TP + FP + FN), computed for the spam class only. Predicting all ham gives recall 0 and F1 0.00, which is the recorded baseline. Predicting all spam gives precision 5/20 and recall 1, for F1 = 0.40. With only 5 hidden spam rows, the 0.85 target allows exactly one mistake of either kind: 4 TP + 0 FP scores 0.889, but 4 TP + 1 FP drops to 0.80 and misses.

What a fitted classifier sees

Logistic regression on standardized counts separates the hidden rows cleanly in an internal run (F1 1.0 at threshold 0.5). The fitted weights are telling: roughly +3.1, +1.9, +1.5 on the first three keyword columns and -0.9, -0.9, -1.9 on the last three, so spam is high on the early keywords and low on the later ones. That is the kind of structure a keyword-count model is supposed to find.

Common pitfalls

Optimizing accuracy. Predicting all ham is 75% accurate on the training split and scores F1 0.00, which is also the baseline.
Fitting on raw counts. A column with large counts produces large gradients, and the same learning rate that suits it oversteps the small columns.
Judging on the training split. F1 1.0 on 32 rows can hide a threshold that misses one of the five hidden spam rows, and one miss plus one false alarm is already below target.

Rules of the run

Define predict(train_X, train_y, test_X) and return exactly one value per test row.
Pure Python or numpy both run; no external data or network access.
Return a spam probability or 0/1 label; values are thresholded at 0.5.
F1 is computed for the spam class (label 1).

Hint

Logistic regression on standardized counts works. Recall matters as much as precision, so do not default every message to ham.

Reference solution

Revealing doesn't affect your record — but try the hint first.

Timed run

10 min limit

25 points

not started · 10:00

The clock starts when you press start. Your predict(train_X, train_y, test_X) is then scored on 20 held-out rows within 10 minutes.

Starter codePython

def predict(train_X, train_y, test_X):
    # Baseline: predict the majority class.
    # Replace with your classifier, then return one value per test row.
    rate = sum(train_y) / len(train_y)
    label = 1 if rate > 0.5 else 0
    return [label for _ in test_X]

Tab inserts 4 spaces · ⌘+Enter runs your code