Table of Contents

Error Analysis

General

Fehlklassifizierte Tupel betrachten, prüfen

Schwierige Beispiele identifizieren.

Skewed classes

Problem:

99% Accuracy, aber nur 0,5 % der Fälle true. Immer false, würde bessere Accuracy bringen.

Andere Evaluationsmetrik:

Precision/Recall

y = 1, wenn true

Predicted | 1 | True positive | False positive class | 0 | False negative | True negative

Precision: Anteil der tatsächlichen true positives von allen als true vorhergesagten.

$\frac{TP}{TP + FP}$

Recall: Von allen Patienten, welchen Anteil wurde korrekt als true erkannt?

$\frac{TP}{TP + FN}$

Für y = 0 ⇒ Recall würde 0 sein.

Precision and recall Tradeoff

Angenommen, $h_\theta(x) >= 0.7$ anstelle von 0.5 und h_\theta(x) < 0.7 Dann hohe Präzision, niedrigerer Recall

Und umgekehrt wenn z.B. $h_\theta(x) >= 0.3$

$F_1$ Score

Zum Vergleich von Precision/Recall.

Durchschnitt: $(P+R)/2$ nicht gut, da es möglich ist immer 1 oder 0 zu tippen.

$F_1$ Score : $2 * \frac{P*R}{P+R}$

Bias / Variance

Plot: Error / Degree of Polynom (with Training and cross validation error)

Regularisierung

Strategie: Increase regularization parameter stepwise (x2), and check what leads to lowest CV error. Then check for test set.

Learning Curve

Plot: Error/m (training set size)

High bias:

Wenn von High bias betroffen, dann helfen mehr Trainingsdaten i.d.R nicht.

High variance:

Wenn von High bias betroffen, dann helfen mehr Trainingsdaten i.d.R.

Basic recipe for ML

  1. High Bias:
    • Additional features
    • Additional polynomial features
    • Decrease Lambda (regularization parameter)
  2. High Variance:
    • More data
    • Smaller number of features
    • Increase Lambda (regularization parameter)

Basic recipe for training NNs

Recommended order:

  1. High bias (look at train set performance):
    • Bigger network (more hidden layers / units)
    • Train longer
    • Advanced optimization algorithms
    • Better NN architecture
  2. High variance (look at dev set performance)
    • More data (won't help for high bias problems)
    • Regularization
    • Better NN architecture

Bigger network almost always improves bias and more data improves variance (not necessarily a tradeoff between the two).

Working on most promising problems

Best case performance if no false positives?

E.g. 100 mislabeled dev set examples, how many are dog images (when training a cat classifier). When 50% could be worth to work on problem (if error is currently at 10% ⇒ 5%).

Evaluate multiple ideas in parallel - Fix false positives - Fix false negatives - Improve performance on blurry images

Create spread sheet: Image / Problem

Result: Calc percentage of problem category (potential improvement “ceiling”)

General rule: Build your first system quickly, then iterate (dev/test setup, build system, bias/variance & error analyis)

Misslabeled data

DL algos: If % or errors is low and errors are random, they are robust

Add another col “incorrectly labeled” in error analysis spread sheet.

Principles when fixing labels:

Missmatched train and dev/test set

Problems with different train and dev/test set dist

Not always good idea to use different dist in train and dev

Training-dev set: same distribution as training set, but not used for training

Still high gap between train and train-dev ⇒ variance problem

If Train and Train-dev would be closer ⇒ data-mismatch problem.

Summary:

Data mismatch problems