Lecture 04: Artificial Intelligence & Machine Learning Foundations

Course: Introduction to Data Analytics (CE KMITL)
Source Material: DA06-07 AI-ML-Intro.pdf (140 slides)
Key Topics: AI vs ML vs DL hierarchy, Supervised vs Unsupervised learning, Model formulations, Regression evaluation metrics (MAE, MSE, RMSE, MAPE, $R^2$), Classification evaluation intro (Confusion Matrix, Accuracy), End-to-end ML workflow, Linear Regression derivation, Decision Trees, Logistic Regression, Naive Bayes, K-Nearest Neighbors, Artificial Neural Networks, MSE Cost function, and Gradient Descent parameter updates.

1. The Artificial Intelligence Hierarchy

Artificial Intelligence represents a broad spectrum of computational systems designed to perform tasks that typically require human intelligence. Within this overarching field, Machine Learning and Deep Learning form nested paradigms.

1.1 Core Definitions & Taxonomy

• Artificial Intelligence (AI): The overarching discipline concerned with building smart machines capable of performing cognitive tasks. AI encompasses both:
  • Rule-Driven Systems: Explicit hand-crafted if-then heuristics, expert systems, and symbolic logic engines.
  • Data-Driven Systems: Empirical statistical models that infer decision rules directly from data.
• Machine Learning (ML): A subfield of AI that provides systems the ability to automatically learn and improve from experience without being explicitly programmed. A computer program is said to learn from experience $E$ with respect to task $T$ and performance measure $P$ if its performance at task $T$, as measured by $P$, improves with experience $E$.
• Deep Learning (DL): A specialized subset of ML based on Artificial Neural Networks with representation learning. Deep learning architectures pass inputs through dozens or hundreds of hidden layers, extracting abstract features hierarchically (e.g., edges $\to$ textures $\to$ object parts $\to$ full objects).

1.2 Comparison: AI vs Machine Learning vs Deep Learning

1.3 Notable Real-World Milestones

• Google AlphaGo (2016): Defeated world champion Go master Lee Sedol using deep convolutional neural networks combined with Monte Carlo Tree Search (MCTS) and reinforcement learning.
• Autonomous Driving: Multi-modal computer vision, LiDAR sensor fusion, and real-time path planning.
• Natural Language Processing & Speech: Voice assistants, real-time speech-to-text, neural machine translation, and modern conversational Large Language Models (LLMs).
• Computer Vision & Healthcare: Automated facial recognition, diagnostic medical imaging (tumor segmentation, diabetic retinopathy), and automated defect inspection in manufacturing.

2. Machine Learning Paradigms & Data Taxonomy

Machine learning systems are broadly categorized according to the nature of their supervisory feedback signal during training.

2.1 Supervised Learning

In supervised learning, the training algorithm receives a dataset consisting of input features paired with corresponding ground-truth target outputs:

$$\mathcal{D} = \left\{ (\mathbf{x}^{(1)}, y^{(1)}), (\mathbf{x}^{(2)}, y^{(2)}), \dots, (\mathbf{x}^{(m)}, y^{(m)}) \right\}$$

The objective is to learn a mapping function $f: \mathcal{X} \to \mathcal{Y}$ such that $f(\mathbf{x})$ accurately predicts the true label $y$ on novel, unseen data.

• Regression: The target variable $y$ is continuous ($y \in \mathbb{R}$). Examples: predicting house sale prices, temperature, stock valuation, product demand.
• Classification: The target variable $y$ is discrete and categorical ($y \in \{0, 1\}$ or $y \in \{1, 2, \dots, C\}$). Examples: binary spam detection (spam vs ham), medical diagnosis (disease present vs absent), multi-class handwritten digit recognition ($0$ through $9$).

2.2 Unsupervised Learning

In unsupervised learning, the training dataset consists exclusively of input features without target labels:

$$\mathcal{D} = \left\{ \mathbf{x}^{(1)}, \mathbf{x}^{(2)}, \dots, \mathbf{x}^{(m)} \right\}$$

The algorithm must discover intrinsic patterns, latent structures, cluster boundaries, or lower-dimensional representations on its own:

• Clustering: Partitioning data into cohesive groups where intra-cluster similarity is maximized and inter-cluster similarity is minimized (e.g., K-Means, DBSCAN).
• Dimensionality Reduction: Compacting high-dimensional feature spaces while preserving maximum variance (e.g., Principal Component Analysis, SVD).

2.3 Mathematical Matrix Representation of Datasets

A dataset with $m$ sample instances and $n$ input features is represented as an $m \times n$ feature matrix $\mathbf{X}$ and an $m \times 1$ target vector $\mathbf{y}$:

$$\mathbf{X} = \begin{bmatrix} x_{1}^{(1)} & x_{2}^{(1)} & \dots & x_{n}^{(1)} \\ x_{1}^{(2)} & x_{2}^{(2)} & \dots & x_{n}^{(2)} \\ \vdots & \vdots & \ddots & \vdots \\ x_{1}^{(m)} & x_{2}^{(m)} & \dots & x_{n}^{(m)} \end{bmatrix} \in \mathbb{R}^{m \times n}, \quad \mathbf{y} = \begin{bmatrix} y^{(1)} \\ y^{(2)} \\ \vdots \\ y^{(m)} \end{bmatrix} \in \mathbb{R}^{m}$$

• Independent Variables (Features / Inputs): Columns $x_1, x_2, \dots, x_n$ representing measured attributes.
• Dependent Variable (Label / Ground Truth): Column $y$ representing the target outcome to be predicted.

3. What is a Model? Candidate Model Comparison (Toy Dataset Study)

A model is a mathematical or algorithmic construct that formalizes the relationship between input features $\mathbf{x}$ and the target outcome $y$:

$$\hat{y} = f(\mathbf{x})$$

Where $\hat{y}$ denotes the predicted value produced by the model, to be contrasted with the true label $y$.

3.1 The 3-Point Toy Problem

Consider a minimal one-dimensional toy training dataset with 3 observation pairs:

$$\mathcal{D}_{\text{toy}} = \left\{ (x=1, y=0), \; (x=2, y=0), \; (x=3, y=1) \right\}$$

In the lecture slides (slides 29–36), seven different mathematical and algorithmic formulations are proposed to fit these three data points. Below, progressive animation reveals are unified into one single comparative reference table.

3.2 Unified Candidate Model Comparison Table

3.3 Generalizing to Multi-Feature Spaces

While 1D toy problems allow intuitive visualization, real-world data involves multiple features of mixed types (numerical, categorical, textual):

$$\mathbf{x} = \left[ x_1, x_2, x_3, \dots, x_n \right]^T \implies \hat{y} = f(\mathbf{x}; \mathbf{W})$$

The challenge in machine learning is not merely memorizing the training samples, but selecting a model architecture and learning its parameter weights $\mathbf{W}$ such that it generalizes accurately to previously unseen data points.

4. End-to-End Machine Learning Workflow

An end-to-end supervised machine learning pipeline consists of systematic phases: data ingestion, data splitting, training, inference, and metric evaluation.

4.1 Pipeline Architecture Diagram

4.2 Detailed Workflow Stages

1. Dataset Partitioning (Train-Test Split):

   • The original dataset $(\mathbf{X}, \mathbf{y})$ is partitioned into two disjoint subsets:
     • Training Set $(\mathbf{X}_{\text{train}}, \mathbf{y}_{\text{train}})$: Used exclusively by the learning algorithm to adjust model parameters.
     • Test Set $(\mathbf{X}_{\text{test}}, \mathbf{y}_{\text{test}})$: Held out strictly as unseen evaluation data. No training information may leak into this set.
   • Standard split ratios include $70/30$, $75/25$, or $80/20$.

2. Model Training (.fit()):

   • The estimator takes $\mathbf{X}_{\text{train}}$ and $\mathbf{y}_{\text{train}}$, minimizes a chosen loss or cost function, and learns optimal internal parameters (coefficients, decision splits, or neural synaptic weights).

3. Inference / Prediction (.predict()):

   • The fitted model receives $\mathbf{X}_{\text{test}}$ without labels and generates predicted output values $\hat{\mathbf{y}}_{\text{pred}}$.

4. Performance Evaluation:

   • The predicted values $\hat{\mathbf{y}}_{\text{pred}}$ are quantitatively compared against actual ground-truth labels $\mathbf{y}_{\text{test}}$ using domain-appropriate error metrics.

5. Cross-Validation Synergy:

   • While a single train-test split provides a quick estimate, it is subject to sampling variance. K-Fold Cross-Validation (detailed in the course personal notes at notes/k-fold-cross-validation.md) partitions data into $K$ equal folds, iteratively training on $K-1$ folds and testing on the remaining fold, then averaging test metrics $e_1, e_2, \dots, e_K$ across all $K$ rounds.

5. Regression Evaluation Metrics

In regression problems, predictions $\hat{y}_i$ and actual targets $y_i$ are continuous real numbers. The discrepancy between actual and predicted values for sample $i$ is the residual error:

$$e_i = y_i - \hat{y}_i$$

Evaluation metrics aggregate these residual errors across all $n$ samples in the evaluation set.

5.1 Standard Mathematical Formulas

Mean Absolute Error (MAE)

Computes the arithmetic average of absolute errors. Linear penalty; robust to outliers:

$$\text{MAE} = \frac{1}{n} \sum_{i=1}^{n} |y_i - \hat{y}_i|$$

Mean Squared Error (MSE)

Computes the arithmetic average of squared errors. Penalizes larger deviations much more severely than small ones due to the quadratic power:

$$\text{MSE} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$$

Root Mean Squared Error (RMSE)

The square root of MSE. Restores error units to the original physical scale of the target variable $y$, making it directly interpretable:

$$\text{RMSE} = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2}$$

Mean Absolute Percentage Error (MAPE)

Measures the relative error percentage with respect to the true magnitude of $y_i$. Invariant to global scale:

$$\text{MAPE} = \frac{100\%}{n} \sum_{i=1}^{n} \left| \frac{y_i - \hat{y}_i}{y_i} \right|$$

Coefficient of Determination ($R^2$ Score)

Measures the proportion of variance in the dependent variable explained by the independent features in the model, relative to a baseline horizontal mean predictor:

$$R^2 = 1 - \frac{SS_{\text{res}}}{SS_{\text{tot}}} = 1 - \frac{\sum_{i=1}^{n} (y_i - \hat{y}_i)^2}{\sum_{i=1}^{n} (y_i - \bar{y})^2}$$

Where:

• $SS_{\text{res}} = \sum_{i=1}^n (y_i - \hat{y}_i)^2$ is the residual sum of squares.
• $SS_{\text{tot}} = \sum_{i=1}^n (y_i - \bar{y})^2$ is the total sum of squares.
• $\bar{y} = \frac{1}{n} \sum_{i=1}^n y_i$ is the empirical sample mean of target values.
• $R^2 = 1.0$ represents a perfect fit ($SS_{\text{res}} = 0$).
• $R^2 = 0.0$ indicates the model performs no better than constantly predicting $\bar{y}$.
• $R^2 < 0.0$ indicates the model performs worse than the simple sample mean.

5.2 Step-by-Step Numerical Walkthrough (Slide 52 Dataset)

To illustrate the exact mechanics of regression error calculation, we walk through the exact 3-case example presented on slide 52 of the lecture deck:

$$\mathcal{D} = \left\{ (y_1 = 10, \hat{y}_1 = 8), \; (y_2 = 20, \hat{y}_2 = 21), \; (y_3 = 30, \hat{y}_3 = 32) \right\}, \quad n = 3$$

Arithmetic Derivation:

1. Mean Absolute Error (MAE):

   $$\text{MAE} = \frac{1}{3} \left( 2 + 1 + 2 \right) = \frac{5}{3} \approx 1.667$$

2. Mean Squared Error (MSE):

   $$\text{MSE} = \frac{1}{3} \left( 4 + 1 + 4 \right) = \frac{9}{3} = 3.0$$

3. Root Mean Squared Error (RMSE):

   $$\text{RMSE} = \sqrt{\text{MSE}} = \sqrt{3.0} \approx 1.732$$

   (Verifies the verbatim value $\text{RMSE} = 1.732$ on Slide 52).

4. Mean Absolute Percentage Error (MAPE):

   $$\text{MAPE} = \frac{100\%}{3} \left( 0.200 + 0.050 + 0.0667 \right) = \frac{100\%}{3} (0.3167) \approx 10.56\%$$

5. Coefficient of Determination ($R^2$):

   $$\bar{y} = \frac{10 + 20 + 30}{3} = 20$$

   $$SS_{\text{tot}} = (10 - 20)^2 + (20 - 20)^2 + (30 - 20)^2 = (-10)^2 + 0^2 + (10)^2 = 100 + 0 + 100 = 200$$

   $$SS_{\text{res}} = 4 + 1 + 4 = 9$$

   $$R^2 = 1 - \frac{9}{200} = 1 - 0.045 = 0.955$$

6. Classification Evaluation Introduction

In classification, predictions represent discrete qualitative labels. Evaluating a classifier involves comparing true category membership against model predictions.

6.1 The $2 \times 2$ Confusion Matrix

For a binary classification task (e.g. classifying whether a sample is a Human vs Not Human, or Positive vs Negative), predictions fall into four mutually exclusive categories:

• True Positive ($TP$): Model correctly predicted positive condition.
• True Negative ($TN$): Model correctly predicted negative condition.
• False Positive ($FP$): Type I error; model predicted positive when actual was negative (false alarm).
• False Negative ($FN$): Type II error; model predicted negative when actual was positive (missed detection).

6.2 Accuracy Metric & Slide 59 Numerical Drill

Accuracy computes the ratio of all correctly classified instances over the total population size:

$$\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}$$

Slide 59 Worked Matrix:

The lecture presents the following confusion counts:

Calculation:

$$\text{Accuracy} = \frac{4 + 3}{4 + 3 + 1 + 2} = \frac{7}{10} = 0.70 \quad (70\%)$$

(Note: Slide 59 contains a CambriaMath shifted font rendering in the original slide deck, which precisely decodes to the numbers $\frac{4 + 3}{4 + 3 + 1 + 2} = 0.70$).

6.3 Associated Core Classification Metrics

• Precision (Positive Predictive Value): What proportion of predicted positives were truly positive?

  $$\text{Precision} = \frac{TP}{TP + FP} = \frac{4}{4 + 2} = \frac{4}{6} \approx 0.667$$

• Recall (Sensitivity / True Positive Rate): What proportion of actual positives were successfully found?

  $$\text{Recall} = \frac{TP}{TP + FN} = \frac{4}{4 + 1} = \frac{4}{5} = 0.800$$

• $F_1$-Score: The harmonic mean of Precision and Recall:

  $$F_1 = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}} = 2 \cdot \frac{0.667 \cdot 0.800}{0.667 + 0.800} \approx 0.727$$

7. Key Machine Learning Algorithms Overview

The lecture introduces six foundational machine learning algorithms spanning regression and classification paradigms.

7.1 Linear Regression

Linear regression models the relationship between dependent scalar $y$ and one or more explanatory variables $\mathbf{x}$ as a linear combination.

• Univariate Formulation:

  $$y = w_1 x + w_0 \quad \text{or} \quad y = mx + c$$

  Where $w_1$ (or $m$) represents slope, and $w_0$ (or $c$) represents the y-intercept.
• Multivariate Formulation:

  $$y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \dots + \beta_k x_k = \mathbf{w}^T \mathbf{x} + b$$

• Feature Selection: Prior to fitting, pairwise Pearson correlation coefficients $r_{xy} \in [-1, 1]$ are examined to identify input variables with high explanatory power and eliminate redundant multi-collinear features.

7.2 Decision Tree Classifier

Decision trees partition the input feature space into rectangular decision regions through recursive hierarchical if-else threshold tests.

• Internal nodes represent attribute evaluation conditions.
• Leaf nodes represent terminal class assignments.
• Non-parametric; requires minimal feature scaling; fully interpretable. (Covered in full depth in Lecture 05: 05-decision-tree.md).

7.3 Logistic Regression

Despite its name, Logistic Regression is a binary classification algorithm. It maps linear combinations of input features to bounded probabilities using the non-linear Sigmoid Activation Function:

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

Where the logit $z$ is a linear model:

$$z = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \dots + \beta_n x_n = \mathbf{w}^T \mathbf{x} + b$$

Substituting $z$ into the sigmoid equation yields:

$$P(y = 1 \mid \mathbf{x}) = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x_1 + \dots + \beta_n x_n)}}$$

• When $z \to +\infty$, $\sigma(z) \to 1$.
• When $z \to -\infty$, $\sigma(z) \to 0$.
• When $z = 0$, $\sigma(0) = 0.5$ (the default decision threshold: predict $\hat{y} = 1$ if $P \ge 0.5$).

7.4 Naïve Bayes

Naïve Bayes is a probabilistic classifier founded upon Bayes’ Theorem, with the “naïve” assumption of conditional independence among features given the class label.

• Bayes’ Theorem:

  $$P(y \mid \mathbf{x}) = \frac{P(\mathbf{x} \mid y) P(y)}{P(\mathbf{x})}$$

• Conditional Independence Assumption:

  $$P(x_1, x_2, \dots, x_n \mid y) = \prod_{i=1}^{n} P(x_i \mid y)$$

• Classification Decision Rule:

  $$\hat{y} = \arg\max_{y \in \mathcal{Y}} P(y) \prod_{i=1}^{n} P(x_i \mid y)$$

• Fast, scalable, highly effective for text classification and high-dimensional categorical problems.

7.5 K-Nearest Neighbors (KNN)

KNN is an instance-based (lazy learning), non-parametric algorithm. It makes no explicit assumptions about the underlying distribution of data and delays all computation until query time.

• Distance Metric: The proximity between a query point $\mathbf{x}_q$ and training instance $\mathbf{x}_i$ is typically calculated via Euclidean distance:

  $$d(\mathbf{x}_q, \mathbf{x}_i) = \sqrt{\sum_{j=1}^{n} \left( x_{q,j} - x_{i,j} \right)^2}$$

• Decision Behavior Dependent on $K$ (Slides 119–122):
  • $K = 1$: Assigns the class of the single nearest neighbor. Extremely flexible boundary, but highly sensitive to noise and outliers.
  • $K = 2$: Identifies 2 nearest neighbors. If each neighbor belongs to a different class, a tie occurs ($\text{Class} = \text{???}$), requiring tie-breaking rules.
  • $K = 3$: An odd $K$ avoids binary ties; majority vote determines the winning class. Smoother, more robust decision boundary.

7.6 Artificial Neural Networks (ANN)

Artificial Neural Networks are biologically inspired computational networks composed of interconnected processing units (neurons / perceptrons) arranged in layers.

• The Single Perceptron Model (Slide 127): A single neuron computes a weighted sum of inputs plus a bias term, followed by an activation function:

  $$h(\mathbf{X}) = w_0 + w_1 x_1 + w_2 x_2 = \mathbf{w}^T \mathbf{x} + b$$

  $$\hat{y} = g(h(\mathbf{X})) = \frac{1}{1 + e^{-h(\mathbf{X})}}$$

• Layered Multi-Layer Perceptron (MLP) Architecture (Slide 128):
  • Input Layer: Transmits raw feature attributes $x_1, x_2, \dots, x_n$.
  • Hidden Layers: One or more intermediate layers where each neuron extracts non-linear feature combinations.
  • Output Layer: Produces final prediction $\hat{y}$ (scalar for regression, probability vector for classification).
• Feedforward Propagation: Information travels strictly forward from input nodes $\to$ hidden layers $\to$ output nodes.
• Backpropagation: Error gradients are propagated backwards through the network to iteratively update synaptic weights.

8. Optimization: Cost Functions and Gradient Descent

Learning in parameterized models (Linear Regression, Logistic Regression, Neural Networks) is formulated as an optimization problem: finding the weight vector $\mathbf{W}$ that minimizes an empirical cost function across all training instances.

8.1 The Mean Squared Error Cost Function $J(\mathbf{W})$

For a dataset of $m$ training samples, the Mean Squared Error (MSE) cost function $J(\mathbf{W})$ is defined as:

$$J(\mathbf{W}) = \frac{1}{2m} \sum_{i=1}^{m} \left( \hat{y}^{(i)} - y^{(i)} \right)^2$$

Where:

• $m$ is the total number of training samples.
• $y^{(i)}$ is the actual target value of the $i$-th sample.
• $\hat{y}^{(i)}$ is the model’s predicted value for the $i$-th sample:

  $$\hat{y}^{(i)} = w_0 + w_1 x_1^{(i)} + w_2 x_2^{(i)} + \dots + w_n x_n^{(i)} = \sum_{j=0}^{n} w_j x_j^{(i)} \quad (\text{with } x_0^{(i)} = 1)$$

• The coefficient $\frac{1}{2}$ is an algebraic convenience: when differentiating $\left(\hat{y}^{(i)} - y^{(i)}\right)^2$, the exponent $2$ cancels cleanly with $\frac{1}{2}$.

8.2 Gradient Descent Parameter Update Rule

Gradient Descent is an iterative first-order optimization algorithm that steps parameter weights in the direction opposite to the gradient of the cost surface:

$$w_j := w_j - \alpha \frac{\partial}{\partial w_j} J(\mathbf{W})$$

Where:

• $w_j$ is the $j$-th parameter weight.
• $\alpha > 0$ is the learning rate (step size). If $\alpha$ is too small, convergence is slow; if $\alpha$ is too large, the algorithm can oscillate or diverge.
• $\frac{\partial}{\partial w_j} J(\mathbf{W})$ is the partial derivative of the cost function with respect to weight $w_j$.
• Updates across all weights $j \in \{0, 1, \dots, n\}$ must be performed simultaneously at each iteration.

8.3 Chain Rule Derivation of the Gradient (Slide 132)

Applying the multivariable calculus chain rule to derive $\frac{\partial}{\partial w_j} J(\mathbf{W})$:

$$\frac{\partial}{\partial w_j} J(\mathbf{W}) = \frac{\partial}{\partial w_j} \left[ \frac{1}{2m} \sum_{i=1}^{m} \left( \hat{y}^{(i)} - y^{(i)} \right)^2 \right]$$

Pass the derivative inside the summation:

$$\frac{\partial}{\partial w_j} J(\mathbf{W}) = \frac{1}{2m} \sum_{i=1}^{m} \frac{\partial}{\partial w_j} \left( \hat{y}^{(i)} - y^{(i)} \right)^2$$

Apply the chain rule: $\frac{\partial}{\partial u} (u^2) = 2u \cdot \frac{\partial u}{\partial w_j}$:

$$\frac{\partial}{\partial w_j} J(\mathbf{W}) = \frac{1}{2m} \sum_{i=1}^{m} 2 \left( \hat{y}^{(i)} - y^{(i)} \right) \cdot \frac{\partial}{\partial w_j} \left( \hat{y}^{(i)} - y^{(i)} \right)$$

Cancel the factor of 2:

$$\frac{\partial}{\partial w_j} J(\mathbf{W}) = \frac{1}{m} \sum_{i=1}^{m} \left( \hat{y}^{(i)} - y^{(i)} \right) \cdot \frac{\partial \hat{y}^{(i)}}{\partial w_j}$$

Since $\hat{y}^{(i)} = w_0 x_0^{(i)} + w_1 x_1^{(i)} + \dots + w_j x_j^{(i)} + \dots + w_n x_n^{(i)}$, the partial derivative with respect to $w_j$ is simply the feature value $x_j^{(i)}$:

$$\frac{\partial \hat{y}^{(i)}}{\partial w_j} = \begin{cases} 1, & \text{for } j = 0 \text{ (bias term } w_0) \\ x_j^{(i)}, & \text{for } j \ge 1 \end{cases}$$

Substituting back into the gradient formula yields the canonical update equation:

$$\frac{\partial}{\partial w_j} J(\mathbf{W}) = \frac{1}{m} \sum_{i=1}^{m} \left( \hat{y}^{(i)} - y^{(i)} \right) x_j^{(i)}$$

And the complete iterative weight update becomes:

$$w_j := w_j - \alpha \frac{1}{m} \sum_{i=1}^{m} \left( \hat{y}^{(i)} - y^{(i)} \right) x_j^{(i)}$$

9. Scikit-Learn Python Implementation Guide

The Scikit-Learn API provides a standardized interface across all learning algorithms:

# ==============================================================================
# End-to-End Supervised Learning Template (Scikit-Learn)
# ==============================================================================
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import mean_squared_error, accuracy_score

# 1. Ingest Data
df = pd.read_csv('dataset.csv')
X = df[['feature_1', 'feature_2', 'feature_3']]
y = df['target']

# 2. Train-Test Split (Holdout Validation)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.30, random_state=42
)

# 3. Instantiate and Train Estimator
reg_model = LinearRegression()
reg_model.fit(X_train, y_train)

# 4. Generate Predictions on Held-Out Test Set
y_pred_reg = reg_model.predict(X_test)

# 5. Evaluate Performance Metric
mse = mean_squared_error(y_test, y_pred_reg)
rmse = mean_squared_error(y_test, y_pred_reg, squared=False)
print(f"Test MSE: {mse:.4f}, Test RMSE: {rmse:.4f}")

# Classification Alternative:
clf_model = DecisionTreeClassifier(random_state=42)
clf_model.fit(X_train, y_train)
y_pred_clf = clf_model.predict(X_test)
acc = accuracy_score(y_test, y_pred_clf)
print(f"Classification Accuracy: {acc * 100:.2f}%")

10. Summary & Key Concept Checklist

• [x] AI / ML / DL Hierarchy: AI $\supset$ ML $\supset$ DL. AI includes rule-based heuristics; ML learns patterns from data; DL utilizes deep neural representations.
• [x] Learning Paradigms: Supervised (labels provided: Regression for continuous $y$, Classification for discrete $y$), Unsupervised (no labels: Clustering and Dimensionality Reduction).
• [x] Candidate Model Deduplication: 7 mathematical models for $\mathcal{D}_{\text{toy}} = \{(1,0), (2,0), (3,1)\}$ consolidated into 1 unified comparative specification.
• [x] Regression Metrics: MAE ($\frac{1}{n}\sum |y - \hat{y}|$), MSE ($\frac{1}{n}\sum (y - \hat{y})^2$), RMSE ($\sqrt{\text{MSE}}$), MAPE ($\frac{100\%}{n}\sum |\frac{y - \hat{y}}{y}|$), and $R^2$ ($1 - \frac{SS_{\text{res}}}{SS_{\text{tot}}}$).
• [x] Classification Evaluation: $2 \times 2$ Confusion Matrix ($TP, TN, FP, FN$), Accuracy ($\frac{TP+TN}{\text{Total}}$), Precision, Recall, and $F_1$-Score.
• [x] Core Algorithms: Linear Regression ($y = \mathbf{w}^T \mathbf{x} + b$), Decision Tree, Logistic Regression Sigmoid ($\frac{1}{1 + e^{-z}}$), Naive Bayes, KNN ($K=1, 2, 3$ tie behavior), and Neural Networks.
• [x] Cost Optimization: MSE Cost Function $J(\mathbf{W}) = \frac{1}{2m}\sum (\hat{y}^{(i)} - y^{(i)})^2$ and simultaneous Gradient Descent update $w_j := w_j - \alpha \frac{1}{m}\sum (\hat{y}^{(i)} - y^{(i)}) x_j^{(i)}$.