Lecture 03: Data Exploration & Visualization

Course: Introduction to Data Analytics (CE KMITL)
Lecturer: Asst. Prof. Dr. Rathachai Chawuthai, Department of Computer Engineering, Faculty of Engineering
Source Slide Deck: DA05 - Data Exploration.pdf

1. Overview of Exploratory Data Analysis (EDA)

1.1 What is Exploratory Data Analysis?

Exploratory Data Analysis (EDA) is the preliminary, iterative investigation of a dataset to discover underlying structures, assess statistical distributions, detect anomalies and outliers, test foundational hypotheses, and verify assumptions using summary statistics and graphical visualizations.

"Exploratory data analysis is an attitude, a state of flexibility, 
a willingness to look for those things that we believe are not there, 
as well as those we believe to be there." 
— John W. Tukey

1.2 Core Motivations of Data Exploration

1. Tool and Algorithm Selection: Understanding whether data is linear or non-linear, normal or skewed, dense or sparse determines whether linear regression, decision trees, neural networks, or clustering methods are appropriate.
2. Leveraging Human Visual Perception: Computational statistical summaries alone (such as mean and variance) can mask critical patterns. As demonstrated by Anscombe’s Quartet, vastly different geometric datasets can share identical summary statistics. Human visual cognition can instantly identify clusters, curvilinear trends, and local outliers that aggregate metrics fail to capture.
3. Data Quality Verification: Uncovering sensor failures, erroneous zero-readings, impossible measurement values, and unexpected encoding issues prior to formal modeling.

2. Common Data Formats in Analytics

Data exists across varied physical and logical structures, each requiring distinct ingestion strategies:

                            +--------------------------+
                            |   Common Data Formats    |
                            +--------------------------+
                                         |
            +----------------------------+----------------------------+
            |                            |                            |
    [Tabular / Flat]            [Semi-Structured]           [Relational / Graph]
   - CSV (.csv)                - JSON (.json)              - Relational Tables (SQL)
   - TSV (.tsv)                - XML (.xml)                - Graph Adjacency Matrix
   - Parquet (.parquet)        - YAML (.yaml)              - Network Edge Lists

2.1 Tabular Data (CSV, TSV, Data Tables)

• Structure: Rectangular grid organized into rows (observations/records) and columns (variables/attributes).
• CSV (Comma-Separated Values): The most ubiquitous exchange format. Human-readable ASCII/UTF-8 text where delimiter commas separate fields.
  • Strengths: Simple, universal software support, highly compressible.
  • Limitations: Lacks built-in schema definitions; data types (integer, float, datetime) must be inferred during parsing.
• Example Data Table (Iris Dataset Benchmark):

2.2 Semi-Structured Hierarchical Data (JSON)

• JSON (JavaScript Object Notation): Text-based format representing data as nested key-value pairs, arrays, and dictionaries.
• Strengths: Flexible schema, supports nested relationships (e.g., an order containing an array of multiple items), native standard for RESTful APIs and NoSQL document stores (e.g., MongoDB).
• Parsing: Flattend into tabular structures using tools like pandas.json_normalize().

2.3 Relational SQL Databases

• Structure: Structured, strongly typed schemas enforcing entity-relationship integrity, primary keys, foreign key constraints, and ACID transactional guarantees.
• Querying: Interrogated using Structured Query Language (SQL) via queries (SELECT, JOIN, GROUP BY, HAVING).

2.4 Graph and Network Structures

• Structure: Modeled as nodes (vertices $V$) and links (edges $E$).
• Representation: Stored as node-edge pair lists or mathematical Adjacency Matrices where $A_{ij} = 1$ if an edge connects entity $i$ and entity $j$, and $0$ otherwise.
• Applications: Social networks, protein interaction networks, routing topologies, fraud detection rings.

3. Descriptive Statistics & The 5-Number Summary

Descriptive statistics condense high-dimensional data vectors into scalar numerical indicators that summarize distribution location, spread, and shape.

3.1 Measures of Central Tendency

• Mean ($\mu$ or $\bar{x}$): The arithmetic average.

  $$\bar{x} = \frac{1}{n} \sum_{i=1}^n x_i$$

  Caveat: Highly sensitive to extreme outliers.
• Median ($Q_2$ / 50th Percentile): The physical center value when data points are sorted in ascending order. If $n$ is even, it is the average of the two middle values. Advantage: Robust against extreme outliers and severe distribution skewness.
• Mode: The most frequently occurring observation. Useful for categorical attributes.

3.2 Measures of Dispersion (Spread)

• Range: $\text{Range} = x_{\max} - x_{\min}$. Sensitive to sample extremes.
• Variance ($s^2$ or $\sigma^2$): The average squared deviation from the mean.

  $$\sigma^2 = \frac{1}{n} \sum_{i=1}^n (x_i - \mu)^2$$

• Standard Deviation ($s$ or $\sigma$): The square root of variance, restoring the metric to the original units of measurement:

  $$\sigma = \sqrt{\frac{1}{n} \sum_{i=1}^n (x_i - \mu)^2}$$

• Interquartile Range ($IQR$): $IQR = Q_3 - Q_1$. Captures the spread of the middle 50% of the dataset.

3.3 The 5-Number Summary and Boxplot Anatomy

The 5-Number Summary provides an information-dense non-parametric overview of any continuous distribution. It comprises:

1. Minimum ($\min$): The lowest observed value (excluding statistical outliers).
2. First Quartile ($Q_1$ / 25th Percentile): Value below which 25% of observations reside.
3. Median ($Q_2$ / 50th Percentile): Value dividing the upper and lower halves of data.
4. Third Quartile ($Q_3$ / 75th Percentile): Value below which 75% of observations reside.
5. Maximum ($\max$): The highest observed value (excluding statistical outliers).

Schematic Anatomy of a Boxplot (Box-and-Whisker Plot)

       Outlier                                                    Outlier
          o                                                          *
                  Lower Whisker       Box        Upper Whisker
              |------------------[=====|======]------------------|
                                 ^     ^      ^
                                Q1   Median   Q3
                                 <--- IQR --->
              |                  |            |                  |
         Lower Fence             Q1           Q3            Upper Fence
      (Q1 - 1.5 * IQR)                                   (Q3 + 1.5 * IQR)

Structural Components of a Boxplot

1. The Interquartile Box:
   • Spans from the First Quartile ($Q_1$) to the Third Quartile ($Q_3$).
   • The length (height) of the box directly represents the Interquartile Range ($IQR = Q_3 - Q_1$).
   • Contains the central 50% of all data points.
2. The Median Line:
   • A distinct bar drawn through the interior of the box at $Q_2$.
   • Assessing Distribution Skewness:
     • If the median line is centered within the box: The distribution is approximately symmetric.
     • If the median line is situated closer to $Q_1$: The distribution is positively skewed (right-skewed).
     • If the median line is situated closer to $Q_3$: The distribution is negatively skewed (left-skewed).
3. The Whiskers:
   • Lines extending outward from $Q_1$ and $Q_3$.
   • Whiskers typically extend to the most extreme data point located within the Tukey boundaries $[\text{Lower Fence}, \text{Upper Fence}]$:

     $$\text{Lower Bound} = \max(\min(x), Q_1 - 1.5 \times IQR)$$

     $$\text{Upper Bound} = \min(\max(x), Q_3 + 1.5 \times IQR)$$

4. Outlier Points (Fliers):
   • Data points lying outside the 1.5 IQR fences are plotted individually as standalone dots or asterisks.

4. Statistical Visualizations & Exploration Scenarios

4.1 Univariate Distributions: Histograms and Boxplots

       Histogram (Density/Count)                Boxplot (Quartile Distribution)
           +---+                                        +---+
           |   |   +---+                                |   |
       +---+   |   |   |                            |---| = |---|  *
       |   |   +---+   +---+                            +---+
       +---+---+---+---+---+                        Min  Q1 M Q3 Max Outlier

Histograms

• Mechanism: Groups continuous data into consecutive, non-overlapping bins along the horizontal axis and plots vertical bars proportional to the frequency count of samples within each bin.
• Analytical Utility:
  • Identifies distribution modality: Unimodal (single peak), Bimodal (two distinct subpopulations), or Multimodal.
  • Visualizes skewness, heavy tails, and truncation effects.
• Bin Width Selection: Choosing too few bins obscures variance; choosing too many produces noisy, jagged bars. Standard heuristics include Freedman-Diaconis rule:

  $$\text{Bin Width} = 2 \times \frac{IQR}{\sqrt[3]{n}}$$

Histogram vs. Boxplot Complementarity

• A histogram excels at displaying the exact probability density shape, peaks, and troughs.
• A boxplot excels at side-by-side comparative evaluations across categorical groups (e.g., comparing test score distributions across 5 distinct classes) and pinpointing outliers.

4.2 Bivariate Relationships: Scatter Plots

A scatter plot displays values for two continuous variables ($X$ and $Y$) as a collection of Cartesian points $(x_i, y_i)$.

• Analytical Objectives:
  • Assess relationship direction: Positive association ($Y$ increases as $X$ increases), Negative association ($Y$ decreases as $X$ increases), or Zero association.
  • Assess linearity: Linear, exponential, logarithmic, or non-linear polynomial relationships.
  • Spot clustering: Visual separation into distinct spatial groupings.
  • Identify heteroscedasticity: Unequal variance of residuals across the range of $X$.

4.3 Multivariate Relationships: Correlation Matrices & Heatmaps

When exploring datasets with $d$ continuous features, inspecting individual scatter plots becomes cumbersome. A Correlation Matrix computes the pairwise Pearson correlation coefficient ($r \in [-1, 1]$) between all feature pairs:

$$r_{jk} = \frac{\sum_{i=1}^n (x_{ij} - \bar{x}_j)(x_{ik} - \bar{x}_k)}{\sqrt{\sum_{i=1}^n (x_{ij} - \bar{x}_j)^2} \sqrt{\sum_{i=1}^n (x_{ik} - \bar{x}_k)^2}}$$

Interpreting Correlation Values

• $r = +1.0$: Perfect positive linear correlation.
• $r = 0.0$: No linear association.
• $r = -1.0$: Perfect negative linear correlation.
• High correlation ($|r| > 0.8$) indicates severe multicollinearity, warning practitioners that redundant features should be eliminated or regularized.

Correlation Heatmap Visualization

A Heatmap maps the correlation matrix into a colored grid where each cell $(j, k)$ displays a color proportional to $r_{jk}$, accompanied by numerical annotation:

                  [Sepal L]   [Sepal W]   [Petal L]   [Petal W]
      [Sepal L]     1.00       -0.11        0.87        0.82
      [Sepal W]    -0.11        1.00       -0.42       -0.36
      [Petal L]     0.87       -0.42        1.00        0.96
      [Petal W]     0.82       -0.36        0.96        1.00

(Strong correlation between Petal Length and Petal Width indicates redundant descriptive capacity in the Iris dataset).

4.4 Visual Chart Selection Framework

To systematically determine the appropriate plot for any analytical scenario:

5. Python Visualization Ecosystem: Matplotlib, Pandas, & Seaborn

The Python scientific data visualization stack is organized in three collaborative abstraction layers:

+--------------------------------------------------------------------------+
|                 Python Visualization Architecture Layers                 |
+--------------------------------------------------------------------------+
| Layer 3: High-Level Statistical Visuals   -> Seaborn (sns)              |
| Layer 2: DataFrame Convenience Wrapper     -> Pandas Plot (df.plot)       |
| Layer 1: Core Foundation & Canvas Engine   -> Matplotlib (plt)           |
+--------------------------------------------------------------------------+

5.1 Matplotlib Core Architecture: Figure vs. Axes Hierarchy

Understanding the Matplotlib object-oriented hierarchy is essential to building multi-panel figures without state-machine bugs.

+-------------------------------------------------------------+
| Figure Canvas (fig)                                         |
|                                                             |
|   +--------------------------+  +-------------------------+ |
|   | Axes Subplot 1 (ax1)     |  | Axes Subplot 2 (ax2)    | |
|   |                          |  |                         | |
|   |   Title                  |  |   Title                 | |
|   |   Y-Axis [Ticks, Labels] |  |   Y-Axis                | |
|   |   Data Line / Points     |  |   Data Line / Points    | |
|   |   Grid, Legend           |  |   Grid, Legend          | |
|   |   X-Axis [Ticks, Labels] |  |   X-Axis                | |
|   +--------------------------+  +-------------------------+ |
+-------------------------------------------------------------+

• Figure (fig): The overarching top-level window or canvas container holding all subplots, colorbars, global titles, and export parameters.
• Axes (ax): The individual coordinate system / plotting area where data is actually rendered. A single Figure can contain one or dozens of distinct Axes subplots.
• Axis: The numerical coordinate rulers (ax.xaxis, ax.yaxis) managing ticks, tick marks, and axis limits.

5.2 The 4-Step Standard Plotting Workflow

Every robust Matplotlib visualization follows a disciplined 4-stage pattern:

1. Step 1: Import Required Libraries:

   import matplotlib.pyplot as plt
   import seaborn as sns
   import pandas as pd
   import numpy as np
   

2. Step 2: Load or Prepare Dataset:

   # Load structured DataFrame
   df = pd.read_csv('iris.csv')
   

3. Step 3: Construct Plot Elements and Configure Parameters:

   # Set aesthetics, styles, titles, labels, linewidth, and colors
   plt.style.use('ggplot')
   fig, ax = plt.subplots(figsize=(8, 5))
   ax.scatter(df['sepal_length'], df['sepal_width'], color='green', label='Iris Samples')
   ax.set_title('Sepal Length vs. Sepal Width', fontsize=14)
   ax.set_xlabel('Sepal Length (cm)', fontsize=12)
   ax.set_ylabel('Sepal Width (cm)', fontsize=12)
   ax.legend()
   ax.grid(True)
   

4. Step 4: Render and Save Display:

   plt.tight_layout()
   plt.savefig('sepal_scatter.png', dpi=300)
   plt.show()
   

5.3 Multi-Panel Visualizations: The Subplots Workflow

When inspecting multiple features simultaneously, building multi-axis grids using plt.subplots() prevents visual clutter:

import matplotlib.pyplot as plt
import numpy as np

# Create a 2x2 subplot grid
fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(10, 8))

# Data generation
x = np.linspace(0, 10, 100)

# Subplot 0,0: Sine wave
axes[0, 0].plot(x, np.sin(x), color='blue')
axes[0, 0].set_title('Sine Wave')

# Subplot 0,1: Cosine wave
axes[0, 1].plot(x, np.cos(x), color='red')
axes[0, 1].set_title('Cosine Wave')

# Subplot 1,0: Linear relationship
axes[1, 0].plot(x, 2 * x + 1, color='green')
axes[1, 0].set_title('Linear: 2x + 1')

# Subplot 1,1: Exponential relationship
axes[1, 1].plot(x, np.exp(x / 3), color='purple')
axes[1, 1].set_title('Exponential Growth')

# Adjust layout to eliminate overlapping labels
plt.tight_layout()
plt.show()

5.4 High-Level Plotting with Pandas (df.plot)

Pandas integrates directly with Matplotlib, enabling immediate exploratory charting from DataFrames:

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'Metric_A': np.random.randn(100),
    'Metric_B': np.random.randn(100) + 2
})

# 1. Line plot
df.plot(title='Metric Trends')

# 2. Stacked Bar Plot
df.iloc[:10].plot.bar(stacked=True, title='Stacked Breakdown')

# 3. Histogram
df['Metric_A'].plot.hist(bins=15, title='Metric A Distribution')

# 4. Boxplot
df.plot.box(title='Distribution Comparison')

Pairwise Scatter Matrix via Pandas

To observe all bivariate interactions with density estimation along the diagonal:

from pandas.plotting import scatter_matrix

# Generates N x N pairwise scatter matrix with Kernel Density on diagonal
axes = scatter_matrix(df, alpha=0.5, figsize=(8, 8), diagonal='kde')
plt.show()

5.5 Statistical Visualization with Seaborn

Seaborn operates directly on tidy DataFrames, automatically handling statistical aggregation, confidence interval calculation, and color palette assignment.

Key Advantages of Seaborn

• Built-in publication-ready styling themes (darkgrid, whitegrid, ticks).
• Native handling of semantic mappings (e.g., coloring points by category using hue='species').
• Specialized routines for linear regression lines (sns.lmplot(), sns.regplot()).
• High-level multi-plot faceted grids (sns.FacetGrid, sns.pairplot).

Creating a Annotated Correlation Heatmap

import seaborn as sns
import matplotlib.pyplot as plt
import pandas as pd

# Compute correlation matrix
corr_matrix = df.corr()

# Configure figure
plt.figure(figsize=(8, 6))

# Plot heatmap with correlation coefficients annotated
sns.heatmap(
    corr_matrix, 
    annot=True,          # Print numerical correlation values inside cells
    cmap='coolwarm',     # Diverging colormap: Red = +1, Blue = -1
    fmt=".2f",           # Format to 2 decimal places
    vmin=-1.0, 
    vmax=1.0, 
    linewidths=0.5
)

plt.title('Feature Correlation Heatmap', fontsize=14)
plt.tight_layout()
plt.show()

6. Chapter Summary & Best Practices Checklist

• Look Before You Model: Never apply machine learning models to a dataset before conducting thorough visual and statistical EDA.
• The 5-Number Summary: Always inspect $[\min, Q_1, \text{Median}, Q_3, \max]$ to evaluate skewness and detect extreme values.
• Understand the Boxplot:
  • Box height equals $IQR = Q_3 - Q_1$.
  • Median offset signals left or right skewness.
  • Whiskers encompass points within $1.5 \times IQR$ fences; fliers represent statistical outliers.
• Chart Selection Discipline:
  • Univariate Continuous: Histogram (shape/peaks) + Boxplot (spread/outliers).
  • Categorical Counts: Bar chart.
  • Bivariate Continuous: Scatter plot.
  • Multivariate Continuous: Pairwise scatter matrix + Correlation Heatmap.
• Object-Oriented Matplotlib: Always distinguish between Figure (the canvas) and Axes (the individual subplots); use plt.subplots() and plt.tight_layout() for robust, reproducible figures.