Lecture 01: Introduction to Data Analytics

Course: Introduction to Data Analytics (CE KMITL)
Lecturer: Asst. Prof. Dr. Rathachai Chawuthai, Department of Computer Engineering, Faculty of Engineering
Source Slide Deck: DA01 - Introduction.pdf

1. Overview and Foundations of Data Analytics

1.1 What is Data Analytics?

Data Analytics is the scientific discipline and operational process of inspecting, cleansing, transforming, and modeling data to discover actionable insights, identify underlying patterns, suggest conclusions, and support strategic decision-making.

In modern engineering and enterprise environments, organizations collect vast quantities of data from transactions, sensors, user interactions, and external logs. Raw data alone, however, does not possess inherent operational utility until it is processed into structured information, evaluated to extract knowledge, and operationalized as actionable intelligence.

1.2 The Triad of Data Science and Analytics

Modern data analytics lies at the intersection of three foundational domains:

1. Mathematics and Statistics: Provides the mathematical rigor, probability theory, hypothesis testing, linear algebra, calculus, and statistical distributions required to build valid inference models.
2. Computer Science and Programming: Supplies the algorithmic foundations, computational efficiency, database architectures, software engineering practices, and data structures necessary to scale computations across large datasets.
3. Domain and Business Knowledge: Informs problem formulation, variable relevance, domain-specific constraints, and the contextual interpretation of results so that quantitative findings translate into real-world business value.

       +---------------------------------------------+
       |           The Data Science Triad            |
       +---------------------------------------------+
       |                                             |
       |             [Mathematics & Stats]           |
       |                     /   \                   |
       |                    /     \                  |
       |                   /   *   \                 |
       |                  / Data    \                |
       |                 /  Science  \               |
       |   [Computer Science] ------- [Domain Expertise] |
       +---------------------------------------------+

1.3 The DIKW Pyramid and Insight Generation

The progression from unstructured observation to strategic action is characterized by the DIKW Hierarchy (Data, Information, Knowledge, Wisdom):

• Data: Raw, uncontextualized facts, signals, symbols, or numerical readings without implicit meaning (e.g., "39.5", "high", "YES").
• Information: Data structured, categorized, aggregated, and endowed with contextual relevance (e.g., "Patient A has a body temperature of 39.5 °C with a high headache score").
• Knowledge: Synthesized information evaluated against historical benchmarks, patterns, and principles (e.g., "A high body temperature combined with severe headache indicates symptoms consistent with viral influenza").
• Wisdom / Action: The application of synthesized knowledge, judgment, and ethical/operational considerations to make proactive, optimal decisions (e.g., "Prescribe antiviral medication within 48 hours to minimize complication risks").

“The goal is to turn data into information, and information into insight.”
— Carly Fiorina

2. The Data Decision Chain

The core imperative of data analytics is to transition organizations from intuitive guesswork to data-backed decisions. Intuitive decision-making without empirical backing frequently results in systematic bias, uncalibrated risk, and costly misallocations.

"Most of the world will make decisions by either guessing or using their gut. 
They will be either lucky or wrong." 
— Suhail Doshi (Mixpanel)

In an analytical system, data flows through a closed-loop feedback pipeline:

Key Stages in the Decision Chain

1. Raw Data Ingestion: Capturing events, logs, telemetry, and transactional state changes across operational systems.
2. Analytical Transformation: Applying statistical summaries, predictive models, and rule-based heuristics to identify trends and anomalies.
3. Decision Making: Selecting the optimal operational strategy based on quantitative trade-offs rather than subjective assumptions.
4. Action Execution: Deploying business interventions, marketing updates, clinical treatments, or routing adjustments.
5. Performance Measurement: Measuring post-intervention outcomes against historical baselines and feeding telemetry back into the analytics engine for continuous refinement.

3. Four Types of Data Analytics (Gartner Maturity Spectrum)

Gartner categorizes analytical capability into four sequential tiers based on the trade-off between Degree of Complexity (Difficulty) and Business Value Delivered:

Value  ^
       |                                          [4. Prescriptive]
       |                                           "How can we make
       |                                            it happen?"
       |                                 [3. Predictive]
       |                                  "What will
       |                                   happen?"
       |                        [2. Diagnostic]
       |                         "Why did
       |                          it happen?"
       |             [1. Descriptive]
       |              "What
       |               happened?"
       +------------------------------------------------------------> Difficulty

Comparative Summary Table

3.1 Descriptive Analytics

Descriptive analytics is the foundational tier. It interprets historical data to summarize past states, quantify operational volumes, and present baseline performance indicators.

• Primary Objective: Condense high-volume transactional logs into understandable, human-interpretable metrics.
• Statistical Techniques:
  • Measures of Central Tendency: Mean, Median, Mode.
  • Measures of Dispersion: Range, Variance, Standard Deviation ($\sigma$), Interquartile Range ($IQR$).
  • Distribution Shapes: Frequency histograms, cumulative frequency curves, quantiles.
  • Data Aggregation: Multidimensional grouping, pivot tables, cross-tabulations.
• Typical Questions:
  • Which product lines generated the highest revenue in Q3?
  • What is the average resolution time for customer support tickets?
  • Who are our top 10% highest-value customers by gross transaction value?

3.2 Diagnostic Analytics

Diagnostic analytics examines historical anomalies, trends, and deviations identified by descriptive analytics to uncover root causes and contributing factors.

• Primary Objective: Isolate dependencies and explain variance across dimensions.
• Analytical Techniques:
  • Drill-down and roll-up operations across hierarchical dimensions (e.g., country $\to$ province $\to$ city).
  • Correlation analysis and scatter matrix evaluations.
  • Anomaly and outlier detection.
  • Cohort analysis and segmentation analysis.
• Typical Questions:
  • Why did mobile app transaction volume drop sharply following the v2.4 release?
  • Why do customers in Region B churn at twice the rate of Region A?
  • What specific component failure caused the manufacturing outage?

3.3 Predictive Analytics

Predictive analytics leverages statistical models, historical patterns, and machine learning algorithms to estimate future outcomes, forecast numerical values, or assess class probabilities.

• Primary Objective: Quantify uncertainty and estimate the likelihood of future events.
• Important Caveat: Predictive models provide probabilistic estimates based on observed patterns; they do not guarantee deterministic certainty.
• Core Techniques:
  • Regression Analysis: Continuous target estimation (e.g., Linear Regression, Polynomial Regression, Ridge/Lasso).
  • Classification Analysis: Discrete target categorization (e.g., Logistic Regression, Decision Trees, Random Forests, Support Vector Machines, Neural Networks).
  • Time-Series Analysis: Autoregressive Integrated Moving Average (ARIMA), Exponential Smoothing, Prophet.
• Typical Questions:
  • What is the expected monthly sales demand for inventory items next winter?
  • Which banking transactions are likely to represent fraudulent activity?
  • Will a patient with high body temperature and nausea test positive for viral flu?

3.4 Prescriptive Analytics

Prescriptive analytics represents the pinnacle of the analytical maturity curve. Beyond predicting what will occur, it evaluates candidate interventions, operational constraints, and projected outcomes to prescribe the optimal course of action.

• Primary Objective: Optimize complex decisions under resource limitations, competing objectives, and stochastic variability.
• Core Techniques:
  • Mathematical Programming & Operations Research: Linear programming (LP), integer programming (MIP), constraint satisfaction.
  • Recommendation Engines: Collaborative filtering, content-based filtering, matrix factorization.
  • Decision Support Systems (DSS): Expert rule engines, heuristic search algorithms.
  • Autonomous Systems & Reinforcement Learning: Real-time dynamic pricing, automated trading agents.
• Typical Questions:
  • What is the optimal flight schedule and crew assignment that minimizes operational downtime while complying with labor regulations?
  • Which personalized discount should be offered to this user to maximize retention while safeguarding profit margin?
  • Which navigation route should an emergency vehicle take to minimize travel time through live traffic congestion?

4. Big Data Foundations: The 4 V’s

When data volume, ingestion rate, and structural heterogeneity exceed the storage and computational capacity of traditional relational database management systems (RDBMS), it is classified as Big Data.

                           +----------------------+
                           |  The 4 V's of Data   |
                           +----------------------+
                           | 1. Volume (Scale)    |
                           | 2. Velocity (Speed)  |
                           | 3. Variety (Format)  |
                           | 4. Veracity (Trust)  |
                           +----------------------+

1. Volume (Scale): The astronomical size of datasets generated every second. Traditional architectures struggle with terabyte-to-petabyte scale datasets, necessitating distributed file systems (e.g., HDFS, cloud object storage) and distributed compute engines (e.g., Apache Spark).
2. Velocity (Speed): The rapid rate at which data is created, ingested, and processed. Applications such as fraud detection, autonomous navigation, and IoT sensor monitoring require streaming ingestion and sub-second analytical response times.
3. Variety (Diversity of Format): Data arrives across varied formats:
   • Structured: Tabular schemas with predefined types (SQL tables, CSV).
   • Semi-Structured: Self-describing schema-flexible trees (JSON, XML, YAML).
   • Unstructured: Binary and free-form artifacts without formal schema (audio, video, natural language text, PDF documents, raw imagery).
4. Veracity (Quality and Trustworthiness): The fidelity, cleanliness, and accuracy of data. Real-world data is inherently subject to sensor noise, missing measurements, human entry typos, duplicated records, and fraudulent manipulation. Data preprocessing is required to ensure downstream model validity.

5. The CRISP-DM Process and Data Science Lifecycle

To execute data analytics initiatives systematically, practitioners adhere to standardized process frameworks. The most widely adopted industry standard is CRISP-DM (Cross-Industry Standard Process for Data Mining).

The Six Phases of CRISP-DM

1. Business Understanding:
   • Define project objectives, business constraints, and success criteria from an organizational perspective.
   • Formulate the business dilemma as an empirical data science problem (e.g., translating “reduce churn” into “predict customer churn probability with precision $\ge 85\%$”).
2. Data Understanding:
   • Collect raw initial data and assess data availability.
   • Perform exploratory data analysis (EDA) to understand distributions, detect anomalies, inspect missingness, and verify initial hypotheses.
3. Data Preparation (Pre-processing):
   • Typically consumes 70% to 80% of project time and effort.
   • Involves data cleansing, missing value imputation, outlier handling, record deduplication, schema integration, feature encoding, scaling, and transformation.
4. Modeling:
   • Select candidate algorithms (e.g., Decision Trees, Linear Regression, K-Means).
   • Partition data into training and validation sets, tune hyperparameters, and fit model parameters to the prepared data.
5. Evaluation:
   • Evaluate model metrics (e.g., RMSE, MAE, Confusion Matrix, Precision, Recall, ROC-AUC) against the business criteria defined in Phase 1.
   • Verify whether the model satisfies operational robustness and fairness requirements before capital deployment.
6. Deployment:
   • Package models into production services, automated batch scoring pipelines, or real-time microservice APIs.
   • Establish ongoing monitoring pipelines to detect data drift, model decay, and system latency.

6. The End-to-End Data Science Pipeline

In technical execution, the data lifecycle moves from messy physical reality to automated data products through a disciplined engineering pipeline:

Pipeline Stage Details

1. Reality & Logging: Real-world entities (users, machines, sensors, medical patients) generate events and physical readings.
2. Raw Data Ingestion: Gathering events via batch queries, API endpoints, web scraping, or telemetry queues into operational storage.
3. Data Processing & Cleansing: Transforming raw, dirty data into consistent, machine-readable records. Resolving missing values, normalizing formats, eliminating duplicate tuples, and filtering extreme noise.
4. Clean Dataset: Standardized, validated, tidy tabular matrices ready for mathematical computation.
5. Exploratory Data Analysis (EDA): Profiling features via visual charts (boxplots, scatter plots, correlation heatmaps) and summary statistics to identify patterns.
6. Modeling & Algorithms:
   • Algorithm: The general computational method or optimization procedure (e.g., ID3, Ordinary Least Squares, Backpropagation).
   • Model: The concrete artifact resulting from training an algorithm on a specific dataset (e.g., a fitted regression equation $\hat{y} = 2.4x_1 - 1.1x_2 + 5.0$, a fitted decision tree structure).
7. Evaluation & Visualization: Assessing generalization ability using validation splits, cross-validation, and performance metrics.
8. Data Products & Decision Support: Embedding trained models into operational software (e.g., spam filters, recommendation widgets, route planners).

7. Machine Learning Paradigms: An Introductory Taxonomy

Data modeling techniques are broadly separated into learning paradigms based on the presence or absence of supervising labels:

                          +-------------------------------+
                          |   Machine Learning Overview   |
                          +-------------------------------+
                                         |
                 +-----------------------+-----------------------+
                 |                                               |
        [Supervised Learning]                           [Unsupervised Learning]
     (Features X + Known Labels y)                      (Features X without Labels)
                 |                                               |
        +--------+--------+                             +--------+--------+
        |                 |                             |                 |
  [Regression]     [Classification]               [Clustering]     [Association]
(Continuous target) (Discrete classes)         (Grouping samples)  (Itemset rules)

Supervised vs. Unsupervised Comparison

8. Roles and Tool Ecosystem in Modern Analytics

8.1 Professional Roles

• Data Analyst: Focuses on descriptive and diagnostic analytics; formulates SQL queries, builds BI dashboards, conducts EDA, and communicates business insights to stakeholders.
• Data Scientist: Formulates hypothesis tests, engineers complex features, builds predictive and prescriptive machine learning models, and evaluates algorithmic performance.
• Data Engineer: Designs, constructs, and maintains scalable data architectures, ingestion pipelines, ETL jobs, database warehouses, and data lakes.
• Machine Learning Engineer: Bridges data science and software engineering; optimizes models for latency, containerizes pipelines, and builds deployment microservices.

8.2 The Python Scientific Tool Ecosystem

Python is the predominant programming language for modern data science due to its clean syntax and rich ecosystem of open-source libraries:

+-------------------------------------------------------------------------+
|                        Python Analytics Stack                           |
+-------------------------------------------------------------------------+
| Interactive IDEs:    Jupyter Notebook, Google Colab, VS Code, Spyder    |
| Machine Learning:    Scikit-learn, XGBoost, PyTorch, TensorFlow        |
| Data Visualization:  Matplotlib, Seaborn, Plotly                        |
| Data Manipulation:   Pandas (DataFrames, Series, Groupby, Pivot)        |
| Numerical Core:      NumPy (N-dimensional arrays), SciPy (Optimization)  |
+-------------------------------------------------------------------------+

1. NumPy: Provides high-performance multidimensional array structures (ndarray) and vectorized mathematical operations implemented in C.
2. Pandas: Offers high-level DataFrame and Series abstractions for structured data manipulation, slicing, joining, grouping, and handling missing data.
3. Scikit-learn: The premier library for classical machine learning algorithms, featuring unified APIs for data preprocessing, estimators (fit), and predictors (predict).
4. Matplotlib & Seaborn: Fundamental libraries for statistical data visualization, plotting distributions, heatmaps, and publication-quality figures.
5. Interactive Environments (Jupyter / Colab): Cell-based literate programming environments that integrate documentation, runnable code, equations, and inline visual plots into reproducible research documents.

9. Chapter Summary and Key Concepts

• Data Analytics Purpose: Transform raw observations into information, extract knowledge, and operationalize insight for proactive decision-making.
• Four Analytical Types:
  1. Descriptive (Historical summaries, “What happened?”)
  2. Diagnostic (Root cause analysis, “Why did it happen?”)
  3. Predictive (Probabilistic future forecasting, “What will happen?”)
  4. Prescriptive (Action optimization, “What should we do?”)
• Big Data Characteristics: The 4 V’s — Volume, Velocity, Variety, and Veracity.
• Process Frameworks: CRISP-DM standardizes the iterative cycle across Business Understanding, Data Understanding, Data Preparation, Modeling, Evaluation, and Deployment.
• Algorithms vs. Models: An algorithm is the procedural recipe for learning; a model is the trained mathematical representation resulting from executing an algorithm on a dataset.