AIPrimer.AI
  • 馃殾AI Primer In Transportation
  • CHAPTER 1 - INTRODUCTION TO MACHINE LEARNING
    • Machine Learning in Transportation
    • What is Machine Learning?
    • Types of Machine Learning
      • Supervised Learning
      • Unsupervised Learning
      • Semi-supervised Learning
      • Reinforced Learning
    • Fundamental concepts of machine learning
      • Model Training and Testing
      • Evaluating the Model鈥檚 Prediction Accuracy
      • The Underfitting and Overfitting Problems
      • Bias-Variance Tradeoff in Overfitting
      • Model Validation Techniques
      • Hyperparameter Tuning
      • Model Regularization
      • The Curse of Ddimensionality
    • Machine Learning versus Statistics
  • CHAPTER 2 - SUPERVISED METHODS
    • Supervised Learning_Complete Draft
    • K-Nearest Neighbor (KNN) Algorithm
    • Tree-Based Methods
    • Boosting
    • Support Vector Machines (SVMs)
  • CHAPTER 3 - UNSUPERVISED LEARNING
    • Principal Component Analysis
      • How Does It Work?
      • Interpretation of PCA result
      • Applications in Transportation
    • CLUSTERING
      • K-MEANS
      • SPECTRAL CLUSTERING
      • Hierarchical Clustering
    • REFERENCE
  • CHAPTER 4 - NEURAL NETWORK
    • The Basic Paradigm: Multilayer Perceptron
    • Regression and Classification Problems with Neural Networks
    • Advanced Topologies
      • Modular Network
      • Coactive Neuro鈥揊uzzy Inference System
      • Recurrent Neural Networks
      • Jordan-Elman Network
      • Time-Lagged Feed-Forward Network
      • Deep Neural Networks
  • CHAPTER 5 - DEEP LEARNING
    • Convolutional Neural Networks
      • Introduction
      • Convolution Operation
      • Typical Layer Structure
      • Parameters and Hyperparameters
      • Summary of Key Features
      • Training of CNN
      • Transfer Learning
    • Recurrent Neural Networks
      • Introduction
      • Long Short-Term Memory Neural Network
      • Application in transportation
    • Recent Development
      • AlexNet, ZFNet, VggNet, and GoogLeNet
      • ResNet
      • U-Net: Full Convolutional Network
      • R-CNN, Fast R-CNN, and Faster R-CNN
      • Mask R-CNN
      • SSD and YOLO
      • RetinaNet
      • MobileNets
      • Deformable Convolution Networks
      • CenterNet
      • Exemplar Applications in Transportation
    • Reference
  • CHAPTER 6 - REINFORCEMENT LEARNING
    • Introduction
    • Reinforcement Learning Algorithms
    • Model-free v.s. Model-based Reinforcement Learning
    • Applications of Reinforcement Learning to Transportation and Traffic Engineering
    • REFERENCE
  • CHAPTER 7 - IMPLEMENTING ML AND COMPUTATIONAL REQUIREMENTS
    • Data Pipeline for Machine Learning
      • Introduction
      • Problem Definition
      • Data Ingestion
      • Data Preparation
      • Data Segregation
      • Model Training
      • Model Deployment
      • Performance Monitoring
    • Implementation Tools: The Machine Learning Ecosystem
      • Machine Learning Framework
      • Data Ingestion tools
      • Databases
      • Programming Languages
      • Visualization Tools
    • Cloud Computing
      • Types and Services
    • High-Performance Computing
      • Deployment on-premise vs on-cloud
      • Case Study: Data-driven approach for the implementation of Variable Speed Limit
      • Conclusion
  • CHAPTER 8 - RESOURCES
    • Mathematics and Statistics
    • Programming, languages, and software
    • Machine learning environments
    • Tools of the Trade
    • Online Learning Sites
    • Key Math Concepts
  • REFERENCES
  • IMPROVEMENT BACKLOG
Powered by GitBook
On this page
  1. CHAPTER 1 - INTRODUCTION TO MACHINE LEARNING
  2. Fundamental concepts of machine learning

Model Regularization

The goal of regularization is to avoid overfitting by penalizing more complex models. Regularization significantly reduces the variance of the model without a substantial increase in its bias. In general, to penalized more complex models, additional penalty term(s) is added to the cost function (Cost function + 位 * R, where R is coefficients or the weights and 位 is a tuning parameter). The tuning parameter controls the impact on the bias and variance and lets the researcher adjust the impact of coefficient (in machine learning) or weight matrices of the nodes (in deep learning) to overcome the overfitting problem. As the value of 位 increases, it reduced the value of coefficients and thus reducing the variance (hence avoid overfitting). Although, after certain value, the increase in 位 may lead to increase in bias in the model (hence result in underfitting). Two of the most common regularization methods are (1) Lasso regression (also known as L1 penalty), which penalize the absolute value of the coefficients/weights; and (2) Ridge regression (also known as L2 penalty), which penalize the squared value of the coefficients/weights. L2 regularization forces the weights to decay toward zero (but not exactly zero), while the weights in L1 regularization may be reduced to zero.

Another regularization techniques that can only be applied to neural networks is a drop-out. Drop-out regularization works by randomly selecting some nodes and remove them along with all of their incoming and outgoing connections at every iteration. This approach is similar to the idea of ensemble technique in machine learning and works very well when dealing with a large neural networks structure. The other techniques that help to avoid overfitting in the context of neural networks are data augmentation and early stopping. The former approach is more common in image processing where the data scientist try to increase the size of training data by adding some noise to the data(here image), such as rotating the image, flipping, scaling, shifting, blurring, etc.

PreviousHyperparameter TuningNextThe Curse of Ddimensionality

Last updated 1 year ago