# How Does It Work?

The goal of PCA method is to find out the principal components of the dataset. The principal components are the values of this dataset projected into the principal directions. The direction which has the largest variance of projected value is taken as the first principal direction, then is the second principal direction and so on. Each principal direction is orthogonal to each other.&#x20;

Therefore, PCA method can be explained by answering the following two questions, a) what is the principal components and principal directions, b) how to find out the principal components and principal directions



**Principal Components**

The first principal component is defined in such a way that if we project the dataset into a vector linearly, the projected value of the samples has the largest possible variance, then the projected value of each sample is called principal component, and this vector is the principal direction. By making the variance of projected values as large as possible, we can guarantee that samples are spitted as much as possible in this direction.  Each succeeding principal components in turn has the largest projected variance under the constraint that this principal component is orthogonal to the preceding components.

The first principal component is defined in such a way that if we project the dataset into a vector linearly, the projected value of the samples has the largest possible variance, then the projected value of each sample is called principal component, and this vector is the principal direction. By making the variance of projected values as large as possible, we can guarantee that samples are spitted as much as possible in this direction. Each succeeding principal components in turn has the largest projected variance under the constraint that this principal component is orthogonal to the preceding components.

**Principal Components Calculation**

There are many approaches to calculate the principal components of the dataset, such as gradient descent method, eigen-decomposition of covariance/correlation matrix method, singular value decomposition of covariance/correlation method and so on(Shamir, 2016, Smith, 2002, Wall et al., 2003). In this section, we will give a step by step tutorial of the eigen-decomposition method and singular value decomposition method. The last method is also famous for its high efficiency and is widely used by different machine learning packages.

**Step 1. Data preprocessing – standardization**

Standardizing the data prior to the PCA process is crucial for most dataset. By subtracting the mean value and dividing the standard deviation of each column, each feature of the dataset can be transformed into unit scale (mean=0 and variance =1). This process is quite useful especially when the features are measured on different scales. Because if one feature’s scale is much more than others, the variance of this feature will significantly larger than others, which made this feature has a higher weight for the calculation of principal components. Mathematically, each data in the dataset can be standarized by the following equation:&#x20;

<figure><img src="https://lh7-us.googleusercontent.com/6TpHLu6JaB7azK4VWUgU_BYtQSB5jtoRZvPHTkbwxrSNEPfqsOMI-ILhDBaNYxy-UcAuXzHhclEdP8bOYAZc7HSmd0cAY96prE_AHxqdzWD0eMkSpcbIKtymgFCZjtCv3s8sC2gPl56bkpkYF7Wb" alt="" width="188"><figcaption></figcaption></figure>

where <img src="https://lh7-us.googleusercontent.com/5liHzz3mAeZnKQY5cs5vGKS6HGfMRlMyV8pbTLE5sza-2C4hUZjzVsb9oSazy7Yl7fRPkvX263POduewbK4jBPLZXPgcIKWsuJdL06vIqlOZgF3BWOBrYV85CTyEbN_YHQmAyTSq-Tjmy7zHjBKi" alt="" data-size="line"> is the standardized value of <img src="https://lh7-us.googleusercontent.com/8HWPdbvRwFlGW4QFP5eDvuusOB7zUYvkBRgWl1m6Xzt61WIH26mBtn8v7ugdzly8k2uQ1i_UxpyqyckJpzY2OhZNqYiOJ7ctR0QSW5MzBPHwCWUZwyXJglDSizVvu964hN8UA6GSfHfKtTd0opVi" alt="" data-size="line">th sample <img src="https://lh7-us.googleusercontent.com/XpO6xCJRS3heUrJXmJYVeQpLY_kHSgf2TLkVR9iE5-IfbiFpGBa4pOZfMwb0W2s-iOocgkGW-AXCrvgGFwYPnFhMrllYE-hr-x9WPFZ95ArCW3gEjbqGfhMe29QPRKeX9l-0H9V9oLnGvTNAGu2m" alt="" data-size="line">th feature;<img src="https://lh7-us.googleusercontent.com/jbFj-NjlKz7FaQumyBxb5wIB-cS45332zsv9By4m5wjRUXdQ5qxraTZgz6FrS6HPU3rPAykicJM8t5MoU9pdhu1JioJynCTW_n59L_Gczv6tzGS1ZsY1eQnVWvR9OTm6KETd8D18t_ZyyLHnhJbv" alt="" data-size="line"> is the data of the dataset; <img src="https://lh7-us.googleusercontent.com/xxleV3BCuIaYJkACnMiD4UygcqfeV_fi5M7PAs5G_aVoKj9ed4kXWT5Zol50ZvBR6iocVY5dy7GEIaKnaUo4QACpoWxWgmuWE8raHCIHboSVPEjR-AyMMzHGl4g5ag8EE7scJZ5BVnBKIx88V7Qh" alt="" data-size="line"> is the mean value of <img src="https://lh7-us.googleusercontent.com/ZxJMqcLWOG1NG6t-dMCIZggq3aaF41ZqOtnpozrobtCJYUCzP_rKWzJH5XqQh_OJBfuGqBri0BNZVcuRkw7OA1fYlk_81FaMekzl298Ajb-QBsLYUEUb1GbmNjYcKGfpzJ8Y8YFOW8tVJ-eDnc3w" alt="" data-size="line">th feature of the dataset; <img src="https://lh7-us.googleusercontent.com/W3VI1FaRiIIam755MYH5w6edwHPlBndDrDrvuZq9_W2BcdDd4DDl74JdB2HjbLN0ckdMvRv5NPRgrmf1oy4cfJ6tXF_7lwlhNlKFImhwAlvy32uENIZFh3eLYtBhpc-uOu8KF9CZEEeoOk6Z1U9R" alt="" data-size="line"> is the variance of the <img src="https://lh7-us.googleusercontent.com/xEU9-dK17SBCWdTIZEyJxRdLG-dqgD9g01h6GpKfc3h8yzcfOKG6-W2PvGGvpNnO7nyGW7s73pbOxBL8j82yMnNMPzl5NXm2_VlikpR9X18KXAlS0mzUMe-t0lP0e65D0wcgupq4cgYl0YwRWASl" alt="" data-size="line">th feature of the dataset.

**Step 2, Covariance/correlation matrix Calculation**

The classic approach to PCA is applying eigen-decomposition on its covariance /correlation matrix. The eigenvectors are the principal directions, while the eigenvalues represent the variance of the dataset along this vector. In other words, the eigenvector with highest eigenvalue is the first principal direction. Then the second largest eigenvalue corresponds to the second principal direction and so on.

**Step 3, Eigen-decomposition of the covariance matrix or correlation matrix.**

Since both covariance matrix and correlation matrix are symmetric matrices with a dimension <img src="https://lh7-us.googleusercontent.com/OUS1PZ6jGqFg8EWBXSrKXe88Sm-QDjXe4hCq3cBnrFMNqCaqSrfvvRELchnU5sdKiedtrFom6yU1pM1Y04wWvr_JcS_Soqd4KFVxXa7s5zlsN1xq6qvyBYPJfFTji3Rx8W0Wd_NK1h2U4F8BknHg" alt="" data-size="line">. The eigen decomposition for real symmetric matrices can be represented as

Different from the eigen decomposition of covariance/correlation matrix, the SVD method can compute the principal directions and principal components at the same time, which is very efficient for big dataset computation.&#x20;

\
\
\
\
\
\


\
\
\
\
\
\
