Stochastic Gradient Descent

The full form of SGD is Stochastic Gradient Descent. It is an iterative optimization algorithm that is used to find the minimum of a function. SGD works by randomly selecting one data point at a time and updating the parameters of the model in the direction of the negative gradient of the function at that data point.

SGD is a popular algorithm for training machine learning models, especially neural networks. It is relatively simple to implement and can be used to train models on large datasets. However, SGD can be slow to converge and may not always find the global minimum of the function. 

I can explain how SGD works with an example. Let's say we have a neural network that is trying to learn to predict the price of a stock. The neural network has a set of parameters, such as the weights and biases of the individual neurons. The goal of SGD is to find the values of these parameters that minimize the error between the predicted prices and the actual prices.

SGD works by iteratively updating the parameters of the neural network. At each iteration, SGD randomly selects one training example and calculates the gradient of the error function with respect to the parameters. The gradient is a vector that points in the direction of the steepest descent of the error function. SGD then updates the parameters in the opposite direction of the gradient, by a small amount called the learning rate.

This process is repeated for many iterations until the error function converges to a minimum. The following diagram illustrates how SGD works:

The blue line represents the error function, and the red line represents the path taken by SGD. As you can see, SGD starts at a random point and gradually moves towards the minimum of the error function.

The learning rate is a hyperparameter that controls the size of the updates to the parameters. A larger learning rate will cause SGD to converge more quickly, but it may also cause the algorithm to overshoot the minimum and oscillate around it. A smaller learning rate will cause SGD to converge more slowly, but it will be less likely to overshoot the minimum.

The number of iterations is another hyperparameter that controls the convergence of SGD. A larger number of iterations will usually result in a more accurate model, but it will also take longer to train the model.

SGD is a simple but effective optimization algorithm that is widely used in machine learning. It is often used to train neural networks, but it can also be used to train other types of models.

photos from researchgate