Chapter 1: Understanding Momentum in Gradient Descent

In our previous discussion on optimization techniques, we delved into various methods, particularly gradient descent. We noted that traditional gradient descent can suffer from slow convergence, especially when dealing with flat gradients. To tackle this issue, we introduced conjugate gradient descent as a potential solution. In this article, we will further examine momentum gradient descent, another powerful technique for enhancing optimization.

When we consider optimization algorithms such as gradient descent, we can visualize momentum as a ball rolling down a gently inclined surface. As the ball descends, it accumulates momentum, similar to how past gradient influences affect the current update direction. In regions where the gradient is shallow, traditional gradient descent may advance sluggishly. However, by incorporating momentum, the optimization process can retain a 'memory' of prior gradients, enabling it to maintain or even increase its speed—much like a rolling ball gains velocity. This allows momentum gradient descent to traverse flat areas or shallow local minima more effectively, leading to quicker convergence towards the optimal solution.

This provides a basic understanding, but I suspect you're looking for more practical insights. Let's illustrate this with some Python examples:

import numpy as np

import matplotlib.pyplot as plt

# Function definition

def J(theta):

return theta**4 - 3 * theta**3 + 2

# Gradient of the function

def grad_J(theta):

return 4 * theta**3 - 9 * theta**2

# Generate theta values

theta_values = np.linspace(-2, 3, 400)

J_values = J(theta_values)

grad_values = grad_J(theta_values)

# Plot the function and its gradient

plt.figure(figsize=(10, 6))

plt.plot(theta_values, J_values, label='J(theta)', color='b')

plt.plot(theta_values, grad_values, label='Gradient of J(theta)', color='r')

plt.xlabel('Theta')

plt.ylabel('Value')

plt.title('Function and its Gradient')

plt.legend()

plt.grid(True)

plt.show()

The graph of the function J(theta) and its gradient reveals that within the range of [-0.5, 1], there is minimal variation in both the function and its gradient. Hence, we can anticipate that gradient descent will proceed slowly in this segment.

In traditional gradient descent, the model parameters (weights) are updated at each iteration in the direction opposite to the gradient of the loss function. This update can be represented as follows:

# Gradient Descent

def gradient_descent(alpha, theta_init, num_iterations):

theta = theta_init

theta_history = [theta_init]

for _ in range(num_iterations):

gradient = grad_J(theta)

theta = theta - alpha * gradient

theta_history.append(theta)

return np.array(theta_history)

Momentum enhances this approach by adding a portion of the previous update vector to the current one. This modification accelerates learning in the relevant direction while reducing oscillations. The update rule for momentum becomes:

# Momentum Gradient Descent

def momentum_gradient_descent(alpha, beta, theta_init, num_iterations):

theta = theta_init

v = 0

theta_history = [theta_init]

for _ in range(num_iterations):

gradient = grad_J(theta)

v = beta * v + (1 - beta) * gradient

theta = theta - alpha * v

theta_history.append(theta)

return np.array(theta_history)

Here, beta represents the momentum term (usually between 0 and 1), alpha is the learning rate, and theta_init is the initial guess for theta.

Let's execute both functions and visualize the results:

# Parameters

alpha = 0.001 # Learning rate

beta = 0.8 # Momentum parameter

theta_init = -5 # Initial guess for theta

num_iterations = 40

# Run Gradient Descent

theta_history_gd = gradient_descent(alpha, theta_init, num_iterations)

# Run Momentum Gradient Descent

theta_history_mgd = momentum_gradient_descent(alpha, beta, theta_init, num_iterations)

# Plot convergence

plt.figure(figsize=(10, 6))

plt.plot(J(theta_history_gd), label="Gradient Descent", linestyle='-', marker='o')

plt.plot(J(theta_history_mgd), label="Momentum Gradient Descent", linestyle='-', marker='o')

plt.xlabel('Iterations')

plt.ylabel('Loss (J)')

plt.title('Convergence Comparison')

plt.legend()

plt.grid(True)

plt.show()

Initially, traditional gradient descent converges rapidly; however, by the 10th iteration, momentum gradient descent achieves a lower loss than its counterpart. In fact, traditional gradient descent only reaches the same loss as momentum around the 30th iteration.

Mathematically, we can express the concepts of gradient descent and momentum gradient descent as follows:

In conclusion, momentum gradient descent significantly enhances traditional gradient descent by introducing momentum, which allows the optimization process to maintain its course even when the gradient is flat or when encountering local minima.

Thank you for reading! Be sure to subscribe for updates on future publications. If you enjoyed this article, consider following me for more insights. Alternatively, if you're eager to learn more, check out my book "Data-Driven Decisions: A Practical Introduction to Machine Learning." It offers comprehensive information to get you started in Machine Learning at an affordable price.

Chapter 2: Practical Examples of Momentum Gradient Descent

This first video explains the intuition and mathematics behind gradient descent with momentum, providing a detailed understanding of the concept.

In this second video, the application of momentum gradient descent is demonstrated in a practical context, enhancing your grasp of the technique.