Surprise#

Which of these two statements conveys the most information?

• I will eat some food tomorrow.

• I will see a giraffe walking by my apartment.

A measure of information (whatever it may be) is closely related to the element of… surprise!

• has very high probability and so conveys little information,

• has very low probability and so conveys much information.

If we quanitfy suprise we will quantify information

Addititivity of Information#

Knowledge leads to gaining information

Which is more surprising (contains more information)?

• E1: The card is heart? $P(E_1)=\frac{1}{4}$

• E2:The card is Queen? $P(E_2)=\frac{4}{52}=\frac{1}{13}$

• E3: The card is Queen of hearts? $P(E_{1}and{E}_2)=\frac{1}{52}$

Knowledge of event should add up our information: $I(E)\ge 0$

1. We learn the card is heart $I(E_1)$

2. We learn the card is Queen $I(E_2)$

3. $I(E_{1}and{E}_2)=I(E_1)+I(E_2)$

A logarithm of probability is a good candidate function for information!

• What about the sign?

Why bit (base two)#

• Consider symmetric a 1D random walk with equal jump probabilities. We can view Random walk = string of Yes/No questions.

• Imagine driving to a location how many left/right turn informations you need to reach destination?

• You gain one bit of information when you are told Yes/No answer

• To decode N step random walk trajectory we need N bits.

Shannon Entropy and Information#

• If we want to understand the overall uncertainty in the system, we need to consider all possible outcomes weighted by their probability of occurrence.

• This means that rather than looking at the surprise of a single event, we consider the average surprise one would experience over many trials drawn from $p$.

• Thus, we take the expectation of surprise over the entire distribution $⟨-logp⟩$ arriving at a formula known as Shaonon’s expression of Entropy.

• One often uses $H$ to denote Shanon Entropy ($lo{g}_2$) and letter $S$ with $lo{g}_e$ for entropy in units of Boltzman constant $k_B$

• For now lets just roll with $k_B=1$ we wont be doing any thermodynamics in here.

• John von Neumann advice to [To Calude Shanon], “You should call it Entropy, for two reasons. In the first place you uncertainty function has been used in statistical mechanics under that name. In the second place, and more importantly, no one knows what entropy really is, so in a debate you will always have the advantage.”

Show code cell source Hide code cell source

import numpy as np
import matplotlib.pyplot as plt

def binary_entropy(p):
    """
    Compute the binary Shannon entropy for a given probability p.
    Avoid issues with log(0) by ensuring p is never 0 or 1.
    """
    return -p * np.log2(p) - (1 - p) * np.log2(1 - p)

# Generate probability values, avoiding the endpoints to prevent log(0)
p_vals = np.linspace(0.001, 0.999, 1000)
H_vals = binary_entropy(p_vals)

# Create a figure with two subplots side-by-side
fig, ax = plt.subplots(1, 2, figsize=(14, 6))

# Plot 1: Binary Shannon Entropy Function
ax[0].plot(p_vals, H_vals, lw=4, color='midnightblue',
           label=r"$H(p)=-p\log_2(p)-(1-p)\log_2(1-p)$")
ax[0].set_xlabel(r"Probability $p$)", fontsize=14)
ax[0].set_ylabel("Entropy (bits)", fontsize=14)
ax[0].set_title("Binary Shannon Entropy",  fontsize=16)
ax[0].legend(fontsize=12)
ax[0].grid(True)

# Plot 2: Example Distributions and Their Entropy
# Define a few example two-outcome distributions:
distributions = {
    "Uniform (0.5, 0.5)": [0.5, 0.5],
    "Skewed (0.8, 0.2)": [0.8, 0.2],
    "Extreme (0.99, 0.01)": [0.99, 0.01]
}

# Colors for each distribution
colors = ["skyblue", "salmon", "lightgreen"]

# For visual separation, use offsets for the bars
width = 0.25
x_ticks = np.arange(2)  # positions for the two outcomes

for i, (label, probs) in enumerate(distributions.items()):
    # Compute the Shannon entropy for the distribution
    entropy_val = -np.sum(np.array(probs) * np.log2(probs))
    # Offset x positions for clarity
    x_positions = x_ticks + i * width - width
    ax[1].bar(x_positions, probs, width=width, color=colors[i],
              label=f"{label}\nEntropy = {entropy_val:.2f} bits")
    
# Set labels and title for the bar plot
ax[1].set_xticks(x_ticks)
ax[1].set_xticklabels(["Outcome 1", "Outcome 2"], fontsize=12)
ax[1].set_ylabel("Probability", fontsize=14)
ax[1].set_title("Example Distributions", fontsize=16)
ax[1].legend(fontsize=12)
ax[1].grid(True, axis='y')

plt.tight_layout()
plt.show()

• To reiterate once again from Shannon formula $S=\sum _i-p_{i}log{p}_i$ we see clearly that Entropy is a statistical quantity characterized by the probability distribution over all possibilities.

• The more uncertain (unpredictable) the outcome the higher the entropy of the probability distribution that generates it.

Show code cell source Hide code cell source

import numpy as np
import scipy.stats as stats
import matplotlib


# Create figure and axes
fig, axes = plt.subplots(2, 2, figsize=(10, 8))

# Define x range
x = np.linspace(-5, 5, 1000)

# Define different distributions
distributions = {
    "Flat (Uniform)": stats.uniform(loc=-2.5, scale=5),
    "Gaussian (Normal)": stats.norm(loc=0, scale=1),
    "Narrow Gaussian": stats.norm(loc=0, scale=0.5),
    "Exponential": stats.expon(scale=1)
}

# Plot distributions and annotate entropy
for ax, (name, dist) in zip(axes.flatten(), distributions.items()):
    pdf = dist.pdf(x)
    ax.plot(x, pdf, label=name, color='black')
    ax.fill_between(x, pdf, alpha=0.2, color='gray')
    
    # Compute differential entropy
    entropy = dist.entropy()
    ax.text(0, max(pdf) * 0.8, f"Entropy: {entropy:.2f}", fontsize=12, ha='center')
    
    ax.set_title(name)
    ax.set_yticks([])
    ax.legend()

# Adjust layout and show plot
plt.tight_layout()
plt.show()

Entropy, micro and macro states#

• When all $\mathrm{\Omega }$ number of microstates of the system have equal probability $p_i=\frac{1}{\mathrm{\Omega }}$ the entropy of the system is:

• We arrive at an expression of Entropy first obtained by Boltzmann where thermal energy units (Boltzman’s) constant were used.

• Entropy for a macrostate A which has $\mathrm{\Omega }(A)$ number of microstates can be written in terms of macrostate probability

• Entropy of a macrostate quantifies how probable that macrostate is!. This is yet another manifestation of Large Deviation Theorem we encountered before

Interpretations of Entropy#

Is Information Physical?#

• Wheeler’s “It from Bit”:

  • Every “it” — every particle, every field, every force, and even the fabric of space-time — derives its function, meaning, and very existence from binary answers to yes-or-no questions. In essence, Wheeler’s idea of “It from Bit” posits that at the most fundamental level, the physical universe is rooted in information. This perspective implies that all aspects of reality are ultimately information-theoretic in origin.

• Maxwell’s Demon:

  • Maxwell’s Demon is a thought experiment that challenges the second law of thermodynamics by envisioning a tiny being capable of sorting molecules based on their speeds. By selectively allowing faster or slower molecules to pass through a gate, the demon appears to reduce entropy without expending energy. However, the act of gathering and processing information incurs a thermodynamic cost, ensuring that the overall entropy balance is maintained. This paradox underscores that information is a physical quantity with measurable effects on energy and entropy.

Maximum Entropy (MaxEnt) Principle#

• Probability represents our incomplete information. Given partial knowledge about some variables how should we construct a probability distribution that is unbiased beyond what we know?

• The Maximum Entropy (MaxEnt) Principle provides the best approach: we choose probabilities to maximize Shannon entropy while satisfying given constraints.

• This ensures the least biased probability distribution possible, consistent with the available information.

• We seek to maximize $S(p)$ subject to the constraints:

• To enforce these constraints, we introduce Lagrange multipliers ${\lambda }_0,{\lambda }_1,{\lambda }_2,\dots$, leading to the Lagrangian (also called the objective function):

  $$J[p]=-\sum _k{p}_k\log p_k-{\lambda }_0(\sum _k{p}_k-1)-{\lambda }_1(\sum _k{p}_k{x}_k-⟨x⟩)-{\lambda }_2(\sum _k{p}_k{y}_k-⟨y⟩)-\dots$$

• To maximize $J[p]$, we take its functional derivative with respect to $p_k$:

• Since $e^{-1-{\lambda }_0}$ is simply a constant normalization factor, we define $Z=e^{1+{\lambda }_0}$ and arrie at final expression for probability distribution:

• Interperation of MaxEnt

  • The probability distribution takes an exponential form in the constrained variables $x_k,y_k,\dots$.

  • The normalization constant $Z$ (also called the partition function) ensures that the probabilities sum to 1.

  • The Lagrange multipliers ${\lambda }_1,{\lambda }_2,\dots$ encode the specific constraints imposed on the system.

  • This result is fundamental in statistical mechanics, where it leads to Boltzmann distributions, and in machine learning, where it underpins maximum entropy models.

Application MaxEnt: Biased Die Example#

• If we are given a fair die MaxENt would predict $p_i=1/6$ as there are no constraints.

• But suppose we are given a biased die the average outcome of which is rolling on average a number $⟨x⟩=5.5$. The entropy function to maximize becomes:

• Solving the variational equation, we find that the optimal probability distribution follows an exponential form:

• where $Z$ is the partition function ensuring normalization:

• To determine $B$, we use the constraint $⟨x⟩=5.5$:

• This equation can be solved numerically for $B$. In many cases, Newton’s method or other root-finding techniques can be employed to find the exact value of $B$. This distribution resembles the Boltzmann factor in statistical mechanics, where higher outcomes are exponentially less probable.

import numpy as np
from scipy.optimize import root
import matplotlib.pyplot as plt

x = np.arange(1, 7)
target_mean = 5.5

def maxent_prob(lambdas):
    '''
    Input: array of lambdas [lambda1, lambda2, ...]
    Return: array of probabilities [p1, p2, ...]
    '''

    lambda1= lambdas[0]
    Z = np.sum(np.exp(-lambda1 * x))
    return np.exp(-lambda1 * x) / Z

def constraint(lambdas):
    '''
    Input: array of lambdas [lambda1, lambda2, ...]
    Return: array of constraints [constraint1, constraint2, ...]
    '''

    p = maxent_prob(lambdas)
    mean = np.sum(x * p)
    return [mean - target_mean]

# Use root with initial guess
sol = root(constraint, [0.0])
lambda1 = sol.x[0]
p_opt = maxent_prob([lambda1]) # more then one lambda should go into array [lambda1, lambda2, ...]

print(f"λ1 = {lambda1_opt:.4f}")
for i, p in enumerate(p_opt, 1):
    print(f"P({i}) = {p:.4f}")

# Plot
plt.bar(x, p_opt, edgecolor='black')
plt.title("MaxEnt Distribution (Mean Constraint Only, using root)")
plt.xlabel("Die Outcome")
plt.ylabel("Probability")
plt.grid(axis='y', linestyle='--', alpha=0.6)
plt.xticks(x)
plt.show()

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[3], line 33
     30 lambda1 = sol.x[0]
     31 p_opt = maxent_prob([lambda1]) # more then one lambda should go into array [lambda1, lambda2, ...]
---> 33 print(f"λ1 = {lambda1_opt:.4f}")
     34 for i, p in enumerate(p_opt, 1):
     35     print(f"P({i}) = {p:.4f}")

NameError: name 'lambda1_opt' is not defined

Relative Entropy#

• We mentioned that in MaxEnt derivation we maximized entropy relative to an underlying assumption of equal-probability microstates. We make this idea precise here

• Consider example of 1D brownian particle starting at $x_0=0$ and diffusing freely. Its probability distribution after time $t$ follows a Gaussian:

• The Shannon entropy of this distribution is given by:

• Evaluating the integral yields nice compact formulla showing that entropy grows with time as molecule diffuse and sparead all over the container.

Problem: Grid Dependence of Shannon Entropy

• All is good but if we tried to evaluate the integral we would run into serious problem!

• Also how to interpret integral expression in terms of binary yes/no quesionts?

• A major issue with Shannon entropy is that it depends on the choice of units. If we refine the grid by choosing a smaller $\mathrm{\Delta }x$, the computed entropy does not converge to a well-defined value—it diverges! This makes it unsuitable for studying entropy change in diffusion.

• To avoid this issue, one can instead use relative entropy (Kullback-Leibler divergence), which remains well-defined and independent of discretization.

• The Kullback-Leibler (KL) divergence, or relative entropy, measures how much information is lost when using a reference distribution Q

• KL is non-negative and equals zero if and only if $P=Q$ everywhere.

• KL divergence is widely used in statistical mechanics, information theory, machine learning and thermodynamics as a measure of information loss when approximating one distribution with another.

Assymetry of KL and irreversibility#

• Given two normal distributions:

• We can compute KL divergence analytically showing that the function is assymteric

• Computing KL divergence between two gaussians describing diffusion at times $t_1$ and $t_2$, where their variances are ${\sigma }_1^2=2D{t}_1$ and ${\sigma }_2^2=2D{t}_2$ results in:

• For a diffusion process, if we compare the forward evolutionof diffusion where variance is spreading over time $t_1=t$ with the hypothetical reversed process where guassian contraints into a peak over time $t_2=T-t$ we can see that:

• This indicates that diffusion is an irreversible process in the absence of external driving forces (since it tends to increase entropy). In contrast, a time-reversed diffusion process (all particles contracting back into the initial state) would violate the second law of thermodynamics.

• This statistical distinguishability of time-forward and time-reversed processes is often referred to the “”thermodynamic arrow of time” showing that the forward flow of events is distinguishable from its reverse.

import numpy as np
import matplotlib.pyplot as plt

# Define time points
t_forward = np.linspace(0.1, 2, 100)
t_reverse = np.linspace(2, 0.1, 100)

# Define probability distributions for forward and reverse processes
x = np.linspace(-3, 3, 100)
sigma_forward = np.sqrt(t_forward[:, None])  # Diffusion spreads over time
sigma_reverse = np.sqrt(t_reverse[:, None])  # Reverse "contracts" over time

P_forward = np.exp(-x**2 / (2 * sigma_forward**2)) / (np.sqrt(2 * np.pi) * sigma_forward)
P_reverse = np.exp(-x**2 / (2 * sigma_reverse**2)) / (np.sqrt(2 * np.pi) * sigma_reverse)

# Compute Kullback-Leibler divergence D_KL(P_forward || P_reverse)
D_KL = np.sum(P_forward * np.log(P_forward / P_reverse), axis=1)

# Plot the distributions and KL divergence
fig, ax = plt.subplots(1, 2, figsize=(12, 5))

# Plot forward and reverse distributions
ax[0].imshow(P_forward.T, extent=[t_forward.min(), t_forward.max(), x.min(), x.max()], aspect='auto', origin='lower', cmap='Blues', alpha=0.7, label='P_forward')
ax[0].imshow(P_reverse.T, extent=[t_reverse.min(), t_reverse.max(), x.min(), x.max()], aspect='auto', origin='lower', cmap='Reds', alpha=0.5, label='P_reverse')

ax[0].set_xlabel("Time")
ax[0].set_ylabel("Position")
ax[0].set_title("Forward (Blue) vs Reverse (Red) Diffusion")
ax[0].arrow(0.2, 2.5, 1.5, 0, head_width=0.3, head_length=0.2, fc='black', ec='black')  # Forward arrow
ax[0].arrow(1.8, -2.5, -1.5, 0, head_width=0.3, head_length=0.2, fc='black', ec='black')  # Reverse arrow

# Plot KL divergence over time
ax[1].plot(t_forward, D_KL, label=r'$D_{KL}(P_{\mathrm{forward}} || P_{\mathrm{reverse}})$', color='black')
ax[1].set_xlabel("Time")
ax[1].set_ylabel("KL Divergence")
ax[1].set_title("Arrow of Time and Irreversibility")
ax[1].legend()

plt.tight_layout()
plt.show()

import numpy as np
import matplotlib.pyplot as plt
import ipywidgets as widgets
from ipywidgets import interact
from scipy.stats import norm

def plot_gaussians(mu1=0, sigma1=1, mu2=1, sigma2=2):
    x_values = np.linspace(-5, 5, 1000)  # Define spatial grid
    P = norm.pdf(x_values, loc=mu1, scale=sigma1)  # First Gaussian
    Q = norm.pdf(x_values, loc=mu2, scale=sigma2)  # Second Gaussian
    
    # Avoid division by zero in KL computation
    mask = (P > 0) & (Q > 0)
    D_KL_PQ = np.trapz(P[mask] * np.log(P[mask] / Q[mask]), x_values[mask])  # D_KL(P || Q)
    D_KL_QP = np.trapz(Q[mask] * np.log(Q[mask] / P[mask]), x_values[mask])  # D_KL(Q || P)
    
    # Plot the distributions
    plt.figure(figsize=(8, 6))
    plt.plot(x_values, P, label=fr'$P(x) \sim \mathcal{{N}}({mu1},{sigma1**2})$', linewidth=2)
    plt.plot(x_values, Q, label=fr'$Q(x) \sim \mathcal{{N}}({mu2},{sigma2**2})$', linewidth=2, linestyle='dashed')
    plt.fill_between(x_values, P, Q, color='gray', alpha=0.3, label=r'Difference between $P$ and $Q$')
    
    # Annotate KL divergences
    plt.text(-4, 0.15, rf'$D_{{KL}}(P || Q) = {D_KL_PQ:.3f}$', fontsize=12, color='blue')
    plt.text(-4, 0.12, rf'$D_{{KL}}(Q || P) = {D_KL_QP:.3f}$', fontsize=12, color='red')
    
    # Labels and legend
    plt.xlabel('$x$', fontsize=14)
    plt.ylabel('Probability Density', fontsize=14)
    plt.title('Interactive KL Divergence Between Two Gaussians', fontsize=14)
    plt.legend(fontsize=12)
    plt.grid(True)
    plt.show()

interact(plot_gaussians, 
         mu1=widgets.FloatSlider(min=-3, max=3, step=0.1, value=0, description='μ1'),
         sigma1=widgets.FloatSlider(min=0.1, max=3, step=0.1, value=1, description='σ1'),
         mu2=widgets.FloatSlider(min=-3, max=3, step=0.1, value=1, description='μ2'),
         sigma2=widgets.FloatSlider(min=0.1, max=3, step=0.1, value=2, description='σ2'))

Problems#

import numpy as np
import matplotlib.pyplot as plt

# Define mean values
mu_x, mu_y = 0, 0  # Mean of x and y

# Define standard deviations
sigma_x, sigma_y = 1, 1  # Standard deviation of x and y

# Define different correlation coefficients
correlations = [0.0, 0.5, 0.9]  # Varying degrees of correlation

# Generate and plot 2D Gaussian samples for different correlations
fig, axes = plt.subplots(1, 3, figsize=(15, 5))

for i, rho in enumerate(correlations):
    # Construct covariance matrix
    covariance_matrix = np.array([[sigma_x**2, rho * sigma_x * sigma_y],
                                   [rho * sigma_x * sigma_y, sigma_y**2]])
    
    # Generate samples from the 2D Gaussian distribution
    samples = np.random.multivariate_normal([mu_x, mu_y], covariance_matrix, size=1000)
    
    # Plot scatter of samples
    axes[i].scatter(samples[:, 0], samples[:, 1], alpha=0.5)
    axes[i].set_title(f"2D Gaussian with Correlation ρ = {rho}")
    axes[i].set_xlabel("X")
    axes[i].set_ylabel("Y")

plt.tight_layout()
plt.show()