Entropy and Information

Surprise¶

The sequences

have exactly the same probability for a fair coin:

• So why does one feel shocking and the other boring?

• How would you describe the sequence to a friend?

• Frame problem in terms of micro and macro states.

A measure of information (whatever it may be) is closely related to the element of... surprise!

• has very high probability and so conveys little information,

• has very low probability and so conveys much information.

If we quanitfy suprise we will quantify information

Addititivity of Information¶

Knowledge leads to gaining information

Which is more surprising (contains more information)?

• E1: The card is heart? $P(E_1) = \frac{1}{4}$

• E2:The card is Queen? $P(E_2) = \frac{4}{52} = \frac{1}{13}$

• E3: The card is Queen of hearts? $P(E_1 \, and\, E_2) = \frac{1}{52}$

Knowledge of event should add up our information: $I(E) \geq 0$

1. We learn the card is heart $I(E_1)$

2. We learn the card is Queen $I(E_2)$

3. $I(E_1 and E_2) = I(E_1) + I(E_2)$

A logarithm of probability is a good candidate function for information!

• What about the sign?

Why bit (base two)¶

• Consider symmetric a 1D random walk with equal jump probabilities. We can view Random walk = string of Yes/No questions.

• Imagine driving to a location how many left/right turn informations you need to reach destination?

• You gain one bit of information when you are told Yes/No answer

• To decode N step random walk trajectory we need N bits.

Shannon Entropy and Information¶

• If we want to understand the overall uncertainty in the system, we need to consider all possible outcomes weighted by their probability of occurrence.

• This means that rather than looking at the surprise of a single event, we consider the average surprise one would experience over many trials drawn from $p$.

• Thus, we take the expectation of surprise over the entire distribution $\langle -log p \rangle$ arriving at a formula known as Shaonon’s expression of Entropy.

• One often uses $H$ to denote Shanon Entropy ($log_2$) and letter $S$ with $log_e$ for entropy in units of Boltzman constant $k_B$

• For now lets just roll with $k_B=1$ we wont be doing any thermodynamics in here.

• John von Neumann advice to [To Calude Shanon], "You should call it Entropy, for two reasons. In the first place you uncertainty function has been used in statistical mechanics under that name. In the second place, and more importantly, no one knows what entropy really is, so in a debate you will always have the advantage.”

• To reiterate once again from Shannon formula $S=\sum_i -p_ilogp_i$ we see clearly that Entropy is a statistical quantity characterized by the probability distribution over all possibilities.

• The more uncertain (unpredictable) the outcome the higher the entropy of the probability distribution that generates it.

Entropy, micro and macro states¶

• When all $\Omega$ number of microstates of the system have equal probability $p_i=\frac{1}{\Omega}$ the entropy of the system is:

• We arrive at an expression of Entropy first obtained by Boltzmann where thermal energy units (Boltzman’s) constant were used.

• Entropy for a macrostate A which has $\Omega(A)$ number of microstates can be written in terms of macrostate probability

• Entropy of a macrostate quantifies how probable that macrostate is!. This is yet another manifestation of Large Deviation Theorem we encountered before

Interpretations of Entropy¶

1. Entropy as a Measure of Information¶

• Entropy quantifies the amount of information needed to specify the exact microstate of a system.

• This can be understood as the number of yes/no (binary) questions required to identify a specific microstate.

• Examples:

  • Determining the exact trajectory of an $N$-step random walk.

  • Identifying the detailed molecular distribution of gas particles in a container.

2. Entropy as a Measure of Uncertainty¶

• A more uniform (flat) probability distribution of microstates corresponds to higher entropy, whereas a more concentrated (narrow) distribution leads to lower entropy.

• Higher entropy implies greater uncertainty about microstates.

  • In a high-entropy system, identifying the exact microstate is much harder.

  • In contrast, a low-entropy system is more predictable, as fewer microstates are accessible.

• When all microstates are equally probable, entropy simplifies to be just the logarithm of the number of microstates, $S = k_B \log \Omega$.

• A system will naturally tend to evolve toward macrostates with higher entropy because they correspond to a larger number of available microstates, making them statistically more likely.

3. Physical and Thermodynamic Implications¶

• Systems with high entropy have a vast number of possible microstates, meaning more “work” is needed—in an informational sense—to pinpoint a specific one.

• Reducing entropy requires physical work:

  • Example: To reduce the number of yes/no questions needed to specify the position of gas molecules, one must compress the gas, which requires energy.

• Spontaneous and irreversible processes tend to increase entropy:

  • Entropy naturally increases in isolated systems because evolution toward more probable (higher entropy) macrostates is statistically favored.

  • This principle underlies the Second Law of Thermodynamics, which states that entropy never decreases in a closed system.

Is Information Physical?¶

• Wheeler’s “It from Bit”:

  • Every “it” — every particle, every field, every force, and even the fabric of space-time — derives its function, meaning, and very existence from binary answers to yes-or-no questions. In essence, Wheeler’s idea of “It from Bit” posits that at the most fundamental level, the physical universe is rooted in information. This perspective implies that all aspects of reality are ultimately information-theoretic in origin.

• Maxwell’s Demon:

  • Maxwell’s Demon is a thought experiment that challenges the second law of thermodynamics by envisioning a tiny being capable of sorting molecules based on their speeds. By selectively allowing faster or slower molecules to pass through a gate, the demon appears to reduce entropy without expending energy. However, the act of gathering and processing information incurs a thermodynamic cost, ensuring that the overall entropy balance is maintained. This paradox underscores that information is a physical quantity with measurable effects on energy and entropy.

Maximum Entropy (MaxEnt) Principle¶

• Probability represents our incomplete information. Given partial knowledge about some variables how should we construct a probability distribution that is unbiased beyond what we know?

• The Maximum Entropy (MaxEnt) Principle provides the best approach: we choose probabilities to maximize Shannon entropy while satisfying given constraints.

• This ensures the least biased probability distribution possible, consistent with the available information.

• We seek to maximize $S(p)$ subject to the constraints:

• To enforce these constraints, we introduce Lagrange multipliers $\lambda_0, \lambda_1, \lambda_2, \dots$, leading to the Lagrangian (also called the objective function):

  $$J[p] = - \sum_k p_k \log p_k - \lambda_0 \left( \sum_k p_k - 1 \right) - \lambda_1 \left( \sum_k p_k \, x_k - \langle x \rangle \right) - \lambda_2 \left( \sum_k p_k \, y_k - \langle y \rangle \right) - \dots$$

  (23)

• To maximize $J[p]$, we take its functional derivative with respect to $p_k$:

• Since $e^{-1 - \lambda_0}$ is simply a constant normalization factor, we define $Z = e^{1+\lambda_0}$ and arrie at final expression for probability distribution:

• Interperation of MaxEnt

  • The probability distribution takes an exponential form in the constrained variables $x_k, y_k, \dots$.

  • The normalization constant $Z$ (also called the partition function) ensures that the probabilities sum to 1.

  • The Lagrange multipliers $\lambda_1, \lambda_2, \dots$ encode the specific constraints imposed on the system.

  • This result is fundamental in statistical mechanics, where it leads to Boltzmann distributions, and in machine learning, where it underpins maximum entropy models.

Application MaxEnt: Biased Die Example¶

• If we are given a fair die MaxENt would predict $p_i=1/6$ as there are no constraints.

• But suppose we are given a biased die the average outcome of which is rolling on average a number $\langle x \rangle = 5.5$. The entropy function to maximize becomes:

• Solving the variational equation, we find that the optimal probability distribution follows an exponential form:

• where $Z$ is the partition function ensuring normalization:

• To determine $B$, we use the constraint $\langle x \rangle = 5.5$:

• This equation can be solved numerically for $B$. In many cases, Newton’s method or other root-finding techniques can be employed to find the exact value of $B$. This distribution resembles the Boltzmann factor in statistical mechanics, where higher outcomes are exponentially less probable.

Relative Entropy¶

• We mentioned that in MaxEnt derivation we maximized entropy relative to an underlying assumption of equal-probability microstates. We make this idea precise here

• Consider example of 1D brownian particle starting at $x_0 = 0$ and diffusing freely. Its probability distribution after time $t$ follows a Gaussian:

• The Shannon entropy of this distribution is given by:

• Evaluating the integral yields nice compact formulla showing that entropy grows with time as molecule diffuse and sparead all over the container.

Problem: Grid Dependence of Shannon Entropy

• All is good but if we tried to evaluate the integral we would run into serious problem!

• Also how to interpret integral expression in terms of binary yes/no quesionts?

• A major issue with Shannon entropy is that it depends on the choice of units. If we refine the grid by choosing a smaller $\Delta x$, the computed entropy does not converge to a well-defined value—it diverges! This makes it unsuitable for studying entropy change in diffusion.

• To avoid this issue, one can instead use relative entropy (Kullback-Leibler divergence), which remains well-defined and independent of discretization.

• $D_{\text{KL}}(P || Q)$ quantifies information cost of using Q instead of P.

• KL is non-negative and equals zero if and only if $P = Q$ everywhere.

• KL divergence is widely used in statistical mechanics, information theory, machine learning and thermodynamics as a measure of information loss when approximating one distribution with another.

Assymetry of KL and irreversibility¶

• Given two normal distributions:

• We can compute KL divergence analytically showing that the function is assymteric

• Computing KL divergence between two gaussians describing diffusion at times $t_1$ and $t_2$, where their variances are $\sigma_1^2 = 2D t_1$ and $\sigma_2^2 = 2D t_2$ results in:

• For a diffusion process, if we compare the forward evolutionof diffusion where variance is spreading over time $t_1=t$ with the hypothetical reversed process where guassian contraints into a peak over time $t_2=T-t$ we can see that:

• This indicates that diffusion is an irreversible process in the absence of external driving forces (since it tends to increase entropy). In contrast, a time-reversed diffusion process (all particles contracting back into the initial state) would violate the second law of thermodynamics.

• This statistical distinguishability of time-forward and time-reversed processes is often referred to the “"thermodynamic arrow of time” showing that the forward flow of events is distinguishable from its reverse.

Relative Entropy in Machine learning¶

1. If $Q(x)$ assigns very low probability to a region where $P(x)$ is high, the term $\log \frac{P(x)}{Q(x)}$ becomes large, strongly penalizing $Q$ for underestimating $P$.

2. If $Q(x)$ is broader than $P(x)$, assigning extra probability mass to unlikely regions, this does not significantly affect $D_{\text{KL}}(P || Q)$, because $P(x)$ is small in those regions.

• This asymmetry explains why KL divergence is not a true distance metric. It penalizes underestimation of true probability mass much more than overestimation, making it particularly useful in machine learning where models are trained to avoid assigning near-zero probabilities to observed data.

Problems¶

Problem-1: Compute Entropy for gas partitioning¶

• A container is divided into two equal halves. It contains 100 non-interacting gas particles that can freely move between the two sides. A macrostate is defined by the fraction of particles in the right half $f$.

1. Write down an exprssion for entropy S(f)

   • Consult the section on Random Walk and Combinatorics formula

2. Evaluate $S(f)$ for:

   • $f_L = 0.1$ (10% left, 90% right)

   • $f_L = 0.25$ (25% left, 75% right)

   • $f_L = 0.5$ (50% left, 50% right)

3. Relate entropy to probability using Large Deviation Theory and answer the following questions:

   • Which macrostate is most probable?

   • How does entropy influence probability?

   • Why are extreme fluctuations (e.g., $f = 0.1$) unlikely?

Problem-2¶

• Below we generate three 2D gaussians $P(x,y)$ with varying degree of correltion betwene $x$ and $y$. Run the code to visualize

import numpy as np
import matplotlib.pyplot as plt

# Define mean values
mu_x, mu_y = 0, 0  # Mean of x and y

# Define standard deviations
sigma_x, sigma_y = 1, 1  # Standard deviation of x and y

# Define different correlation coefficients
correlations = [0.0, 0.5, 0.9]  # Varying degrees of correlation

# Generate and plot 2D Gaussian samples for different correlations
fig, axes = plt.subplots(1, 3, figsize=(15, 5))

for i, rho in enumerate(correlations):
    # Construct covariance matrix
    covariance_matrix = np.array([[sigma_x**2, rho * sigma_x * sigma_y],
                                   [rho * sigma_x * sigma_y, sigma_y**2]])
    
    # Generate samples from the 2D Gaussian distribution
    samples = np.random.multivariate_normal([mu_x, mu_y], covariance_matrix, size=1000)
    
    # Plot scatter of samples
    axes[i].scatter(samples[:, 0], samples[:, 1], alpha=0.5)
    axes[i].set_title(f"2D Gaussian with Correlation ρ = {rho}")
    axes[i].set_xlabel("X")
    axes[i].set_ylabel("Y")

plt.tight_layout()
plt.show()

• Compute entropy using probability distributions $P(x,y), P(x|y=0), P(x)$ and $P(y)$

• Compute relative entropy $D_{KL}(P(x,y)|| p(x) p(y))$ which is known as Mutual Information. How can you interpret this quantity

• You will need these functions. Also check out scipy.stats.entropy

Problem-3: Gambling with ten sided die¶

• Take a ten sided die described by random variable $X =1,2....10$

• Experiment shows that after many throughs the average number die shows is $E[x]=5$ with variance $V[X] = 1.5$.

• Determine probabilities of ten die sides $p(x)$. You can use the code for six sided die in this notebook. But remember there are now two constraints one on mean and one on variance!

Problem-3 Biased Random walker in 2D¶

A random walker moves on a 2D grid, taking steps in one of four possible directions:

Each step occurs with unknown probabilities:

which satisfy the normalization condition:

Experimental measurements indicate the expected displacement per step:

Using the Maximum Entropy (MaxEnt) principle, determine the probabilities $p^*$ by numerically solving an optimization problem.

• You will need scipy.minimize(entropy, initial_guess, bounds=bounds, constraints=constraints, method='SLSQP') to do the minimization. Check the documentation to see how it works

Problem-5: Arrow of time in 2D diffusion¶

1. Simulate 2D diffusion using a Gaussian distribution

   • Model diffusion in two dimensions with a probability distribution:

     $$P(x, y, t) = \frac{1}{4\pi D t} \exp\left(-\frac{x^2 + y^2}{4 D t}\right)$$

     (66)

   • Generate samples at different time steps $t$ using numpy.random.normal with time-dependent variance.

2. Compute the relative entropy as a function of time $S_{\text{rel}}(t)$

   • Use relative entropy (Kullback-Leibler divergence) to compare the probability distributions at time $t$ with a reference time $t_0 = 0.1$:

     $$D_{\text{KL}}(P(x, y, t) || P(x, y, t_0))$$

     (67)

3. Compare forward and reverse evolution of diffusion

   • Compute the relative entropy between forward ($t \to t + \Delta t$) and reverse ($t \to t - \Delta t$) diffusion distributions.

   • Quantify how diffusion leads to irreversible evolution by evaluating:

     $$D_{\text{KL}}(P_{\text{forward}} || P_{\text{reverse}})$$

     (68)

4. Interpret the results

   • How does relative entropy change with time?

   • Why is the forward process distinguishable from the reverse process?

   • What does this tell us about the arrow of time?