Entropy and Information

Surprise (self-information)¶

The sequences

have exactly the same probability for a fair coin:

• So why does one feel shocking and the other boring?

• How would you describe the sequence to a friend?

• Frame problem in terms of micro and macro states.

A measure of information (whatever it may be) is closely related to the element of... surprise!

• has very high probability and so conveys little information,

• has very low probability and so conveys much information.

If we quanitfy suprise we will quantify information

Addititivity of Information¶

Knowledge leads to gaining information

Which is more surprising (contains more information)?

• E1: The card is heart? $P(E_1) = \frac{1}{4}$

• E2:The card is Queen? $P(E_2) = \frac{4}{52} = \frac{1}{13}$

• E3: The card is Queen of hearts? $P(E_1 \, and\, E_2) = \frac{1}{52}$

Knowledge of event should add up our information: $I(E) \geq 0$

1. We learn the card is heart $I(E_1)$

2. We learn the card is Queen $I(E_2)$

3. $I(E_1 and E_2) = I(E_1) + I(E_2)$

A logarithm of probability is a good candidate function for information!

• What about the sign?

• This quantifies surprise or self-information associated with one specific event or microstate with probability $p_i$

Why bit (base two)¶

• Consider symmetric a 1D random walk with equal jump probabilities. We can view Random walk = string of Yes/No questions.

• Imagine driving to a location how many left/right turn informations you need to reach destination?

• You gain one bit of information when you are told Yes/No answer

• To decode N step random walk trajectory we need N bits.

Shannon Entropy and Information¶

• If we want to understand the overall uncertainty in the system, we need to consider all possible outcomes weighted by their probability of occurrence.

• This means that rather than looking at the surprise of a single event, we consider the average surprise one would experience over many trials drawn from $p$.

• Thus, we take the expectation of surprise over the entire distribution $\langle -log p \rangle$ arriving at a formula known as Shaonon’s expression of Entropy.

• One often uses $H$ to denote Shanon Entropy ($log_2$) and letter $S$ with $log_e$ for entropy in units of Boltzman constant $k_B$

• For now lets just roll with $k_B=1$ we wont be doing any thermodynamics in here.

• John von Neumann advice to [To Calude Shanon], "You should call it Entropy, for two reasons. In the first place you uncertainty function has been used in statistical mechanics under that name. In the second place, and more importantly, no one knows what entropy really is, so in a debate you will always have the advantage.”

• To reiterate once again from Shannon formula $S=\sum_i -p_ilogp_i$ we see clearly that Entropy is a statistical quantity characterized by the probability distribution over all possibilities.

• The more uncertain (unpredictable) the outcome the higher the entropy of the probability distribution that generates it.

Entropy, micro and macro states¶

• When all $\Omega$ number of microstates of the system have equal probability $p_i=\frac{1}{\Omega}$ the entropy of the system is:

• We arrive at an expression of Entropy first obtained by Boltzmann where thermal energy units (Boltzman’s) constant were used.

• Entropy for a macrostate A which has $\Omega(A)$ number of microstates can be written in terms of macrostate probability

• Entropy of a macrostate quantifies how probable that macrostate is!. This is yet another manifestation of Large Deviation Theorem we encountered before

1. Entropy as Information¶

Entropy measures the information required to specify the exact microstate of a system. Equivalently, it is the number of binary (yes/no) questions needed to uniquely identify that microstate.

Examples include specifying:

• the full trajectory of an (N)-step random walk,

• the detailed molecular configuration of a gas.

2. Entropy as Uncertainty¶

Entropy quantifies uncertainty over microstates.

• Broad, uniform distributions over microstates correspond to high entropy.

• Narrow, concentrated distributions correspond to low entropy.

When all microstates are equally likely,

• where $\Omega$ is the number of accessible microstates.

• Systems naturally evolve toward macrostates with higher entropy because these correspond to overwhelmingly more microstates and are therefore statistically favored.

3. Physical Meaning¶

High entropy means that many microstates are consistent with the same macrostate, making the exact microstate hard to identify.

• Lowering entropy requires physical work, such as compressing a gas to reduce the number of accessible configurations.

• In isolated systems, entropy tends to increase, reflecting the statistical bias toward more probable macrostates.

This statistical tendency is the content of the Second Law of Thermodynamics.

Is Information Physical?¶

• Maxwell’s Demon:

  • Maxwell’s Demon is a thought experiment that challenges the second law of thermodynamics by envisioning a tiny being capable of sorting molecules based on their speeds. By selectively allowing faster or slower molecules to pass through a gate, the demon appears to reduce entropy without expending energy. However, the act of gathering and processing information incurs a thermodynamic cost, ensuring that the overall entropy balance is maintained. This paradox underscores that information is a physical quantity with measurable effects on energy and entropy.

Maximum Entropy (MaxEnt) Principle¶

• Probability represents our incomplete information. Given partial knowledge about some variables how should we construct a probability distribution that is unbiased beyond what we know?

• The Maximum Entropy (MaxEnt) Principle provides the best approach: we choose probabilities to maximize Shannon entropy while satisfying given constraints.

• This ensures the least biased probability distribution possible, consistent with the available information.

• We seek to maximize $S(p)$ subject to the constraints:

• To enforce these constraints, we introduce Lagrange multipliers $\lambda_0, \lambda_1, \lambda_2, \dots$, leading to the Lagrangian (also called the objective function):

  $$J[p] = - \sum_k p_k \log p_k - \lambda_0 \left( \sum_k p_k - 1 \right) - \lambda_1 \left( \sum_k p_k \, x_k - \langle x \rangle \right) - \lambda_2 \left( \sum_k p_k \, y_k - \langle y \rangle \right) - \dots$$

  (24)

• To maximize $J[p]$, we take its functional derivative with respect to $p_k$:

• Since $e^{-1 - \lambda_0}$ is simply a constant normalization factor, we define $Z = e^{1+\lambda_0}$ and arrie at final expression for probability distribution:

• Interperation of MaxEnt

  • The probability distribution takes an exponential form in the constrained variables $x_k, y_k, \dots$.

  • The normalization constant $Z$ (also called the partition function) ensures that the probabilities sum to 1.

  • The Lagrange multipliers $\lambda_1, \lambda_2, \dots$ encode the specific constraints imposed on the system.

  • This result is fundamental in statistical mechanics, where it leads to Boltzmann distributions, and in machine learning, where it underpins maximum entropy models.

Application MaxEnt: Biased Die Example¶

• If we are given a fair die MaxENt would predict $p_i=1/6$ as there are no constraints.

• But suppose we are given a biased die the average outcome of which is rolling on average a number $\langle x \rangle = 5.5$. The entropy function to maximize becomes:

• Solving the variational equation, we find that the optimal probability distribution follows an exponential form:

• where $Z$ is the partition function ensuring normalization:

• To determine $B$, we use the constraint $\langle x \rangle = 5.5$:

• This equation can be solved numerically for $B$. In many cases, Newton’s method or other root-finding techniques can be employed to find the exact value of $B$. This distribution resembles the Boltzmann factor in statistical mechanics, where higher outcomes are exponentially less probable.

Relative Entropy¶

• In the MaxEnt derivation, we implicitly maximized entropy relative to an underlying assumption of equal-probability microstates. We now make this idea precise.

• Consider a 1D Brownian particle starting at $x_0=0$ and diffusing freely. Its probability distribution at time (t) is Gaussian:

• The Shannon entropy of this distribution is

• which evaluates to

• suggesting that entropy grows as the particle spreads in space.

Problem: Grid Dependence of Shannon Entropy¶

• For continuous variables, Shannon entropy is not absolute: it depends on the choice of spatial resolution (\Delta x). Refining the grid causes (S) to diverge, making it unsuitable for comparing entropy across different distributions or times.

• To obtain a well-defined, resolution-independent measure, we instead use relative entropy.

• Relative entropy quantifies the information cost of using $Q$ when the system is actually distributed as $P$. It is non-negative and vanishes if and only if $P=Q$.

Asymmetry of KL Divergence and Irreversibility¶

• Consider two normal distributions,

• The KL divergence can be computed analytically:

$$ D_{\mathrm{KL}}(P|Q) = \ln\frac{\sigma_2}{\sigma_1}

\cdot \frac{\sigma_1^2+(\mu_1-\mu_2)^2}{2\sigma_2^2}

• \frac{1}{2}. $$

• This expression is not symmetric under $P\leftrightarrow Q$, highlighting that KL divergence defines a directed notion of distance between distributions.

• For diffusion, the probability distribution at time $t$ is Gaussian with variance $\sigma^2(t)=2Dt$. Comparing distributions at two times $t_1$ and $t_2$ yields:

• This quantity is strictly positive unless (t_1=t_2).

Physical Interpretation of relative entropy¶

KL divergence measures the statistical distinguishability between probability distributions. For diffusion, the forward process (spreading in time) is statistically distinguishable from the hypothetical time-reversed process (spontaneous contraction).

• This asymmetry reflects irreversibility: diffusion naturally increases entropy, while its time-reversed counterpart is overwhelmingly improbable.

• This statistical asymmetry underlies the thermodynamic arrow of time.

Relative Entropy in Machine Learning¶

• If $Q(x)$ assigns very low probability where $P(x)$ is large, the term $\log \frac{P(x)}{Q(x)}$ becomes large, strongly penalizing underestimation of the data distribution.

• If $Q(x)$ is broader than $P(x)$ and assigns probability to unlikely regions, the penalty is mild because $P(x)$ is small there.

• As a result, KL divergence is asymmetric and not a true distance metric.

• It penalizes missing probability mass far more than wasting probability mass, which is why it is widely used in machine learning and statistical inference.

Problems¶

Problem-1: Compute Entropy for gas partitioning¶

• A container is divided into two equal halves. It contains 100 non-interacting gas particles that can freely move between the two sides. A macrostate is defined by the fraction of particles in the right half $f$.

1. Write down an exprssion for entropy S(f)

   • Consult the section on Random Walk and Combinatorics formula

2. Evaluate $S(f)$ for:

   • $f_L = 0.1$ (10% left, 90% right)

   • $f_L = 0.25$ (25% left, 75% right)

   • $f_L = 0.5$ (50% left, 50% right)

3. Relate entropy to probability using Large Deviation Theory and answer the following questions:

   • Which macrostate is most probable?

   • How does entropy influence probability?

   • Why are extreme fluctuations (e.g., $f = 0.1$) unlikely?

Problem-2: Guassians in 2D¶

• Below we generate three 2D gaussians $P(x,y)$ with varying degree of correltion betwene $x$ and $y$. Run the code to visualize

import numpy as np
import matplotlib.pyplot as plt

# Define mean values
mu_x, mu_y = 0, 0  # Mean of x and y

# Define standard deviations
sigma_x, sigma_y = 1, 1  # Standard deviation of x and y

# Define different correlation coefficients
correlations = [0.0, 0.5, 0.9]  # Varying degrees of correlation

# Generate and plot 2D Gaussian samples for different correlations
fig, axes = plt.subplots(1, 3, figsize=(15, 5))

for i, rho in enumerate(correlations):
    # Construct covariance matrix
    covariance_matrix = np.array([[sigma_x**2, rho * sigma_x * sigma_y],
                                   [rho * sigma_x * sigma_y, sigma_y**2]])
    
    # Generate samples from the 2D Gaussian distribution
    samples = np.random.multivariate_normal([mu_x, mu_y], covariance_matrix, size=1000)
    
    # Plot scatter of samples
    axes[i].scatter(samples[:, 0], samples[:, 1], alpha=0.5)
    axes[i].set_title(f"2D Gaussian with Correlation ρ = {rho}")
    axes[i].set_xlabel("X")
    axes[i].set_ylabel("Y")

plt.tight_layout()
plt.show()

• Compute entropy using probability distributions $P(x,y), P(x|y=0), P(x)$ and $P(y)$

• Compute relative entropy $D_{KL}(P(x,y)|| p(x) p(y))$ which is known as Mutual Information. How can you interpret this quantity

• You will need these functions. Also check out scipy.stats.entropy

Problem-3: Gambling with ten sided die¶

• Take a ten sided die described by random variable $X =1,2....10$

• Experiment shows that after many throughs the average number die shows is $E[x]=5$ with variance $V[X] = 1.5$.

• Determine probabilities of ten die sides $p(x)$. You can use the code for six sided die in this notebook. But remember there are now two constraints one on mean and one on variance!

Problem-3 Biased Random walker in 2D¶

A random walker moves on a 2D grid, taking steps in one of four possible directions:

Each step occurs with unknown probabilities:

which satisfy the normalization condition:

Experimental measurements indicate the expected displacement per step:

Using the Maximum Entropy (MaxEnt) principle, determine the probabilities $p^*$ by numerically solving an optimization problem.

• You will need scipy.minimize(entropy, initial_guess, bounds=bounds, constraints=constraints, method='SLSQP') to do the minimization. Check the documentation to see how it works

Problem-5: Arrow of time in 2D diffusion¶

1. Simulate 2D diffusion using a Gaussian distribution

   • Model diffusion in two dimensions with a probability distribution:

     $$P(x, y, t) = \frac{1}{4\pi D t} \exp\left(-\frac{x^2 + y^2}{4 D t}\right)$$

     (58)

   • Generate samples at different time steps $t$ using numpy.random.normal with time-dependent variance.

2. Compute the relative entropy as a function of time $S_{\text{rel}}(t)$

   • Use relative entropy (Kullback-Leibler divergence) to compare the probability distributions at time $t$ with a reference time $t_0 = 0.1$:

     $$D_{\text{KL}}(P(x, y, t) || P(x, y, t_0))$$

     (59)

3. Compare forward and reverse evolution of diffusion

   • Compute the relative entropy between forward ($t \to t + \Delta t$) and reverse ($t \to t - \Delta t$) diffusion distributions.

   • Quantify how diffusion leads to irreversible evolution by evaluating:

     $$D_{\text{KL}}(P_{\text{forward}} || P_{\text{reverse}})$$

     (60)

4. Interpret the results

   • How does relative entropy change with time?

   • Why is the forward process distinguishable from the reverse process?

   • What does this tell us about the arrow of time?