Probability theory

Probability theory#

What you need to know

Sample space \(\Omega\) is a set of elementary outcomes or events.
Events can contain more than one elementary event and can be constructed by forming subsets (\(A\), \(B\), \(C\) etc) of \(\Omega\)
Probability function P(A) assigns a numeric value to each event, A quantifying certainty of an event happening on a 0-1 scale.
Venn diagrams visualize P(A) as a “volume fraction” of our confidence in the event expressed on 0-1 scale.
Probability axioms define a set of logical rules for creating composite events from trivial ones.
Bayesian approach: In physical sciences and modeling one often deals with situations where counting is impossible. Hence, probability is interpreted as a degree of belief.

Probabilistic world of complex many particle systems#

compton — Fig. 1 Characterizing many particle complex systems is best done via probabilistic approach.#

What is the probability to find a gas in upper right corner cell?
What is the probability that all gas atoms will be in left side of the box?
What is the probability distribution of velocities of the gas?

What is the probability of finding molecule away from center by n steps out of N?
How do we obtain probability distribution after N steps given probability for 1 step?
Why is there tendency for probability distributions to evolve towards Gaussian?

Sample space#

The sample space, often signified by an \(\Omega\) is a set of all possible elementary events.
Elementary means the events can not be further broken down into simpler events. For instance, rolling a die can result in any one of six elementary events.
States of \(\Omega\) are sampled during a system trial, which could be done via an experiment or simulation.

If our trial is a single roll of a six-sided die, then the sample has size \(n(\Omega) = 6\)

\[\Omega =\{1, 2, 3, 4, 5, 6 \}\]

A fair coin is tossed once has a sample size \(n(\Omega) = 2\)

\[\Omega = \{H, T \}\]

If a fair coin is tossed three times in a row, we will have sample space of size \(n(\Omega) = 2^3\)

\[\Omega = \{H,T\}^{3}= {HTT, THT, TTH, THH, HTH, HHT, HHH, TTT}\]

Position of an atom in a container of size \(L_x\) along x. \(n(\Omega)\) is a huge number. We will need some special tools to quantify.

\[\Omega = \{0... L_x\}\]

Events, micro and macro states#

An event in probability theory refers to an outcome of an experiment. Event can contain one or more elementary events from \(\Omega\).
Event can be getting getting 5 on a die or getting any number less than 5.
In the context of statistical mechanics, we are going to call elementary events in \(\Omega\) microstates and events containing multiple microstates as macrostates
- If we roll a single die there are six micostates. We can define a macrostate as an event \(A\) of getting any number less than 4
\[A= \{1, 2, 3 \}\]
- Or we can create a macrostate \(B\) containing only even numbers
\[B = \{2, 4, 6 \}\]
- IF we roll toss two coins microstates are HT, TH, HH, TT. We can define a macrostate \(D\) of having 50% H and 50% T
\[D = \{TH, HT\}\]
- A microstate of a gas atom in 1D container could be its position x. A macrostate could be finding atom anywehere in the second half of the container
\[C = \{L_x/2, ..., L_x \} \]

Compute probabilities through counting#

probabilities of events as fractions in the sample space

\[ { P(A) = \frac{n(A)}{n(\Omega)}} \]

\(n(A)\) probability of event, e.g rolling an even number. The size of the event space is 3
\(n(\Omega)\) size of sample space. In the context of single die roll is equal 6

Visualizing events as Venn diagrams#

# COllab has this but in local notebook you may want to install it
#!pip install matplotlib-venn #install if running locally
import matplotlib_venn as venn
import matplotlib.pyplot as plt

Omega = {1,2,3,4,5,6}

A = {1, 2, 3, 4, 5}

B = {4, 5, 6}

venn.venn2([A, B], set_labels=('A','B'))

print(len(A)/len(Omega))
print(len(B)/len(Omega))
print(len(A & B)/len(Omega))
print(len(A | B)/len(Omega))

8333333333333334
5
3333333333333333
0

../_images/add9e7dce330173df816b143145198cbe89a8c9807e20b0b83645a8db6cef64a.png

Probability Axioms#

Positivity and Normalization

\[P(A) \ge 0\]

\[P(\Omega)=1\]

Probability of rolling each number is 1/6 and rolling any number is 1.

Addition rule

For any sequence of mutually exclusive events, \(A_i \cap A_j = \emptyset \), the probability of their union is the sum of their probabilities,

\[P\left( A_1 \cup A_2 \cup \ldots \right) = P\left(A_1\right) + P\left(A_2\right) + \ldots\]

Probability of die rolling even number is: \(1/6+1/6+1/6\)

Product rule

When independent events \(A_i \cap A_j = \emptyset\), the probability of their intersection is a product of their probabilities

\[P\left( A_1 \cap A_2 \cap \ldots \right) = P\left(A_1\right) \cdot P\left(A_2\right) \cdot \ldots\]

Probability of rolling twice getting 3 and 5 is: \(\frac{1}{6}\cdot \frac{1}{6}\)

Complement

Given that \(A \cap \bar A=\emptyset\) and \(A \cup \bar A=\Omega\).

\[P(\bar A)=1-P(A)\]

the probability of not rolling a number: \(1-\frac{1}{6}\)

Conditional probability and Bayes Theorem

Knowledge of past events may change the probability of future events

\[P(A,B)=P(A|B)P(B)=P(B|A)P(A)\]

the probability of getting 4 given that we have rolled an even number: \(p(4|even) = \frac{p(even|4)p(4)}{p(even)}=\frac{1\cdot 1/6}{1/2}=1/3\)

Bayes Theorem#

Example of Using Bayes Formula to Test Hypothesis

A test for cancer is known to be 90% accurate either in detecting cancer if present or in giving an all-clear if cancer is absent.
The prevalence of cancer in the population is 1%. How worried should you be if you test positive? Try answering this question using Bayes’ theorem.

Prior, Posterior, and Likelihood

Bayes’ theorem provides a powerful framework for testing hypotheses or learning model parameters from data. While the mathematical formulation remains the same, the terminology used in Bayesian inference differs from the standard probability notation:

\[ P(\theta | D) = \frac{P(D | \theta) \cdot P(\theta)}{P(D)} \]

where:

Prior: \(P(\theta)\) represents our initial belief about the hypothesis or parameter before observing the data. For example, if we are tossing a coin, a reasonable prior might be a gaussian centered at \(1/2\) or take unifrom distribution in absence of information.
Evidence: \(P(D)\) is the probability of the observed data, also known as the marginal likelihood. It accounts for all possible parameter values and normalizes the posterior. For example, it is the probability of obtaining a specific sequence, such as \(HTHH\), given all possible biases of the coin.
Likelihood: \(P(D | \theta)\) describes how probable the observed data is for a given parameter \(\theta\). E.g for sequence of \(HTHH\) it will be \(L(\theta)=\theta^3(1-\theta)\) giving probability of landing three H and 1 T.
Posterior: \(P(\theta | D)\) is the updated probability of the hypothesis after incorporating the observed data. This is the key quantity in Bayesian inference, as it represents our revised belief about \(\theta\) given the data. We can take value of \(\theta\) corresponding to maximum of posterior to be most likely value of our parameter. For our case of uniform prior and likelhood the maxima will be \(\theta = 3/4\) as we may expect.

Computing number of microstates via combinatorics#

Binomial Distribution (Two-State Systems)

When molecules can be in two states (e.g., adsorbed vs. free, spin-up vs. spin-down), the number of ways to arrange \(N\) molecules into state A \(k\) and state B \(N-k\) follows:

\[ W(N, n) = \binom{N}{n} = \frac{N!}{N!(N-n)!} \]

For example, if \(N\) gas molecules distribute between two parts of the box or spins occupying two energy levels, this formula gives the number of microstates for a given occupation \(k\).

Multinomial Distribution (Multiple States)

For systems with more than two states, such as molecules distributed among \(m\) energy levels, the number of ways to assign \(N\) molecules into states \(n_1, n_2, ..., n_m\) with \(\sum n_i = N\) is:

\[ W(n_1, n_2, ..., n_m) = \frac{N!}{n_1! n_2! \cdots n_m!}. \]

Example: partitioning gas particles

Consider a container filled with 1000 atoms of Ar.

Example: spins

Solid metal has 100 atoms. Magnetic measurements show that there are 10 atoms with spin down. If ten atoms are chosen at random, what is the probability that they all have spin up?

def gas_partition(k1=30, k2=30, k3=30):
    '''partitioning N gas molecules into regions k1, k2 and k3'''

    from scipy.special import factorial

    N = k1+k2+k3 

    return factorial(N) / (factorial(k1) * factorial(k2)* factorial(k3))

print( gas_partition(k1=50, k2=50, k3=0) )

print( gas_partition(k1=50, k2=49, k3=1) )

print( gas_partition(k1=50, k2=25, k3=25) )

print( gas_partition(k1=34, k2=33, k3=33) )

0089134454556417e+29
0445672272782094e+30
2753736048324953e+43
19244792425583e+45

Strinling approximation of factorial and binomials#

Stitrling approximation of N!

This is the crude version of Stirling approximation that works out for \(N\gg 1\)

\[logN! \approx \sum log N_i = \int log N dN \approx NlogN-N \]

\[{N! \approx N^N e^{-N}}\]

A more accurate version is:

\[{N! \approx N^N e^{-N} \sqrt{2\pi N}}\]

../_images/5f52d037b52033af16e9b8e1fbf34a4168773afa0c95692a3b09d7461912f4e7.png

Random Walk#

Consider a problem with a binary outcome, with fixed probabilities \(p_{+} + p_{-} = 1\).
A clssic example is Random walk of N steps where molecules jumps right (\(+1\)) or left (\(-1\)) with fixed probabilities.
- Other examples are tossing \(N\) coins or counting \(N\) non-interacting molecules in the left vs right hand side of a container.
Each experiment generates a sequence—e.g., \(+1, -1, -1, -1, +1\) for a random walk or \(HTHTTT\) for coin flips.
Such a sequence represents a single microstate in the sample space of all possible sequences which is \(\Omega=2^N\).
- For unbiased random walk \(p_{+} = p_{-} = 1/2\), all microstates are equally probable and equal to \(\frac{1}{2^N}\).
- For biased random walk \(p_{+} \neq p_{-}\) the probability of microstates (sequence) is determined by the product of step probabilities (becasue steps are independent)

Probability of a sequence (microstate)

\[P(sequence) = p_{+}^{N_{+}} \cdot p_{-}^{N_{-}}\]

A more interesting question is the probability of taking \(N_{+}\) steps to the right, or having a net displacement of \(\Delta N \), regardless of the sequence of events.

\[ N_{+} + N_{-} = N \]

\[ N_{+} - N_{-} = \Delta N \]

Probability of net number of steps or displacements (macrostate)

Probability of \(N_{+}\) Steps to the Right

\[ P(N_{+} | N, p_{+}) = \frac{N!}{N_{+}! N_{-}!} \cdot p_{+}^{N_{+}} \cdot p_{-}^{N_{-}} \]

Probability of \(\Delta N\) net displacement from the origin

\[ P(\Delta N | N, p_{+}) = \frac{ N!}{ \Big(\frac{N+\Delta N}{2}\Big)! \Big(\frac{N-\Delta N}{2}\Big)!} \cdot p_{+}^{(N+\Delta N)/2} \cdot p_{-}^{(N-\Delta N)/2} \]

../_images/375ea7c1aa73b9357997004d512949250564771eb934870081c0048e40861069.png

Log of Macrostate Probability, Entropy, and fluctuations#

The logarithm of the probability of a given macrostate, \( P(N_{+} | N, p_{+}) \), can be written as:

\[ \log P(N_{+} | N, p_{+}) = \bigg[ \log \frac{N!}{N_{+}! N_{-}!} \bigg] + \log \bigg[ p_{+}^{N_{+}} \cdot p_{-}^{N_{-}} \bigg] \]

In simulations, we measure the fraction of steps

\[f_{\pm} = \frac{N_{\pm}}{N}\]

Fractions will fluctuate but, in the limit of an infinitely long simulation, should converge to the true probabilities \(f_{\pm} \to p_{\pm}\)
Since \( f_{+} + f_{-} = 1 \) and \( p_{+} + p_{-} = 1 \), we introduce the \(f = f_{+}, \quad p = p_{+}\) notation to simplify subsequent expressions

\[ \log P(f| N, p) = S(f) - E(f) \]

where:

\( S \) represents the entropy term, related to the number of ways to distribute steps,
\( E \) is an energy-like function governing the bias in step distribution.

Energy as a Measure of Bias#

Taking the logarithm of the probability factor results in an energy-like function \( \epsilon \), which introduces a bias that shifts the distribution left or right depending on the probabilities of left/right steps, \( p_{\pm} \), determined by microscopic details of the random walk.

\[ E = - \log \big[ p_{+}^{N_{+}} \cdot p_{-}^{N_{-}} \big] = - \log \big[p^{Nf} \cdot (1-p)^{N(1-f)} \big] \]

\[E= -N \big[ f \log p + (1-f) \log (1-p) \big] = -N \epsilon(f)\]

Where \(\epsilon=E/N\) is energy term per step of a random walk.
When \( p = \frac{1}{2} \), there is no bias, and the energy simplifies to \(E = N \log 2\)

Entropy as the Logarithm of Microstates in a Macrostate#

The entropy term, related to the number of ways to distribute steps. Using Stringling approximation for the log we obtain:

\[ S(f) = \log \frac{N!}{N_{+}! N_{-}!} \approx NlogN - N - [N_{+}logN_{+} - N_{+}] - [N_{-}logN - N_{-}] = NlogN - N_{+}logN_{+} - N_{-}logN_{-} \]

\[S(f) = N\Big[- \frac{N_{+}}{N} \log \frac{N_{+}}{N} - \frac{N_{-}}{N} \log \frac{N_{-}}{N} \Big] = N\Big[- f \log f - (1-f) \log (1-f) \Big]\]

where \( s(f)=S/N \) is the entropy per step of a random walk.

Entropy of a macrostate in \(N\)-Step Random Walk

Entropy of a macrostate defined by a fraction of steps \(f\) is:

\[S(f) = N \Big[-f \log f - (1-f) \log (1-f)\Big]\]

Alternatively, in terms of the net displacement fraction \(x = 2f - 1\):

\[S(x) = N \Bigg[\frac{1+x}{2} \log \frac{1+x}{2} + \frac{1-x}{2} \log \frac{1-x}{2}\Bigg]\]

Large Deviation Theory#

By taking a log of macrostate probability \(P(f)\) we find that it scales linearly with \(N\) and is proportional to a \(I(f)\) which is independent of N

\[ \log P_{N}(f) \approx -N \Big[\epsilon(f) - s(f) \Big] = -N I(f). \]
When there are \(N\) steps, molecules, or components, the probability distribution over the fraction of steps \(f\) tends to concentrate near the minima of \(I(f)\) caled Large deviation function
Large deviation function \(I\) dictates both the shape and decay of probability distributions in the large-\(N\) limit.
This general result applicable is known under the name of Large Deviation Theorem (LDT).

Large Deviation Theorem (LDT)

For a macrostate probability \(P(f)\) where \(f=n/N\) is some empirical fraction or mean quantity over \(N\) components:

\[ P_N (f) \sim e^{-N I(f)}. \]

\(I(f)\) is the rate function, which quantifies the likelihood of fluctuations away from the most probable value.
Example of \(I(f)\) for Random Walk: Determines how deviations of \(f_+\) (empirical fractions) from \( p_+\) (true or exact probabilities) are exponentially suppressed as \(N\) increases.

\[ I(f) = f_+ \log \frac{f_+}{p_+} + f_- \log \frac{f_-}{p_-} \]

Show code cell source Hide code cell source

# Re-import required libraries since execution state was reset
import numpy as np
import matplotlib.pyplot as plt

# Define parameters
p_plus = 0.7  # Biased probability
p_minus = 1 - p_plus  # Complementary probability

# Define range for f_+
f_plus_values = np.linspace(0.01, 0.99, 200)  # Avoid log(0) issues
f_minus_values = 1 - f_plus_values  # f_- = 1 - f_+

# Compute entropy component s(f_+)
s_values = -(f_plus_values * np.log(f_plus_values) + f_minus_values * np.log(f_minus_values))

# Compute energy component ε(f_+)
epsilon_values = - (f_plus_values * np.log(p_plus) + f_minus_values * np.log(p_minus))

# Compute large deviation rate function I(f_+)
I_values = f_plus_values * np.log(f_plus_values / p_plus) + f_minus_values * np.log(f_minus_values / p_minus)

# Compute probability P_N(f_+) using large deviation approximation
N = 50  # Arbitrary large N
P_x_values = np.exp(-N * I_values)  # Exponential suppression
P_x_values /= np.trapz(P_x_values, f_plus_values)  # Normalize for probability density

# Create subplots
fig, axes = plt.subplots(1, 2, figsize=(12, 5))

# First subplot: Entropy and Energy Components
axes[0].plot(f_plus_values, s_values, label=r"$s(f)$ (Entropy)", color="blue")
axes[0].plot(f_plus_values, epsilon_values, label=r"$\epsilon(f_+)$ (Energy)", color="green")
axes[0].plot(f_plus_values, I_values, label=r"$I(f)$ (Rate Function)", color="red")
axes[0].axvline(p_plus, linestyle="--", color="black", label=r"$f_+ = p_+$")
axes[0].set_xlabel(r"$f_+$")
axes[0].set_ylabel("Value")
axes[0].set_title("Entropy, Energy, and Rate Function")
axes[0].legend()
axes[0].grid()

# Second subplot: Probability Distribution P_N(f_+)
axes[1].plot(f_plus_values, P_x_values, label=r"$P_N(f_+)$", color="purple")
axes[1].axvline(p_plus, linestyle="--", color="black", label=r"$f_+ = p_+$")
axes[1].set_xlabel(r"$f_+$")
axes[1].set_ylabel(r"$P_N(f_+)$")
axes[1].set_title("Probability Distribution")
axes[1].legend()
axes[1].grid()

plt.tight_layout()
plt.show()

../_images/1b070db7d2c2db090e6e87f152af35add1baf61c9b4a97fce4a84f57a3a5578f.png

../_images/235aaa3eba81c6cdf206bc97d7a11add286f3abfccc41168936cbe5d4f5536fc.png

Gaussian Nature of Fluctuations#

The Taylor expansion of the large deviation function around its minimum leads to a Gaussian distribution in the limit of small fluctuations. This fundamental result in large deviation theory explains why fluctuations in equilibrium statistical mechanics are often Gaussian.
Let \( I(f) \) be the large deviation function, which attains a minimum at \( f_{\text{min}} \). Expanding around \( f_{\text{min}} \):

\[ I(f) = I(f_{\text{min}}) + \frac{1}{2} I''(f_{\text{min}}) (f - f_{\text{min}})^2 + \mathcal{O}((f - f_{\text{min}})^3). \]
Since \( I(f_{\text{min}}) = 0 \) (by definition, probability is maximal at \( f_{\text{min}} \)), we obtain:

\[ I(f) \approx \frac{1}{2} I''(f_{\text{min}}) (f - f_{\text{min}})^2. \]
Substituting this into the large deviation form:

\[ P(f) \approx e^{-N I(f)} \approx e^{-N \frac{1}{2} I''(f_{\text{min}}) (f - f_{\text{min}})^2} = e^{-\frac{(f - f_{\text{min}})^2}{2 \sigma^2}} \]
where the variance is:

\[ \sigma^2 = \frac{1}{N I''(f_{\text{min}})}. \]
This is a Gaussian distribution centered at \( f_{\text{min}} \), with variance \( \sigma^2 \). Here, \( f \) represents the fractional quantity \( f = n/N \), though one may also express the distribution in terms of absolute particle numbers, \( P(n) \).
Thus, the Taylor expansion of \( I(f) \) near its minimum shows that, for large \( N \), fluctuations become Gaussian, explaining why equilibrium statistical physics often exhibits Gaussian distributions.

Appendix: Explicit derivations#

Appendix A. Gaussian or large \(N\) limit of Binomial Distribution

The binomial distribution for large values of \(N\) has a sharply peaked distribution around its maximum (most likely) value \(\tilde{n}\). This motivates us to seek a continuous approximation by Taylor expanding the probability distribution around its maximum value \(\Delta n = n - \tilde{n}\) and keeping terms up to quadratic order.

\[ P_N(n) = \frac{N!}{n! (N - n)!} p^{n} (1-p)^{N-n} \]

Thus, from the onset, we aim for a Gaussian distribution. The task is to find the coefficients and justify that the third term in the Taylor expansion is negligible compared to the second.

\[ \log P(n) = \log P(\tilde{n}) + \frac{1}{2} B_2 \Delta n^2 + O(\Delta n^3) \]

\[ \log P(n) = \log N! - \log n! - \log( N - n)! + n \log(p) + N_{-} \log(1-p) \]

We evaluate the derivative of \(\log n!\) in the limit of \(n \gg 1\) as:

\[ \frac{d}{dn} \log n! = \frac{\log(n+1)! - \log n!}{n+1 - n} \approx \log(n+1) \approx \log(n) \]

We could also arrive at the same result by using Stirling’s approximation \(\log N! \approx N \log N - N\).
Taking the first derivative of the Taylor expansion for the binomial distribution, we find the peak of the distribution around which we expand:

\[ \frac{d}{dn} \log P(n) \Big|_{n = \tilde{n}} = - \log n + \log(N - n) + \log(p) - \log(1-p) = 0 \]

\[ \log \left( \frac{N - n}{n} \cdot \frac{p}{1 - p} \right) = 0 \quad \Rightarrow \quad \tilde{n} = N p \]

We recall that \(\tilde{n} = N p\) is also the mean of the binomial distribution!
Having found the peak of the distribution and knowing the first derivative, we now proceed to compute the second derivative:

\[ B_2 = \frac{d^2}{d n^2} \log P(n) \Big|_{n = \tilde{n}} \]

\[ = \frac{d}{dn} \log \left( \frac{N - n}{n} \cdot \frac{p}{1 - p} \right) = \left( -\frac{1}{N - n} - \frac{1}{n} \right) \Bigg|_{n = \tilde{n}} = -\frac{1}{N p(1 - p)} \]

While the first derivative gave us the mean of the binomial distribution, we notice that the second derivative produces the variance \(\sigma^2 = N p (1 - p)\).
Now, all that remains is to plug the coefficients into our approximated probability distribution and normalize it. Why normalize? The binomial was already properly normalized, but since we made an approximation by neglecting higher-order terms, we must re-normalize.

\[ P(n) \approx P(\tilde{n}) e^{-(n - \tilde{n})^2 / 2N p (1 - p)} \]

Normalizing the Gaussian distribution is done via the following integral:

\[ \int_{-\infty}^{+\infty} e^{-a x^2} dx = \left(\frac{\pi}{a}\right)^{1/2} \]

\[ \int P(\tilde{n}) e^{-(n - \tilde{n})^2 / 2N p (1 - p)} dn = P(\tilde{n}) (2\pi N p (1 - p))^{1/2} = 1 \]

Finally, we obtain the normalized Gaussian approximation to the binomial distribution:

\[ P(n) \approx \frac{1}{(2\pi N p (1 - p))^{1/2}} e^{-(n - \tilde{n})^2 / 2N p (1 - p)} = \frac{1}{(2\pi \sigma^2)^{1/2}} e^{-(n - \mu)^2 / 2\sigma^2} \]

Appendix B. Poisson limit or the limit of large \(N\) and small \(p\) such that \(Np=const\)

This is a situation of rare events like rains in forest or radioactive decay of uranium where each individual event has small chance of happening \(p \rightarrow 0\) yet there are large number of samples \(N\rightarrow \infty\) such that one has a constant average rate of events \(\lambda = pN = const\)
In this limit distirbution is no longer well described by the gaussian as the shape of distribution is heavily skewed due to tiny values of p.

\[P_N(n) = \frac{N!}{n! (N-n)!} p^n (1-p)^{(N-n)}\]

Writing factorial \(N!/(N-n)!\) explicitely we realize that it is dominated \(N^n\) and also \(N-n \approx N\)

\[P_N(n) = \frac{N(N-1)...(N-1+1))}{n!} p^n (1-p)^{(N-n)} \approx \frac{N^n}{n!} p^n (1-p)^{N}\]

Next let us plug in \(\lambda = pN = const\) and recall the definition of exponential \(lim_{x\rightarrow \infty }(1-1/x)^x = e^{-x}\)

\[P(n) = \frac{N^n}{n!} \Big( \frac{\lambda}{N} \Big)^n \Big( 1-\frac{\lambda}{N} \Big)^{N} = \frac{\lambda^n}{n!} \Big( 1-\frac{\lambda}{N} \Big)^{N} \approx \frac{\lambda^n}{n!} e^{-\lambda}\]

Example of Gaussian limit of Large Deviation function for random walk

The large deviation rate function for a simple random walk is given by:

\[ I(f) = f_+ \log \frac{f_+}{p_+} + f_- \log \frac{f_-}{p_-} \]

where \(f_+\) and \(f_-\) are empirical step probabilities, and \(p_+\), \(p_-\) are their expected values with \(p_+ + p_- = 1\).

Expansion Around the Minimum

The function \(I(f)\) is minimized at \(f_+ = p_+\), \(f_- = p_-\). Introducing small deviations \(\delta f\):

\[ f_+ = p_+ + \delta f, \quad f_- = p_- - \delta f. \]

Expanding the logarithms:

\[ \log \frac{p_+ + \delta f}{p_+} \approx \frac{\delta f}{p_+} - \frac{(\delta f)^2}{2 p_+^2}, \quad \log \frac{p_- - \delta f}{p_-} \approx -\frac{\delta f}{p_-} - \frac{(\delta f)^2}{2 p_-^2}. \]

Substituting into \(I(f)\), the linear terms cancel, and we obtain:

\[ I(f) \approx \frac{(\delta f)^2}{2} \left( \frac{1}{p_+} + \frac{1}{p_-} \right). \]

Gaussian Limit

By the large deviation principle:

\[ P(f) \approx e^{-N I(f)} = e^{-N \frac{(\delta f)^2}{2} \left( \frac{1}{p_+} + \frac{1}{p_-} \right)} = e^{- \frac{(\delta f)^2}{2 p_{+}p_{-}/N} } \]

This is a Gaussian with variance:

\[ \sigma^2 = \frac{p_+ p_-}{N}. \]

Thus, the empirical frequency \(f_+\) follows a Gaussian distribution in the large \(N\) limit.

Problems#

Problem 1: Counting Dies and coins#

You flip a coin 10 times and record the data in the form of head/tails or 1s and 0s

What would be the probability of ladning 4 H’s?
What would be the probability of landing HHHTTTHHHT sequence?
In how many ways can we have 2 head and 8 tails in this experiments?
Okay, now you got tired of flipping coins and decide to play some dice. You throw die 10 times what is the probability of never landing number 6?
You throw a die 3 times what is the probability of obtaining a combined sum of 7?

Problem 2: Counting gas molecules#

\[C^{N}_{k}\]

A container of volume \(V\) contains \(N\) molecules of a gas. We assume that the gas is dilute so that the position of any one molecule is independent of all other molecules. Although the density will be uniform on the average, there are fluctuations in the density. Divide the volume \(V\) into two parts \(V_1\) and \(V_2\), where \(V = V_1 + V_2\).

What is the probability p that a particular molecule is in each part?
What is the probability that \(N_1\) molecules are in \(V_1\) and \(N_2\) molecules are in \(V_2\)?
What is the average number of molecules in each part?
What are the relative fluctuations of the number of particles in each part?

Project Porosity of materials #

A simple model of a porous rock can be imagined by placing a series of overlap- ping spheres at random into a container of fixed volume \(V\) . The spheres represent the rock and the space between the spheres represents the pores. If we write the volume of the sphere as v, it can be shown the fraction of the space between the spheres or the porosity \(\phi\) is \(\phi =e^{-Nv/V}\), where \(N\) is the number of spheres.

For simplicity, consider a 2D system, (e.g \(v=\frac{1}{4}\pi d^2\), see wiki if you forgot the formula). Write a python function which place disks of \(d=1\) into a square box. The disks can overlap. Divide the box into square cells each of which has an edge length equal to the diameter of the disks. Find the probability of having 0, 1, 2, or 3 disks in a cell for \(\phi\) = 0.03, 0.1, and 0.5.

You will need np.random.uniform() to randomly place N disks of volume v into volume V. Check out this cool python lib for porosity evaluation of materials R Shkarin, et al Plos Comp Bio 2019