The MaxEnt (Best Guess) Machine

If one has information about ``expected averages'', it can be used to modify the assignment of equal a priori probabilities used for the simple maximization of S=k Log[Ω] in the microcanonical ensemble.  One first writes entropy in terms of probabilities by defining for each probability a ``suprisal'' , in units determined by the value of k (e.g. nats for k=1, bits for k=1/Log[2]).    The average value of this suprisal reduces to S=k Log[Ω] when the are all equal.  Note:  Here Log refers to the natural and not the base-10 log, following the Mathematica convention.  Also the relations described here will likely translate seamlessly into quantum mechanical applications, as Jaynes in Phys Rev 108 (1956) 171 and others have shown.

The problem

Our job is to maximize average suprisal, a natural generalization of Log[Multiplicity] when all Ω accessible states are not equally probable...

...subject to the usual normalization requirement that the probabilities add to 1, i.e. that...

along with the "expected average" constraints which for the rth of R constraints might take the form...

The solution

The Lagrange method of undetermined multipliers tells us that the solution for the ith of Ω probabilities is simply...

where partition function Z is defined to normalize probabilities as...

Here is the Lagrange (or ``heat'') multiplier for the rth constraint, and is the value of the rth parameter when the system is in the ith accessible state.  For example, when is the energy E, is often written as .  Values for these multipliers can be calculated by substituting the two equations above back into the constraint equations, or from the differential relations derived below.

A few ``big picture'' quantitites

The resulting maximized entropy is...

We can rearrange this expression by defining the dimensionless availability in natural units as A=-Log[Z].  This quantity in turn can be seen as the common numerator behind a range of dimensioned availabilities (one for each extensive variable of type r) defined as...

For example, the energy availability is Helmholtz free energy E-TS, if R=1 and is a constraint on energy E.  It turns out that our maximization also minimizes these dimensioned availabilities for a given value of their corresponding heat multiplier , assuming of course that the coefficients for all values of r and i are held constant themselves (cf. page 46 of Betts and Turner).

Familiar ensemble contexts for this calculation include the Microcanonical Ensemble (R=0 so that Z=S/k), the Canonical Ensemble (R=1 and is energy E so that = 1/kT and =E-TS), the Pressure Ensemble (R=2 with energy, = 1/kT, volume, =P/kT, and = E+PV-TS=Gibbs Free Energy), and the Grand Canonical Ensemble (same as pressure except that =N, = -μ/kT, and = E - μN - TS = = The Grand Potential).  

There is much more to go, as the process of transcribing and synthesizing disparate notes continues.  Along the way, I suspect that a better understanding of the validity and limits of ``altered looks'' at heat (and other) capacities, as bits of information lost per two-fold increase in the corresponding extensive variable or it's Lagrange multipler, will emerge.  

Contact and Copyright Information

This note represents insights provided by numerous colleages, and has benefited in particular from discussions with, and notes provided by, the late E. T. Jaynes.  The person responsible for mistakes (and to whom you can forward suggestions) is P. Fraundorf in Physics and Astronomy at UM-St. Louis (pfraundorf@umsl.edu).