A pointwise information-theoretic measure of how unexpected a linguistic unit is given its context, defined as the negative logarithm of the conditional probability of the unit (−log P(unit | context)); typically expressed in bits when log base 2 is used.

Principle

Surprisal quantitatively links probability and processing: lower conditional probability (higher surprisal) corresponds to greater information content and, in many models, increased cognitive processing effort or longer reaction times.

Demonstration

Illustrative calculation → Context: preceding sentence fragment. Two candidate next words with P=0.7 and P=0.05 respectively. Surprisal: −log2(0.7) ≈ 0.51 bits versus −log2(0.05) ≈ 4.32 bits. Recognition: the lower-probability word carries substantially higher surprisal and, all else equal, is predicted to elicit higher processing cost.

Misapplication

Using unigram frequency as a substitute for surprisal in context-sensitive settings. The error is ignoring conditional information: a frequent word can be highly surprising in a particular context, and a rare word can be predictable given preceding context.

Consequence

Surprisal provides a testable predictor for human sentence-processing difficulty and a diagnostic for language models’ predictive behavior; it also guides choices in compression and communication modeling.

Reversal

In communicative situations with strong redundancy, prosodic cues, or multimodal signals, measured surprisal may overpredict processing cost because listeners use additional sources of information to reduce uncertainty.

Boundary

Clearly within: computing −log P(word | preceding context) for tokens in a probabilistic model. Boundary case: using corpus-based conditional estimates with sparse data, where smoothing and model assumptions crucially affect surprisal values. Clearly outside: corpus frequency counts without conditioning or measures of global lexical rarity.

Semantic Tension

Predictability ↔ Informativeness: speakers and listeners balance producing informative (high surprisal) content and maintaining predictability for efficient communication.

Synthesis

Surprisal operationalizes unexpectedness as conditional information content; its utility depends on reliable context-sensitive probability estimates and awareness of extra-linguistic cues that modulate effective unpredictability.