If I have a fair coin (parameter value) then the probability that it will come up heads is 0.5.
If I flip a coin 100 times and it comes up heads 52 times then it has a high likelihood of being fair (the numeric value of likelihood potentially taking a number of forms).
The difference between probability and likelihood is closely related to the difference between probability and statistics.
In a sense probability and statistics concern themselves with problems that are opposite or inverse to one another.
- probability attaches to possible results;
- likelihood attaches to hypotheses.
Probability quantifies anticipation (of outcome), likelihood quantifies trust (in model).
Unlike probability density functions, likelihoods aren’t normalized.
The area under their curves does not have to add up to 1.