Suppose that you are playing tennis and that upon missing the opponent’s serve the tennis ball lands in a bunch of shrubs behind the court. How would you decide where to start searching for the ball? There are two approaches in statistics to answer the above question: Frequentist and Bayesian. A plain English explanation of the differences between these two approaches is given below.
In the frequentist approach, you would draw upon your mental model of how tennis balls behave in different situations in order to identify which areas to start the search from. For example, where a tennis ball lands depends on its speed, the nature of the surface it encounters, wind speed, ground conditions etc. Thus, using the above mental model you would start the search in the area where the tennis ball has the highest likelihood of landing.
In the Bayesian approach, you would not only use the mental model outlined above but also incorporate prior information about where the tennis ball landed in the past. In other words, the Bayesian approach would combine your prior information and your mental model of how tennis balls behave in order to identify the areas to search.
Nature of Solutions
The interesting thing about the Bayesian approach is that if there is no prior information available (e.g., you did not miss any serves in the past) then you only have the mental model to rely on. Thus, in the absence of prior information, the Bayesian approach and the frequentist approach lead to identical solutions. In other words, in the absence of prior information, you would start the search from the same area irrespective of which approach you choose.
However, you may have very strong prior information. You have missed several serves and in all these instances the ball was found in a specific area. In such a situation, you would disregard your mental model completely and start your search in the area where the ball was found on previous occasions.
In contrast, if your prior information is weak (e.g., you did not miss a lot of serves), then you would combine the inferences from your mental model and your prior information to identify areas to search. Under these circumstances, the search area represents a compromise of sorts between your mental model and the prior information you have.
I hope the above gives you a sense of the differences and similarities between the Bayesian and the frequentist approaches in statistics. In a future post, we will take a simple context (e.g., a coin toss) and see how the above intuitions are backed up mathematically.