{"id":90,"date":"2016-09-10T08:13:49","date_gmt":"2016-09-10T08:13:49","guid":{"rendered":"http:\/\/www.aarondefazio.com\/tangentially\/?p=90"},"modified":"2023-10-13T10:32:30","modified_gmt":"2023-10-13T14:32:30","slug":"a-complete-guide-to-the-bayes-factor-test","status":"publish","type":"post","link":"https:\/\/www.aarondefazio.com\/tangentially\/?p=90","title":{"rendered":"A complete guide to the bayes factor test"},"content":{"rendered":"

(PDF version<\/a> of this article is available)<\/p>\n

<\/p>\n

The Bayes factor test is an interesting thing. Some Bayesians advocate it unequivalently, whereas others reject the notion of
\ntesting altogether, Bayesian or otherwise. This post takes a critical look at the Bayes factor, attempting to tease apart the
\nideas to get to the core of what it’s really doing. If you’re used to frequentist tests and want to understand Bayes factors, this
\npost is for you.\n<\/p>\n

<\/p>\n

The Bayes factor test goes all the way back to Je\ufb00reys’ early book on the Bayesian approach to statistics [Je\ufb00reys<\/a>,\u00a01939<\/a>]. Evidence for an
\nalternative hypothesis H<\/mi><\/mrow>1<\/mn><\/mrow><\/msub \n><\/math> against
\nthat of the null hypothesis H<\/mi><\/mrow>0<\/mn><\/mrow><\/msub \n><\/math>
\nis summarized by a quantity known as the Bayes factor. The Bayes factor is just the ratio of the data likelihoods, under
\nboth hypotheses and integrating out any nuisance parameters:\n<\/p>\n
\n B<\/mi><\/mrow>1<\/mn>0<\/mn><\/mrow><\/msub \n> : =<\/mo> p<\/mi>(<\/mo>D<\/mi>|<\/mo>H<\/mi><\/mrow>1<\/mn><\/mrow><\/msub \n><\/mrow>)<\/mo><\/mrow><\/mrow> \np<\/mi>(<\/mo>D<\/mi>|<\/mo>H<\/mi><\/mrow>0<\/mn><\/mrow><\/msub \n><\/mrow>)<\/mo><\/mrow><\/mrow><\/mfrac>.<\/mo>\n<\/mrow><\/math><\/div>\n

<\/p>\n

\n

<\/p>\n

If this ratio is large, we can conclude that there is strong evidence for the alternative hypothesis. In contrast, if the
\ninverse of this ratio is large, we have evidence supporting the null hypothesis. The de\ufb01nition of what is strong evidence is
\nsubjective, but usually a Bayes factor of 10 or more is considered su\ufb03cient.\n<\/p>\n

<\/a>The standard Bayes factor test against the point-null<\/h3>\n

<\/p>\n

Although Bayes factors are sometimes used for testing simple linear regression models against more complex ones, by far
\nthe most common test in practice is the analogue to the frequentist t-test, the Bayes factor t-test. Under the
\nassumption of normality with unknown variance, it tests a null hypothesis of zero mean against non-zero
\nmean.\n<\/p>\n

<\/p>\n

This test is implemented in the BayesFactor<\/a> R package with the ttestBF method. For the pairwise case it
\ncan be invoked with bf <- ttestBF(x = xdata, y=ydata)<\/span>. For a non-pairwise test just x can be passed
\nin.\n<\/p>\n

<\/p>\n

<\/a>Statistical details of the test<\/span>
\n To apply the test we must formally de\ufb01ne the two hypotheses in the form of statistical models. Je\ufb00reys’ book is very
\npragmatic, and the modelling choices for this test re\ufb02ect that. Since this is a t-test, we of course assume a normal
\ndistribution for the data for both hypothesis, so it just remains to just de\ufb01ne the priors on it’s parameters in both
\ncases.\n<\/p>\n

<\/p>\n

We have essentially 3 parameters we have to place priors on: the mean
\n\u03bc<\/mi><\/math> of the alternative, the variance
\nunder the alternative \u03c3<\/mi><\/mrow>1<\/mn><\/mrow>2<\/mn><\/mrow><\/msubsup \n><\/math> and
\nthe variance under the null \u03c3<\/mi><\/mrow>0<\/mn><\/mrow>2<\/mn><\/mrow><\/msubsup \n>.<\/mo><\/math>
\nThe \u03bc<\/mi><\/math> under the null
\nis assumed to be 0<\/mn><\/math>.
\nWe have to be extremely careful about the priors we place on these parameters. Reasonable seeming choices can lead to
\ndivergent integrals or non-sensical results very easily. We discuss this further below.<\/p>\n

<\/p>\n

<\/a>Variance<\/span><\/span>
\n Because the variance appears in both the numerator and denominator of the Bayes factor
\n(the null and the alternative), we can use a \u201cnon-informative\u201d improper prior of the form
\np<\/mi>(<\/mo>\u03c3<\/mi><\/mrow>2<\/mn><\/mrow><\/msup \n><\/mrow>)<\/mo><\/mrow> =<\/mo> 1<\/mn>\u2215<\/mo>\u03c3<\/mi><\/mrow>2<\/mn><\/mrow><\/msup \n><\/math> for both.
\nThis prior causes the marginal likelihood integrals to go to in\ufb01nity, but this cancels out between the numerator and
\ndenominator giving a \ufb01nite value.\n<\/p>\n

<\/p>\n

<\/a>Mean<\/span><\/span>
\n The choice of a prior for the mean is a little more subjective. We can’t use an improper prior here, as the mean prior
\nappears only in the numerator, and without the convenient cancelation with factors in the denominator, the Bayes factor
\nwon’t be well de\ufb01ned. Je\ufb00reys argues for a Cauchy prior, which is just about the most vague prior you can use that still
\ngives a convergent integral. The scale parameter of the Cauchy is just \ufb01xed to the standard deviation in his formulation.
\nThe BayesFactor package uses 0.707 times the standard deviation instead as the default, but this can be
\noverridden.\n<\/p>\n

<\/p>\n

<\/a>What are the downsides compared to a frequentist t<\/mi><\/math>-test?<\/span>
\n The main downside is that the BF test is not as good by some frequentist measures than the tests that are designed to
\nbe as good as possible with respect to those measures. In particular, consider the standard measure of power at
\n\u03b3<\/mi><\/math>: the probability of
\npicking up an e\ufb00ect of size \u03b3<\/mi><\/math>
\nunder repeated replications of the experiment. Depending on the e\ufb00ect size and the number of samples
\nn<\/mi><\/math>, a Bayesian
\nt<\/mi><\/math>-test
\noften requires 2-4 times as much data to match the power of a frequentist
\nt<\/mi><\/math>-test.\n<\/p>\n

<\/p>\n

<\/a>Implementation details<\/span><\/span>
\n The Bayes factor requires computing marginal likelihoods, which is a quite distinct problem from the usual posterior
\nexpectations we compute when performing Bayesian estimation instead of hypothesis testing. The marginal likelihood is
\njust an integral over the parameter space, which can be computed using numerical integration when the
\nparameter space is small. For this t-test example, the BayesFactor package uses Gaussian quadrature for
\nthe alternative hypothesis. The null hypothesis doesn’t require any integration since it consists of a single
\npoint.\n<\/p>\n

<\/p>\n

Je\ufb00erys lived in the statistical era of tables and hand computation, so he derived several
\nformulas that can be used to approximate the Bayes factor. The following formula in terms of the
\nt<\/mi><\/math>-statistic
\nt<\/mi> =<\/mo> x<\/mi><\/mrow>\u0304<\/mo><\/mover>(<\/mo>n<\/mi> \u2212<\/mo> 1<\/mn><\/mrow>)<\/mo><\/mrow>\u2215<\/mo>s<\/mi><\/mrow>2<\/mn><\/mrow><\/msup \n><\/mrow><\/msqrt><\/math> works
\nwell for n<\/mi><\/math>
\nin the hundreds or thousands:\n<\/p>\n
\n B<\/mi><\/mrow>1<\/mn>0<\/mn><\/mrow><\/msub \n> \u2248<\/mo> 2<\/mn><\/mrow>\n\u03c0<\/mi>n<\/mi><\/mrow><\/mfrac><\/mrow><\/msqrt>exp<\/mo> t<\/mi><\/mrow>2<\/mn><\/mrow><\/msup \n>\u2215<\/mo>2<\/mn><\/mrow><\/mfenced>.<\/mo>\n<\/mrow><\/math><\/div>\n

<\/p>\n

\n

<\/p>\n

\n

<\/a>The sensitivity to priors<\/h3>\n

<\/p>\n

There are two types of priors that appear in the application of Bayes factors; The priors on the hypothesis
\np<\/mi>(<\/mo>H<\/mi><\/mrow>0<\/mn><\/mrow><\/msub \n><\/mrow>)<\/mo><\/mrow><\/math>,
\np<\/mi>(<\/mo>H<\/mi><\/mrow>1<\/mn><\/mrow><\/msub \n><\/mrow>)<\/mo><\/mrow><\/math>, and the priors on parameters that
\nappears in marginal likelihoods, the p<\/mi>(<\/mo><\/mi><\/mrow>0<\/mn><\/mrow><\/msub \n>|<\/mo>H<\/mi><\/mrow>0<\/mn><\/mrow><\/msub \n><\/mrow>)<\/mo><\/mrow>,<\/mo><\/math>
\np<\/mi>(<\/mo><\/mi><\/mrow>1<\/mn><\/mrow><\/msub \n>|<\/mo>H<\/mi><\/mrow>1<\/mn><\/mrow><\/msub \n><\/mrow>)<\/mo><\/mrow><\/math>. Your prior
\nbelief ratio p<\/mi>(<\/mo>H<\/mi><\/mrow>1<\/mn><\/mrow><\/msub \n><\/mrow>)<\/mo><\/mrow>\u2215<\/mo>p<\/mi>(<\/mo>H<\/mi><\/mrow>0<\/mn><\/mrow><\/msub \n><\/mrow>)<\/mo><\/mrow><\/math> is
\nupdated to form your posterior beliefs using the simple equation:\n<\/p>\n
\n p<\/mi>(<\/mo>H<\/mi><\/mrow>1<\/mn><\/mrow><\/msub \n>|<\/mo>D<\/mi><\/mrow>)<\/mo><\/mrow><\/mrow> \np<\/mi>(<\/mo>H<\/mi><\/mrow>0<\/mn><\/mrow><\/msub \n>|<\/mo>D<\/mi><\/mrow>)<\/mo><\/mrow><\/mrow><\/mfrac> =<\/mo> \u222b\n <\/mo>p<\/mi>(<\/mo>D<\/mi>|<\/mo><\/mi><\/mrow>1<\/mn><\/mrow><\/msub \n><\/mrow>)<\/mo><\/mrow>p<\/mi>(<\/mo><\/mi><\/mrow>1<\/mn><\/mrow><\/msub \n>|<\/mo>H<\/mi><\/mrow>1<\/mn><\/mrow><\/msub \n><\/mrow>)<\/mo><\/mrow>d<\/mi><\/mi><\/mrow>1<\/mn><\/mrow><\/msub \n><\/mrow>\n \u222b\n <\/mo>p<\/mi>(<\/mo>D<\/mi>|<\/mo><\/mi><\/mrow>0<\/mn><\/mrow><\/msub \n><\/mrow>)<\/mo><\/mrow>p<\/mi>(<\/mo><\/mi><\/mrow>0<\/mn><\/mrow><\/msub \n>|<\/mo>H<\/mi><\/mrow>0<\/mn><\/mrow><\/msub \n><\/mrow>)<\/mo><\/mrow>d<\/mi><\/mi><\/mrow>0<\/mn><\/mrow><\/msub \n><\/mrow><\/mfrac><\/mrow><\/mfenced> \u22c5<\/mo>p<\/mi>(<\/mo>H<\/mi><\/mrow>1<\/mn><\/mrow><\/msub \n><\/mrow>)<\/mo><\/mrow><\/mrow> \np<\/mi>(<\/mo>H<\/mi><\/mrow>0<\/mn><\/mrow><\/msub \n><\/mrow>)<\/mo><\/mrow><\/mrow><\/mfrac>.<\/mo>\n<\/mrow><\/math><\/div>\n

<\/p>\n

\n

<\/p>\n

The ratio p<\/mi>(<\/mo>H<\/mi><\/mrow>1<\/mn><\/mrow><\/msub \n><\/mrow>)<\/mo><\/mrow>\u2215<\/mo>p<\/mi>(<\/mo>H<\/mi><\/mrow>0<\/mn><\/mrow><\/msub \n><\/mrow>)<\/mo><\/mrow><\/math>
\nis not formally considered part of the Bayes factor, but it plays an important role. It encodes your prior
\nbeliefs about the hypotheses. If you are using Bayes factors in a subjective fashion, to update your
\nbeliefs in a hypothesis after seeing data, then it is crucial to properly encode your prior belief in
\nwhich hypothesis is more likely to be true. However, in practice everybody always takes this ratio to be
\n1<\/mn><\/math>,
\nfollowing the ideals of objective Bayesian reasoning, where one strives to introduce as little prior knowledge as possible into
\nthe problem. If your writing a scienti\ufb01c paper, it’s reasonable to use 1 as anybody reading the paper can apply there own
\nratio as a correction to your Bayes factor, re\ufb02ecting there own prior beliefs.\n<\/p>\n

<\/p>\n

The prior ratio of the hypotheses plays an interesting part in the calibration of the Bayes factor
\ntest. One way of testing Bayes factors is to simulate data from the priors, and compare the empirical
\nBayes factors over many simulations with the true ratios. This involving sampling a hypothesis,
\nH<\/mi><\/mrow>0<\/mn><\/mrow><\/msub \n><\/math> or
\nH<\/mi><\/mrow>1<\/mn><\/mrow><\/msub \n><\/math>, then sampling<\/p>\n

<\/mi><\/math> then D. If the
\nprobability of sampling H<\/mi><\/mrow>0<\/mn><\/mrow><\/msub \n><\/math>
\nand H<\/mi><\/mrow>1<\/mn><\/mrow><\/msub \n><\/math>
\nis not equal at that \ufb01rst stage, then the observed Bayes factor will be o\ufb00 by the corresponding
\np<\/mi>(<\/mo>H<\/mi><\/mrow>1<\/mn><\/mrow><\/msub \n><\/mrow>)<\/mo><\/mrow>\u2215<\/mo>p<\/mi>(<\/mo>H<\/mi><\/mrow>0<\/mn><\/mrow><\/msub \n><\/mrow>)<\/mo><\/mrow><\/math>. This is
\nsomewhat of a tautology, as we are basically just numerically verifying the Bayes factor equation as written. Nevertheless, it
\nshows that if the priors on the hypotheses don’t re\ufb02ect reality, then the Bayes factor won’t either. Blind application of a
\nBayes factor test won’t automatically give you interpretable \u201cbeliefs\u201d.\n<\/p>\n

<\/p>\n

\n

<\/a>Parameter priors<\/h4>\n

<\/p>\n

The dependence of Bayes factors on the parameter priors
\np<\/mi>(<\/mo><\/mi><\/mrow>1<\/mn><\/mrow><\/msub \n>|<\/mo>H<\/mi><\/mrow>1<\/mn><\/mrow><\/msub \n><\/mrow>)<\/mo><\/mrow><\/math> and
\np<\/mi>(<\/mo><\/mi><\/mrow>0<\/mn><\/mrow><\/msub \n>|<\/mo>H<\/mi><\/mrow>0<\/mn><\/mrow><\/msub \n><\/mrow>)<\/mo><\/mrow><\/math> is really
\nat the core of understanding Bayes factors. It is a far more concrete problem than the hypothesis likelihoods, as there is no
\nclear \u201cobjective\u201d choices we can make. Additionally, intuitions from Bayesian estimation problems do not carry over to
\nhypothesis testing.\n<\/p>\n

<\/p>\n

Consider the simple BF t-test given above. We described the test using the Cauchy prior on the location
\nparameter of the Gaussian. Often in practice people use large vague Gaussian priors instead when doing
\nBayesian estimation (as opposed to hypothesis testing), as they tend to be more numerically stable than the
\nCauchy. However, it has been noted by several authors [Berger and Pericchi<\/a>,\u00a02001<\/a>] that simply replacing the
\nCauchy with a wide Gaussian in the BF t-test causes serious problems. Suppose we use a standard deviation of
\n\u03bd<\/mi><\/math> in this wide
\nprior (\u03bd<\/mi><\/math>
\nlarge). It turns out that Bayes factor scales asymptotically like:\n<\/p>\n
\n 1<\/mn><\/mrow> \n\u03bd<\/mi>n<\/mi><\/mrow><\/msqrt><\/mrow><\/mfrac>exp<\/mo> t<\/mi><\/mrow>2<\/mn><\/mrow><\/msup \n>\u2215<\/mo>2<\/mn><\/mrow><\/mfenced>.<\/mo>\n<\/mrow><\/math><\/div>\n

<\/p>\n

\n

<\/p>\n

where t<\/mi> =<\/mo> x<\/mi><\/mrow>\u0304<\/mo><\/mover>(<\/mo>n<\/mi> \u2212<\/mo> 1<\/mn><\/mrow>)<\/mo><\/mrow>\u2215<\/mo>s<\/mi><\/mrow>2<\/mn><\/mrow><\/msup \n><\/mrow><\/msqrt><\/math> is the standard
\nt-statistic of the data. The large \u03bd<\/mi><\/math>
\nthat was chosen with the intent that it would be ’non-informative’ (i.e. minimally e\ufb00ect the result) turns out to directly
\ndivide the BF!\n<\/p>\n

<\/p>\n

This is the opposite situation from estimation, where as
\n\u03bd<\/mi><\/math> increases is behaves more and
\nmore like the improper \ufb02at prior p<\/mi>(<\/mo><\/mi><\/mrow>)<\/mo><\/mrow> \u221d<\/mo> 1<\/mn><\/math>.
\nWe can’t use the \ufb02at prior in the BF t-test, but if we assume that the sample standard deviation
\ns<\/mi><\/math>
\nmatches the true standard deviation, we can perform a BF z-test instead. Using a \ufb02at prior on
\n<\/mi><\/math>, we get a
\nBF of:\n<\/p>\n
\n \u2248<\/mo> s<\/mi>2<\/mn>\u03c0<\/mi><\/mrow>\n n<\/mi><\/mrow><\/mfrac><\/mrow><\/msqrt> exp<\/mo> z<\/mi><\/mrow>2<\/mn><\/mrow><\/msup \n>\u2215<\/mo>2<\/mn><\/mrow><\/mfenced>,<\/mo>\n<\/mrow><\/math><\/div>\n

<\/p>\n

\n

<\/p>\n

Because the \ufb02at prior appears in the numerator only there is no clear scaling choice for it, so this expression is not in
\nany sense calibrated (Several approaches to calibration appear in the literature, see Robert<\/a>\u00a0[1993<\/a>]).\n<\/p>\n

<\/p>\n

It is interesting to see what happens when we apply the Cauchy prior, as used in the standard form of the test. In that
\ncase we see approximate scaling like [Robert et\u00a0al.<\/a>,\u00a02009<\/a>]:\n<\/p>\n

\n \u2248<\/mo> 2<\/mn><\/mrow>\n\u03c0<\/mi>n<\/mi><\/mrow><\/mfrac><\/mrow><\/msqrt>exp<\/mo> t<\/mi><\/mrow>2<\/mn><\/mrow><\/msup \n>\u2215<\/mo>2<\/mn><\/mrow><\/mfenced>.<\/mo>\n<\/mrow><\/math><\/div>\n

<\/p>\n

\n

<\/p>\n

A similar formula appears when the data standard deviation is assumed to be known also:\n<\/p>\n

\n \u2248<\/mo> 2<\/mn><\/mrow>\n\u03c0<\/mi>n<\/mi><\/mrow><\/mfrac><\/mrow><\/msqrt> 1<\/mn> +<\/mo> t<\/mi><\/mrow>2<\/mn><\/mrow><\/msup \n><\/mrow> \nn<\/mi><\/mrow><\/mfrac> <\/mrow><\/mfenced>exp<\/mo> t<\/mi><\/mrow>2<\/mn><\/mrow><\/msup \n>\u2215<\/mo>2<\/mn><\/mrow><\/mfenced>.<\/mo>\n<\/mrow><\/math><\/div>\n

<\/p>\n

\n

<\/p>\n

The 1<\/mn> +<\/mo> t<\/mi><\/mrow>2<\/mn><\/mrow><\/msup \n><\/mrow> \nn<\/mi><\/mrow><\/mfrac> <\/math> part can
\nalso be written 1<\/mn> +<\/mo> x<\/mi><\/mrow>\u0304<\/mo><\/mover><\/mrow> \ns<\/mi><\/mrow>2<\/mn><\/mrow><\/msup \n><\/mrow><\/mfrac> <\/math>
\nin terms of the known standard deviation and the empirical mean, and clearly it is constant asymptotically; the expression
\non the whole scales similarly as in the unknown variance case.<\/p>\n

<\/p>\n

\n

<\/a>Lindley’s paradox<\/h3>\n

<\/p>\n

Lindley’s paradox refers to a remarkable di\ufb00erence in the behavior of Bayes factor tests compared
\nto frequentist tests as the sample size increases. Under a frequentist t-test, we compute the
\nt<\/mi><\/math> statistic and compare it against
\na threshold that depends on n<\/mi><\/math>,
\nand that decreases as n<\/mi><\/math> increases,
\n(Recall the de\ufb01nition of the t<\/mi><\/math>
\nstatistic: t<\/mi> =<\/mo> x<\/mi><\/mrow>\u0304<\/mo><\/mover>(<\/mo>n<\/mi> \u2212<\/mo> 1<\/mn><\/mrow>)<\/mo><\/mrow>\u2215<\/mo>s<\/mi><\/mrow>2<\/mn><\/mrow><\/msup \n><\/mrow><\/msqrt><\/math>). In contrast the e\ufb00ective
\nthreshold used by the BF t<\/mi><\/math>-test
\nactually increases as n<\/mi><\/math> increases,
\nalthough slowly, following roughly log<\/mo> n<\/mi><\/mrow><\/msqrt><\/math>.
\nThis di\ufb00erence is illustrated in the following plot lifted from Rouder et\u00a0al.<\/a>\u00a0[2009<\/a>]:\n<\/p>\n

\n<\/p>\n

\n

<\/p>\n

\"PIC\"\n<\/p>\n<\/div>\n

<\/p>\n

The e\ufb00ect of this di\ufb00erence is that for large sample sizes, a frequentist test can reject the null with small
\np<\/mi><\/math> (say
\np<\/mi> =<\/mo> 0<\/mn>.<\/mo>0<\/mn>5<\/mn><\/math>), while a
\nBF t<\/mi><\/math>-test
\nrun on the same data can strongly support the null hypothesis! As a concrete example, if we generate
\nn<\/mi> =<\/mo> 5<\/mn>0<\/mn>,<\/mo>0<\/mn>0<\/mn>0<\/mn><\/math> synthetic data points
\nwith mean 2<\/mn>\u2215<\/mo>5<\/mn>0<\/mn>0<\/mn>0<\/mn>0<\/mn><\/mrow><\/msqrt><\/math>, variance
\n1, then the BF t<\/mi><\/math>-test
\nyields 0.028 for the alternative hypothesis (i.e. 36:1 odds for the null being true), whereas a frequentist
\nt<\/mi><\/math>-test rejects
\nthe null with p<\/mi> =<\/mo> 0<\/mn>.<\/mo>0<\/mn>4<\/mn>7<\/mn>8<\/mn><\/math>
\n(two-sided test).\n<\/p>\n

<\/p>\n

It’s all well and good to argue the philosophical di\ufb00erences between Bayesian and frequentist approaches, but when
\nthey disagree so strongly, it becomes a practical issue. The key issue is the handling of small-e\ufb00ects. As
\nn<\/mi><\/math> increases and
\ns<\/mi><\/mrow>2<\/mn><\/mrow><\/msup \n><\/math> stays \ufb01xed, a
\n\ufb01xed t<\/mi><\/math>
\nvalue of say 2 represents a smaller and smaller measured e\ufb00ect, namely:\n<\/p>\n
\n x<\/mi><\/mrow>\u0304<\/mo><\/mover> \u221d<\/mo> 2<\/mn><\/mrow> \nn<\/mi><\/mrow><\/msqrt><\/mrow><\/mfrac>.<\/mo>\n<\/mrow><\/math><\/div>\n

<\/p>\n

\n

<\/p>\n

Detecting small e\ufb00ects requires a large sample size, and the Bayesian
\nt<\/mi><\/math>-test
\ngreatly prefers the null hypothesis over that of a small e\ufb00ect. This is usually described positively as a Occam’s razor e\ufb00ect,
\nbut it does have real consequences when we are attempting to \ufb01nd true small e\ufb00ects. Essentially, the BF test will require
\nmuch more data pick up a small e\ufb00ect then a corresponding frequentist test. Often, this is viewed from the opposite side;
\nthat frequentist tests too easily reject the null hypothesis for large sample sizes. This is more of an issue
\nwhen testing more complex models such as linear regressions with multiple coe\ufb03cients. Unless the data was
\nactually generated from a linear equation with Gaussian noise, large samples will inevitably reject simple
\nnull models. This is in contrast on simple point null hypothesis testing, where we can appeal to asymptotic
\nnormality.\n<\/p>\n

<\/p>\n

\n

<\/a>Possible changes from the default Bayes factor test<\/h3>\n

<\/p>\n

Here we attempt to answer the following question: are the properties of the default Bayes factor test indicative of BF tests
\nin general? As we see it, there are two main ways to modify the test: We can use a non-point hypothesis as the null, or we
\ncan change the alternative hypothesis in some fashion. We discuss these possibilities and their combination
\nseparately.\n<\/p>\n

<\/p>\n

\n

<\/a>Changing the alternative hypothesis<\/h4>\n

<\/p>\n

The default Bayes factor t-test is formulated from a objectivist point of view: it’s designed so that the data
\ncan \u201cspeak for its self\u201d, the priors used are essentially as vague as possible while still yielding a usable test.
\nWhat about if we choose the prior for the alternative that gives the strongest evidence for the alternative
\ninstead? This obviously violates the likelihood principle as we choose the prior after seeing the data, but it is
\nnevertheless instructive in that it indicates how large the di\ufb00erence between point NHST tests and BF tests
\nare.\n<\/p>\n

<\/p>\n

A reasonable class of priors to consider is those that are non-increasing in terms of
\n|<\/mo><\/mi>|<\/mo><\/math> , as they have a
\nsingle mode at <\/mi> =<\/mo> 0<\/mn><\/math>.
\nWe discuss other priors below. Table 1<\/a> shows a comparison of p-values, likelihood ratios from the Neyman-Pearson
\nlikelihood ratio test, and a bound on BFs from any possible prior of this form. It is immediately obvious that according to
\nthis bound, any reasonable choice of prior yields much less evidence for the alternative then that suggested
\nby the p-value. For example, the di\ufb00erence at p=0.001 is 18 fold, and the di\ufb00erence is larger for smaller
\np<\/mi><\/math>.\n<\/p>\n
\n

<\/p>\n

<\/a><\/p>\n

\n
<\/p>\n
\n<\/p>\n\n<\/colgroup>\n\n<\/colgroup>\n\n<\/colgroup>\n\n<\/colgroup>\n\n<\/colgroup>\n<\/p>\n<\/p>\n<\/p>\n<\/p>\n<\/p>\n<\/p>\n<\/p>\n<\/p>\n<\/p>\n<\/p>\n<\/p>\n<\/p>\n<\/p>\n
\n
\n<\/td>\n		\n
\n<\/td>\n		\n
\n<\/td>\n		\n
\n<\/td>\n		\n
\n<\/td>\n<\/tr>\n
Method <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\n<\/tr>\n
\n
\n<\/td>\n		\n
\n<\/td>\n		\n
\n<\/td>\n		\n
\n<\/td>\n		\n
\n<\/td>\n<\/tr>\n
\n
\n<\/td>\n		\n
\n<\/td>\n		\n
\n<\/td>\n		\n
\n<\/td>\n		\n
\n<\/td>\n<\/tr>\n
t-statistic<\/td>\n 1.645 <\/td>\n 1.960 <\/td>\n 2.576 <\/td>\n 3.291<\/td>\n<\/tr>\n
\n
\n<\/td>\n		\n
\n<\/td>\n		\n
\n<\/td>\n		\n
\n<\/td>\n		\n
\n<\/td>\n<\/tr>\n
p-value <\/td>\n 0.1 <\/td>\n 0.05 <\/td>\n 0.01 <\/td>\n 0.001 <\/td>\n<\/tr>\n
\n
\n<\/td>\n		\n
\n<\/td>\n		\n
\n<\/td>\n		\n
\n<\/td>\n		\n
\n<\/td>\n<\/tr>\n
LR <\/td>\n0.258:1<\/td>\n0.146:1<\/td>\n0.036:1<\/td>\n0.0044:1<\/td>\n<\/tr>\n
\n
\n<\/td>\n		\n
\n<\/td>\n		\n
\n<\/td>\n		\n
\n<\/td>\n		\n
\n<\/td>\n<\/tr>\n
BF bound<\/td>\n0.644:1<\/td>\n0.409:1<\/td>\n0.123:1<\/td>\n 0.018:1 <\/td>\n<\/tr>\n
\n
\n<\/td>\n		\n
\n<\/td>\n		\n
\n<\/td>\n		\n
\n<\/td>\n		\n
\n<\/td>\n<\/tr>\n
<\/td>\n<\/tr>\n<\/table>\n<\/div>\n
Table\u00a01: <\/span>Comparison of methods for point-null normal data tests, reproduced from Berger and Wolpert<\/a>\u00a0[1988<\/a>].<\/span><\/div>\n

<\/p><\/div>\n

\n <\/div>\n

<\/p>\n

Although we disregarded multi-modal alternative priors above, they actually form an interesting class of tests. In terms of
\nconservativeness, their BF is easily seen to be bounded by the likelihood ratio when using a prior concentrated on the maximum
\nlikelihood solution <\/mi><\/mrow>\u0302<\/mo><\/mover><\/math>:\n<\/p>\n
\n p<\/mi>(<\/mo>D<\/mi>|<\/mo><\/mi> =<\/mo> 0<\/mn><\/mrow>)<\/mo><\/mrow><\/mrow> \np<\/mi>(<\/mo>D<\/mi>|<\/mo><\/mi> =<\/mo> <\/mi><\/mrow>\u0302<\/mo><\/mover><\/mrow>)<\/mo><\/mrow><\/mrow><\/mfrac>.<\/mo>\n<\/mrow><\/math><\/div>\n

<\/p>\n

\n

<\/p>\n

This ratio is simplify the Neyman-Pearson likelihood ratio for composite hypothesis,
\nand it’s values are also shown in Table 1<\/a>. This values are still more conservative than
\np<\/mi><\/math>-values
\nby a factor of 2-5. Note that in a Neyman-Pearson test such as the frequentist
\nt<\/mi><\/math>-test, this
\nratio is not read-o\ufb00 directly as an odds ratio as we are doing here; that’s why this NP odds ratio doesn’t agree directly with
\nthe p-value.\n<\/p>\n

<\/p>\n

The interesting aspect of multi-modal priors is that they allow you to formulate a test that is more powerful
\nfor \ufb01nding evidence of the null hypothesis. It can be shown that under the default BF t-test, evidence for
\nthe null accumulates very slowly when the null is true, where as evidence for the alternative accumulates
\nexponentially fast when the alternative is true. Multi-modal priors can be formulated that \ufb01x this imbalance
\n[Johnson and Rossell<\/a>,\u00a02010<\/a>]. One simple case is a normal or Cauchy prior with a notch around the null
\n<\/mi> =<\/mo> 0<\/mn><\/math> set to
\nhave low probability.\n<\/p>\n

<\/p>\n

If we go a step further and consider the limits of sequences of improper priors, the Bayes factor
\ncan take essentially any value. Some particular \u201creasonable\u201d sequences actually yield values very similar to
\nt<\/mi><\/math>-test’s
\np-values [Robert<\/a>,\u00a01993<\/a>] as their posterior probabilities, at least between 0.1 and 0.01. Their construction also places 0
\nprobability mass in the neighborhood of the null point in the limit, so it is similar to the multi-modal priors discussed in the
\nprevious paragraph. Although this improper prior does actually avoid Lindley’s paradox, it is more of a theoretical
\nconstruction; it is not recommend in practice, even by it’s discoverer [Robert<\/a>,\u00a01993<\/a>].\n<\/p>\n
<\/a>Consequences for Lindley’s paradox<\/h5>\n

<\/p>\n

As discussed above, when restricting ourselves to well behaved proper priors, \u201cLindley’s paradox\u201d as such can not be
\nrecti\ufb01ed by a careful choice of a proper prior on the alternative hypothesis. When one takes a step back and considers the
\nconsequences of this, Lindley’s paradox becomes much clearer, as a statement about objective priors rather than BF tests in
\ngeneral.\n<\/p>\n

<\/p>\n

The central idea is this: It requires a lot of evidence to detect small e\ufb00ects when you use vague priors. The idea of running a
\nlong experiment when you use a vague prior is nonsensical from an experimental design point of view: large amounts of data are
\nuseful for picking up small e\ufb00ects, and if you expect such a small e\ufb00ect you should encode this sensibly in your prior. The default
\nCauchy prior used in the BF t-test essentially says that you expect a 50% chance that the absolute e\ufb00ect size (Cohen’s
\nd<\/mi><\/math>) is larger
\nthan 1<\/mn><\/math>,
\nwhich is a completely implausible in many settings. For instance, if you are trying to optimize conversion rates on a website<\/p>\n

with say a 10% base conversion rate, an e\ufb00ect size of \u201c1\u201d is a 4x increase in sales! Some software defaults to 0.707 instead of
\n1 for the Cauchy scale parameter, but the e\ufb00ect is largely the same\n<\/p>\n

<\/p>\n

Experimental design considerations are important in the choice of a prior, and the Bayesian justi\ufb01cation of the Bayes
\nfactor depends on our priors being reasonable. The choice of using a particular BF test is not a full experimental
\ndesign, just as for a frequentist it is not su\ufb03cient to just decide on particular test without regard to the
\neventual sample size. It is necessary to examine other properties of the test, so determine if it will behave
\nreasonably in the experimental setting in which we will apply it. So in e\ufb00ect, Lindley’s paradox is just the
\nstatement that a poorly designed Bayesian experiment won’t agree with a less poorly designed frequentist
\nexperiment.\n<\/p>\n

<\/p>\n

Put another way, if the Bayes factor is comparing two hypothesis, both of which are unlikely under the data, you
\nwon’t get reasonable results. You can’t identify this from examining the Bayes factor, you have to look at
\nthe experiment holistically. The extreme case of this is when neither hypothesis includes the true value of
\n<\/mi><\/math>. In such
\ncases, the BF will generally not converge to any particular value, and may show strong evidence in either
\ndirection.\n<\/p>\n

<\/p>\n

\n

<\/a>An example<\/h5>\n

<\/p>\n

It is illustrative to discuss an example in the literature of this problem: a high pro\ufb01le case where the Bayesian
\nand frequentist results are very di\ufb00erent under a default prior, and more reasonable under a subjective
\nprior. One such case is the study by Bem<\/a>\u00a0[2011<\/a>] on precognition. The subject matter is controversial;
\nBem gives results for 9 experiments, in which 8 show evidence for the existence of precognition at the
\np<\/mi> =<\/mo> 0<\/mn>.<\/mo>0<\/mn>5<\/mn><\/math>
\nlevel! This obviously extraordinary claim lead to extensive examination of the results from the Bayesian
\nview-point.\n<\/p>\n

<\/p>\n

In Wagenmakers et\u00a0al.<\/a>\u00a0[2011a<\/a>], the raw data of Bem<\/a>\u00a0[2011<\/a>] is analyzed using the default Bayesian t-test. They report
\nBayes factors towards the alternative (in decreasing order) of 5.88, 1.81, 1.63, 1.05, 0.88, 0.58, 0.47, 0.32, 0.29, 0.13 . So only
\none experiment showed reasonable evidence of precognition (5.88:1 for), and if you combine the results you get 19:1 odds
\nagainst an e\ufb00ect.\n<\/p>\n

<\/p>\n

In contrast, Bem et\u00a0al.<\/a>\u00a0[2011<\/a>] performed a Bayes factor analysis using informative priors on e\ufb00ect sizes, uses values
\nconsidered reasonable under non-precognitive circumstances. The 3 largest Bayes factors for the existence of precognition
\nthey got were 10.1, 5.3 and 4.9, indicating substantial evidence!\n<\/p>\n

<\/p>\n

The interesting thing here is that the vague prior used by Wagenmakers et\u00a0al.<\/a>\u00a0[2011a<\/a>] placed a large prior
\nprobability on implausibly large e\ufb00ect sizes. Even strong proponents of ESP don’t believe in such large e\ufb00ects,
\nas they would have been easily detected in other past similar experiments. Somewhat counter-intuitively,
\nthis belief that ESP e\ufb00ects are strong if they exist at all yields a weaker posterior belief in the existence of
\nESP!\n<\/p>\n

<\/p>\n

It should be noted that Wagenmakers et\u00a0al.<\/a>\u00a0[2011b<\/a>] responded to this criticism, but somewhat unsatisfactory in my
\nopinion. Of course, even with the use of correct subjective Bayesian priors on the alternative hypothesis, these experiments
\nare not su\ufb03cient to convince a skeptic (such as my self) of the existence of ESP! If one factors in the prior
\np<\/mi>(<\/mo>H<\/mi><\/mrow>1<\/mn><\/mrow><\/msub \n><\/mrow>)<\/mo><\/mrow>\u2215<\/mo>p<\/mi>(<\/mo>H<\/mi><\/mrow>0<\/mn><\/mrow><\/msub \n><\/mrow>)<\/mo><\/mrow><\/math> and
\nupdates their beliefs correctly, you may go from a 1:1-million belief in ESP existing to a 1:76 belief (if you use the naive
\ncombined BF of 13,669 from Bem et\u00a0al.<\/a>\u00a0[2011<\/a>]).\n<\/p>\n

<\/p>\n

Most reasonable people would consider the chance that a systematic unintentional error was made during the
\nexperiments to be greater than 1 in a thousand, and so not update their beliefs quite as severely. Regardless, several
\npre-registered replications of Bem’s experiments have failed to show any evidence of precognition [Ritchie
\net\u00a0al.<\/a>,\u00a02012<\/a>].\n<\/p>\n

<\/p>\n

\n

<\/a>Recommendations when using point-nulls<\/h5>\n

<\/p>\n

The general consensus in the literature is that single Bayes factors should not be reported. Instead the BFs corresponding to
\nseveral di\ufb00erent possible priors for the alternative hypothesis should be reported. Only if the conclusions are stable for a<\/p>\n

wide range of priors can they be reported as de\ufb01nitive. If the results are not stable, then it must be argued why a particular
\nprior, chosen subjectively, is the best choice.\n<\/p>\n

<\/p>\n

Bounds giving the largest or smallest possible Bayes factor over reasonable classes of priors, such as the bounds given in
\nBerger and Je\ufb00erys<\/a>\u00a0[1991<\/a>], can also be reported. The consideration of multiple possible priors is generally called \u201crobust\u201d
\nBayesian analysis.\n<\/p>\n

<\/p>\n

\n

<\/a>Non-point nulls<\/h4>\n

<\/p>\n

As we showed above, under essentially any point-null Bayes factor t-test, the results are much more conservative
\nthan frequentist t-tests on the same data. But is this indicative of BF tests in general? It turns out Bayes
\nfactor tests that don’t use point-nulls behave much more like their frequentists counterparts [Casella and
\nBerger<\/a>,\u00a01987<\/a>].\n<\/p>\n

<\/p>\n

But \ufb01rst, lets just point out that point-null hypotheses are fundamentally weird things. When we start assigning positive
\nprobability mass to single values, we move from the comfortable realm of simple distribution functions we can plot on a
\nwhiteboard, into the Terra pericolosa <\/span>that is measure theory. As anybody who has done a measure theory course knows,
\ncommon sense reasoning can easily go astray even on simple sounding problems. The \u201csurprising\u201d Occam’s razor
\ne\ufb00ect seen in point-null BF tests is largely attributable to this e\ufb00ect. It can be argued that any prior that
\nconcentrates mass on a single point is not \u201cimpartial\u201d, e\ufb00ectively favoring that point disproportionately [Casella and
\nBerger<\/a>,\u00a01987<\/a>].\n<\/p>\n

<\/p>\n

\n

<\/a>One-sided tests<\/h5>\n

<\/p>\n

The simplest alternative to a point null test is a postive versus negative e\ufb00ect test, where the null is taken to be the entire
\nnegative (or positive) portion of the real line. This test is very similar in e\ufb00ect to a frequentist one sided test. Unlike the
\npoint-null case, both the alternative and null hypotheses spaces are the same dimension; in practice this means we can use
\nthe same unnormalized priors on both without running into issues. This test can be preformed using the BayesFactor<\/a> R
\npackage with the ttestBF method, althought somewhat indirectly, by taking the ratio of two seperate point null
\ntests:\n<\/p>\n

<\/p>\n

<\/a><\/span>bfInterval\u00a0<\u2212<\/span>\u00a0ttestBF(x\u00a0=\u00a0ldata,\u00a0y=rdata,\u00a0nullInterval=c(\u2212<\/span>Inf,0))\u00a0
<\/a><\/span>bfNonPoint\u00a0<\u2212<\/span>\u00a0bfInterval[2]\u00a0\/\u00a0bfInterval[1]\u00a0
<\/a><\/span>print(bfNonPoint)<\/p><\/div>\n

<\/p>\n

Applying one-sided tests in the frequentist setting is routine (in fact often advocated [Cho and Abe<\/a>,\u00a02013<\/a>]), it
\nis surprising then that their theoretical properties are very di\ufb00erent to two-sided tests. If we look at both
\nBayesian and frequentist approaches from the outside, using decision theory (with quadratic loss), we see a
\nfundamental di\ufb00erence. For the one sided case, the Fisher p-value approach and obeys a weak kind of optimality
\nknown as admissibility [Hwang et\u00a0al.<\/a>,\u00a01992<\/a>]: no approach strictly dominates it. In contrast, in the two-sided
\ncase neither the frequentist or the standard Bayesian approach is admissible, and neither dominates the
\nother!\n<\/p>\n

<\/p>\n

<\/a>What does this tell us about sequential testing settings?<\/span>
\n The use of Bayes factor tests is sometimes advocated in the sequential testing setting, as their interpretation is not
\na\ufb00ected by early stopping. \u201cSequential testing\u201d is when a test of some form of test is applied multiple times during the
\nexperiment, with the experiment potentially stopping early if the test indicates strong evidence of an e\ufb00ect. Using
\nfrequentist tests sequentially is problematic; the false-positive rate is ampli\ufb01ed and special considerations
\nmust be made to account for this. See my previous post<\/a> for details on how to perform the frequentist tests
\ncorrectly.<\/p>\n

<\/p>\n

The interpretation of the Bayes factor in contrast is una\ufb00ected by early stopping. Naive application of a point-null BF
\ntest does seem to perform reasonable in a sequential setting, as it’s naturally conservative nature results in few false
\npositives being detected. However, as we pointed out above, one-sided non-point null tests are equally valid, and they are
\nnot particularly conservative. Just applying one sequentially gives you high false-positive rates comparable to the frequentist
\ntest. The false-positive rate is a frequentist property, and as such it is quite unrelated to the Bayesian interpretation of the
\nposterior distribution.\n<\/p>\n

<\/p>\n

Indeed, in most sequential A\/B testing situations, such as website design tests, the one-sided non-point test seems the
\nmost reasonable, as the two hypothesis correspond more directly to the two courses of action that will be taken after the
\ntest concludes: Switch all customers to design A or design B.\n<\/p>\n

<\/p>\n

\n

<\/a>So, should I use a point null hypothesis?<\/h3>\n

<\/p>\n

Here we start to move outside the realm of mathematics and start to meander into the realm of philosophy. You have to
\nconsider the purpose of your hypothesis test. Point-nulls enforce a strong Occam’s razor e\ufb00ect, and depending on the
\npurpose of your test, this may or may not be appropriate.\n<\/p>\n

<\/p>\n

The examples and motivation used for null hypothesis testing in the original formulation of the problem by Je\ufb00reys
\ncame from physics. In physics problems testing for absolute truth is a much clearer proposition. Either an exotic particle
\nexists or it doesn’t, there is no in-between. This is far cry from the psychology, marketing and economics problems that are
\nmore commonly dealt with today, where we always expect some sort of e\ufb00ect, although potentially small and possibly in the
\nopposite direction from what we guessed apriori.\n<\/p>\n

<\/p>\n

You need a real belief in the possibility of zero e\ufb00ect in order to justify a point null test. This is a real
\nproblem in practice. In contrast, by using non-point null tests you will be in closer agreement with frequentist
\nresults, and you will need less data to make conclusions, so there is a lot of advantages to avoiding point null
\ntests.\n<\/p>\n

<\/p>\n

It is interesting to contrast modern statistical practice in high energy physics (HEP) with that of other areas where statistics
\nis applied. It is common practice to seek \u201c\ufb01ve sigma\u201d of evidence before making a claim of a discovery in HEP \ufb01elds. In contrast
\nto the usual p<\/mi> \u2264<\/mo> 0<\/mn>.<\/mo>0<\/mn>5<\/mn><\/math>
\nrequirements, the di\ufb00erence is vast. Five sigma corresponds to about a 1-in-1million
\np<\/mi><\/math> value. However,
\nif you view the \ufb01ve sigma approach as a heuristic approximation to the point-null BF test, the results are much more comparable.
\nThe equivalent t<\/mi><\/math>
\nstatistic needed for rejection of the point null under the Bayesian test scales with the amount of data. HEP experiments often
\naverage over millions of data points, and it is in that multiple-million region where the rejection threshold, when rephrased in
\nterms of the t<\/mi><\/math>
\nstatistic, is roughly \ufb01ve.\n<\/p>\n

<\/p>\n

\n

<\/a>Concluding guidelines for applying Bayes factor tests<\/h3>\n