4 Calculating p-values
The main feature of the myFitter library is the numerical computation of p-values in likelihood ratio tests of nested and non-nested models. As discussed in [arXiv:1207.1446], p-values in likelihood ratio tests have to be computed numerically when parameters are bounded or when the two models to be compared are not nested, meaning that one model can not be obtained from the other by fixing some of its parameters.
In a likelihood ratio test one compares the performance of two models A and B in desribing observed data. The test is performed under the null hypothesis that one of the models, say, model B, is realised with its best-fit parameters. Then one uses the difference \Delta\chi^2=\chi^2_B-\chi^2_A of the minimal chi-square values of the two models as test statistic. Then one considers a large ensemble of “toy measurements” which are randomly distributed about their true values (as predicted by model B) according to their experimental errors. The p-value is the probability that a toy measurement leads to a \Delta\chi^2 value which is bigger (i.e. more in favour of model A) than the \Delta\chi^2 value obtained from the measured data.