A new Bayesian procedure for testing a point null hypothesis is introduced, by which the Lindley’s paradox (Lindley [13]) in testing a normal mean does not occur any more. The problem of testing a location family is also considered. It is shown that the Bayesian evidence derived by this new testing procedure turns out to be very reasonable measure of evidence against the null hypothesis. Particularly, our evidence derived under the noninformative or uniform prior is usually coincident with the corresponding observed significance level.