Bayesian methods are often applied for estimating the event rate from a series of event occurrences. However, the Bayesian posterior distribution requires the computation of the marginal likelihood which generally involves an analytically intractable integration. As an event rate is defined in a very high dimensional space, it is computationally demanding to obtain the Bayesian posterior distribution for the rate. We estimate the rate underlying a sequence of event counts by deriving an approximate Bayesian inference algorithm for the time-varying binomial process. This enables us to calculate the posterior distribution analytically. We also provide a method for estimating the prior hyperparameter, which determines the smoothness of the estimated event rate. Moreover, we provide an efficient method to compute the upper and lower bounds of the marginal likelihood, which evaluate the approximation accuracy. Numerical experiments demonstrate the effectiveness of the proposed method in terms of the estimation accuracy.