How to reduce the horizontal pressure gradient error is a key issue in terrain-following coastal models. The horizontal pressure gradient splits into two parts, and incomplete cancellation of the truncation errors of those parts cause the error. Use of the finite volume discretization leads to a conserved scheme for pressure gradient computation that has better truncation properties with high accuracy. The analytical coastal topography and seamount test cases are used to evaluate the new scheme. The accuracy of the new scheme is comparable to the sixth-order combined compact scheme (with an error reduction by a factor of 70 comparing to the second-order scheme) with mild topography and much better than the sixth-order combined compact scheme with steep topography. The computational efficiency of the new scheme is comparable to the second-order difference scheme. The two characteristics, high accuracy and computational efficiency, make this scheme useful for the sigma coordinate ocean models.