Some studies of global paleomagnetic data have found an offset of the magnetic pole during the Plio-Pleistocene which has been interpreted as indicating a period of rapid True Polar Wander, with a rate of movement comparable to the present-day rate of polar motion deduced from astronomical observations. We show that much of the polar offset determined from the paleomagnetic data may be due to deviations in pole position caused by persistent non-dipole zonal components of the geomagnetic field. A correction of paleomagnetic poles for the long-term non-dipole field reduces the polar offset and thus suggests a slower or shorter episode of True Polar Wander over the past 5 million years.