Major strike slip faults juxtapose geologically dissimilar terrain which may vary in mechanical properties, leading to an asymmetric pattern of strain accumulation. We present new GPS data on the Carrizo segment of the San Andreas Fault, separating the Salinian block southwest of the fault from Franciscan terrane northeast of the fault, to better quantify asymmetric strain accumulation. We also present a series of finite element models to investigate the possible role of variable elastic layer thickness and material properties of the upper crust. The geodetic data are well fit with a simple model comprising a weak upper crustal zone 10--25 km wide northeast of the fault. This model is also consistent with geologic data on the distribution of major rock types and corresponding laboratory data on their material properties, as well as paleoseismic, seismic and magnetotelluric data. Using this model, we estimate a "long-term" (average over several seismic cycles) slip rate for the San Andreas Fault of 36-1.5+2 mm/yr in agreement with the known Holocene rate within uncertainties, and a viscosity for the combined lower crust/upper mantle of 2--5 ¿ 1019 Pa s.