We utilize a scattering-based seismic tomography technique to constrain crustal structure around the southern San Francisco Bay region (SFBR). This technique is based on coupled traveling wave scattering theory, which has usually been applied to the interpretation of surface waves in large regional-scale studies. Using fully three-dimensional kernels, this technique is here applied to observed P, S, and surface waves of intermediate period (3--4 s dominant period) observed following eight selected regional events. We use a total of 73 seismograms recorded by a U.S. Geological Survey short-period seismic array in the western Santa Clara Valley, the Berkeley Digital Seismic Network, and the Northern California Seismic Network. Modifications of observed waveforms due to scattering from crustal structure include (positive or negative) amplification, delay, and generation of coda waves. The derived crustal structure explains many of the observed signals which cannot be explained with a simple layered structure. There is sufficient sensitivity to both deep and shallow crustal structure that even with the few sources employed in the present study, we obtain shallow velocity structure which is reasonably consistent with previous P wave tomography results. We find a depth-dependent lateral velocity contrast across the San Andreas fault (SAF), with higher velocities southwest of the SAF in the shallow crust and higher velocities northeast of the SAF in the midcrust. The method does not have the resolution to identify very slow sediment velocities in the upper approximately 3 km since the tomographic models are smooth at a vertical scale of about 5 km.