A practical time domain reflectometry (TDR) measurement system consists of a cable tester and a nonuniform transmission line. The transmission line includes a coaxial cable, a transitional device, and a measurement probe. Dielectric spectroscopy in a nonuniform transmission line requires a wave propagation model that accounts for multiple reflections in a nonuniform and dispersive TDR measurement system. Spectral analysis and the concept of input impedance were utilized to develop such a model. The system parameters were calibrated using the measured TDR waveform in an aqueous material with known dielectric permittivity. The excellent match between the simulated and measured waveforms validated the wave propagation model. A layer-peeling algorithm was developed, based on the new formulation of the wave propagation model to remove the effect of the cable and transitional device on the measurement by TDR. The dielectric spectrum obtained by solving the scatter function was compared with that obtained by matching waveforms based on a dielectric relaxation model. The latter method was confirmed by the experimental results as being superior to the former at high frequencies if an appropriate dielectric relaxation model was used. Both the Debye relaxation model and the volumetric mixing model could model the dielectric dispersion of soils in the TDR frequency range. The latter method outperforms the former in modeling the dielectric dispersion of soils over the full range of frequencies.