Large-scale tectonic structures such as orogens and rifts commonly display regularly spaced faults and/or localized shear zones. To understand how fault sets organize with a characteristic spacing, we present a semianalytical instability analysis of an idealized lithosphere composed of a brittle layer over a ductile half-space undergoing horizontal shortening or extension. The rheology of the layer is characterized by an effective stress exponent, ne. The layer is pseudoplastic if 1/ne = 0 and forms localized structures if 1/ne < 0. Two instabilities grow simultaneously in this model: the buckling/necking instability that produces broad undulations of the brittle layer as a whole, and the localization instability that produces a spatially periodic pattern of faulting. The latter appears only if the material in the brittle layer weakens in response to a local perturbation of strain rate, as indicated by 1/ne < 0. Fault spacing scales with the thickness of the brittle layer and depends on the efficiency of localization. Localization is more efficient for more negative 1/ne, leading to more widely spaced faults. The fault spacing is related to the wavelength at which different deformation modes within the layer enter a resonance that exists only if 1/ne < 0. Depth-dependent viscosity and the model density offset the instability wavelengths by an amount aL that we determine empirically. The wave number of the localization instability, is kjL = π(j + aL)(-1/ne)-1/2/H, with H the thickness of the brittle layer, j an integer, and 1/4 < aL < 1/2 if the strength of the layer increases with depth and the strength of the substrate decreases with depth.