Understanding causes of dune irregularity, especially dune trough scour, is important for the modeling of vertical sorting of sediment mixtures in morphological models of rivers with sediment mixtures. Sediment in dunes is generally sorted in a fining-upward manner, which affects the sediment transport composition depending on the scour depth distribution of the dunes. Why dunes become more irregular and develop deep scour holes in some conditions is only partially understood. Moreover, there is a feedback between vertical sorting and dune irregularity. In gravelly sands, erosion-resistant coarse layers may form that decrease or inhibit dune trough scour. The causes of dune irregularity and the feedback by coarse sediment layers are explored in experiments and are demonstrated to be related partly to the upstream sediment boundary condition determined by the experimental setup: sediment feeding or recirculating. Sediment recirculation flumes promote fining of the transported sediment and formation of a less or immobile gravel lag layer in the dune troughs. Sediment feed flumes may force the transported sediment at the flume entry to be equal to the bed sediment (equal mobility condition) and hence allow no lag layer formation. Experiments show that the dune trough scour in recirculation is less deep than in feed flumes, and the vertical sorting and transport sediment composition are different. The experiments also indicate that dune irregularity in feed flumes is related to dimensionless shear stress similarly as in uniform sediments. Interpretation of experiments of dune dynamics in sediment mixtures should therefore account for differences in the upstream sediment supply condition. More in general, it is hypothesized that the upstream sediment supply condition (resembling either recirculation or feed flumes) in rivers may affect dune and vertical sorting dynamics.