There is evidence that solar activity variations can affect the cloud cover at Earth. However, it is still unclear which solar driver plays the most important role in the cloud formation. Here we use partial correlations to distinguish between the effects of two solar drivers (cosmic rays and the UV irradiance) and the mutual relations between clouds at different altitudes. We find that the solar influence on cloud cover is not uniquely defined by one solar driver, but both seem to play a role depending on the climatic conditions and altitude. In particular, low clouds are mostly affected by UV irradiance over oceans and dry continental areas and by cosmic rays over some mid-high latitude oceanic areas and moist lands with high aerosol concentration. High clouds respond more strongly to cosmic ray variations, especially over oceans and moist continental areas. These results provide observational constraints on related climate models.