The timescales and phases of auroral substorm activity are quantified in this study using the hemispheric power computed from Polar Ultraviolet Imager (UVI) observations. With this approach we are able to determine substorm phase self consistently from the UVI optical measurements without needing any ancillary metrics for auroral activity. This technique also allows us to combine a quantitative measurement of substorm timescales with the qualitative description of auroral morphology during each event. We show how the characteristics of substorms vary with IMF BZ orientation and season for several hundred substorm events. Substorm intensity, as measured by the rate of energy deposition by electron precipitation, exhibits greater variation with IMF BZ orientation than with season. The timescale of substorm expansion shows comparable variation with both IMF BZ orientation and season. On the other hand, substorm recovery times vary more strongly with season than with IMF BZ direction. The recovery time for substorm activity is well ordered by whether or not the nightside auroral region is sunlit: substorms occurring in the winter and equinox periods have similar recovery timescales which are both roughly a factor of two longer than that for summer when the auroral oval is sunlit. However, IMF BZ orientation is shown to have a strong modulating effect on both substorm expansion and recovery timescales during the summer. The winter-summer asymmetries in the aurora described by previous studies are more completely explained by the seasonal variations of substorm recovery timescales characterized in this work. Our results support the hypothesis that the ionosphere plays an active role in governing the dynamics of the aurora.