Real-time forecasts of PM2.5 aerosol mass from seven air quality forecast models (AQFMs) are statistically evaluated against observations collected in the northeastern United States and southeastern Canada from two surface networks and aircraft data during the summer of 2004 International Consortium for Atmospheric Research on Transport and Transformation (ICARTT)/New England Air Quality Study (NEAQS) field campaign. The AIRNOW surface network is used to evaluate PM2.5 aerosol mass, the U.S. EPA STN network is used for PM2.5 aerosol composition comparisons, and aerosol size distribution and composition measured from the NOAA P-3 aircraft are also compared. Statistics based on midday 8-hour averages, as well as 24-hour averages are evaluated against the AIRNOW surface network. When the 8-hour average PM2.5 statistics are compared against equivalent ozone statistics for each model, the analysis shows that PM2.5 forecasts possess nearly equivalent correlation, less bias, and better skill relative to the corresponding ozone forecasts. An analysis of the diurnal variability shows that most models do not reproduce the observed diurnal cycle at urban and suburban monitor locations, particularly during the nighttime to early morning transition. While observations show median rural PM2.5 levels similar to urban and suburban values, the models display noticeably smaller rural/urban PM2.5 ratios. The ensemble PM2.5 forecast, created by combining six separate forecasts with equal weighting, is also evaluated and shown to yield the best possible forecast in terms of the statistical measures considered. The comparisons of PM2.5 composition with NOAA P-3 aircraft data reveals two important features: (1) The organic component of PM2.5 is significantly underpredicted by all the AQFMs and (2) those models that include aqueous phase oxidation of SO2 to sulfate in clouds overpredict sulfate levels while those AQFMs that do not include this transformation mechanism underpredict sulfate. Errors in PM2.5 ammonium levels tend to correlate directly with errors in sulfate. Comparisons of PM2.5 composition with the U.S. EPA STN network for three of the AQFMs show that sulfate biases are consistently lower at the surface than aloft. Recommendations for further research and analysis to help improve PM2.5 forecasts are also provided.