Evaluation and Comparison of Global Ensemble Prediction Systems for Probabilistic Forecasting of Heavy Rainfalls (Case Study: Kan Basin, Iran)

Introduction
Heavy rainfalls in small basins can lead to devastating flash flood with fatalities and tremendous damages. Thus, forecasting of heavy rainfall is an important step in development of a flood warning system. Various models were used for rainfall forecasting such as artificial neural network (Moustris et al. 2011), time series models (Sapmson et al, 2013), wavelet theory (Partal and Kişi, 2007), and regression tree model (Fallahi et al, 2011). In recent decades, the Numerical Weather Prediction (NWP) models were widely applied for weather prediction. Numerical weather predictions (NWPs) usually have uncertainties in initial conditions and model structures. In recent decades, Ensemble Prediction Systems (EPS) have been increasingly used to capture the uncertainties in NWPs. Several operational centers, including the European Centre for Medium-Range Weather Forecasts (ECMWF), the National Centers for Environmental Prediction (NCEP), the Japan Meteorological Agency (JMA), and the United Kingdom Meteorological Office (UKMO) offer valuable operational numerical predictions at a global scale (Hsiao et al, 2013).
The purpose of the present study is the comparison of the ECMWF, UKMO, and NCEP global ensemble prediction systems for forecasting of heavy rainfalls in Kan watershed, Tehran, Iran.
Materials and Methods
In this paper, the performance of the global ensemble prediction models has been evaluated for heavy rainfall forecasting in Kan Basin, Tehran, Iran. This research was conducted for 8 heavy rainfalls (flood producer) in the study area using two different precipitation thresholds including 5 and 10 mm. For this purpose, the UKMO, NCEP and ECMWF ensemble predictions are archived in the TIGGE database. Other forecast centers were not used in this study for various reasons, such as the unavailability ensemble forecasts of some centers in 0000 UTC. It is worth noting that interpolated predictions on 0.125 degree resolution were used in this study.
Then, the heavy rainfalls predicted by UKMO, NCEP and ECMWF were compared with the observed rainfall. Three criteria including the accuracy, reliability and sharpness were applied to assess the predictive efficiency of ensemble forecasters. The Brier Score, reliability diagram and average width of 50% and 90% prediction intervals were respectively used to assess the three mentioned criteria.
The Brier score is widely applied in meteorology to assess the probability and ensemble forecasts. This score is presented as following equation:



 

(2)




In the above equation,  is the forecast probability of the event,  is the observational value equal to 1 or 0 depending on whether the event occurred or not, and N is the number of forecast-observation pairs. A minimum Brier score is equal to zero for a perfect forecaster.
Reliability diagrams are a graph of the observed frequency plotted against the forecast probability of the event. For perfect reliability, the forecast probability and the frequency of event is equal. Thus, the closer the reliability curve to the diameter is the higher the reliability.
Sharpness is a feature of the forecasts that refers to the concentration of the predictive distributions.
The more concentrated the predictive distributions are, the sharper the forecasts and thus the better the predictive model.
Results and Discussion  
The results showed that for 5 and 10 mm rainfall thresholds, UKMO’s ensembles were the least efficient, reliable and sharp. Thus, UKMO’s ensembles are not suitable for heavy rainfall forecasting in the study area. It was also observed that for 5 mm rainfall threshold, there was not a significant difference between the accuracy and reliability of NCEP and ECMWF ensembles but with increasing the level of threshold to 10 mm, NCEP’s ensembles had higher efficiency and were more reliable. In terms of sharpness, NCEP’s ensembles were also the most sharp, followed by ECMWF and UKMO.
Conclusion
Since the higher threshold is necessary for heavy rainfall prediction, so the 10 mm rainfall threshold was used in assessment of the predictions by the criteria. Analysis of the results based on the three mentioned criteria showed that NCEP’s ensembles had the best performance compared with the other predictions. Therefore, it is recommended to study the NCEP’s ensembles for prediction of heavy and flood producing rainfalls in mountainous watersheds like Kan Basin.