Evaluation and Comparison of Global Ensemble Prediction Systems for Probabilistic Forecasting of Heavy Rainfalls (Case Study: Kan Basin, Iran)

Document Type : Full length article


1 PhD Candidate in Water Resources Engineering, College of Aburaihan, University of Tehran

2 Associate Professor of Irrigation and Drainage, College of Aburaihan, University of Tehran, Iran

3 Assistant Professor of Iranian National Institute for Oceanography and Atmospheric Science, Iran

4 Assistant Professor of Irrigation and Drainage, College of Aburaihan, University of Tehran, Iran


Heavy rainfalls in small basins can lead to devastating flash flood with fatalities and tremendous damages. Thus, forecasting of heavy rainfall is an important step in development of a flood warning system. Various models were used for rainfall forecasting such as artificial neural network (Moustris et al. 2011), time series models (Sapmson et al, 2013), wavelet theory (Partal and Kişi, 2007), and regression tree model (Fallahi et al, 2011). In recent decades, the Numerical Weather Prediction (NWP) models were widely applied for weather prediction. Numerical weather predictions (NWPs) usually have uncertainties in initial conditions and model structures. In recent decades, Ensemble Prediction Systems (EPS) have been increasingly used to capture the uncertainties in NWPs. Several operational centers, including the European Centre for Medium-Range Weather Forecasts (ECMWF), the National Centers for Environmental Prediction (NCEP), the Japan Meteorological Agency (JMA), and the United Kingdom Meteorological Office (UKMO) offer valuable operational numerical predictions at a global scale (Hsiao et al, 2013).
The purpose of the present study is the comparison of the ECMWF, UKMO, and NCEP global ensemble prediction systems for forecasting of heavy rainfalls in Kan watershed, Tehran, Iran.
Materials and Methods
In this paper, the performance of the global ensemble prediction models has been evaluated for heavy rainfall forecasting in Kan Basin, Tehran, Iran. This research was conducted for 8 heavy rainfalls (flood producer) in the study area using two different precipitation thresholds including 5 and 10 mm. For this purpose, the UKMO, NCEP and ECMWF ensemble predictions are archived in the TIGGE database. Other forecast centers were not used in this study for various reasons, such as the unavailability ensemble forecasts of some centers in 0000 UTC. It is worth noting that interpolated predictions on 0.125 degree resolution were used in this study.
Then, the heavy rainfalls predicted by UKMO, NCEP and ECMWF were compared with the observed rainfall. Three criteria including the accuracy, reliability and sharpness were applied to assess the predictive efficiency of ensemble forecasters. The Brier Score, reliability diagram and average width of 50% and 90% prediction intervals were respectively used to assess the three mentioned criteria.
The Brier score is widely applied in meteorology to assess the probability and ensemble forecasts. This score is presented as following equation:



In the above equation,  is the forecast probability of the event,  is the observational value equal to 1 or 0 depending on whether the event occurred or not, and N is the number of forecast-observation pairs. A minimum Brier score is equal to zero for a perfect forecaster.
Reliability diagrams are a graph of the observed frequency plotted against the forecast probability of the event. For perfect reliability, the forecast probability and the frequency of event is equal. Thus, the closer the reliability curve to the diameter is the higher the reliability.
Sharpness is a feature of the forecasts that refers to the concentration of the predictive distributions.
The more concentrated the predictive distributions are, the sharper the forecasts and thus the better the predictive model.
Results and Discussion  
The results showed that for 5 and 10 mm rainfall thresholds, UKMO’s ensembles were the least efficient, reliable and sharp. Thus, UKMO’s ensembles are not suitable for heavy rainfall forecasting in the study area. It was also observed that for 5 mm rainfall threshold, there was not a significant difference between the accuracy and reliability of NCEP and ECMWF ensembles but with increasing the level of threshold to 10 mm, NCEP’s ensembles had higher efficiency and were more reliable. In terms of sharpness, NCEP’s ensembles were also the most sharp, followed by ECMWF and UKMO.
Since the higher threshold is necessary for heavy rainfall prediction, so the 10 mm rainfall threshold was used in assessment of the predictions by the criteria. Analysis of the results based on the three mentioned criteria showed that NCEP’s ensembles had the best performance compared with the other predictions. Therefore, it is recommended to study the NCEP’s ensembles for prediction of heavy and flood producing rainfalls in mountainous watersheds like Kan Basin.


Main Subjects

آزادی، م.؛ واشانی، س. و حجام، س. (1391). پیش‏بینی احتمالاتی بارش با استفاده از پس‌پردازش برون‏داد یک سامانة همادی، مجلة فیزیک زمین و فضا، 38 : 203ـ 216.
بنی‏حبیب، م.ا. و عربی، ا. (1389). ارزیابی اثر عملیات آبخیزداری بر زمان پیش‏هشدار حوضة آبخیز گلابدره‏- دربند، علوم و تکنولوژی محیط زیست، 12: 77ـ81.
تقوی، ف.؛ نیستانی، ا. و قادر، س. (1392). ارزیابی پیش‏بینی‏های کوتاه‏مدت بارش مدل عددی WRF در منطقة ایران، مجلة فیزیک زمین و فضا، 39(2): 145ـ170.
دارند، د. و زندکریمی، س. (1394). واکاوی سنجش دقت زمانی- مکانی بارش پایگاه دادة مرکز پیش‏بینی میان‏مدت جوی اروپایی بر روی ایران‏زمین،  پژوهش‏های جغرافیای طبیعی، 47: 651ـ675.
ذوالجودی، م.؛ قاضی میرسعید، م. و سفیری، ز. (1392). بررسی صحت و دقت طرحواره‏های مختلف مدل WRF و ارزیابی پیش‏بینی بارش در ایران‏زمین، فصل‏نامة تحقیقات جغرافیایی، 28: 187ـ194.
فتحی، م.؛ آزادی، م.؛ ارکیان، ف.؛ کفاش‏زاده، ن. و امیرطاهری افشار، م. (1392). واسنجی پیش‏بینی احتمالی بارش به دو روش بافت‏نگار رتبه‏ای و لجستیک روی ایران، نشریة پژوهش‏های اقلیم‏شناسی، 12: 23ـ34.
فلاحی، م.ر.؛ وروانی، ه. و گلیان، س. (1390). پیش‏بینی بارش با استفاده از مدل رگرسیون درختی به منظور کنترل سیل، پنجمین کنفرانس سراسری آبخیزداری و مدیریت منابع آب و خاک کشور، کرمان، انجمن مهندسی آبیاری و آب ایران.
معیری، م. و انتظاری، م. (1387). سیلاب و مروری بر سیلاب‏های استان اصفهان، فصل‏نامة چشم‏انداز جغرافیایی، 3(6): 109ـ123.
Azadi, M.; Vashani, S. and Hajjam, S. (2012). Probabilistic precipitation forecast using post processing of output of ensemble forecasting system, Earth and Space Physics, 38(3): 203-216. (In Persian).
Banihabib, M.E.; Arabi, A. and Salha, A. (2015). A dynamic artificial neural network for assessment of land-use change impact on warning lead-time of flood, International Journal of Hydrology Science and Technology, 5(2): 163-178.
Banihabib, M. and Arabi, A. (2010). Evaluation of the effects of watershed management practices on the lead time, Environmental Science and Technology, 12(1): 77-81. (In Persian).
Crochemore, L.; Ramos, M.H. and Pappenberger, F. (2016). Bias correcting precipitation forecasts to improve the skill of seasonal streamflow forecasts, Hydrology and earth system science journal, 20(9), 3601-3618.
Darand, M. and Zande Karimi, S. (2015). Evaluation of Spatio-Temporal Accuracy of Precipitation of European Center for Medium-Range Weather Forecasts (ECMWF) over Iran. Physical Geography Research, 47: 651-675. (In Persian).
 Duan, M.; Ma, J. and Wang, P. (2012). Preliminary comparison of the CMA, ECMWF, NCEP, and JMA ensemble prediction systems, Acta Meteorologica Sinica, 26: 26-40.
Falahi, M.; Varvani, H. and Golian, S. (2011). Rainfall forecasting using the regression tree model inordet to flood management,  5th National Conference on Watershed Management and Soil and Water Resources Management, Kerman. (In Persian).
Fathi, M.; Azadi, M.; Arkian, F.; Kafashzadeh, N. and Amirtaheri Afshar, M. (2012). Precipitation Probabilistic Forecast Calibration by two approaches Rank histogram and Logistics, Climatology research, 7(3): 23-34. (In Persian).
Gneiting, T.; Balabdaoui, F. and Raftery, A.E. (2007). Probabilistic forecasts, calibration and sharpness, Journal of the Royal Statistical Society: Series B (Statistical Methodology), 69(2): 243-268.
Hamill, T.M. (2012). Verification of TIGGE multimodel and ECMWF reforecast-calibrated probabilistic precipitation forecasts over the contiguous United States, Monthly Weather Review, 140(7): 2232-2252.
Hsiao, L.F.; Yang, M.J.; Lee, C.S.; Kuo, H.C.; Shih, D.S.; Tsai, C.C. and Lin, G.F. (2013). Ensemble forecasting of typhoon rainfall and floods over a mountainous watershed in Taiwan, Journal of Hydrology, 506: 55-68.
Komma, J.; Reszler, C.; Blöschl, G. and Haiden, T. (2007). Ensemble prediction of floods–catchment non-linearity and forecast probabilities, Natural Hazards and Earth System Science, 7(4): 431-444.
Moayeri, M. and Entezari, M. (2007). A review of floods in Isfahan province, Geographical landscape, 3(6): 100-123. (In Persian).
Moustris, K.P.; Larissi, I.K.; Nastos, P.T. and Paliatsos, A.G. (2011). Precipitation forecast using artificial neural networks in specific regions of Greece, Water resources management, 25(8): 1979-1993.
Partal, T. and Kişi, Ö. (2007). Wavelet and neuro-fuzzy conjunction model for precipitation forecasting, Journal of Hydrology, 342(1): 199-212.
Sampson, W.; Suleman, N. and Gifty, A.Y. (2013). Proposed Seasonal Autoregressive Integrated Moving Average Model for Forecasting Rainfall Pattern in the Navrongo Municipality, Ghana, Journal of Environment and Earth Science, 3(12):80-85.
Sodoudi, S.; Noorian, A.; Geb, M. and Reimer, E. (2010). Daily precipitation forecast of ECMWF verified over Iran, Theoretical and applied climatology, 99(2): 39-51.
Taghavi, F.; Neyestani, A. and Ghader, S. (2013). Short range precipitation forecasts evaluation of WRF model over IRAN, J. of Earth and Space Physics, 39(2): 145-170. (In Persian).
Ye, J.; He, Y.; Pappenberger, F.; Cloke, H.L.; Manful, D.Y. and Li, Z. (2014). Evaluation of ECMWF medium‐range ensemble forecasts of precipitation for river basins, Quarterly Journal of the Royal Meteorological Society, 140(682): 1615-1628.
Young, R.M.B. (2010). Decomposition of the Brier score for weighted forecast‐verification pairs, Quarterly Journal of the Royal Meteorological Society, 136(650): 1364-1370.
Zoljoodi, M.; Ghazimirsaeid, S. and Seifari, Z. (2013). Evaluation of physics scheme of WRF model in precipitation forecasting in Iran. J of Geographical Research, 28(2): 187-194. (In Persian).