|
Abstract.
The study focuses on reviewing quantitative validation methods for their applicability to the stages of developing digital models of complex socio-economic and socio-technical systems. Conducting validation and verification checks improves the adequacy, accuracy, and robustness of the developed models, considering their specific features, such as complex dynamic structures, multi-agent systems, uncertainty, and variability. The stages of model validation, model data validation, and the final quality assessment of modeling were identified to define the scope of the research and highlight the specifics of the validation process and its distinction from the final quality assessment of the model in the context of decision-making. The results of the study provide a correspondence between the application of selected model evaluation tools and the stages of validating digital mathematical models based on key characteristics evaluated during validation, namely: correlation, variability, information, form, and delay.
Keywords:
decision support, validation and verification, modelling complex systems, the accuracy of simulation results.
DOI 10.14357/20718632250209
EDN EHVGDG
PP. 100-112.
References
1. Lambiotte R., Rosvall M., Scholtes I. From networks to optimal higher-order models of complex systems. Nature physics. 2019; (15): 313–320. doi: 10.1038/s41567-019-0459-y. 2. Bagdasaryan A. System approach to synthesis, modeling and control of complex dynamical systems. arXiv preprint arXiv:0902.3541. 2009. doi: 10.48550/arXiv.0902.3541. 3. Schwaninger M., Grösser S. System dynamics modeling: validation for quality assurance. System dynamics: Theory and applications. 2020; 119–138. doi: 10.1007/978-3-642-27737-5_540-3. 4. Gintciak, A. M et al. Cifrovoe modelirovanie sociotekhniche-skih i social'no-ekonomicheskih sistem [Digital modeling of socio-technical and socio-economic systems]. Saint Petersburg, POLITEKH-PRESS. 2023 (In Russ). doi: 10.18720/SPBPU/2/i23-253. 5. Salnikov A.V. et al. Digital Simulation Verification and Validation. BMSTU Journal of Mechanical Engineering. 2022; 9 (750): 100–115 (In Russ). doi: 10.18698/0536-1044-2022-9-100-115. 6. Sargent R. G. Verification and validation of simulation models. Proceedings of the 2010 winter simulation conference, IEEE. 2010; 166-183. doi: 10.1109/WSC.2010.5679166. 7. Law A. M. How to build valid and credible simulation models. Procedings of 2022 Winter Simulation Conference (WSC), IEEE. 2022; 1283-1295. 8. Sansana, J. et al. Recent trends on hybrid modeling for Industry 4.0. Computers & Chemical Engineering. 2021; (151): 107365. doi: 10.1016/j.compchemeng.2021.107365. 9. Bolsunovskaya M. V. et al. The opportunities of using a hybrid approach for modeling socio-economic and sociotechnical systems. Proceed-ings of Voronezh State University. Series: Systems Analysis and Information Technologies. 2022; (3): 73-86 (In Russ). doi:10.17308/sait/1995-5499/2022/3/73-86. 10. Sally C. B. et al. Hybrid simulation modelling in operational research: A state-of-the-art review. European Journal of Operational Research. 2019; 278(3): 721-737. doi:1016/j.ejor.2018.10.025. 11. Balci O. Verification, validation and accreditation. Proceedings of the 29th conference on Winter simulation, IEEE Computer Society, USA. 1997; 135-141. doi: 10.1109/WSC.1998.744897. 12. Botchkarev A. A new typology design of performance metrics to measure er-rors in machine learning regression algorithms. Interdisciplinary Journal of Information, Knowledge, and Management. 2019; 14: 045-076 (In Russ). doi:10.28945/4184. 13. Asmelash M. et al. Simulation modeling of a manufacturing process using Tecnomatix plant simulation software. Journal of Modern Manufacturing Systems and Technology. 2021; 5(1): 56-62. doi:10.15282/jmmst.v5i1.6083. 14. Beloglazov I., Krylov K. An interval-simplex approach to determine technological parameters from experimental data. Mathematics. 2022; 10(16): 2959 (In Russ). 15. Singh M. et al. Digital twin: Origin to future. Applied System Innovation. 2021; 4(2): 36. doi:10.3390/asi4020036. 16. Ilin V.A., Kiryushow N. P. Method of testing the training models for the adequacy. Software & Systems. 2021; 34(1): 061-066 (In Russ). 17. Hora J., Campos P. A review of performance criteria to validate simulation models. Expert Systems. 2015; 32(5): 578-595. doi:10.1111/exsy.12111. 18. Zheleznyakova A. L. Verification and validation technologies for gas dynamic simulations. Fiz.-Khim. Kinet. Gaz. Din. 2018; 19(2): 1-62 (In Russ). doi:.33257/PhChGD.19.2.687. 19. Nikolić B., Popović T. Hypothesis testing and statistical test selection: Fundamentals of statistics in clinical studies-part II.Medicinski pregled. 2024; 77(1-2): 49-54. doi: 10.2298/MPNS2402049N. 20. do Amaral J. V. S. et al. Metamodeling-based simulation optimization in manufacturing problems: a comparative study. The International Journal of Advanced Manufacturing Technology. 2022; 120( 7):5205-5224. doi: 10.1007/s00170-022-09072-9. 21. Asadi R., Zamaniannejatzadeh M., Eilbeigy M. Assessing the impact of human activities and climate change effects on groundwater quantity and quality: a case study of the Western Varamin Plain, Iran. Water. 2023; 15(18): 3196. doi: 10.3390/w15183196. 22. Velichko A. et al. Neural Network Entropy (NNetEn): Entropy-Based EEG Signal and Chaotic Time Series Classification, Python Package for NNetEn Calculation. Algorithms. 2023; 16(5): 255. doi: 10.3390/a16050255. 23. Gorbunov D. V., Gavrilenko T. V. Simulation of dynamic processes in the human body using differential equations with discontinuous right-hand side. Russian Journal of Cybernetics. 2023; 4(1): 15-20 (In Russ). doi:10.51790/2712-9942-2023-4-1-02. 24. Arkov V. Yu., Sharipova A. M., Kulikov G. G. Uncertainty estimation in machine learning. Bulletin of the South Ural State University. Ser. Computer Technologies, Automatic Control, Radio Electronics. 2023; 23(3): 48-58 (In Russ). doi: 10.14529/ctcr230305. 25. Zapechnikov S. V. Models and algorithms of privacy-preserving machine learning. IT Security. 2020; 27(1): 51-67 (In Russ). 26. McCoy L. G. et al. Believing in black boxes: machine learning for healthcare does not need explainability to be evidence-based. Journal of clinical epidemiology. 2022; 142: 252-257. doi:10.1016/j.jclinepi.2021.11.001. 27. Gusev A.V., et al. Machine learning based on laboratory data for disease prediction. FARMAKOEKONOMIKA. Modern Pharmacoeconomics and Pharmacoepidemiology. 2021; 14(4): 581-592 (In Russ). doi: 10.17749/2070-4909/farmakoekonomika.2021.115. 28. Isaev D. V. Strategy for finding an effective machine learning method based on the example of credit scoring. Economic problems and legal practice. 2020; 16(6): 132-138 (In Russ). 29. Gurtova K. S. Method for Information Protection of Digital Documents Using Invisible Digital Watermarks and Its Implementation. Sovremennye informacionnye tehnologii i IT-obrazovanie = Modern Information Technologies and IT-Education. 2022; 18(1): 152-166 (In Russ). 30. Maulud D., Abdulazeez A. M. A review on linear regression comprehensive in machine learning Journal of Applied Science and Technology Trends. 2020; 1 (4): 140-147. doi: 10.38094/jastt1457. 31. Bisaso K. R. et al. A comparative study of logistic regression based machine learning techniques for prediction of early virological suppression in antiretroviral initiating HIV patients. BMC medical informatics and decision making. 2018; (18): 1-10. doi: 10.1186/s12911-018-0659-x. 32. Lebedev I. S. Adaptive application of machine learning models on separate segments of a data sample in regression and classification problems. Informatsionno-upravliaiushchie sistemy [Information and Control Systems]. 2022; 3 (118): 20-30 (In Russ). doi:10.31799/1684-8853-2022-3-20-30. 33. Botchkarev A. A new typology design of performance metrics to measure errors in machine learning regression algorithms. Interdisciplinary Journal of Information, Knowledge, and Management. 2019; (14):045-076. doi: 10.28945/4184. 34. Hodson T. O. Root-mean-square error (RMSE) or mean absolute error (MAE): When to use them or not. Geoscientific Model Development. 2022; 15(14): 5481-5487. doi: 10.5194/gmd-7-1247-2014. 35. Handoyo S. et al. The varying threshold values of logistic regression and linear discriminant for classifying fraudulent firm. Mathematics and Statistics. 2021; 9(2): 135-143. 36. Grandini M., Bagli E., Visani G. Metrics for multi-class classification: an overview. arXiv preprint arXiv:2008.05756. 2020. doi: 10.48550/arXiv.2008.05756. 37. Kipkogei F. et al. Business success prediction in Rwanda: a comparison of tree-based models and logistic regression classifiers. SN Business & Economics. 2021; (1): 1-19. 38. Borovkov A., Bolsunovskaya M., Gintciak A., Rakova V., Efremova M., Akbarov R. COVID-19 Spread Modeling Considering Vaccination and Re-Morbidity. International Journal of Technology. 2022; 13 (7): 1463-1472. doi: 10.14716/ijtech.v13i7.6186. 39. Beketov S.M., Fedyaevskaya D.E., Burlutskaya Z.V., Gintciak A.M. Intelligent decision-making support system architecture for oil production enterprises. Automation and informatization of the fuel and energy complex. 2024: 11(616):1-26.
|
