With the improvement in reliability of modern aircraft systems, an increase in their complexity led to the fact that failures became non-standard and problematic to predict. As a result, IATA guided the development of Evidence-Based Training (EBT), which is an innovative approach to the training and assessment of pilots’ competencies based on the evidence obtained from their performance. Yet, operators planning to use this framework should develop line-oriented scenarios based on EBT methodology, and a grading system for a clear assessment. In this paper, a method for modelling a quantitative assessment of pilots’ performance is proposed. The method may be used as part of an EBT assessment programme due to its adaptability to different grading strategies according to quantifiable error criteria, the number of occurrences during the evaluation phase, and prior data on the distribution of pilots by their competency levels.