A Rough Set-Based Evaluation System for Information Technology Teachers in Middle Schools

With the rapid development and widespread application of information technology, cultivating students' information literacy and innovative ability has become one of the important goals of educational reform. Taking senior high school IT teachers as an example, the proportion of IT teachers among all teachers increased from 2.31% in 2020 to 2.92% by 2022. However, current teacher evaluation mainly serves professional-title appraisal, promotion and salary adjustment, and emphasizes appraisal as well as reward-and-punishment functions [1]. Meanwhile, traditional evaluation either relies on self-evaluation, peer evaluation, democratic appraisal and leader assessment, or introduces KPI-based performance appraisal; the conclusions are often biased or fail to achieve the intended goals, and can hardly reflect the professional competence and teaching characteristics of IT teachers [2]. Evaluating IT teachers can provide effective guidance for their professional development and improve the quality of IT education [3]. Some developed countries, such as the United States and Japan, have established relatively complete evaluation systems for primary and secondary school teachers. Tennessee proposed the Student Growth Portfolio Models (SGPM) to address the problems of the TVAAS value-added evaluation system [4]. In 2010, New York implemented “The New Teacher Project,” which emphasizes collecting evidence from different channels, time points and perspectives, and attaches equal importance to both the evaluation process and the results [5]. Between 2003 and 2005, Japan promoted its new teacher evaluation nationwide; by April 2011, 65 prefectures and ordinance-designated cities had implemented it for all teachers in all schools, combining goal-management-based self-evaluation with evaluations by principals and other evaluators [6]. Japan's PDCA-based management cycle consists of four repeated steps—plan, do, check and act—in which teachers set personal goals at the beginning of the year and adjust them during teaching practice, and school leaders provide guidance and feedback through interviews [7]. Many Chinese scholars have also studied teacher evaluation. For example, Yang proposed a PDCA quality framework for provincial college teacher evaluation policies, improving their quality through PMC index analysis and qualitative text analysis [8]. However, most of these studies focus on college and vocational teachers. Evaluation of primary and secondary school teachers in China has long concentrated on the four aspects of “virtue, diligence, performance and ability,” which are applicable to any profession and cannot sufficiently reflect the professional characteristics of teachers [9]. Few studies address evaluation methods specifically for IT teachers. IT differs from traditional subjects, and IT teachers also differ from teachers of traditional subjects in education and teaching; conventional methods can hardly evaluate IT teachers' teaching objectively. Rough set theory is relatively objective in describing and handling uncertainty: it can directly analyze and reason over data, discover implicit knowledge and reveal potential rules, making it suitable for exploring the currently under-developed evaluation of IT teachers. This paper selects four first-level indicators—practicing teacher ethics, learning to teach, learning to educate and learning to develop—with seven second-level indicators and twenty-three third-level indicators, so as to comprehensively evaluate the overall quality of IT teachers. Data were collected from 108 IT teachers in primary and secondary schools across the country by questionnaire, including basic information, the perceived importance of each indicator and the score of each indicator. The rough set theory is used to reduce the number of evaluation indicators, simplify the evaluation process and retain the most important indicators, avoiding distorted results caused by redundant indicators. Meanwhile, the discernibility-matrix-based heuristic attribute reduction algorithm and the BP neural network model are used for comparison to evaluate their performance and interpretability in teacher evaluation. The overall framework is shown in Fig. 1.

A. Information System and Indiscernibility Relation Rough set theory, proposed by the Polish scholar Pawlak in 1982, is an important data analysis method specially designed to handle uncertain and fuzzy data [10]. Definition 1 [11]: An information system is a quadruple S = (U, A, V, f), where U is a non-empty finite set of objects, A = C ∪ D is a non-empty finite set of attributes with C the conditional attributes and D the decision attributes, V is the union of the value domains of all attributes, and f : U × A → V is an information function assigning a value f(x, a) ∈ Vₐ to each object x ∈ U and attribute a ∈ A. Table I gives a simple example of an information system. Definition 2 [11]: For an information system S = (U, A, V, f) and any attribute set B ⊆ A, two objects x, y ∈ U are indiscernible with respect to B (denoted Ind(B)) if b(x) = b(y) for all b ∈ B. The indiscernibility relation Ind(B) induces a partition of U, denoted U/Ind(B). In Table I, for example, Ind({a}) partitions U into {0,3,4}, {1,7} and {2,5,6}. TABLE I EXAMPLE OF AN INFORMATION SYSTEM x ∈ U a b c d e 0 S R T T R 1 R S S S T 2 T R R S S 3 S S R T T 4 S R T R S 5 T T R S S 6 T S S S T 7 R S S R S For a given decision problem, rough set theory aims to find a minimal set of decision rules to classify data and make decisions among different attribute values. In this process, the indiscernibility relation is used to describe the uncertainty of data, and the positive, negative and boundary regions are used to describe the precision of decision rules; the core idea is to classify and decide based on rough approximations over different attributes. B. Rough Sets and Attribute Reduction To characterize fuzziness, rough set theory uses a pair of precise concepts, the lower approximation and the upper approximation, to represent each imprecise concept. Definition 3: For an information system S = (U, A, V, f) and any object set X ⊆ U, the lower and upper approximations of X with respect to an attribute set B ⊆ A are: R̲B(X) = { x ∈ U | [x]B ⊆ X } (1) R̄B(X) = { x ∈ U | [x]B ∩ X ≠ ∅ } (2) Definition 4: For any object set X ⊆ U, the positive region POSB(X) is the set of objects whose equivalence classes are fully contained in X, the negative region NEGB(X) is the set of objects whose equivalence classes are definitely not contained in X, and the boundary region BNDB(X) is the set of objects whose equivalence classes are partially contained in X: POSB(X) = R̲B(X), NEGB(X) = U − R̄B(X), BNDB(X) = R̄B(X) − R̲B(X) (3) Fig. 2 illustrates the approximation relation of the rough set. Fig. 2. Approximation of a rough set. Definition 5: For an information system S = (U, A, V, f) and any object set X ⊆ U, the dependency degree of X with respect to an attribute set B ⊆ A is: γB(X) = |R̲B(X)| / |U| (4) The dependency degree determines the extent to which X depends on the attribute set B; classical rough set theory uses significance as the criterion to realize attribute reduction [12]. Definition 6: For an information system S = (U, A, V, f), any object set X ⊆ U and attribute set B ⊆ A, the significance of an attribute a with respect to X and B is: SigB(a) = γB(X) − γB−{a}(X) (5) The significance reflects the importance of an attribute; when it is zero, the attribute can be reduced. Through reduction, the efficiency and accuracy of evaluation can be improved, the key indicators highlighted, and teachers' performance assessed more quickly, which in turn better guides teaching and teacher development.

A. Construction of Evaluation Indicators To establish indicators more suitable for IT teachers, the evaluation indicators in existing systems were statistically analyzed. Researchers have proposed, through AHP or fuzzy evaluation, a series of key indicators such as teaching quality, basic quality, teacher ethics, teaching objectives, teaching content, teaching methods, teaching process and teaching effect, which provide an important reference for building the evaluation system [13]–[17]. In 2017, the Chinese Ministry of Education issued the “Professional Accreditation Standards for Secondary Education (Level 3),” which requires secondary school teachers to practice teacher ethics, learn to teach, learn to educate and learn to develop. Combining the existing studies with these standards, this paper adopts the four dimensions as first-level indicators aᵢ, seven second-level indicators aᵢⱼ, and twenty-three third-level indicators aᵢⱼₖ, as shown in Table II. TABLE II EVALUATION INDEX SYSTEM OF PRIMARY AND SECONDARY SCHOOL IT TEACHERS Practicing teacher ethics a₁ Ethics norms a₁₁ Political conviction a₁₁₁; Fostering virtue a₁₁₂; Teaching according to law a₁₁₃ Educational affection a₁₂ Professional identity a₁₂₁; Professional quality a₁₂₂; Caring for students a₁₂₃ Learning to teach a₂ Professional competence a₂₁ Subject competence a₂₁₁; Professional knowledge a₂₁₂; Application ability a₂₁₃; Club ability a₂₁₄; Competition guidance a₂₁₅; Information literacy a₂₁₆ Teaching competence a₂₂ Teaching ability a₂₂₁; Teaching principles a₂₂₂; Teaching-research ability a₂₂₃ Learning to educate a₃ Education ability a₃₁ Moral-education concept a₃₁₁; Subject-based education a₃₁₂; Cultural education a₃₁₃ Learning to develop a₄ Self-reflection a₄₁ Reflective learning a₄₁₁; Autonomous development a₄₁₂ Communication & cooperation a₄₂ Communication a₄₂₁; Teamwork a₄₂₂; Sharing & exchange a₄₂₃ This index system aims to comprehensively reflect teachers' ethics, teaching, education and personal development, so as to better understand teachers' teaching level and professional quality, and to provide an important reference for teacher training and development. B. Evaluation Model and Algorithm Based on Rough Set The model includes four steps: constructing the initial information system from the evaluation indicators, reducing the information system to remove redundant attributes, analyzing and computing the attribute weights, and performing the comprehensive evaluation of teachers. 1) Initial information system: All third-level indicators in Table II are taken as the attribute set and the teachers to be evaluated as the object set. The value of each third-level indicator is rated on a 1–10 scale (1 = worst, 10 = best), forming an information system S. 2) Attribute reduction: Directly using the total score of attribute values as the criterion may make the evaluation too complex, because the number of attributes is large and some attributes may be redundant and unable to distinguish teachers effectively. Reduction is therefore performed before evaluation. An attribute a ∈ A is redundant if Ind(A) = Ind(A − {a}); otherwise it is independent. A set B ⊆ A is a reduct of A if B is independent and Ind(B) = Ind(A). In the illustrative system, since U/Ind(A − {a₁₁₁}) = U/Ind(A), the attribute a₁₁₁ (political conviction) is redundant and can be reduced. Repeating this procedure yields the final reduction. 3) Attribute weights and comprehensive evaluation: Since different attributes are not equally important, an entropy-based importance measure is introduced to determine the weight of each attribute, and the total score is computed by a weighted formula. For the partition induced by a knowledge P, the information entropy is H(P) = −Σ p(Xᵢ) log₂ p(Xᵢ); the importance of an attribute a in A is Sigₐ(a) = H(A) − H(A − {a}); and the weight of a is wₐ = Sigₐ(a) / Σ Sigₐ(b), b ∈ A. The comprehensive score of an object is the weighted sum of the secondary- and tertiary-indicator weights multiplied by the corresponding attribute values. Applying this procedure to the illustrative system of ten teachers, the evaluation results can distinguish all teachers: u₆ and u₉ rank first, and u₈ ranks last. 4) Algorithm description: The evaluation process consists of three steps. First, reduction loops over the columns of the dataset, records the name and ordinal of each column, temporarily removes the current column, and compares the indiscernibility of the original dataset with that of the reduced dataset to judge whether the column should be reduced. Second, the objects with identical attribute values are screened out and classified. Third, the weights of the reduced tertiary and secondary indicators are computed, and the final scores are obtained by the weighted comprehensive-evaluation formula.

A. Data Collection and Processing The evaluation data were collected by designing and distributing a questionnaire on the Wenjuanxing online survey platform. The questionnaire covered IT teachers in primary and secondary schools across the country and contained their teaching information and indicator ratings, including basic information (Q1–Q5), the perceived importance of each professional skill (Q6) and a self-evaluation of each professional skill on a 1–10 scale (Q7). Invalid responses (e.g., too-short completion time or unreasonable scores) were removed, and 108 valid questionnaires were finally obtained. The data of Q7 were organized to build the information system, while the data of Q1–Q6 were analyzed together with the evaluation results. B. Case Analysis According to the data of Q7, the collected data were organized into an information system according to Table II. After reduction, nine key indicators were obtained: a₁₁₃ (teaching according to law), a₁₂₁ (professional identity), a₁₂₃ (caring for students), a₂₁₃ (application ability), a₂₁₆ (information literacy), a₂₂₃ (teaching-research ability), a₃₁₁ (moral-education concept), a₄₁₂ (autonomous development) and a₄₂₂ (teamwork). Table III compares these with the nine most important indicators rated in Q6. TABLE III COMPARISON OF IMPORTANCE RATINGS AND REDUCTION RESULTS Source Indicators (codes) Indicators (meanings) Reduction a₁₁₃, a₁₂₁, a₁₂₃, a₂₁₃, a₂₁₆, a₂₂₃, a₃₁₁, a₄₁₂, a₄₂₂ Teaching according to law; professional identity; caring for students; application ability; information literacy; teaching-research ability; moral-education concept; autonomous development; teamwork Importance a₁₁₃, a₁₂₁, a₁₂₃, a₂₁₂, a₂₁₃, a₂₂₃, a₂₁₄, a₄₂₁, a₄₂₂ Teaching according to law; professional identity; caring for students; professional knowledge; application ability; teaching-research ability; club ability; communication; teamwork Two thirds of the indicators are identical in Table III, indicating that the indicators generally considered important by teachers are also the key indicators for evaluating teachers. Meanwhile, teachers may pay insufficient attention to information literacy, moral-education concept and autonomous development; schools and education departments can strengthen these aspects in teacher training. Using the entropy weights of the key indicators and the weighted comprehensive-evaluation formula, the final scores of the 108 teachers were obtained. The lowest score is 12.89 (u₁₁), the highest is 32.22 (u₂, u₅, u₇ and u₈), and the mean is 25.72. Fig. 3 shows the score distribution: about 70% of the teachers score in the (25–30] interval and about 20% in the (20–25] interval, i.e., most teachers are above the average and the scores are approximately normally distributed. Compared with the normal distribution computed from the mean and standard deviation, there are more teachers in the (20–25] interval and fewer in the (30–35] interval, indicating that excellent teachers scoring above 30 are relatively scarce and most teachers are at the middle level. Fig. 3. Score distribution of the 108 IT teachers. To analyze regional differences, the basic information of high-scoring teachers (score above 27) was examined. Most high-scoring teachers come from junior/senior high schools in tier-3 or higher cities; there are few excellent teachers in rural and township areas, and only one primary-school teacher. This may be because primary-school students are younger and their teaching objectives and methods differ greatly from those of junior/senior high schools, so the evaluation indicators used are less fair for primary-school teachers; the proposed evaluation system is therefore more suitable for the middle-school stage. In addition, the imbalance of education between urban and rural areas is significant, and education departments need to coordinate. C. Comparison with Other Methods To verify the advantages of applying rough set theory to teacher evaluation, we compared it with discernibility-matrix-based heuristic attribute reduction and the BP neural network. Common heuristic reduction algorithms include those based on information entropy, discernibility matrix and positive region. The discernibility-matrix-based algorithm is popular because it is concise and easy to understand; it records the occurrence frequency of each attribute while generating the discernibility matrix, and attributes with higher frequency are considered more important and selected with priority [18]. The BP (back-propagation) neural network is a multi-layer feedforward network proposed by Rumelhart and McClelland in 1986 and trained by the error back-propagation algorithm [20]; it is one of the most widely used neural network models. Since education departments need to evaluate teachers over an entire region, the efficiency of the methods is also an important reference. Fig. 4 compares the time efficiency of the three methods. The evaluation time of the discernibility matrix increases exponentially with the number of teachers and is the largest in large-scale tests, whereas the time of rough set grows slowly and is small. For small samples, the training time of BP is short; for large-scale training, BP is slightly slower than rough set. Overall, rough set and BP take comparable and short times, indicating that both can be effectively applied to teacher evaluation. Fig. 4. Time efficiency of rough set, discernibility matrix and BP neural network. The optimality of the reduction and the accuracy of the results are also important criteria. The discernibility-matrix reduction retains 19 indicators, about twice the 9 indicators retained by rough set, which shows that rough set can more precisely identify the important indicators and remove redundant attributes. As for the ranking results, the difference between rough set and the discernibility matrix is not obvious: the top six teachers are identical and the last five are the same with only slight changes. In contrast, although seven teachers are ranked in the top 10% by both rough set and BP, the two rankings differ significantly. Overall, rough set and the discernibility matrix achieve similar evaluation results, but rough set has higher efficiency and more precise reduction and is more suitable for teacher evaluation. Rough set computes the weights of the indicators and obtains the evaluation results through a weighted formula, so the principle and process of the evaluation can be concretely explained. A major drawback of BP is its lack of interpretability: although it can provide relatively accurate results, it cannot clearly explain the evaluation process, which limits its application in decision-making that requires transparency and interpretability, such as teacher evaluation. Therefore, considering time, precision and interpretability comprehensively, the rough set method is more suitable for teacher evaluation.

Under the background of educational informatization, the evaluation of IT teachers affects both teachers' teaching level and students' learning and growth. Focusing on rough set theory, this paper explores its applicability to the evaluation of primary and secondary school IT teachers, systematically constructs an evaluation index system, builds an evaluation model and algorithm, demonstrates the evaluation process with examples, and conducts an empirical study and a comparative analysis. Future work includes extending the rough set-based evaluation method to other subjects or other types of teachers; further improving the evaluation index system to make it more comprehensive and scientific; enlarging the sample size with more data and real cases to further validate and optimize the model; and combining new technologies such as artificial intelligence and big data to make the evaluation process more intelligent, accurate, efficient and convenient.