key: cord-0047134-ohqnj4pp
authors: Mani, A.
title: Towards Student Centric Rough Concept Inventories
date: 2020-06-10
journal: Rough Sets
DOI: 10.1007/978-3-030-52705-1_19
sha: 8a5e9c58ad6f094a002bab9637d562b94d58f8c9
doc_id: 47134
cord_uid: ohqnj4pp

In the context of education research, a concept inventory is an instrument (that consists of a number of multiple-choice questions) designed to test the understanding of concepts (and possibly the reasons for failure to understand) by learners. Subject to a few caveats they are known to be somewhat effective in non student centric learning environments. In this research the issue of adapting the subject/concept-specific instruments to make room for diverse response patterns (including vague ones) is explored in some detail by the present author. It is shown that higher granular operator spaces (or partial algebras) with additional temporal and key operators are well suited for handling them. An improved version of concept inventory called rough concept inventory that can handle vague subjective responses is also proposed in this research.

A test that focuses on evaluating a student's competence in a specific skill is a criterion-referenced test. Usually a person's test scores are intended to suggest a general statement about their capabilities and behavior. Concept inventories (CIs) are criterion-referenced test designed to test a student's functional understanding of concepts. However they are mostly used by education researchers to assess the effectiveness of pedagogical methods.

The standard way to construct concept inventories is as follows:

• select a number of key concepts in a subject or topic;

• formulate multiple choice questions (MCQs) that aim to test key aspects of applications of the chosen concepts; • each question is required to have at least one correct answer and a number of incorrect answers (distractors) based on student misconceptions or alternative conceptions. Individual steps may require plenty of additional work as can be seen in [1] also because the stakeholders views may not be clear in the first place.

They have been used in studying the effectiveness of a number of pedagogical methods that target concept maturity. These involve pre and post tests that concern assessing students or participants before and after the implementation of the pedagogical procedure (see for example [2] ). Like most other practices in education research, a number of concerns have been raised about the methodology and its supposed effectiveness in measuring all that it claims to measure (see Section 3 for some details). It is accepted that descriptive explanations by students for their choice of answers can significantly boost the quality of assessment offered by the concept inventory. Usually these are evaluated by instructors or researchers and the scores from MCQs are accordingly modified.

Apart from the sheer volume of responses generated in specific studies involving the use of concept inventories, in some learning environments it is often the case that instructors themselves may not be sufficiently competent in handling concepts [1] . Therefore automated methods are relevant for evaluation of CIs and for extending their use to provide relevant feedback to learners and instructors.

In student-centered learning students are put at the center of the learning process, and are encouraged to learn through active methods. Arguably, students become more responsible for their learning in such environments. In traditional teacher-centered classrooms, teachers have the role of instructors and are intended to function as the only source of knowledge. By contrast, teachers are typically intended to perform the role of facilitators in student-centered learning contexts. A number of best practices for teaching in such contexts [3] have evolved over time. These methodologies are naturally at odds with concept inventories.

Granular operator spaces and variants [4] [5] [6] [7] are abstract frameworks for extending granularity and parthood in the context of general rough sets, and are also variants of rough Y-systems studied by the present author [8] . It has been recently shown by her that all types of granular operator spaces and variants can be transformed into partial algebras that satisfy additional conditions. General rough sets are used in knowledge representation in a number of contexts [5, 6, [9] [10] [11] [12] [13] [14] [15] [16] . But the problem of knowledge representation in the present context is more complicated. Because the concepts associated with concept inventories have some ontology associated, it is not a good idea to directly reduce them to information table format. Reduction of additional information from descriptive responses may be reasonable in supervised perspective, but it is far more easier to reduce them to higher granular operator perspectives or abstract operator perspectives. Learning contexts (especially constructive learning) adopt perspectives that are most compatible with axiomatic granular perspective because of the hierarchies imposed on any body of knowledge. For example, it is usually imposed that multiplication of natural numbers should be taught only after addition has been taught (and therefore this corresponds to an instance of context dependency). It can be shown that in most constructive teaching contexts, teachers stick to an approximately fixed hierarchy of concept dependence and that student centric activities are pursued (if at all) within a relatively looser variant of the same.

In this research, aspects of concept inventories are explained, a rough variant is introduced and it is shown that higher granular operator spaces with additional temporal operators are optimal for representing related knowledge. The paper is organized as follows: in the next section necessary background and recent results on partial groupoids are mentioned, essential aspects of concept inventories are explained and variants are proposed in the third section, higher granular operator spaces are explained and enhanced versions introduced in the next section, rough concept inventories are proposed in the fifth, and an example is explored in the sixth section.

An information table I, is a tuple of the form

with S, A and V a being sets of objects, attributes and values respectively. Information tables generate various types of relational or relator spaces which in turn relate to approximations of different types and form a substantial part of the problems encountered in general rough sets.

The rough domain corresponds to rough objects of specific type, while the classical and hybrid one correspond to all and mixed types of objects respectively [8] . Boolean algebra with approximation operators forms a classical rough semantics [9] . This fails to deal with the behavior of rough objects alone. The scenario remains true even when R in the approximation space is replaced by arbitrary binary relations. In general, ℘(S) can be replaced by a set with a parthood relation and some approximation operators defined on it as in [8] . The associated semantic domain is the classical one for general Rough sets. The domain of discourse associated with roughly equivalent sets is a rough semantic domain. Hybrid domains can also be generated and have been used in the literature [6] .

The problem of reducing confusion among concepts from one semantic domain in another is referred to as the contamination problem. Use of numeric functions like rough membership and inclusion maps based on cardinalities of subsets are also sources of contamination. The rationale can also be seen in the definition of operations like in pre-rough algebra (for example) that seek to define interaction between rough objects but use classical concepts that do not have any interpretation in the rough semantic domain. Details can be found in [17] . In machine learning practice, whenever inherent shortcomings in algorithmic framework being used are the source of noise then the frameworks may be said to be contaminated.

Key concepts used in the context of general rough sets (and also high granular operator spaces [4, 6] ) are mentioned next.

• A crisp object is one that has been designated as crisp or is an approximation of some other object.

• A vague object is one whose approximations do not coincide with itself or that which has been designated as a vague object. • An object that is explicitly available for computations in a rough semantic domain (in a contamination avoidance perspective) is a discernible object. • Many definitions and representations are associated with the idea of rough objects. From the representation point of view these are usually functions of definite or crisp or approximations of objects. Objects that are invariant relative to an approximation process are said to be definite objects. In rough perspectives of knowledge [5, 9] , algebraic combinations of definite objects (in some sense) or granules are assumed to correspond to crisp concepts, and knowledge to specific collections of crisp concepts. It should be mentioned that non algebraic definitions are excluded in the present author's axiomatic approach [4, 6, 8] .

A partial algebra (see [18] )P is a tuple of the form

with P being a set, f i 's being partial function symbols of arity r i . The interpretation of f i on the set P should be denoted by f P i , but the superscript will be dropped in this paper as the application contexts are simple enough. If predicate symbols enter into the signature, then P is termed a partial algebraic system.

In this paragraph the terms are not interpreted. For two terms s, t, s ω = t shall mean, if both sides are defined then the two terms are equal (the quantification is implicit). ω = is the same as the existence equality (also written as e =) in the present paper. s ω * = t shall mean if either side is defined, then the other is and the two sides are equal (the quantification is implicit). Note that the latter equality can be defined in terms of the former as

In relational approach to general rough sets various granular, pointwise or abstract approximations are defined, and rough objects of various kinds are studied [6, 8, 12, [19] [20] [21] . These approximations may be derived from information tables or may be abstracted from data relating to human (or machine) reasoning. A general approximation space is a pair of the form S = S, R with S being a set and R being a binary relation (S and S will be used interchangeably throughout this paper ). Approximations of subsets of S may be generated from these and studied at different levels of abstraction in theoretical approaches to rough sets. In relational approaches to rough sets a number of types of knowledge are representable starting from those by classical rough sets [9] to general rough sets as in [4, 11, 12, 22] . However those based on relational approaches are not always applicable in evaluation and representation of academic data.

Mereology, the study of parts and wholes, has been studied from philosophical, logical, algebraic, topological and applied perspectives. In the literature on mereology [11, 23, 24] , it is argued that most ideas of binary part of relations in human reasoning are at least antisymmetric and reflexive. A major reason for not requiring transitivity of the parthood relation is because of the functional reasons that lead to its failure (see [23] ), and to accommodate apparent parthood [24] . The study of mereology in the context of rough sets can be approached in at least two essentially different ways. In the approach aimed at reducing contamination by the present author [4] [5] [6] 8] , the primary motivation is to avoid intrusion into the data by way of additional assumptions about the data and to capture rough reasoning at the level. In numeric function based approaches [25] , the strategy is to base definitions of parthood on the degree of rough inclusion or membership -this differs substantially from the former approach. Rough Y-systems and granular operator spaces, introduced and studied extensively by the present author [4] [5] [6] 8, 24] , are essentially higher order abstract approaches in general rough sets in which the primitives are ideas of approximations, parthood, and granularity.

Under certain conditions, partial or total groupoid operations can correspond to binary relations on a set. This subsection is repeated from a forthcoming paper for Sect. 4.

In a general approximation space S = S, R consider the following conditions:

If S satisfies up-dir, then it shall said to be a up-directed approximation space. If it satisfies the last two then it shall said to be a parthood space and a up-directed parthood space when it satisfies all three.

The condition up-dir is equivalent to the set U R (a, b) = {x : Rax & Rbx} being nonempty for every a, b ∈ S and is also referred to as directed in the literature. It is avoided because it may cause confusion.

If R is a binary relation on S, then a type-1 partial groupoid operation (1PGO) determined by R is defined as follows:

If R is up-directed, then the operation is total. In this case, the collection of groupoids satisfying the condition will be denoted by B(S) and an arbitrary element of it will be denoted by B(S). The term 'a • b' will be written as 'ab' for convenience.

The following results have been proved for relational systems in [26, 27] .

For a groupoid A, the following are equivalent

Morphisms between up-directed approximation spaces are preserved by corresponding groupoids in a nice way. This is an additional reason for investigating the algebraic perspective.

Concept inventories are expected to fulfill a number of requirements for assessment of the effect on learning. In particular, they are expected to • be designed for measuring understanding as opposed to declarative knowledge, • measure what they claim to measure (that is they should be valid), • be standardized for use over diverse educational institutions at the level, and

• be longitudinal (that is they should be amenable for reuse at different points of time for evaluation with relatively less interference).

The well known force concept inventory (FCI) [28] and mechanics diagnostic test (MDT) are among the earliest concept inventories developed. They are used in the context of assessment of teaching procedures in physics and have played a significant role in influencing the development of concept inventories in other subjects. A number of concept inventories for specific subjects or topics in mathematics such as the calculus concept inventory [29] and function CI [30] are known. Some have claimed that FCI is a test of mastery of certain contexts and content relating to force and not a test of the force concept itself [31] . Others have tried to show that conceptual understanding is actually addressed in FCI [32] . Though people differ on their opinions about the thing that is actually being measured by FCI [33] , FCI is known to measure something useful, and has been widely used.

The literature on concept inventories is large, but ontologies are not commonly used in their analysis or evaluation (though in principle much seems to be possible). Computer-based assessment software do use conceptual models such as labeled conceptual graphs and formal concept analysis. But related exercises require careful formalism to avoid misunderstanding and automatic evaluation is known to miss conceptual problems [34, 35] . In the present author's view this is also because they try to avoid (rather than confront) vagueness inherent to the available knowledge.

Most authors agree (see [2] ) that notions of misconceptions or alternative conceptions have a important role in determining measurements of conceptual understanding. In the present author's view alternative conceptions and apparent or real misconceptions have a dialectical relationship with conceptual understanding as a whole. This is corroborated by studies that show that students may or may not consistently apply their understanding of concepts (that they seem to have understood). The idea of consistency is a very relative notion that is typically associated with rigid goals in the teaching perspective. It is also very difficult to explore misconceptions with formal concept analysis and concept maps because of simply misreading the intended interpretation of students. Identification of student misconceptions depends on choice of domains and related specification of distractors that can actually relate to alternative conceptions. Studies [36] suggest that often they are not properly included in concept inventories.

The biggest deficiency of concept inventories that use questions in the MCQ format alone is its incompatibility with student centric approaches to evaluation of understanding. The MCQ in CIs (unlike those used in ordinary MCQs) are formulated after estimating possibilities on range and modalities of student responses. Further they are evaluated to ensure test reliability and validity. It is known that evaluators may not know the exact reasons for students choice of an incorrect answer and that understanding may not correlate with correct response [2, 33] . A number of proposals have been put forward to address these deficiencies. The most popular have been ones that require students to add explanations for their choice (see for example [1] . MCQ scores obtained by students are adjusted based on the teacher's evaluations of the explanations. This immediately suggests the problem of improving the methodology towards minimizing biased evaluation by teachers or evaluators.

Competence levels (relating to a concept inventory) are typically constructed through abstract specification. For example in the function CI proposed in [30] , six levels of understanding are identified. This can be enhanced to the following:

1. the ability to distinguish between functions and equations; 2. the ability to recognize and relate different representations of functions and use them interchangeably; 3. the ability to classify relationships as functions or not functions; 4. the ability to have a working familiarity with properties of functions such as 1 − 1, many-one, increasing, decreasing, linearity; 5. the ability to have a working familiarity with properties of sets of functions such as composition and inverses; 6. the ability to use functions in context, modeling and interpreting; 7. the ability to use functions to preserve relationships across models; 8. the ability to engage with co-variational reasoning; 9. the ability to engage with algebraic reasoning.

Needless to say, the problem of representing data of this form is well beyond the capabilities of relational approach to rough sets. From an general rough set perspective, representing such ideas within the context of collection of relevant concepts subject to the granularities of constructivist ideas of knowledge and human learning are of much interest. While it is not hard to see that a number of abstract granular approximations are involved, it is necessary to identify and classify granules, represent the fine structure of concepts and the process of transformation of concepts by the pedagogical practice.

Not all concepts are constructed equal. Some are more relevant target concepts and can be regarded as key concepts. As can be seen in the list of abstract conceptual states (that may be read as key concepts) relating to the function concept, key concepts need not be simple from a representation point of view.

Granular operator spaces and variants [4] [5] [6] [7] are abstract frameworks for extending granularity and parthood in the context of general rough sets, and are also variants of rough Y-systems studied by the present author [8] . They are well suited for handling approximations of unclear aetiology (relative to construction from information systems) but subject to certain minimal conditions on granularity. In [37] , it is shown by the present author that all types of granular operator spaces and variants can be transformed into partial algebras that satisfy additional conditions. Part of this is repeated for convenience in this section. It is also nontrivial because all covering approximation spaces cannot be transformed in the same way.

A High General Granular Operator Space (GGS) S shall be a partial algebraic system of the form S = S, γ, l, u, P, ≤, ∨, ∧, ⊥, with S being a set, γ being a unary predicate that determines G (by the condition γx if and only if x ∈ G) an admissible granulation(defined below) for S and l, u being operators : S −→ S satisfying the following (S is replaced with S if clear from the context. ∨ and ∧ are idempotent partial operations and P is a binary predicate. Further γx will be replaced by x ∈ G for convenience.):

The conditions defining admissible granulations mean that every approximation is somehow representable by granules in a algebraic way, that every granule coincides with its lower approximation (granules are lower definite), and that all pairs of distinct granules are part of definite objects (those that coincide with their own lower and upper approximations). Special cases of the above are defined next.

then the GGS is said to be a high granular operator space GS. • A higher granular operator space (HGOS) S is a GS in which the lattice operations are total. • In a higher granular operator space, if the lattice operations are set theoretic union and intersection, then the HGOS will be said to be a set HGOS.

In the context of Definition 5, the binary predicates P can be replaced by partial two-place operations 1PGO and γ is replaceable by a total unary operation h defined as follows:

Consequently S + = S, h, l, u, , ∨, ∧, ⊥, is a partial algebra that is semantically (and also in a category-theoretic sense) equivalent to the original GGS S.

Proof. Because of the restriction UL3 on ⊥ and the redundancy of ≤ (because of G5), the result follows.

The partial algebra formed in the above theorem will be referred to a high granular operator partial algebra (GGSo). Problem 1. All covering approximation spaces considered in the rough set literature actually assume partial Boolean or partial lattice theoretical operations. Some authors (especially in modal logic perspectives) [12, 20, 38] presume that all Boolean operations are admissible -this view can be argued against. A natural question is Are the modal logic semantics themselves only a possible interpretation of the actuality? All this suggests the problem of finding minimal operations involved in the context. Because all covering approximation spaces do not use granular approximations in the sense mentioned above, it follows that they do not form GGSo always.

A rough concept inventory is intended to be a concept inventory that can effectively handle vagueness inherent in relatively student centric perspectives through methodological improvements, and representations of approximate evaluations. Relatively, because the central process of concept inventories is not compatible beyond a point with student-centric approaches.

While it would be best if the methodology and the final analysis are all integrated together, it may be useful in practice to separate the two. The methodological aspect would be as follows:

1. select a number of key concepts in a subject or topic; 2. situate them relative to the concepts and granular concepts described in the model in the subsection below (or alternatively situate the concepts relative to a concept map in terms was constructed from and is a part of, and basic well-understood concepts); 3. formulate multiple choice questions that aim to test key aspects of applications of the chosen concepts; 4. each question is required to have at least one correct answer and a number of incorrect answers (distractors) based on student misconceptions or alternative conceptions; 5. require explanation from students for their choice; 6. evaluate explanations relative to model in terms of concept approximations (or alternatively evaluate explanations relative to concepts that are definitely understood and those that are possibly understood).

In the latter case, the methodology would follow the alternatives suggested in the second and the sixth step. The end result in this approach would also include a temporal extension of GGSo described below.

A temporal extension of GGSo is introduced next to model rough concept inventories from a minimalist perspective. Essentially this is an extension of a GGSo with two unary temporal operations for specifying before and after states under few constraints and an additional operation for indicating key concepts. If desired a GGS can also be extended in the same way for simplicity. This is intended to be used for the purpose of constructing a single model for the entire procedure of administering the inventory first, applying the pedagogical practice and then applying the concept inventory in the final stage. formed by adjoining three unary operations k, A and B to the GGSo S + will be said to be a basic temporal high granular operator partial algebra (TGGSo) provided the following properties are satisfied:

Compared with common usage of these temporal operators (see [39] ) this may appear to be very minimalist. But the application context dictates that it would be a good idea to avoid imposing any connections with l, u, h, and . Additional approximation operators may also be needed in practice. An element x that satisfies kx = x will be said to be a key concept.

The most direct interpretation of the different components of the model (or its equivalent formed from a GGS instead) from a practical perspective are as follows:

1. S can be read as the collection of relevant concepts tagged by real or dummy student/instructor names (including those that are not apparently part of the concept inventory); 2. h can be read as a partial function that helps in identifying granules in S (the relatively definite concepts from which other definite concepts are made up of in a simple way). The simple ways must be related to the definitions of other operations; 3.

corresponds to parthood and a perspective of aggregation. 4. a ∨ b = b can be read as b was constructed from a. 5. a ∨ b can be read as that which is constructed out of an aggregation of a and b.

The easiest simple way can be by way of aggregation. When multiple approximations are used then the number of ways can be increased. Note that the model is not tied down to a single idea of concept evolution and has scope for handling the structures generated by the entire sample because of the very definition of S.

A real application requires datasets that include explicit student responses, and because of ethics concerns it is necessary to form synthetic versions of the same. Due to limited time, aspects of the proposed model are considered in relation to secondary information derived from a typical concept inventory (the temporal aspect is not used in the study).

In [1] , the development and analysis of a concept inventory on rotational kinematics is considered. The questions and answers can be found in the appendix of the paper. The authors restrict themselves to questions probing angular velocity of a rigid body, trajectory of an arbitrary particle on a rotating rigid body, angular and linear velocities of particles on a rigid body, angular acceleration of a rigid body, validity of the equation τ = Iα, dependence of angular velocity on the origin, relation between angular acceleration and tangential acceleration, relation between angular acceleration and centripetal acceleration, and finally components of linear acceleration. Thus the concept inventory is focused on a very specific set of key concepts (other key concepts may be latent). Apparently the questions have been optimized for testing the conception of a specific set of students through a number of steps. The authors mention that responses to questions were verified in the light of the explanations offered (if any).

The authors have this to say on the concept maps associated with angular velocity and angular acceleration:

Consider the operational definition of the angular velocity of a rigid body as an illustrative example. Identifying the angle Δθ in ω = Δθ Δt would require the selection of an arbitrary particle on the body, not necessarily the center of mass, drawing a perpendicular line from the particle to the axis, noting the angle traced by this line as the rigid body rotates, etc. As another example, consider the case of α (angular acceleration), which may be nonzero even if the instantaneous angular velocity is zero. Operationally this would entail, among other things, identifying the angular velocities at two different instances and subtracting them. We noted similar intricacies that helped us probe pitfalls in student thinking.

This suggests that the authors have specific ideas of how the concepts being tested must evolve. While, this can be read as an idea of standard suggested conception, it is necessary to look into possible alternative conceptions that may be in the explanations offered by the students. In the methodology adopted, these aspects are to be discovered through pilot studies.

A specific question in the inventory is the following: A ceiling fan is rotating around a fixed axis. Consider the following statements for the particles not on the axis at a given instant. Statement I: Every particle on the fan has the same linear velocity. Statement II: Every particle on the fan has the same angular velocity. The correct statement(s) is (are) The correct answer is the second option. Explanations offered by a student for this choice and others should be approximated by the evaluator. In the approach of [1] , they are simply used for verifying the correctness of the response. Statistical analysis is also used for evaluating the conceptual maturity of the set of participants studied.

In the approach suggested in the present paper, the entire dataset can be recoded as a TGGSo (with possibly multiple lower and upper approximation operators) and studied with minimum intrusion. Example granules can be The angular velocity is computed by ω = Δθ Δt , and angular velocities are computed relative to an axis. Erroneous responses can also be seen as part of the data because objects are all labeled by student/instructor or evaluator names. Last but not in the least explanations can be read as approximation of correct or incorrect concepts and used to help in constructing a vivid characterization of the data set.

For using descriptive statistical methods on the relatively enlarged dataset, it would still be possible to permit additional categories based on the names of sub-concepts introduced or qualifiers on concept names. This would also lead to a descriptive statement on learning as opposed to grades or marks. Thus it leads to less contamination of the essence by numeric simplifications.

In this research rough concept inventories are introduced, and their relation to existing approaches are discussed by present author. This is motivated by the need to make concept inventories more student centric, reduce contamination, and address a number of other known deficiencies. In particular, this can be a step towards answering the deeper question: what does a concept inventory actually measure?.

Rotational kinematics of a rigid body about a fixed axis: development and analysis of an inventory

Using concept inventories to measure understanding

Simple, Powerful Strategies for Student Centered Learning. SE

High granular operator spaces and less-contaminated general rough mereologies. Forthcoming

Knowledge and Consequence in AC Semantics for General Rough Sets

Algebraic methods for granular rough sets

Antichain based semantics for rough sets

Dialectics of counting and the mathematics of vagueness

Rough Sets: Theoretical Aspects of Reasoning About Data

Choice Inclusive General Rough Semantics

Algebraic semantics of proto-transitive rough sets

A Geometry of Approximation: Rough Set Theory: Logic, Algebra and Topology of Conceptual Patterns

Reasoning about concepts by rough mereological logics

Rough-set concept analysis: interpreting rs-definable concepts based on ideas from formal concept analysis

A view on rough set concept approximations

Rough sets and vague concepts

Contamination-free measures and algebraic operations

Partial Algebras and Their Applications. Academic, Kluwer

Algebraic methods for rough approximation spaces by lattice interior-closure operations

Three lessons on the topological and algebraic hidden core of rough set theory

Algebraic methods for orthopairs and induced rough approximation spaces

Algebraic semantics of similarity-based bitten rough set theory

The New Palgrave: Dictionary of Economics. TM

Dialectical rough sets, parthood and figures of opposition-I

Approximate Reasoning by Parts

An algebraic approach to binary relations

Groupoids assigned to relational systems

Force concept inventory. Phys. Teacher

The calculus concept inventory -measurement of the effect of teaching methodology in mathematics

The development of a function concept inventory

What does the force concept inventory actually measure?

Interpreting the force concept inventory: a response to

Analyzing force concept inventory with item response theory

Using FCA for modeling conceptual difficulties in learning processes

Using conceptual structures in the design of computer-based assessment software

Are they all created equal? -a comparison of different concept inventory development methodologies

Functional extensions of knowledge representation in general rough sets

Interface of rough set systems and modal logics: a survey

Temporal Logic