Evaluating the Effectiveness of Teaching Information Systems Courses: A Rasch Measurement Approach

Systems analysis and design (SAND) is an information systems (IS) course that is taught around the world in most higher education management of information systems (MIS) programmes. However, the theoretical nature of this type of course presents challenges for instructors as they devise instructional strategies to convey the abstract concepts that are necessary for their students to understand, such as, how to draw data flow diagrams (DFD) to correctly represent the informational specifications of an IS. Evidence suggests that one of the factors of the low success rates of many IS-design projects in the workforce is due to the graduate recruits’ failure to acquire basic SAND knowledge. While a considerable amount of literature focused on integrating technology into the teaching practices to facilitate the knowledge acquisition, a few investigated its effectiveness to fulfil this particular purpose. This paper reflects on such challenges and proposes an evaluation approach to assess the effectiveness of technology integration in teaching an IS course like SAND. The empirical interpretations represented in this paper are gathered through a series of quasi-experimental 2x3 factorial experiments that were conducted at four higher education institutions and based on the Rasch item response theory and measurement analysis. The preliminary analysis from this study provides reliable evidence to delineate key instructional strategies when designing higher education IS courses.