An Evaluation of Assessment Tools in Outcome-based Education: A Way Forward

: With the existing manual-based of outcome based education (OBE) systems, data complexity appears as the main problem to be solved. In this paper, a critical discussion of the manual-based system and a way forward with the online version of the OBE systems in the Faculty of Electrical and Electronic Engineering (FKEE), Universiti Tun Hussein Onn Malaysia (UTHM) have been discussed. Two assessment tools of programme learning outcomes (PLOs) as well as course learning outcomes (CLOs) have been used to measure and analyse manually both achievements of the PLOs and CLOs. However, results obtained have shown that these processes require a better system, to deal with various parameters such as different cohort of students and various courses. Obviously, the online-based version and further improvement of the manual-based system that have been proposed are expected to resolve these issues, hence it will be fruitful for the FKEEs’ OBE administrator.


Introduction
Outcome-Based Education (OBE) refers to an educational system that focuses on what the students are expected to be able to do within the specified period of learning.This requires having a clear set of essential knowledge and skills that students are expected to have learnt at the end of their learning experience and, then organizing the curriculum, instruction, and assessment to make sure this learning ultimately happens.In Spady's words: "Outcome-Based Education means clearly focusing and organizing everything in an educational system around what is essential for all students to be able to do successfully at the end of their learning experiences.This means starting with a clear picture of what is important for students to be able to do, then organizing the curriculum, instruction, and assessment to make sure this learning ultimately happens" (Spady, 1994).In Malaysia, OBE is being implemented mostly in IPTAs since 2005 to ensure the academic programmes, delivery system and the assessment methods will produce graduates equipped with high quality knowledge and skills.The paradigm shift to OBE in Malaysia is driven by the Quality Assurance Department at the Ministry of Higher Education, known as the Malaysia Quality Agency (MQA).Undergraduate curriculum needs to be transformed into OBE in order to meet the requirements of both the Engineering Accreditation Council (EAC, 2012) and the Malaysian Qualification Framework introduced by the MQA (MQA, 2012).Faculty of Electrical and Electronic Engineering (FKEE) in Universiti Tun Hussein Onn Malaysia (UTHM) has implemented OBE in its seven (7) Bachelor Degree programme.The OBE implementation at FKEE is conducted, evaluated and monitored by OBE committee that has representatives from five (5) departments.There are many assessment tools developed by University's OBE committee as well as FKEE OBE's committee in order to support the process of OBE implementation and ease the work of documenting for each programme (OBE Committee, 2010).Here, in this paper, the discussion is mainly focuses on direct evaluation/assessment tools that have been developed by FKEE OBE's committee to measure the OBE performance for each programme in the faculty.The rest of the paper is organized as follows: The next section discusses on the current OBE implementation based on one of the Bachelor Degree programme offered by FKEE.The next three sections discuss on the assessment tools used to measure the OBE performance for each programme.The last section explains on the future works and, finally the paper is concluded in section 6.

Current OBE Implementation
At fkee, each programme has their curriculum structures that contain three major components to be evaluated which are programme educational objectives (PEO), programme learning outcomes (PLO) and course learning outcomes (CLO).In order to describe each component, bachelor of electronic engineering (computer engineering) with honours or bec is used as an example.The first enrolment for this programme is started in semester 1 session 2010/2011.PEOs describe in general the career and professional accomplishments that the programme is preparing for the graduates to achieve.Table 1 shows the PEOs for BEC programme.

PEO1
competent in the field of electronic and computer engineering to fulfill the needs of industry at the national and international level PEO2 able to deliver, enhance and develop their ability and experience on a continuous basis as professional engineers.PEO3 able to practice and implement ethics and social responsibility to the global community.
PLOs as shown in Table 2 describe the expectations in terms of skills, knowledge or attitude acquired by students after following the programme.

Sustainable Development Affective
Even though the PEOs assessment will begin after at least five (5) years of graduation, inputs obtained from PLOs direct assessment can be used as initial indicator of PEO achievement of the programme.This can be done by knowing the relationship between PEOs and PLOs as shown in Table 3.

Table 3: Relationship between PEOs and PLOs
Currently, in BEC programme there are four (4) categories of courses which are university courses, core courses, specialization courses and elective courses.These courses have been designed to assess the stated PLOs. Figure 1 has taken few BEC specialization courses that have been mapped with the BEC PLOs.
CLOs are designed to specify skills or domains required to be achieved by students at the end of the course.Thus, each course has three (3) CLOs comprises of three (3) domains that reflect to cognitive, psychomotor and affective skills.Using specific direct assessment tools (such as quizzes, assignment etc.) setting up by lecturers/instructors, students' skills/domains will be measured.The result obtained will give indicator on CLOs achievement.In OBE, there are three-level attainment loop shown in Figure 2. The inner-most loop which is the CLO attainment is where it will affect the teaching and learning activities.Direct feedback or measurement especially from the students will be acquired at this level.The results from the assessments and survey shall be used in the CLO analysis and review.The PEO or the outer-loop encompasses the whole OBE implementation.The inputs and feedback from the major stakeholders, which included the alumni, advisory committee, external examiner, faculty and students, would be acquired periodically to be used in the review process.Any changes made at PEO level will cascade down to programme level.The PLO or middle-loop attainment would define the attributes set upon graduation.Again, the relevant feedback will be acquired and shall be used during the PLO analysis and review.The changes at PLO level will go down to the innermost loop which is the CLO or course level attainment loop.

Figure 2: Three-level attainment loop of OBE
There are two (2) types of assessments; direct assessment or indirect assessment.In evaluating the OBE achievement for the program, this paper discusses two (2) direct assessment tools that have been used to assess the CLOs and PLOs.These tools have been developed using Microsoft Excel.

Course Learning Outcomes Tool
CLOs   CLOs should be measureable and observable via cognitive, psychomotor and affective learning domains.In other words, CLOs should reflect essential knowledge, skills and attitudes and finally, represent the minimum performances that must be achieved to successfully complete the course.

Figure 5: CLO analysis for BEC10102 Computer Programming
In the CLO Analysis section, the information need to be entered is the assessment methods/tools used to measure the achievement of cognitive, affective and psychomotor domains.In the following example shown in Figure 5, the cognitive domain for CLO#1 of BEC10102 Computer Programming is evaluated based on Test1, Test2, Final Exam, Quiz and Prelab that carry 10%, 10%, 40%, 5% and 4% weight age.The affective domain for CLO#2 for the course is evaluated based on project report and peers assessment with 7% and 4%.Lab activities (16%) and presentation (4%) are used as the assessment method to evaluate the psychomotor domain for CLO#3 of the same course.A total of 30 students from 52 students have achieved the target set for the Cognitive Domain (57.7%),Affective Domain (94.2%) and Psychomotor Domain (94.25).This shows that this CLO has been achieved based on the KPI that has been set.The CLO achievement for each section/group of students is recorded by the course coordinator in Coordinator section as shown in Figure 6.This is done to get an overview of the overall achievement for the course and then plotted into a graph as shown in Figure 7.As shown in Figure 10, the Matrix CLO-PLO section listed all courses that need to be taken and completed by BEC students.Programme coordinator is a person responsible to record this section by inserting the collected data from CLO analysis tool.The CQI section for the programme is then proposed based on discussion among lecturers/instructors of the department.Figure 11 shows some example of suggested action that can be taken to improve the achievement of each PLO.Therefore, OBE Online System is proposed to automate the task of OBE committee and ease the documentation work for the programme.(Chan et. al, 2007), (Md. Kamal et al., 2009) and (Anuar, et al, 2009) also use online system to manage their document on OBE.This online system will consists of three (3) major components; PEOs, PLOs and CLOs.For a start, a login and registration modules of OBE Online System has been developed, uploaded and tested to meet user's requirements.

Conclusion
In conclusion, an online version of the OBE systems has been proposed and developed using HTML, PHP and SQL Coding.Preliminary results that have been obtained reveal that the online version still requires further analysis and evaluation to cope with massive data and various parameters.However, it is worth mentioning that this system is a significant kick start to solve issues in OBE.

Figure 1 :
Figure 1: Mapping on BEC Specialization Courses and BEC PLOs Specialization Courses PLO1 PLO2 PLO3 PLO4 PLO5 PLO6 PLO7 PLO8 PLO9 PLO10 PLO11 describe the specification of what a student should learn as the result of a period of specified and supported course.CLO ANALYSIS 2012 is a revised version of tool used to measure achievement level of CLO in the course.The tool has five (5) sections; Matrix CLO-PLO, CLO Analysis, Coordinators, CQI and Help as highlighted by the red box in Figure 3.

Figure 3 :
Figure 3: Sections in CLO Analysis Tool

Figure 4 :
Figure 4: Matrix of CLO and PLO

Figure 6 :
Figure 6: Overall CLO achievement for BEC10102 Computer Programming

Figure 7 :
Figure 7: Average of CLO achievement for BEC10102 Computer Programming

Figure 9 :
Figure 9: Sections in PLO Analysis Tool

Figure
Figure 10: Matrix CLO-PLO section for BEC programme

Figure 11 :
Figure 11: CQI Section in PLO Analysis Tool

Figure 12 :
Figure 12: Initial GUI of OBE Online System Figure 13: Login module

Figure 14 :
Figure 14: Sub-module: Manage Programme, Manage Courses, Manage Lecturers, Manage Students and Manage Departments.