China WBI

The project is going along alright — things might get a little tight at the end of this month, but the major hurdle (#2) is out of the way, and we’re ready to begin coding, albeit still not with a web UI yet (it’s not needed in the beginning). I’m getting a better grasp on ASP now, but the concepts are still a little disparate… I’m sure they’ll come together better when I start actual development.

1. Continue learning ASP. I’m continuing to read through the ASP book. I expect to start coding soon, which will help with retaining and applying what I’m learning.
2. Brainstorm and flesh out the concept of mastery. (DONE)
3. Implement/program a model for mastery. Ready to start.
4. Continue development of the problem domain to accommodate a wider range of problems. Still haven’t touched this yet, and might push to next month if #3 takes longer than anticipated.

Upcoming

• Continue to read the ASP books
• Start implementing the programming logic for mastery and structures for site information.

I used to be really big on writing detailed specifications upfront when designing and writing software. It makes logical sense — the more detailed and specific the requirements are, the easier it is to write code exactly with the big picture in mind.

The problem is that this rarely holds true in the real world for complex problems, especially those in a domain that isn’t well-defined and clean cut. The reality is that the vision and direction of a software project changes frequently in both major and minor ways throughout the course of a project. It doesn’t mean that specifications are worthless — just that there isn’t a huge payoff in thinking out every last detail and dreaming up every scenario that the user might face very early in the project, because things are likely to change.

Along those lines, I’ve seen a different sort of specifications in some projects whose purpose is to informally layout a “projected vision” of what the software will do. Some practitioners talk about the idea of “user stories” — telling the story of a user using the software in a real-life situation, and thus anticipating some of the major aspects of the necessary interactions the user has with the software, as authentically contextualized as possible. I’m going to take a similar approach in thinking out how the idea of mastery will be implemented in the system I’m proposing. This “user story specs” will serve as a guide for my eventual software design.

Note: I don’t yet have an official name for the software I’m writing, so for the purposes of these user stories, I’m just referring to it as CWBI.

Ronnie – Instructor

Ronnie is a high school physics teacher in China. He has 2 sections of about 50 students each. Ronnie has been using CWBI software with his students to supplement his instruction since the beginning of the year. Earlier, Ronnie had divided his class randomly into “study groups” of 10 students; so 5 study groups per section. He set up a course and added all of his students to the course, dividing them into their respective study groups.

Ronnie now wants teach a module on kinematics. This module consists of 3 main parts — motion in 1D, vectors, and motion in a plane — with each part taking approximately 3 weeks to complete. In CWBI, he creates a new module, and adds these 3 instructional units from the available supplements provided by CWBI. For each unit, he sets the start date and end date for the unit, as well as a midpoint cutoff date when the 1st assessment in Bloom’s mastery learning model will be taken. Ronnie usually sets the cutoff at 2 weeks, leaving 1 week between the 1st and 2nd assessments. Ronnie also sets his definition of “mastery” by specifying the percentages of problems in each difficulty level that students must answer correctly.

The kinematics module starts, and Ronnie conducts his teaching as usual in the classroom. Ronnie does not give homework to his students; rather, each student is expected to log hours on CWBI to run through practice problems. CWBI provides Ronnie with statistics on the number of practice problems his students perform. Although he does not factor this number into his grading, it enables him to intervene for some students who may be procrastinating, or may be experiencing difficulties. He is also able to view quickly how many of his students are approaching mastery.

As the cutoff date approaches, Ronnie starts to get a sense of how much of his class is understanding the material, and he is able to spend more time on concepts that students seem to be having difficulty on. The cutoff day has finally arrived, and Ronnie’s students take a proctored assessment in the computer lab through CWBI. Ronnie locks the practice problem area for the duration of the assessment. At the end of the assessment, Ronnie looks up the scores in CWBI. Ronnie teaches a challenging class, and as expected, only about 40% of his class achieves mastery at the first assessment. At this point, Ronnie switches gears, and spends most of class time in the computer lab. The 40% of the class who achieve mastery now are peer tutors, helping the students in their study group to understanding the material. The 1 week passes quickly, but by the end, 90% of the class have achieved mastery.

Freda – Student

Freda is a 2nd year high school student in China (US equivalent 11th grade), taking physics from Ronnie. This is Freda’s first course that is supplemented by a web-based instructional tool, but she is looking forward to the time on the computer, as she uses it for chatting with friends online frequently. Freda goes to class and begins a unit in 1D motion, the first part in a larger series of unit in the kinematics module.

Freda listens to Ronnie’s lecture, and reads in her textbook about 1D motion. When she starts getting a feel for what she needs to know, she logs onto CWBI and selects her course and current instructional unit. Noting what kind of performance she needs to achieve mastery as defined by her teacher, she goes to the practice problem area, and tries her hand at several problems. The first few problems are relatively easy, but she stumbles on a few. Looking at her progress, she knows that she is not yet at a mastery level. She has some difficulties with understanding the difference between acceleration and velocity, so she asks some of her fellow students online. Freda is able to understand a few things, but there are some areas which she needs more help in, so the next day, she goes to Ronnie’s office for some help.

After talking with Ronnie, Freda is much more confident on being able to solve the problems she earlier had difficulty on, so she goes back to the practice problem area. This time, she gets many more answers correctly, and is able to build her confidence by going through a variety of problems. She notices that according to CWBI’s projection, her latest session indicates that she should achieve mastery were she given an assessment. Freda is in luck, because the 1st assessment cutoff happens to be the very next day. Freda passes with flying colors, and as such, is now given the task of helping other students in her study group to achieve mastery themselves. The degree to which her study group is able to achieve mastery is part of her grade, so she has extra motivation to help, and she also has plenty of time to do so, now that classroom time has moved to the lab.

To Freda’s delight, 9 out of 10 members of her study group achieve mastery level by the end of the instructional unit. She is also looking forward to the next unit, as she has already read a little about it in her textbook, and has started doing practice problems ahead of time.

I’ve done some more in-depth reading into mastery learning and some of the other mastery-related instructional approaches. It’s actually quite a big area, and as I’ve mentioned earlier, the point of the project is not to study the concept of mastery, but I do want to choose a model.

After a review of some of the literature, I’ve found some key points. Mastery learning, as a theoretical approach, goes back to the work of Benjamin Bloom in 1968, who came up with the “Learning for Mastery” (LFM) method. Bloom was interested in how he could improve traditional classroom instruction by examining what it was about individual tutoring that made it an effective instructional approach.

Bloom contended that most instructors were dividing their instructional material into smaller units of instruction, but that the way the students’ progress was assessed was not helpful for their learning. To be specific, instructors typically had the students take an assessment at the end of the unit of instruction, which served to give the students a grade for their performance, but regardless of how the student did, he or she continued on into the next unit of instruction without any benefit or lesson learned through the assessment.

Bloom’s proposal, instead, was to have 2 formative assessments per unit of instruction. The purpose of the first assessment is similar to the traditional instructional approach; however, this time, the results of this first assessment are not only used to give a grade to the student, but also function as a diagnostic to the instructor as to what particular areas of the instructional unit the student is having difficulty with.

Those students who failed to achieve mastery for the first formative assessment would now be given further instruction using different instructional approaches (Bloom believed that varying instructional methods would help a larger percentage of students achieve, since learners learned effectively in different ways). They would be assessed a second time to determine how far they had progressed.

Meanwhile, students who achieved mastery in the first formative assessment would go on to learn concepts which extended and built upon the unit of instruction, and their second assessment would primarily revolve around these extended concepts. The purpose was to raise the bar higher, while not leaving the rest of the class behind.

There are other similar or derived approaches; the literature also mentions the Personalized System of Instruction (PSI), the Problem Based Learning (PBL), and Outcome-Based Education (OBE) as related concepts. PSI allows students to control the pace of instruction themselves, and use primarily written materials for self-instruction. I don’t think this will fit well into the Chinese educational context. PBL is more concerned with the process of inquiry and authentic learning models, than centrally on the idea of mastery. Fhe focus of OBE seems to be more on the larger curriculum development process, and defining desired “outcomes” or “goals,” then working backwards to instructional strategies. Mastery learning is used frequently in conjunction with OBE, but is not a necessary component.

After brainstorming a little on mastery learning, I’ve come up with a slightly modified model based on Bloom’s. What I plan to do is to have 2 assessments, similar to Bloom’s model, but for students who pass the first assessment, the primary task of the 2nd stage is to tutor and assist other students who did not pass the first assessment. It’s often said that having to teach a subject forces one to learn it in depth. More importantly, I believe that this works relatively well with the collaborative bent of Chinese students, and as long as students’ scores are not comparatively dependent on each others’ success or failure, this could mean a relative win-win situation for all students involved.

There is a question of whether my proposed system will have the idea of group mastery. The idea was that groups would achieve mastery only when all the students in that group achieved individual mastery, and that this would somehow be tied to assessment of the group. While in principle, it could motivate students and provide an opportunity for collaboration, there are negative consequences to consider when a large majority of the group is pressuring an individual who is unmotivated or has social adjustment problems.

But now that I am adapting Bloom’s approach with a collaborative tutoring aspect, I feel more inclined to discard the idea of group mastery altogether, since I am delving into the cultural value of collaboration through different means. If I can think of a different way to approach group mastery that doesn’t have the obvious potential to reduce students’ intrinsic motivations to learn, then I’ll put this back into the site design; otherwise, for now, I’m shelving it.

Here is a list of articles I read, which helped me to understand Bloom’s idea of mastery learning. These are mostly secondary sources, so I’ll need to read up on some primary and peer-reviewed sources later for a proper literature review, but it’s a sufficient start. I’m not taking the time to look up exact references and do APA and all that jazz, since I’ll be getting better references later anyway.

Articles Read

• Anderson, S. A. (1994). Synthesis of research on mastery learning.
• Douglas, C. (2002). The effects of mastery and performance goals on college students’ motivation.
• ERIC Digest E530. (1984). Connecting performance assessment to instruction: A comparison of behavioral assessment, mastery learning, curriculum-based measurement, and performance assessment.
• Ford, B. & Klicka, M. A. (1998). The effectiveness of individualized Computer Assisted Instruction in basic algebra and fundamentals of mathematics courses.
• Guskey, T. R. (2005). Formative classroom assessment and Benjamin S. Bloom: Theory, research, and implications.
• Guskey, T. R. (1994). Outcome-Based Education and Mastery Learning: Clarifying the differences.
• Price, R. (2000). PSI revisited: Designing college courses using the Personalized System of Instruction (PSI) model.

There isn’t a whole lot to say for this update. Mostly, I’ve been chugging slowly along with the milestones for this month.

1. Continue learning ASP. It was a good decision to slow down on this, as there’s just a wealth of information to absorb. This past week I’ve been covering information on debugging and tracing ASP code, and also how to use the built-in validation for form input, which will be a real time-saver.
2. Brainstorm and flesh out the concept of mastery. See previous post.
3. Implement/program a model for mastery. I can’t get started on this until I finish #2.
4. Continue development of the problem domain to accommodate a wider range of problems. Haven’t touched this yet.

Things are taking a little longer than anticipated, so I have some time to look more in detail on what the literature says on mastery, and to make the necessary connections to cultural research. We’ll see how things shape up in a few weeks.

Upcoming

• Continue to read the ASP books
• Read and report on literature on the topic of mastery
• Come up with the proposed definition of mastery, for the purposes of this project

When I first brainstormed for this project, I didn’t think a whole lot on how I will define “mastery.” This is an important part of this project, though, because it is central to determining the progress of each student in their respective units of instruction.

I had assumed that mastery could simply be defined as getting a certain number of problems correct, whether a percentage of total problems, or a percentage of problems within a fixed time period. I remember taking a “unit mastery” course for Sociology, which basically required the student to get something like 85% or better on a multiple choice examination based on a book chapter.

As with most important concepts, though, there’s a lot more to mastery than meets the eye. For example, the system for determining mastery for my Sociology course is more complex than getting a certain percentage correct. There are other issues, such as:

• How is the course broken down into instructional units that can be tested? How granular is this?
• Can you take tests for the instructional units out of order?
• Can you take tests multiple times until you pass? Is there any consequence for failing a test?
• Is there a time limit for when a test must be passed by?
• Does the unit mastery test itself have a time limit?

Actually, these are just issues dealing mostly with practical details. There are other, more philosophical issues to consider as well, such as: what kind of test would appropriately demonstrate a student’s mastery? What kind of record is kept of the student’s progress, and who has access to this? What is the teacher’s role in mastery-based instruction?

After thinking a little more about this, I came to the conclusion that while it would be OK to propose a system that defined mastery as “getting a certain percentage of questions correct,” it would be important to fill in more details, as well as be able to give a basic rationale for the reasoning behind this.

To do this with any kind of credibility in the academic realm, though, I need to read up on the idea of mastery in the literature. A preliminary search on this topic has yielded 3 key approaches under the mastery umbrella that have come up often in my literature searches:

• The Mastery Learning Method. One of the fundamental ideas of this approach is that nearly all students can learn individually or in groups, given the appropriate resources, tools and environment. The equivalent “A” and “B” grades are only accepted, as these demonstrate “mastery.”
• Personalized System of Instruction (PSI). Proposed by Fred Keller in the 1970s, and is based on operant conditioning (behaviorism). The key principles involve self-pacing, the use of written verbal communication between instructor and students, and the use of proctors who score tests and individual tutor students. Also, classroom time is not used to introduce material, but as a time for motivation.
• Problem-Based Learning (PBL). This group-based approach is based on a mix of cognitive and constructivist theory. In this case, the instructor (or facilitator) gives a problem to a group of students. The students may do individual research on the problem, but will come together as a group to exchange ideas.

I will be looking more closely at what the literature shows for these 3 approaches in the coming weeks. Although the focus of this project is not on the concept of mastery itself, I would like to try to match a custom approach which seems like it would work well in Chinese educational culture, and for the practical affordances and limitations presented by a web-based educational tool.

Although I’ve technically met my milestones for February, I’ve decided to push the remaining milestones on my original schedule by at least one month for two main reasons:

1. Learning ASP. I found myself trying to go too quickly through a  956 page in an effort to meet an arbitrary milestone. The reality is that there are a ton of concepts to nail down in ASP; many more than I anticipated, and it’s not a good sign to look at a chapter I read a few days ago and not remember 90% of it. So I’ve decided that I need to go more slowly and digest more, and lessening the time pressure will help.
2. The concept of mastery. Although I’ve mentioned that the idea of mastery of a unit, both by the individual and by the group, is central to making this project relate to cultural issues in web-based educational tools, I’ve barely dealt with what this will really involve. As I was working on my paper prototypes, I realized that I just assumed that mastery was just achieving a certain percentage of problems correctly, specified by the instructor. It may just end up being that, but the thing that is missing is a reasoned rationale for such an approach. What I really need to do is think through what the alternatives are, and the strengths and drawbacks of whatever approaches are available.

The revised schedule:

March

• Continue learning ASP
• Brainstorm and flesh out the concept of mastery
• Implement/program a model for mastery
• Continue development of the problem domain to accommodate a wider range of problems

April

• Setup basic login and user authentication
• Develop drilling and test modules

May

• Prototype drilling module
• User profiles (teacher & student)
• Create and manage students, study groups

June and beyond

• Prototype testing module
• Report and progress display functions
• UI design
• User acceptance and usability testing
• Performance testing
• Live deployment