Abstract
- While student populations in higher education are becoming more heterogeneous, recently several attempts have been made to introduce online peer support to decrease the tutor load of teachers. We propose a system that facilitates synchronous online reciprocal peer support activities for ad hoc student questions: the Synchronous Allocated Peer Support (SAPS) system. Via this system, students with questions during their learning are allocated to competent fellow-students for answering. The system is designed for reciprocal peer support activities among a group of students who are working on the same fixed modular material every student has to finish, such as courses with separate chapters. As part of a requirement analysis of online reciprocal peer support to succeed, this chapter is focused on the second requirement of peer competence and sustainability of our system. Therefore a study was conducted with a simulation of a SAPS-based allocation mechanism in the NetLogo simulation environment and focuses on the required minimum population size, the effect of the addition of extra allocation parameters or disabling others on the mechanism's effectiveness, and peer tutor load spread in various conditions and its influence on the mechanism's effectiveness. The simulation shows that our allocation mechanism should be able to facilitate online peer support activities among groups of students. The allocation mechanism holds over time and a sufficient number of students are willing and competent to answer fellow-students' questions. Also, fine-tuning the parameters (e.g. extra selection criteria) of the allocation mechanism further enhances its effectiveness.
- Keywords:
- Peer Support, Peer Allocation, Computational Simulations, System Dynamics, Distance Learning
Introduction
- 1.1
- Society and (higher) education have changed rapidly in recent decades. The digital revolution has had its influence on the educational process (Sloep & Jochems 2007). For example, students can learn more independent of place and time today. Higher education itself has been subject of change in the last decade as well. Many institutes have transformed their learning approach to one in which students have more control over their own learning process. As a result student populations are less homogeneous, students being increasingly involved in different activities. This leads to increasing tutoring needs, which has had a negative effect on teacher workload (Fox & MacKeogh 2003; Rumble 2001). Most of the tutoring today is in the hands of teachers. However, several researchers have explored whether students could take over (parts of) teachers' tutoring tasks by acting as peer tutors. Not only could this reduce teachers' tutoring load, it also has some additional advantages. Peer tutoring could have a positive effect on the learning process and knowledge construction (Fantuzzo et al. 1989; Gyanani & Pahuja 1995; King et al. 1998; Wong et al. 2003). For example, Fantuzzo et al. (1989) found higher learning outcomes and more social interaction in a peer tutoring setting, as compared to several control groups such as a group that received video-based instruction, which they argue was caused by the element of structured exchange between students subjected to the peer tutoring. Tutors themselves also benefit from tutoring others (Fantuzzo, et al. 1989), a phenomenon known as the self-explanation effect (Ainsworth & Loizou 2003; Chi et al. 1994). Other studies found that peer tutoring stimulates interactions leading to knowledge construction (Gyanani & Pahuja 1995;Slavin 1995), that students become more motivated (Fantuzzo et al. 1989), and that they can gain more self-confidence in their learning (Anderson et al. 2000).
- 1.2
- Teachers indicate the answering of student questions is specifically time-consuming (De Vries et al. 2005). A problem however in using peers to act as tutors for students' questions is selecting peers who are sufficiently competent to answer a specific question. Attempts have been made to make this process more efficient, by introducing systems for online reciprocal peer support (e.g.Van Rosmalen et al. 2006;Sloep et al. 2007). In these cases, questions students have while studying are answered by fellow-students acting as peer tutors via computer applications (De Bakker et al. 2008) or web services (Van Rosmalen et al. 2008). Reciprocal here means that students can be both tutee and tutor, but they can also be one or none of the two. Especially in distance education, such systems to facilitate peer support activities with intervened peer allocation could be beneficial, since students are more isolated and more often do not know which fellow-student to turn to with their questions. Many higher education institutes have introduced forms of peer support over the last few years. Perhaps the most common implementation is that of a bulletin board or web forum, via which students can post their questions. Other students who log on to the bulletin board can read and answer these questions. This is a method in which peer allocation is self-regulated without the intervention of a facilitating allocation system. Although this seems appropriate in many cases, there are definite benefits to mediated peer support based on direct allocation of peers to answer questions, some of which are pointed out by Westera (2007): a) someone gets the responsibility to offer the support, b) the likelihood of support becoming available is increased, c) allocation results in the selection of the most competent peer tutor, d) the time before getting an answer can be reduced, e) peer tutor load can be distributed more evenly over the population. Field experiments with peer support systems with intervened peer allocation as described have shown promising results in terms of user appreciation and effectiveness. For example, Van Rosmalen et al. (2008) found that the majority of students working with such a system were positive towards it and that the majority of students' questions was answered sufficiently according to experts who rated the answers given by peers. Similar previous initiatives for online peer support systems have some important drawbacks. They are either only suitable for larger populations (Westera 2007) or, if the support is given asynchronously, confront tutees with a waiting time (Van Rosmalen et al. 2006). To develop an online reciprocal peer support system that is suitable for smaller population sizes and that provides students with support more quickly, we introduce the SAPS system (Synchronous Allocated Peer Support). This system connects students with study questions to peers. The support is given via instant messaging (IM). Our research is based on an analysis of requirements online reciprocal peer support systems should meet. As part of that, this study is focused on the requirement of sufficient peer competence and sustainability. Such a system should be able to allocate sufficiently competent peers for the support need at hand. Peer competence here means that students are expected to be able to answer fellow-students' questions, based on their competence on the topic of the question. Furthermore, a sufficient number of peers should remain willing to act as peer tutors during the period their support is needed, i.e. that the system should be sustainable.
- 1.3
- The current study focuses on this requirement as it was tested via a model of the SAPS system in a simulation study using the NetLogo simulation environment (Wilensky 1999). First however, the new system will be described.
The SAPS system
- 2.1
- The SAPS system is designed for reciprocal peer support activities among a group of students who are working on the same fixed and stand-alone modular material every student has to finish, such as courses with separate chapters.
Selection quality: tutor competence
- 2.2
- Analogous to existing peer allocation systems (e.g.Van Rosmalen et al. 2006; Westera 2007), the SAPS system in the first place determines a candidate peer's competence by looking at 'proximity'. Students who are working on the same learning unit (e.g. learning task, course unit, module) or who have recently completed it are prioritised as candidate tutors for answering a question on that learning unit, since they are expected to be able to answer questions on the content of the learning unit. As opposed to other systems, SAPS aims to enhance the general competence of peer tutors by introducing two more selection criteria which could be implemented in the allocation algorithm for determining peer tutor competence: 'question type' and 'previous result'. Question types can be 'theoretical questions', 'organisational questions', etc. 'Question type' could be used to prioritise candidate peers who have indicated to be competent in the question type asked for by the student who has a question. Through 'previous result', the algorithm takes into account marks students acquired on learning material on similar topics, such as previous courses. This could be used to prioritise students with high marks on those topics.
Economy principles
- 2.3
- To prevent some tutors (e.g. those with the highest pace) to be selected too often, following Westera (2007) the SAPS allocation mechanism consists of two economy principles that are used to spread the peer tutor load evenly among the student population. The first economy principle prioritises those students who have previously had few tutor turns ('uniformity'), the second prioritises those students who have already asked many questions themselves ('favour-in-return'). The economy principles therefore act as a kind of mediated version of a tit-for-tat mechanism expected to be crucial in cooperativeness in social peer interaction (e.g. Sloep 2008).
- 2.4
- A final selection criterion is 'online/offline'. The SAPS system has been developed to be used with both synchronous and asynchronous communication media (e.g. via instant messaging). Via 'online/offline' candidate peer tutors who are being online (synchronous) or offline (asynchronous) can be prioritised. The current implementation of the system is mainly focused on synchronous communication to speed up the process of answering questions.
Selection procedure: ranking
- 2.5
- For each of the above criteria peer candidates (i.e. all students except for the student asking a question) are given allocation points. Proximity for example is calculated as follows: a student working on the same learning unit as the learning unit the asking student is working on is given 10 points. A student who is one learning unit further gets 9 points, etc. The allocation points given on all selection criteria result in a total score of a candidate peer tutor. The candidate with the highest score is selected as peer tutor and receives an invitation. If the selected peer tutor does not respond to the request, the student with the second highest score is selected. The SAPS system has the ability to assign variable weights to all selection criteria to give more or less priority to them. Due to this ranking procedure it is possible that peers are selected with a score of 0 on all of the three quality selection criteria ('proximity', 'question type' and 'previous result'), which in practice would mean that the peer would not be competent enough to answer a question. The number of cases in which this occurs is essential to the success of the system and is therefore a main focus of the simulation study.
Willingness to answer questions
- 2.6
- Another essential aspect in the effectiveness of a peer allocation mechanism such as one based on the SAPS system, is students' lasting willingness to answer each other's questions. If, in the long run, none of the students would be willing enough, the system is doomed to failure. Students' willingness is therefore an important variable within the simulation, which will be further detailed in next sections.
- 2.7
- Figure 1 shows a schematic representation the SAPS system, displaying the activity sequence from question to peer-support session, as well as the allocation procedure and criteria used for matching students for the peer support activities. Additional to the quality and economy selection criteria, the selection procedure consists of two more parameters. If the ranking procedure leads to more students with the same highest ranking one is randomly selected; also each student asking a question is excluded from the list of candidate peer tutors.
Figure 1. schematic representation of the SAPS system. - 2.8
- To test the peer competence and sustainability of the SAPS peer allocation system we built a simulation model of a SAPS-based allocation mechanism. Simulations offer the possibility to adjust systems developed and to test the effectiveness of improved versions before testing or implementing them in practice. We chose to model the mechanism in the NetLogo simulation environment (Wilensky 1999). NetLogo is especially suitable for modelling complex systems that develop over time, and analysing the connection between micro-level interaction behaviour of the agents (e.g. students) in the model and the macro-level patterns that emerge from these interactions. Furthermore, simulations offer the opportunity to examine behavioural patterns without having the limitations of contextual factors empirical studies can suffer from, such as working with real students (e.g. you cannot test the system at different population sizes within a short period of time).
Research questions
- 3.1
- As part of our requirement analysis for online reciprocal peer support to succeed, this study focused on our second requirement: peer competence and sustainability. The main question was whether the mechanism was actually able to allocate sufficient peer tutors to students with questions based on the SAPS allocation algorithm that were sufficiently competent to do so, and whether peers remained willing to help others over a longer usage period. Furthermore, we had four additional research questions, all of which are aimed at further enhancing the mechanism's effectiveness.
- What is the minimum required population size at which a sufficient number of competent peer tutors are found who are also willing to answer?The study starts with a version of the SAPS mechanism in which only 'proximity' is used to select peer students, since this is the starting point of many of such systems. As stated previously, the selection procedure could result in a peer being selected with a score of 0 on 'proximity' as well as in a situation in which questions remain unanswered because none of the fellow-students is willing to provide an answer. This is assumed to happen in a number of all selection procedures executed within the model. In an empirical study with an online peer allocation mechanism, Van Rosmalen (2008) found that approximately 9% of all questions remained unanswered. Following this outcome, we consider a maximum of 10% of unanswered questions as acceptable. Van Rosmalen also found that 25% of the questions were not solved correctly (as rated by external experts). However, since even high-quality peers (i.e. scoring high on 'proximity') might not give correct answers, we treated a lower threshold of 10% for the percentage of low-quality peers as acceptable, in line with our threshold for unanswered questions. Also, we expect that both percentages decrease when the number of students is increased, since the more students are available to act as peer, the less low-quality and unwilling peers there will be. We also expect there is a minimum population size that yields acceptable results on both.
- Does the introduction of extra quality selection criteria lead to an increased quality of the selection mechanism?In the SAPS system we introduced 'question type' and 'previous result' as extra quality selection criteria in the allocation process in order to enhance the general selection quality of the mechanism. We expected that the introduction of 'question type' leads to a decrease of the percentage of low-quality peers being selected. Since the introduction of these criteria introduces extra staff work (e.g. preferred question types need to be collected), we were interested in how big this difference exactly would be. In other words, we wanted to find out whether the expected quality gain is worth the extra effort. We believe this quality gain should be minimally 10% (i.e. a decrease by the same amount of the percentage of low-quality peers being selected).
- In what way does omitting the economy principles from the mechanism influence tutor load spread? Many peer allocation mechanisms have economy principles incorporated (e.g.Van Rosmalen et al. 2006; Westera 2007). These could be useful to prevent overloading individual peer tutors. It could be questioned, however, whether economy principles are always wanted. For example, while implementing a SAPS-like mechanism among a student population, it might turn out that only a particular fraction of the population is enthusiastic about it and motivated to help each other regularly. Then only a percentage of the entire student population is actively involved in peer support activities. In such a case it would be counter-productive to apply economy principles, since these principles result in involving non-active students and the unnecessary application of load levelling on highly motivated peer tutors. Also, since students themselves benefit from acting as peer tutors (Fantuzzo et al. 1989), incorporating study load levelling would rob motivated students from the opportunity to improve themselves. We therefore wanted to ensure that omitting the economy principles from the allocation mechanism would not lead to an unacceptable overload of individual peer tutors. It would be unacceptable when omitting the principles would show a large spread of tutor turns among students, compared to a condition in which the economy principles are applied. We think an increase of less than 50% of the maximum of tutor turns in a condition in which the economy principles are disabled compared to one in which they are enabled would be acceptable.
- In what way does disabling the economy principles influence the percentage of low-quality peers? Another benefit of omitting the economy principles is that the selection procedure is focused more on the quality criteria, so this would introduce another chance to further enhance the general selection quality of the allocation algorithm. The economy principles consist of two selection criteria with the same weight as the quality selection. As the mechanism concentrates on just competence without these principles we expected a decrease of the percentage of low-quality peers being selected. Since omitting the economy principles would not cost extra effort, any quality gain would be desirable (under the assumption that omitting the economy principles would not lead to a large spread in tutor turns).
The simulation model
-
Model variables, relations, formulas and their implementation within the simulation
- 4.1
- After the introduction of the SAPS system and the most important aspects to be focussed upon in the study, we will now describe the simulation model of an environment with a SAPS-based allocation mechanism we developed to examine our research questions. The simulation model takes into account learner profiles of all students (which can be both tutee and tutor), learning units students are studying, and questions they are asking. When asking questions, students are matched to other students based on their own and the other students' study progress and some additional variables via a tutor selection procedure. For a detailed description of all procedures see the next paragraph. Table 1 presents an overview of all variables, relations, formulas and their implementation within the simulation. Some variables are related by formulas and are further detailed in table 2.
Table 1: Overview of variables and their implementation within the simulation. VARIABLE DESCRIPTION IMPLEMENTATION IN SIMULATION INPUT FOR Range (initialization) Formula LEARNER PROFILES General characteristics available study time The study time a student has in a certain period. M = 1.5 hours/week, SD = 0.5 hours (Normally-distributed) no progress constraints Fatigue, flow, stress, a.s.o. [-0.5, 0, 0.5] (Randomized) for each time unit no progress prior knowledge Prior knowledge of a specific LU [0, 0.33, 0.66, 1] no progress progress (in a learning unit) Progress of a student within a module. [0 - 6.75] hours yes current LU, question corresponding LU question trigger Determines whether a question will be asked. Boolean yes tutor selection, question status, nr of tutee turns Question & answer profile inclination to ask questions the general tendency of students to ask questions Integer [1, 2, 3] no question trigger willingness to help The willingness of students to help fellow-students with their questions. Rational [0 - 5] yes tutor selection nr of tutee turns The number of times a student has had the rol of tutee. Integer [0 - ...] no favour-in-return nr of tutor turns The number of times a student has had the rol of tutor. Integer [0 - ...] no uniformity Peer competence profile current LU The LU the student is working on. Integer [1, x]. The model used in in the experiment runs consisted of 20 LUs. no quality criterion: proximity preferred question type The question type(s) the student is competent in. One or more integers from [1, 2, 3, 4, 5] no quality criterion: question type previous result The mean mark students received for past exams. Integer [1 - 10] no quality criterion: previous result login status Describes whether the student is logged on to the system. Boolean no login status check LEARNING UNIT LU size The time needed to study the LU. Fixed value of 6.75 hours for each LU. no progress in a learning unit LU complexity The complexity level of the LU for each individual learner. [1, 2, 3, 4] (Randomized) for each learner at each LU no question trigger QUESTION question corresponding LU The LU the question corresponds to. Integer [1 - 10] no quality criterion: proximity question type The question type asked for by the tutee. Integer [0 - 10] no quality criterion: question type question status Describes whether a question is answered or remains unanswered. Boolean no - Table 2: Formulas and descriptions for all variables that are changed through formulas by other values in the simulation. VARIABLE
NAMEDESCRIPTION FORMULA IN SIMULATION INPUT FOR progress in a learning unit Progress of a student within a module that describes likeliness of transition to a next module. Progress = previous progress state + available study time + constraints + prior knowledge current LU, question corresponding LU willingness to help The willingness of students to help fellow-students with their questions. IF willingness of student < 3, add 0.33 to his willingness. This procedure is executed at each time interval. tutor selection question trigger Determines whether a question will be asked. question trigger = (inclination to ask questions) + (LU complexity) > 4 and random [0,1] tutor selection, question status, nr of tutee turns - 4.2
- The standard unit of time within the simulation is 1 day. Each student has a Learner Profile consisting of General Characteristics, a Question & Answer Profile and a Peer Competence Profile. The General Characteristics consist of all general parameters each student has within the simulation. Available Study Time is the amount of time each student has available for studying, measured per day. In the simulation, available study time is a normal distribution with a mean of 1.5 hours per day. The amount of actual studied time per day is influenced by constraints and prior knowledge. Constraints reflect the effect of contextual influences a student encounters while studying, which can be positive (being in a study flow), neutral or negative (e.g. suffering from fatigue or stress). Although constraints are likely to be a multi-dimensional construct, following Nadolski et al. (2009), in the simulation we simplified constraints to a unidimensional construct with possible values of -0.5, 0 or 0.5, reflecting the three possible influences described (negative, neutral, positive). Prior Knowledge is defined as an extra time gain a student might have on a specific learning unit because of his prior knowledge of the topic of the LU. For each LU, a student has a prior knowledge of 0, 0.33, 0.66 or 1. The number of possible values chosen provided sufficient variation of prior knowledge among the population. Each student has a certain Progress in a learning unit he is currently studying, which is the result of the sum of available study time, constraints and prior knowledge. Each student has a Question Trigger that determines at each time unit within the simulation whether he will ask a question. A student's Question & Answer Profile consists of his inclination to ask questions, willingness to help and his number of tutee and tutor turns. Each student has a general Inclination to Ask Questions set as a random integer (1, 2 or 3) at the start of the simulation, reflecting the differences between students in the need they have to ask questions while studying. After some test runs with the simulation model, three possible values were found to be sufficient to provide sufficient variation among the population and at the same time to result in a number of questions being asked in the simulation runs that was acceptable. Willingness to Help is a general parameter independent of who is asking a question. It is set at a general value at the start of the simulation, ranging from 0 to 5, but is a variable during the simulation when a student has answered a question. We treated willingness as a much more fine-grained variable by giving it more possible values to arrive at sufficient variation among the population and at the same time to arrive at acceptable percentages of students being willing to answer questions reflecting empirical results found in similar conditions (Van Rosmalen et al. 2008). Each student's Peer Competence profile consists of the current LU he is studying, his preferred question type, his previous result and his login status. Each student has a Preferred Question type, which is set as a parameter at the start of the simulation via one or more random integers ranging from 0 to 5. A student's Previous result reflects the mark a student has acquired on a similar set of learning units (e.g. a previous course), and is set as a random value at the start of the simulation with a mean of 7. Login Status reflects at each time unit whether a student is logged into to the peer support system, set randomly as a Boolean at each time unit. Each Learning Unit (LU) consists of a fixed LU Size of 6.75 hours and a LU Complexity that is student-specific and set randomly for each student at the start of studying each LU at a value ranging from 1 to 4. Each Question belongs to a Corresponding Learning Unit, depending on the LU a student is currently studying. Furthermore it has a specific Question type, which is set at random when a question is asked, and it has a Question Status.
Processes
- 4.3
- In the simulation three main processes take place: studying, question asking and tutor selection. Note that the eventual peer support itself is not part of this simulation, since it only concentrates on the mechanism for peer allocation.
- Studying: at each time unit each student follows the study procedure, which is defined within the simulation as progress within a learning unit. At each time unit a student's progress within the LU he is currently studying is increased based on the following formula:
progress(i,t) = progress(i,t-1) + st(i) + cs(i) + pk(i)
in which:progress(i,t) = the current progress status of student i st(i) = the available study time of i cs(i) = the constraints of i at t pk(i) = the prior knowledge of i for the current LU If progress(i,t) >= LU size, a student proceeds to the next LU.
- Question asking: at each time unit students can ask questions. Whether they do so, depends on the following procedure: if (ask(i) + lucomp(i) > 4) and a random Boolean procedure results in true, then student i asks a question. In this procedure ask(i) is a student's general inclination to ask questions and LUcomp(i) is the LU complexity of the current LU for student i. When a question is asked, the model determines which LU the question is about based on the current LU student i is studying, and a question type is allocated to the question. Then the procedure tutor selection is executed.
- Tutor selection: In the tutor selection procedure the following steps are taken:
- For each student except for the student asking a question their candidate score is computed based on the SAPS algorithm. With six selection criteria in the current model, this is computed as follows:
score(i) = (s1(i) * w1) + (s2(i) * w2) + (s3(i) * w3) + (s4(i) * w4) + (s5(i) * w5) + (s6(i) * w6)
in which:
sj(i) = the selection criterion j for student i
wj = the weight for selection criterion j in the current simulation run - A list is produced with students that have a willingness of 3 or more, reflecting those students who would actually be willing to answer the question.
- From this list, the student with the highest candidate score is chosen as peer tutor. If this list remains empty, the question remains unanswered.
- Normally students' willingness to accept a question would be checked as he receives an invitation to answer a question after being selected as tutor. For model simplification this is now treated in opposite order, but we assume that that would lead to the same results.
- The willingness of the selected peer tutor is decreased by 1, simulating a dead time in which a tutor is not willing to answer new questions. Following that, at each time unit the willingness of a student with a willingness of less than 1 is increased by 0.25.
- For each student except for the student asking a question their candidate score is computed based on the SAPS algorithm. With six selection criteria in the current model, this is computed as follows:
- Studying: at each time unit each student follows the study procedure, which is defined within the simulation as progress within a learning unit. At each time unit a student's progress within the LU he is currently studying is increased based on the following formula:
- 4.4
- Figure 2 shows the interface section of the SAPS model in the NetLogo simulation environment.
Figure 2. Interface section of the SAPS NetLogo model.
The study
- 5.1
- In our study we conducted simulation runs with various parameter and variable values in the simulation model to test how the model reacts under various conditions following our expectations. In order to achieve sufficient stability in the results found, we replicated our simulation conditions in several runs to correct for measurement errors. We used an empirical method to arrive at the number of runs needed for this study. A random variable was chosen and the mean value of it was compared between various population sizes at an increasing number of runs. Above 100 runs no significant differences in outcomes were measured, so we decided to use this number for each condition. Below we describe the parameter and variable values and simulation runs we executed for each research question we had. All simulation runs had fixed values for the following model parameters: 90 days, 20 learning units.
- 5.2
- Research question 1: what is the minimum required population size at which a sufficient number of competent peer tutors are found who are also willing to answer?
Simulation runs: 500 (5 × 100) simulation runs with the following parameter values for population size: 10, 25, 50, 100, 200.
Weight of all selection criteria: 5. Note that the selection criteria 'question type' and 'previous result' are disabled in this part of the study. - 5.3
- Research question 2: does the introduction of extra quality selection criteria lead to an increased quality of the selection mechanism?
Simulation runs: 500 (5 × 100) simulation runs with the following parameter values for population size: 10, 25, 50, 100, 200.
Weight of all selection criteria: 5. In this part of the study, the selection criteria 'question type' and 'previous result' were enabled. - 5.4
- Research question 3: in what way does omitting the economy principles from the SAPS mechanism influence tutor load spread?
Simulation runs: 600 simulation runs with the following parameter values for the economy principles and population size: economy principles disabled/enabled, 50/500 students, 'willingness' variable enabled/disabled.
Weight of all selection criteria: 5. - 5.5
- Research question 4: In what way does disabling the economy principles influence the percentage of low-quality peers?
Simulation runs: 500 (5 × 100) simulation runs with the following parameter values for population size: 10, 25, 50, 100, 200.
Weight of all quality selection criteria: 5. In this part of the study, the economy principles were disabled to test their influence.
Results
-
Research question 1
- 6.1
- In order to test the first hypothesis two analyses were conducted. We first compared the percentage of cases in which a peer was selected with a score of 0 on the selection criterion 'proximity' (i.e. a low-quality peer) in each condition (i.e. number of students in the simulation run). Figure 3 shows the mean percentage of low-quality peers at various population sizes, and the lower and upper bounds of the 95% confidence interval of the means found in the simulation runs. Please note that the values on the x-axis are represented on a logarithmic scale, as they are in all the following result figures.
Figure 3. Mean percentage of low-quality peers at various population sizes and their confidence interval. - 6.2
- The results indicate that, as expected, the percentage of low-quality peers decreases when larger population sizes are used, ranging from 17.3% at a population of 10 students to 5% at a population size of 200. The confidence intervals of the means show that the criterion of no more than 10% of the questions being answered by low-quality peers is reached by population sizes slightly larger than 50 students.
- 6.3
- As the quality of the selection mechanism also depends on peers' willingness, we compared the percentage of questions that remained unanswered in each condition due to lack of willingness among the population to answer questions. Figure 4 shows the mean percentage of unanswered questions in each condition.
Figure 4. Mean percentage of unanswered questions at various population sizes and their confidence interval. - 6.4
- The results indicate that the percentage of unanswered questions decreases when larger population sizes are used, ranging from 18.2% at a population of 10 students to less than 5% at a population size of 200. When taking into account a 95% confidence interval of the means found in the simulation runs, we see that the upper bound of this interval is below 10% from 50 students or up. In other words, the model shows acceptable results with student populations of 50 students. The following tests are aimed to increase the model's effectiveness by further enhancing the outcomes.
Research question 2
- 6.5
- To test whether the introduction of extra selection criteria would enhance the selection quality of the model leading to more competent tutors being selected, we added 'question type' and 'previous result' as extra selection criteria (i.e. they were given the same weight (5) as the other selection criteria in the mechanism). Figure 5 shows the mean percentage of low-quality peers in simulation runs with the extra selection criteria enabled in the mechanism.
Figure 5. : Mean percentage of low-quality peers at various population sizes and their confidence interval in simulation runs with extra selection criteria enabled. - 6.6
- The percentage of low-quality peers decreases when larger population sizes are used, ranging from 15.2% at a population of 10 students to 3% at a population size of 200. The data showed that with the added criteria, the mean percentage of low-quality peers being selected is generally lower than in conditions in which only 'proximity' is used as a quality selection criterion, as shown in table 3. The mean difference is 2.2%, which is equal to 24.7% less low-quality peers when the extra selection criteria are enabled. At the same population size as the first part of the study, 50 students, the mean percentage of low-quality peers has decreased form 9.7% to 7.2%.
Table 3: Difference in mean percentage of low-quality peers at various population sizes. number-of-students Mean percentage with extra criteria disabled Mean percentage with extra criteria enabled Difference percentage-of-low-quality-peers 10 17.32 15.17 2.15 25 12.98 10.84 2.14 50 9.74 7.16 2.58 100 6.78 4.88 1.90 200 5.06 3.04 2.03 Mean difference 2.16 - 6.7
- When comparing the percentage of unanswered questions, no differences were found between conditions in which the criteria were disabled or enabled. This is logical since this percentage is influenced by the 'willingness' variable in the model, which was not altered.
Research question 3
- 6.8
- To test the third hypothesis, we examined the tutor load spreads in two conditions. During the first 100 runs the economy principles ('uniformity' and 'favour-in-return') were disabled, during the second 100 runs they were enabled. In both cases, the student population was made up of 50 students. Figure 6 shows the lowest, the highest and mean number of tutor turns of all students in both conditions.
Figure 6. Lowest, highest and mean number of tutor turns of all students with economy principles disabled and enabled respectively. - 6.9
- The average number of questions answered in both conditions is 674. Although the mean number of tutor turns is similar in both conditions (M=14 in condition 1, M=13 in condition 2), we did find a greater spread in the number of tutor turns in the condition with the economy principles disabled compared to the condition with the principles enabled. There are more students with lower numbers of tutor turns when the economy principles are turned off and the lowest number of tutor turns found was generally lower. Although we expected the same differences to be present among the students with higher number of tutor turns, we found that the maximum number of questions answered in both conditions was 16. With a mean total of 674 questions answered that means that the maximum number of questions a student receives is equal to 2% of all questions, which is acceptable. Also, the maximum number of tutor turns is equal in both conditions. However, since the maximum of tutor turns never exceeded 16 while at the same time we found a larger spread among the students with less tutor turns, the data clearly showed a ceiling effect in the maximum number of tutor turns. After examination of the data and the simulation model we found that this effect was caused by the willingness variable. In the simulation model, after accepting a tutor request a student will not accept any new request for a short period of time; a natural tutor dead time). The effect of this variable was exacerbated by the results of a test run in which the willingness variable was disabled. In a real-life setting this would mean that every student who receives a request accepts it and answers the question. In this case, the variance in the number of tutor turns is now much greater. There are students who do not have any tutor turns, while other students are overloaded with questions, while they receive as many as 134 of all requests (20% of all questions answered). In this scenario the enabling of the economy principles does not influence these results dramatically.
- 6.10
- To test whether larger population sizes would show other patterns in the data, we did the same procedure of 200 simulation runs with a population of 500 students. This showed results similar to those described above.
Research question 4
- 6.11
- To test whether the disabling of the economy principles would lead to an extra decrease of the percentage of low-quality peers, we compared the mean percentage of these simulation runs with the results found previously. Table 4 shows the results combined with the results from the previous executed runs. The mean difference in percentages between the last is 10%.
Table 4: Difference in mean percentage of low-quality peers at various population sizes with a) only 'proximity' enabled, b) with both 'proximity' and 'question type' enabled, and c) the same as b, but now with economy principles disabled. number-of-students Mean percentage with extra criteria disabled Mean percentage with extra criteria enabled Mean percentage with economy principles disabled percentage-of-low-quality-peers 10 17.32 15.17 14.29 25 12.98 10.84 9.79 50 9.74 7.16 6.50 100 6.78 4.88 3.90 200 5.06 3.04 2.74 - 6.12
- Figure 7 shows a graphical representation of the percentages of low-quality peers found in all simulation runs. It shows the quality gains achieved in each of the above-described setups of the allocation mechanism.
Figure 7. Difference in mean percentage of low-quality peers at various population sizes with a) only 'proximity' enabled, b) with both 'proximity' and 'question type' enabled, and c) the same as b, but now with economy principles disabled.
Conclusions and discussion
- 7.1
- The purpose of this study was to explore the feasibility of an allocation mechanism for online reciprocal peer support activities among groups of students working on the same modular material. To improve earlier approaches we developed the Synchronous Allocated Peer Support (SAPS) system. As a first step in examining whether the allocation mechanism of our system could work in practice, this simulation study was conducted.
- 7.2
- Quite generally, given our assumptions for the simulation model, a SAPS-based allocation mechanism should be able to facilitate online peer support activities among groups of students. The allocation mechanism holds over time and a sufficient number of students are willing and competent to answer fellow-students' questions.
- 7.3
- A more detailed look showed that the model reacts differently to various population sizes, and the results give an indication of the minimum population size needed to achieve acceptable results. We defined acceptable as values of lower than 10% on the percentage of low-quality peers as well as the percentage of unanswered questions. In a real-life setting, this would mean that one in ten questions asked remains unanswered or is answered by a peer who has not yet completed the question-specific course unit ('proximity') or who has not indicated to be competent at providing the type of support ('question type') needed. In such cases students could repost their question at a later stage. The study showed that a SAPS-based allocation mechanism operates properly from student populations of 50 or more. However, it should be noted that the more students are added, the more effective the mechanism becomes.
- 7.4
- The aim of the first part of the study was to arrive at a minimum required population size, the second part concentrated on the aim to further enhance the selection quality of the SAPS-based allocation mechanism. We found that introducing extra quality selection criteria increases the quality of the selection mechanism, since it decreases the percentage of low-quality peers. At the minimum required population size of 50 found previously we found that the percentage of questions that was allocated to a low-quality peer tutor decreased by 12.4%. When the number of students was increased, the difference in mean percentage of low-quality peers increased as well. We found a mean 25% decrease of low-quality peers being selected over all population sizes, far more than the 10% we expected. Therefore, we state that adding the extra criteria is recommended. It increases the chance to arrive at more competent peer tutors at fairly low cost, namely those of composing a list of common question types for a domain. This however assumes that such themes are easily defined and clear to all students when applied. Empirical testing should be able to show how this would work in practice.
- 7.5
- In the third part of the study we found that the omission of economy principles to distribute the tutor load evenly among the population did not influence the mean number of tutor turns, but that there occurs a difference in the spread of turns. The spread only occurred in one direction; there were more students with fewer tutor turns in the condition with the economy principles disabled, but the maximum number of tutor turns was similar in both cases. Although this would mean that the acceptable maximum of 50% more tutor turns for certain peer tutors was not exceeded, we found that a ceiling effect - caused by the way in which willingness to answer questions was implemented - influenced the results. Therefore we conducted additional simulation runs with the willingness variable left out of the simulation model. This lead to an overload of some of the peer tutors, since they received 20% of all questions. However, it is to be expected that in peer support activities willingness does play a significant role, so omitting it probably decreases model validity. To prevent complexity issues in the simulation model, willingness was defined as a relatively simple construct. In practice, willingness would be a much more complex construct, with aspects such as selective willingness, tit-for-tat and time constraints likely to be of influence. While it is hard to model such complex constructs within the boundaries of a simulation model, it would be interesting to test empirically how willingness works in practice and following that, if our current conclusion that the economy principles could be omitted actually hold.
- 7.6
- In the last part of the study we examined whether omitting economy principles from peer allocation systems would result in an extra selection quality gain. We examined whether disabling the economy principles in the SAPS-based allocation mechanism would lead to an extra decrease of the percentage of low-quality peers being selected. We argued that since omitting economy principles requires hardly any effort, a mean difference in the percentages of low-quality peers found compared to those found in the previous parts of the study would already be a sufficient gain. This turned out to be the case. We think this needs additional empirical testing. Further research could show whether this would indeed lead to having a more enthusiastic group of students who would be more willing to help each other, thus lowering the percentage of unanswered questions, and consequently lead to students being more satisfied with the answers they receive. This should however be combined with the suggested empirical research based on a more complex definition of willingness.
- 7.7
- Another focus for future research could be to look at different contexts. For this study we limited ourselves by modelling a set of linear modular material that consisted of 20 Learning Units and had a life cycle of 90 days. Future simulations should give insights to see what influence changing these characteristics would have. For example, since we expect the student population to become much more heterogeneous as time increases (e.g. the differences in study pace increase), additional runs could show if a SAPS-based allocation mechanism would work over a longer period of time. Also, to be able to serve educational material organised in a different way, it would be valuable to examine if such a mechanism could be applied to a set of non-linear learning materials.
- 7.8
- The SAPS allocation algorithm does not include didactic competences peers should have in order to be able to tutor fellow-students. A tutor that is competent in terms of the content of a course (e.g. by having completed the course module a tutee has a question about) does not necessarily have this competence. However, in two empirical studies we conducted we found that the majority of tutees' answers are sufficiently answered by peer tutors selected by the SAPS algorithm based on their content competence (De Bakker et al. 2010a; De Bakker et al. 2010b). Van Rosmalen et al. (2008) also found that the majority of tutees' answers were sufficiently answered by peer tutors selected via a similar allocation algorithm. In our view this is an indication of the didactical competence of peer tutors to answer fellow-students' questions.
- 7.9
- Van Rosmalen (2008) points to the importance of the community aspect of online peer support systems. The formation of ad-hoc transient communities could be used as starting points for the formation of longer lasting communities (Fetter, Berlanga & Sloep, in press ). Students in this way could be motivated to continue contact with their peers This offers them the opportunity to develop a more structural support relationship with fellow-students.
Acknowledgements
- The authors would like to thank Bert van den Berg of the Open University for his consulting activities during the development of the simulation model.
References
-
AINSWORTH, S E, & Loizou, A (2003) The effects of self-explaining when learning with text or diagrams. Cognitive Science, 27, 669-681. [doi:10.1207/s15516709cog2704_5]
ANDERSON, A, Cheyne, W, Foot, H, Howe, C, Low, J, & Tolmie, A (2000) Computer support for peer-based methodology tutorials. Journal of Computer Assisted Learning, 16, 41-53. [doi:10.1046/j.1365-2729.2000.00114.x]
DE BAKKER, G, Sloep, P and Jochems, W (2008) Towards a system for allocated peer tutoring via instant messaging: a pilot study. Paper presented at the ALT-C conference 2008, Leeds, UK.
DE BAKKER, G, van Bruggen, J, Sloep, P, Jochems, W, Schellens, T (2010a) The influence of synchronous online reciprocal peer support on answer quality, test performance and student satisfaction with peer support. Unpublished manuscript, Eindhoven University of Technology.
DE BAKKER, G, van Bruggen, J, Sloep, P, Jochems, W (2010b) The quality of online reciprocal peer support in a distance education context without teacher availability. Unpublished manuscript, Eindhoven University of Technology.
CHI, M T H, de Leeuw, N, Chiu, M H, & LaVancher, C (1994) Self-explanations improve understanding. Cognitive Science, 18, 439-477.
DE VRIES, F, Kester, L, Sloep, P, van Rosmalen, P, Pannekeet, K, and Koper, R (2005) Identification of critical time-consuming student support activities in e-learning. ALT-J, Research in Learning Technology 13(3), pp. 219-229. [doi:10.1080/09687760500376488]
FANTUZZO, J W, Dimeff, L A and Fox, S L (1989) Reciprocial peer tutoring: a multimodal assessment of effectiveness with college students. Teaching of Psychology 16(3), pp. 133-135. [doi:10.1207/s15328023top1603_8]
FETTER, S, Berlanga, A, & Sloep, P B (accepted) Using Ad-Hoc Transient Communities to Socialize a Learning Network: A Theoretical Framework and Model. International Journal of Learning Technology.
FOX, S, and MacKeogh, K (2003) Can e-learning promote higher-order learning without tutor overload? Open Learning 18(2), pp. 121-134. [doi:10.1080/02680510307410]
GYANANI, T C, & Pahuja, P (1995) Effects of peer tutoring on abilities and achievement. Contemporary Educational Psychology, 20, 469-475. [doi:10.1006/ceps.1995.1034]
KING, A, Staffieri, A, & Adelgais, A (1998) Mutual peer tutoring: effects of structuring tutorial interaction to scaffold peer learning. Journal of Educational Psychology, 90, 134-152. [doi:10.1037/0022-0663.90.1.134]
NADOLSKI, R J, van den Berg, B, Berlanga, A J, Drachsler, H, Hummel, H G K , Koper, R and Sloep, P (2009) Simulating Light-Weight Personalised Recommender Systems in Learning Networks: A Case for Pedagogy-Oriented and Rating-Based Hybrid Recommendation Strategies. Journal of Artificial Societies and Social Simulation 12(1)4 https://www.jasss.org/12/1/4.html.
RUMBLE, G (2001). The Costs and Costing of Networked Learning. Journal of Asynchronous Learning Networks 5, pp. 75-96.
SLAVIN, R E (1995) When does cooperative learning increase student achievement? Psychological Bulletin, 94, 429-445. [doi:10.1037/0033-2909.94.3.429]
SLOEP, P, Kester, L, Brouns, F, van Rosmalen, P, de Vries, F, de Croock, M and Koper, R (2007) Ad Hoc Transient Communities to Enhance Social Interaction and Spread Tutor Responsibilities. In Uskov, V (Ed.) The Sixth IASTED International Conference on Web-based Education WBE 2007, Calgary, Canada: Acta Press, pp. 549-554.
SLOEP, P, and Jochems, W (2007). De e-lerende burger [The e-learning citizen]. In Steyaert J and de Haan J (Eds.) Jaarboek ICT en samenleving 2007. Gewoon digitaal, Amsterdam: Boom, pp. 171-187.
SLOEP, P (2008). Netwerken voor lerende professionals [Networks for learning professionals]. Develop 2008(4), pp. 84 - 91.
VAN ROSMALEN, P (2008). Supporting the tutor in the design and support of adaptive e-learning, Heerlen: Open Universiteit Nederland.
VAN ROSMALEN, P, Sloep, P, Brouns, F, Kester, L, Koné, M, and Koper, R (2006) Knowledge matchmaking in Learning Networks: Alleviating the tutor load by mutually connecting learning network users. British Journal of Educational Technology 37(6), pp. 881-895. [doi:10.1111/j.1467-8535.2006.00673.x]
VAN ROSMALEN, P, Sloep, P, Kester, L, Brouns, F, de Croock, M, and Pannekeet, K (2008) A learner support model based on peer tutor selection. Journal of Computer Assisted Learning 24, pp. 74-86. [doi:10.1111/j.1365-2729.2007.00245.x]
WESTERA, W (2007) Peer-allocated instant response (PAIR): computational allocation of peer tutors is learning communities. Journal of Artificial Societies and Social Simulation 10(2)5 https://www.jasss.org/10/2/5.html.
WILENSKY, U (1999) NetLogo. Center for Connected Learning and Computer-Based Modeling, Northwestern University. Evanston, IL. http://ccl.northwestern.edu/netlogo/.
WONG, W K, Chan, T W, Chou, C Y, Heh, J S, & Tung, S H (2003) Reciprocal tutoring using cognitive tools. Journal of Computer Assisted Learning, 19, 416-428. [doi:10.1046/j.0266-4909.2003.00046.x]