(Go to the Stencils Project page)
(Try the Stencil Stacking exercise before reading further)
(Review Questionnaire Responses)
(Retrieve Your Results)

Understanding Stencils Stacking Behaviour

External Considerations

An accurate interpretation of Stencil Stacking behaviours requires that a user diligently complete the task. Obviously, distractions while completing the task will affect results. Recall the distractions if you can, or be aware of them when you complete the Stencils Stacking task to help understand how they affect your results. Also be aware of network "gliches" in which your connection to the Internet may waiver. They will also affect your results (see the netlag column in Table 2)

It is also important to understand how prior knowledge affects the possible outcomes of the exercise. Since the possible outcomes are easily determined by reading about them here, it is not difficult to manipulate results. In fact, complete the exercise a second time, keeping in mind what you learned here about metacognition, and compare your results to the first time through. Your goal here, in learning about reflective behaviour, should be to understand, or experience them and apply them in other learning situations.

Viewing Summary and Raw Results

The primary result in the Stencils Stacking Task is termed "post-failure reflectivity". Post-failure reflectivity represents the metacognitive behaviour of attending to errors. Two other metacognitive behaviours can also be demonstrated in the task though they tend to be less consistent than post-failure reflectivity. These other behaviours are termed pre-failure reflectivity and difficulty judgement. The three behaviours are described in more detail below.

Table 1 displays the summary results for User 1925. This user did well with 86% (above average) of the items correct. The distribution of reaction times that appear below the percentage is what might be expected of a user who has diligently completed the exercise. Reaction times between individuals will vary greatly; some people are better than others at this type of task and will be faster overall. In demonstrating metacognitive behaviour however, overall speed is not important; it is the ratios between reaction times that are used to measure metacognitive behaviour.

Table 2 displays the raw results for User 1925 for each of the 60 individual items in the exercise. Studying the raw results can reveal some interesting patterns that help explain the summary results. One such pattern is termed the "first mistake effect". Almost invariably users will take longer to reflect on their first mistake than any of their subsequent mistakes.

It is interesting to question users when they are reflecting on their mistakes to see what they were thinking. The questionnaire following the task, though not nearly as effective as a face-to-face semi-structured interview, asks users about their thinking while doing the task and provides a rich source of information about how people think "metacognitively". Table 3 includes typical responses to the questions in the questionnaire, with a variety of references to metacognitive thought (e.g. reflection, decision making, desire, discovery, disbelief)

Metacognitive Measures:

Of the six measures that follow, those for failed items are of interest. Those for passed items are provided for contrast, helping us understand where or when metacognitive behaviour occurs. The reflective thought associated with failure is the metacognitive behaviour you are learning about here. There are many other forms of metacognitive behaviour. Of the three failure measures described, post-failure reflectivity is the one we are most interested in. It can be thought of as the behaviour of "learning from one's mistakes." One consistent finding in studies that have investigated post-failure reflective behaviour is that those who spend more time attending to errors also perform better overall on a variety of tasks than those who are less attentive to their mistakes (they're better learners).

Error Behaviours:

Pre-Failure Reflectivity: Pre-failure reflectivity is the ratio between the average time it takes to answer failed items and the average time it takes to answer items overall. Pre-failure reflectivity greater than 1.0 suggests reflective behaviour prior to making a mistake. In some cases users do reflect before a failure, but many do not. The user in Table 1 does have a pre-failure reflectivity ratio higher than 1.0, but only moderately higher, suggesting there was a little metacognitive activity occurring before making mistakes. These reflective behaviours might include delaying in anticipation of negative feedback , or delaying because of uncertainly or indecisiveness.

(Pre-failure reflectivity = average reaction time on failed items / average reaction time on all items)

Post Failure Reflectivity: Post-failure reflectivity is the ratio between the average time it takes to "continue" following failed items and the average time it takes to "continue" overall. Post-failure reflectivity greater than 1.0 suggests reflective behaviour after making mistakes. In most cases users do reflect following failures. The user in Table 1 has a post-failure reflectivity ratio much higher than 1.0, suggesting that this user was paying particularly close attention to errors.

(Post-failure reflectivity = average post reaction time on failed items / average post reaction time on all items)

After making a mistake users will often indicate that they were reviewing the items to determine where they went wrong, or were trying to confirm that they had in fact made a mistake. Notice the large difference between post-failure reflectivity and post-success reflectivity ratios, indicating the user was much more reflective on failed items than on passed items. Reviewing the raw results in Table 2 shows that in almost all cases post-reaction times following failed items (i.e. postt) were longer than post-reaction times following passed items.

Post-Failure Difficulty Judgement: Post-failure difficulty judgement is the ratio between the average time it takes to answer "Easy", "So-So", or "Difficult" following failed items and the average time it takes to make the judgement overall. Difficulty judgements greater than 1.0 suggest reflective behaviour when judging the difficulty of failed items. In some cases users do reflect before judging the difficulty of failed items, but many do not. The user in Table 1 does exhibit reflective behaviour when judging the difficulty of the failed items, though this ratio is only moderate.

(Post-failure difficulty judgement = average judgement time on failed items / average judgement time on all items)

A review of the raw results in Table 2 shows that in most cases difficulty judgement reaction times (posttt) were longer for failed items. In some instances however this user took a relatively long time to make the difficulty judgements on passed items as well. Questioning the user about his or her thoughts during the delayed judgments on passed items (i.e. What were you thinking while …?) would likely provide interesting information about the metacognitive thinking that was occurring. Interestingly their appears to be an interaction between post-failure reflectivity and difficulty judgements, with those having higher post-failure reflectivity ratios taking longer to judge on failed items and those with lower post-failure reflectivity ratios taking longer to judge on passed items. It's not yet clear why this interaction occurs.

Success Behaviours:

Pre-Success Reflectivity: Pre-success reflectivity is the ratio between the average time it takes to answer passed items and the average time to answer items overall. Pre-success reflectivity greater than 1.0 suggests reflective behaviour before answering an item correctly. In most cases users do not reflect before a success, and the ratio is not significantly different than 1.0.

Post-Success Reflectivity: Post-success reflectivity is the ratio between the average time it takes to "continue" on passed items and the average time to "continue" overall. Again most users do not reflect following successes, but some do. Instances where a user might reflect following a success could be to give themselves a "pat on the back", or to review the item if they got it right by guessing.

Post-Success Difficulty Judgement: Post-success difficulty judgement is the ratio between the average time it takes to answer Easy, So-So, or Difficult following passed items and the average time it takes to make the judgement overall. In many cases users reflect less when judging the difficulty of passed items, though some do take longer as mentioned above. Again, questioning the user's about his or her thoughts at the time would likely provide interesting information about the metacognitive thinking that was occurring.

Table 1: Summary Results

Table 2: Raw Results

Retrieving Metacognitive Thoughts.

The questionnaire that follows the Stencil Stacking task attempts to retrieve the metacognitive thinking that was occurring while completing the exercise. An online form like that used for the questionnaire is quite inefficient for collecting this type of information, compared to asking a person directly, but it does gather an interesting summary of the conscious metacognitive thinking that occurs during the task. Metacognitive thinking in general, even when asking directly, is often difficult to retrieve. Most people have only a vague recollection of what they were thinking while doing the task, which is why metacognitive behaviour has been so difficult to study.

Table 3 includes the questions from the questionnaire and a typical set of responses provided by user 1925. Question 1 attempts to help us understand what happens when a user makes their first mistake. As mentioned earlier, users will almost always take longer to reflect on their first mistake than on all subsequent mistakes. In the case of user 1925, s/he did spend a relatively long time reflecting after the first mistake but also spent a long time reflecting on several other failed items (see the p/f and postt columns in Table 2)

The three questions that follow (2, 3, 4 ) asks users to recall their thinking as they were completing the task, and after answering items correctly and incorrectly. Words such as "decided", "careful", and "desire" suggest that metacognitive thought was occurring throughout the task. In interpreting these words it is important to understand that retrieving memory of metacognitive thinking is difficult for most and often inaccurate for those who do recall. Although this user indicated various metacognitive thoughts during the task, it is very common for people to answer "I don't know."

Question 5 attempts to retrieve the thinking that was occurring when judging the difficulty of items. People generally provide a response for this question, often indicating time, or pass/fail as a criteria for judging difficulty.

Question 6 attempts to retrieve emotional behaviour that was occurring during the task. Emotions will emerge while completing the task. Many will feel exhilaration when they do well, and discomfort when they fail items. It is helpful to understand how emotions affect learning. Positive emotional arousal often aids learning while negative emotional arousal often impedes learning. Understand that negative feedback does not necessarily evoke negative emotional arousal. Many see negative feedback as a challenge, or an opportunity to learn.

Question 7 attempts to retrieve the user's understanding of "learning from one's mistakes." User 1925 seems to have a good understanding of the strategy. S/he was able to identify specific patterns in the errors s/he made, remembered and applied those patterns, and knew that s/he had learned from her mistakes. In contrast to post-failure reflectivity, which is a metacognitive behaviour, knowing that s/he had learned from her mistakes is metacognitive knowledge. Knowing about metacognitive behaviour has a strong effect on whether these behaviours are applied or not. Knowing about the metacognitive behaviours you are learning about here will increase the likelihood you'll use them in everyday learning.

Finally Question 8 asks the user to judge the overall difficulty of the task. A comparison of pass percentage and this overall difficulty judgement is another indicator of metacognitive knowledge. Those who have more metacognitive knowledge about their own abilities and the characteristics of tasks that challenge those abilities will more accurately judge the overall difficulty of the task. That is, they will perform well and find the task easy, or perform poorly and find the task difficult. It is certainly possible to perform poorly, but be a good learner.

After reviewing your questionnaire responses from the first attempt of the Stencil Stacking task, you may wish to complete the task again, paying closer attention to your thoughts while doing the task a second time, and completing the questionnaire with a greater insight into your thinking.

Table 3: Typical Questionnaire Responses

Conclusion

The goal of the Stencil Stacking exercise has been to understand by experience the behaviours associated with learning from ones mistakes. When failure occurs, learning often occurs right after. Learners use a wide variety of strategies to understand where they went wrong, and learn from that so future failures are prevented. This reflective behaviour is a key behaviour across many learning situations. By experiencing the "feeling" of reflection, your first goal should be to recognize when it happens. Your second goal should be to apply that reflective behaviour to other learning situations.

Appendix: Stencils A through L

A	B	C	D	E	F
G	H	I	J	K	L

Further Reading

• Shafrir, U. (1995). Computer-Based Testing of Reflective Thinking: Executive control of Erroneous Performance in 9 to 12 Year Old Children. In Y. Anzai, K. Ogawa, and H. Mori (Eds), Symbiosis of Human and Artifact. New York: Elsevier Sciences B. V. (pp. 437-442)
• Shafrir, U. & Eagle, M. (1995). Response to Failure, Strategic Flexibility, and Learning. International Journal of Behavioral Development, 18(4), 677-700.
• Shafrir, U., Ogilvie, M., & Bryson, M. (1990). Attention to Errors and Learning: Across-Task and Across-Domain Analysis of the Post-failure Reflectivity Measure. Cognitive Development, 5, 405-425.
• Shafrir, U. and Pascual-Leone, J. (1990). Post-Failure Reflectivity/Impulsivity and Spontaneous Attention to Errors. Journal of Educational Psychology, 2, 378-387.
• Shafrir, U. & Siegel, L. S. (1994). Preference for Visual Scanning Strategies Versus Phonological Rehearsal in University Students with Reading Disabilities. Journal of Learning Disabilities, 27 (9), 583-588.
• Shafrir, U. and Siegel. L. S. & Chee, M (1990). Learning Disability, Inferential Skills, and Post-failure Reflectivity. Journal of Learning Disabilities, 23(3), 506-517.