Explore QTI in depth

Learn about response and result processing for test questions

The IMS Question and Test Interoperability (QTI) specification is a widely used standard for marking up questions and other learning material. Look at the response and result processing for a QTI item and learn what happens when a user answers a question.

Erik Siegel, XML Specialist, Xatapult

Photo of Erik SiegelErik Siegel works as a self-employed XML specialist based in The Netherlands. His career has involved many jobs—researcher, programmer, systems analyst, project manager, systems architect, consultant. During the past five years, XML crept in. His main customers are in publishing, and his XML activities include consulting, training, schema development, and XSLT programming. You can find more information about Erik and his company at www.xatapult.com.



17 January 2012

Also available in Chinese Russian Japanese

The most widely used standard for marking up assessments, tests, and individual questions in XML is currently IMS Question and Test Interoperability (QTI). IMS is an e-learning standards-related organization (see Resources for a link) that works on standards ranging from metadata to transferring learner information. QTI standardizes how to mark up questions in XML, arrange them into tests and assessments, add metadata, and package everything in a ZIP file.

Frequently used acronyms

  • URI: Uniform resource identifier
  • XHTML: Extensible Hypertext Markup Language

QTI XML not only describes the static parts of questions, assessments, and tests (text, layout, order, and so on) but also their dynamic behavior—how, for instance, to behave when the user gives a wrong answer, when to provide certain feedback, and how to calculate the score. You can make this process as complicated as you like: QTI provides you with programming facilities to make it all happen.

Unfortunately, how it all works is a bit of a puzzle. The documentation is structured around defining the standard, not explaining it. Some examples are accompanied by a helpful document (the IMS Question and Test Interoperability Implementation Guide—see Resources for a link), but this still doesn't provide much detail about what goes on inside.

This article explores the response and result processing within QTI's basic building block: the assessment item. How do you rate a user's response, and what do you do with it in terms of scoring?

QTI assessment items

If you are unfamiliar with QTI, this section provides a short overview of assessment items.

Assessment items (or in short, items ) are the basic building blocks of QTI. An item can consist of an arbitrarily complex combination of text (with markup), multiple questions of different types, side information panels, multimedia objects, and more. It also defines its own result and response processing—when the answer is considered correct, scoring, and feedback.

In practice, QTI assessment items are usually not complex. The majority I've seen contain only a single question. Listing 1 shows the anatomy of a QTI assessment item.

Listing 1. Anatomy of a QTI item
<assessmentItem xmlns="http://www.imsglobal.org/xsd/imsqti_v2p1" ...>

<responseDeclaration ...> 
   <!-- Information about the answers to the questions inside -->
</responseDeclaration>

<outcomeDeclaration ...> 
   <!-- Internal variables for computing scores, controlling feedback, etc. -->
</outcomeDeclaration> 

<itemBody ...> 
   <!-- Text and the interactive parts (questions) --> 
</itemBody> 

<responseProcessing ...> 
   <!-- How to handle the user's response --> 
</responseProcessing> 

<modalFeedback ...> 
   <!-- Optional feedback messages to the user --> 
</modalFeedback> 

</assessmentItem>

Here's a breakdown of the markup:

  • A <responseDeclaration> contains information about the answer (the response) to a question: When is it correct, and (optionally) how is it scored? A <responseDeclaration> is required for every separate question in the <itemBody>.
  • An <outcomeDeclaration> defines an internal variable—for instance, for returning the score to the environment or for controlling the feedback to the user.
  • The single <itemBody> element contains the text, markup, and questions for this item. The markup has a remarkable similarity with XHTML, but beware: Although you see familiar elements like <p> and <ul>, everything is in the QTI namespace (http://www.imsglobal.org/xsd/imsqti_v2p1) and not in the XHTML namespace. The standard does not prescribe that you have to render it with a browser, which leaves a lot of room for interpretation. So, consistent rendering is a major problem when exchanging content between different QTI processing engines.
  • The <responseProcessing> section describes the actions to perform when the student is finished with the item. Its content is a simple programming language in which you can perform computations, set <outcomeDeclaration> variables, and more.
  • Finally, <modalFeedback> defines the item's feedback to the student (for instance, display "Well done" when the answer is correct).

This article is about the <responseDeclaration>, <outcomeDeclaration>, and <responseProcessing> sections.

Interaction types

An <itemBody> element in an item can contain zero or more interactions with the user. Most of these interactions are typical questions (for example, multiple choice, order elements, fill in the blanks). You can also add interactions like "upload document," "draw a picture," and "start a film."

Listing 2 provides an example of a multiple-choice question.

Listing 2. Multiple-choice interaction example
<itemBody> 
   <choiceInteraction responseIdentifier="RESPONSE" shuffle="true" maxChoices="1">
      <prompt>What is the top color of the Dutch flag?</prompt> 
      <simpleChoice identifier="A">Red</simpleChoice> 
      <simpleChoice >="B">Orange</simpleChoice> 
      <simpleChoice identifier="C">Black</simpleChoice> 
      <simpleChoice identifier="D">White</simpleChoice> 
   </choiceInteraction> 
</itemBody>

And Listing 3 offers an example of an ordering question.

Listing 3. Ordering interaction example
<itemBody>
   <orderInteraction responseIdentifier="RESPONSE" shuffle="false">
      <prompt>Order these cities from north to south:</prompt> 
      <simpleChoice identifier="A">Tripoli</simpleChoice> 
      <simpleChoice identifier="B">Stockholm</simpleChoice> 
      <simpleChoice identifier="C">Paris</simpleChoice> 
   </orderInteraction> 
</itemBody>

QTI v2.1 defines 20 interaction types. You can find a full set of examples in the IMS Question and Test Interoperability Implementation Guide. You determine which types to use, but note that not all QTI processing engines support all interaction types.

As you can see in the examples, an interaction has a responseIdentifier attribute that binds the interaction to a <responseDeclaration> with the same identifier. This is the link for the result and response processing, and I come back to it later.


The QTI data model

An important prerequisite for understanding QTI's response and result processing is understanding its data model and the reasons behind its apparent complexity.

When a user is finished with a QTI item, the answers need to be checked. For this, answers are represented as data. Sometimes, the data is simple: A multiple-choice, single-answer question just returns the identifier of the given answer, and a numeric slider interaction returns a number. Most interaction types, however, are more complicated. For instance:

  • Multiple-choice, multiple-answer questions return a set of answer identifiers (for example, "The identifiers of the answers checked were A, C, and D.").
  • Ordering things returns an ordered set of answer identifiers (for example, "The identifiers of the sequence created were C, A, and B.").
  • Questions in which you associate words with each other return a set of identifier pairs (for example, "The words were associated as (A, C), (B, D), and (E, F).").
  • Positioning objects on a map returns a set of points (for example, "The objects were placed at locations (102, 10), (125, 653).").

Deciding whether an answer is right or wrong is done through the <responseDeclaration> bound to the interaction. This means that it must have an appropriate data type for the interaction. For instance, Listing 4 provides a declaration for a <positionObjectInteraction>, a question type in which you have to position objects on a graphic (for example, a map). Its data type is a set of points.

Listing 4. A response declaration with a set of points
<responseDeclaration identifier="RESPONSE" baseType="point" cardinality="multiple">
   <correctResponse> 
      <value>118 184</value> 
      <value>150 235</value> 
      <value>96 114</value> 
   </correctResponse> 
</responseDeclaration>

QTI data types are defined by the baseType and cardinality attributes on <responseDeclaration> and <outcomeDeclaration> elements:

  • The baseType attribute defines the basic data type:
    • boolean
    • directedPair
    • duration
    • file
    • float
    • identifier
    • integer
    • pair
    • point
    • string
    • uri
  • The cardinality attribute defines the number of baseType elements and whether the set is ordered. Values are:
    • multiple
    • ordered
    • record (a special, rather complicated type not discussed in this article)
    • single

Response and result processing

Response and result processing takes care of what happens when the user is finished with the item. The QTI processor must perform tasks such as deciding whether the answer is correct, computing a score, and providing feedback. The <responseDeclaration>, <outcomeDeclaration>, and <responseProcessing> sections all have a role in this process.

Response declarations

The identifier of a <responseDeclaration> element always binds it to an interaction in the item's body. Listing 5 provides an example.

Listing 5. Binding an interaction to its response declaration
<responseDeclaration identifier="QUESTION1" cardinality="single"
                     baseType="identifier"> 
   <correctResponse> 
      <value>B</value> 
   </correctResponse> 
</responseDeclaration> 
<itemBody> 
   <choiceInteraction responseIdentifier="QUESTION1" shuffle="false" maxChoices="1">
      <prompt>Is a goldfish a mammal?</prompt> 
      <simpleChoice identifier="A">Yes</simpleChoice> 
      <simpleChoice identifier="B">No</simpleChoice> 
   </choiceInteraction> 
</itemBody>

A <responseDeclaration> is not a simple variable: It's better to think about it as an object in the object-oriented sense. This object has three "methods" that you can use in computations in the <responseProcessing> section of the item:

  • Get the learner's response to the question.
  • Get the correct (optimal) response to the question.
  • Map different answers to points so that an answer can have more nuances than plain right or wrong.

Listing 6 provides an example of a <responseDeclaration> in which a score is calculated depending on the answers given.

Listing 6. Response declaration that calculates a score
<responseDeclaration identifier="QUESTION1" cardinality="multiple"
                     baseType="identifier"> 
   <correctResponse> 
      <value>A</value> 
      <value>D</value> 
   </correctResponse> 
   <mapping defaultValue="0" lowerBound="0" upperBound="1"> 
      <mapEntry mapKey="A" mappedValue="0.5"/> 
      <mapEntry mapKey="B" mappedValue="-0.5"/> 
      <mapEntry mapKey="C" mappedValue="-0.5"/> 
      <mapEntry mapKey="D" mappedValue="0.5"/> 
   </mapping> 
</responseDeclaration> 
<itemBody> 
   <choiceInteraction responseIdentifier="QUESTION1" shuffle="false" maxChoices="0">
      <prompt>Which are colors?</prompt> 
      <simpleChoice identifier="A">Red</simpleChoice> 
      <simpleChoice identifier="B">Small</simpleChoice> 
      <simpleChoice identifier="C">Soft</simpleChoice> 
      <simpleChoice identifier="D">Purple</simpleChoice> 
   </choiceInteraction> 
</itemBody>

So, if the learner provided the answers A (red), B (small), and D (purple), the score would be calculated according to the <mapping> as (A) 0.5 + (B) -0.5 + (D) 0.5 = 0.5.

You might wonder why this process is all done inside a <responseDeclaration>. Determining whether an answer is correct and calculating scores can also be done in the <responseProcessing> section. There seems to be no direct need to declare correct answers and scores when you can also compute them (or vice versa).

The reason behind this is that QTI also wants the standard to be applicable for simple situations—for instance, a rendering engine that simply ignores the response processing and just looks at the defined correct responses. The same reasoning applies for the predefined response processing templates.

Outcome declarations

An <outcomeDeclaration> is the QTI equivalent of a variable. Listing 7 provides an example.

Listing 7. Example of an outcome declaration
<outcomeDeclaration identifier="SCORE" cardinality="single" baseType="float">
   <defaultValue> 
      <value baseType="float">0.5</value> 
   </defaultValue> 
</outcomeDeclaration>

This markup defines a variable called SCORE, with a single float in it and a default value of 0.5. Providing a default is optional. If you don't, numeric variables are initialized to 0.0 and non-numeric variables to NULL.

Note:SCORE is a reserved name. You use it to provide a numeric score representing the candidate's overall performance on the item to the environment.

You can add more intelligence to an <outcomeDeclaration> by providing a structure called <matchTable> or <interpolationTable>. I haven't seen them used, and most rendering engines probably don't support them, so I don't explain them here.

Response processing

The <responseProcessing> section of an item contains instructions for the rendering engine about what to do after the user has provided an answer. The instructions can be in two formats:

  • A program or script in a simple, XML-based programming language
  • A reference to a predefined template that is also a script in the response processing programming language (Because the template is predefined, rendering engines can hard-code it and, if you stick to this option, you don't need to provide a full script interpreter.)

The script language

Let's first examine the response processing script. Listing 8 provides an example of such a script.

Listing 8. Simple example of response processing
<responseDeclaration identifier="RESPONSE_01" cardinality="single" 
                     baseType="string"> 
   <correctResponse> 
      <value>white</value> 
   </correctResponse> 
</responseDeclaration> 
<responseDeclaration identifier="RESPONSE_02" cardinality="single" 
                     baseType="string"> 
   <correctResponse> 
      <value>green</value> 
   </correctResponse> 
</responseDeclaration> 

<outcomeDeclaration identifier="SCORE" cardinality="single" baseType="float"/>

<itemBody> 
   <p>Fill in the right colors:</p> 
   <p>Snow is 
      <textEntryInteraction responseIdentifier="RESPONSE_01" expectedLength="5"/> 
      and grass is 
      <textEntryInteraction responseIdentifier="RESPONSE_02" expectedLength="5"/>.
   </p> 
</itemBody> 

<responseProcessing> 
   <responseCondition> 
      <responseIf> 
         <and> 
            <match> 
               <variable identifier="RESPONSE_01"/> 
               <correct identifier="RESPONSE_01"/> 
            </match> 
            <match> 
               <variable identifier="RESPONSE_02"/> 
               <correct identifier="RESPONSE_02"/> 
            </match> 
         </and> 
         <setOutcomeValue identifier="SCORE"> 
            <baseValue baseType="integer">1.0</baseValue> 
         </setOutcomeValue> 
      </responseIf> 
      <responseElse> 
         <setOutcomeValue identifier="SCORE"> 
            <baseValue baseType="integer">0.0</baseValue> 
         </setOutcomeValue> 
      </responseElse> 
   </responseCondition> 
</responseProcessing>

This response processing examines the values of the learner's two responses to the defined correct (optimal) responses. This is done in the <match> elements. Both must be correct, which is handled by the surrounding <and> element. If they are, the SCORE <outcomeDeclaration> is set to 1.0; otherwise, it is set to 0.0. Listing 9 offers another, somewhat more complicated, example.

Listing 9. More complicated example of response processing
<responseDeclaration identifier="RESPONSE" cardinality="multiple" 
                     baseType="identifier"> 
   <correctResponse> 
      <value>A</value> 
      <value>C</value> 
   </correctResponse> 
   <mapping lowerBound="0.0" upperBound="1.0" defaultValue="0.0"> 
      <mapEntry mapKey="A" mappedValue="0.5"/> 
      <mapEntry mapKey="B" mappedValue="-0.25"/> 
      <mapEntry mapKey="C" mappedValue="0.5"/> 
      <mapEntry mapKey="D" mappedValue="-0.25"/> 
   </mapping> 
</responseDeclaration> 

<outcomeDeclaration identifier="SCORE" cardinality="single" baseType="float"/>
<outcomeDeclaration identifier="FEEDBACK" cardinality="single" baseType="identifier"/>
<outcomeDeclaration identifier="FEEDBACK_TRESHOLD" cardinality="single" 
                    baseType="float"> 
   <defaultValue> 
      <value baseType="float">0.75</value> 
   </defaultValue> 
</outcomeDeclaration> 

<itemBody> 
   <choiceInteraction responseIdentifier="RESPONSE" shuffle="true" maxChoices="0">
      <prompt>Which countries are south of the USA?</prompt> 
      <simpleChoice identifier="A">Mexico</simpleChoice> 
      <simpleChoice identifier="B">Canada</simpleChoice> 
      <simpleChoice identifier="C">Brazil</simpleChoice> 
      <simpleChoice identifier="D">Norway</simpleChoice> 
   </choiceInteraction> 
</itemBody> 

<responseProcessing> 
   <responseCondition> 
      <responseIf> 
         <isNull> 
            <variable identifier="RESPONSE"/> 
         </isNull> 
         <setOutcomeValue identifier="SCORE"> 
            <baseValue baseType="float">0.0</baseValue> 
         </setOutcomeValue> 
         <setOutcomeValue identifier="FEEDBACK"> 
            <baseValue baseType="identifier">FAILURE</baseValue>
         </setOutcomeValue>
      </responseIf>
      <responseElse>
         <setOutcomeValue identifier="SCORE">
            <mapResponse identifier="RESPONSE"/> 
         </setOutcomeValue> 
         <responseCondition> 
            <responseIf>
               <gte> 
                  <variable identifier="SCORE"/> 
                  <variable identifier="FEEDBACK_TRESHOLD"/>
               </gte> 
               <setOutcomeValue identifier="FEEDBACK"> 
                  <baseValue baseType="identifier">ANSWER_CORRECT</baseValue>
               </setOutcomeValue>
            </responseIf>
            <responseElse>
               <setOutcomeValue identifier="FEEDBACK">
                  <baseValue baseType="identifier">FAILURE</baseValue>
               </setOutcomeValue>
            </responseElse> 
         </responseCondition>
      </responseElse> 
   </responseCondition> 
</responseProcessing>

Besides setting a score, this response processing also sets an <outcomeDeclaration> called FEEDBACK to either FAILURE or ANSWER_CORRECT. You can use this declaration to provide the right feedback to the user.

Response processing follows these steps:

  1. Determine whether an answer is present (maybe the user clicked Continue without providing an answer). If so, the <responseDeclaration> value is NULL (checked by the <isNull> element), and SCORE and FEEDBACK are set to the right values.
  2. Set the SCORE to the value provided by the <mapping> from the <responseDeclaration (using the <mapResponse> element).
  3. Compare the SCORE to a fixed-value set by the FEEDBACK_TRESHOLD <outcomeDeclaration> (in the <gte> [greater than or equal] element) and sets the FEEDBACK accordingly.

Besides your own declarations, you can use some predefined variables in the response processing:

  • duration. The time the user spent answering the question in seconds. Use this variable, for instance, to lower the score when it took the user too long to provide an answer.
  • nunAttempts and completionStatus. These variables are handled in the section about adaptive processing.

If you're familiar with programming or scripting, writing response processing is not complicated. You can find a full definition of the language in the QTI document, IMS Question and Test Interoperability Assessment Test, Section, and Item Information Model (see Resources for a link). As you can see, because of the XML, it is all rather verbose, and you do have to write a lot of code to produce meaningful actions. Unfortunately, that's the standard.

Predefined response processing templates

As I already touched on in Response declarations, the QTI standard has built-in options for simple response processing. For instance, by using the correct response in a <responseDeclaration>, it can determine whether a given answer is right or wrong without doing any scripted response processing at all. This approach allows simple content and accompanying rendering engines without violation of the standard.

Another simplifying feature is the response processing templates. QTI predefines three templates for basic response processing. The templates are identified by fixed URIs—for instance:

<responseProcessing 
   template="http://www.imsglobal.org/question/qti_v2p0/rptemplates/match_correct"/>

Actual scripts are coupled to these URIs. A rendering engine can handle them as scripts should. To help rendering engines find the right template, you can provide a location such as:

<responseProcessing template="..." templateLocation="/rptemplates/match_correct.xml/>

You can use the predefined templates if:

  • The action includes only one interaction
  • The <responseDeclaration> for this action is called RESPONSE
  • The action includes an <outcomeDeclaration> called SCORE of type float

The predefined templates are:

  • Match_Correct: Set the score to 0.0 if the answer is absent or wrong and to1.0 if it is right. The URI is:
    http://www.imsglobal.org/question/qti_v2p0/rptemplates/match_correct
  • Map_Response: Set the score to 0.0 if there was no answer; otherwise, use the mapping inside the RESPONSE <responseDeclaration>. The URI is:
    http://www.imsglobal.org/question/qti_v2p1/rptemplates/map_response
  • Map_Response_Point: This is the same as Map_Response but for graphic interaction types. It uses area mapping instead of identifier mapping. The URI is:
    http://www.imsglobal.org/question/qti_v2p1/rptemplates/map_response_point

You can express all predefined templates in the response processing scripting language. If you download the QTI examples, the template scripts are included. For instance, Listing 10 shows the code for Map_Response.

Listing 10. Script for the predefined template Map_Response
<responseProcessing> 
   <responseCondition> 
      <responseIf> 
         <isNull> 
            <variable identifier="RESPONSE"/> 
         </isNull> 
         <setOutcomeValue identifier="SCORE"> 
            <baseValue baseType="integer">0</baseValue> 
         </setOutcomeValue> 
      </responseIf> 
      <responseElse> 
         <setOutcomeValue identifier="SCORE"> 
            <mapResponse identifier="RESPONSE"/> 
         </setOutcomeValue> 
      </responseElse> 
   </responseCondition> 
</responseProcessing>

You don't have to keep to these QTI predefined templates. In a situation where the content producer and rendering engine builder are tightly coupled, nothing stops you from defining your own.

Adaptive and non-adaptive processing

Up to this point, I have talked about the response processing of a QTI item, such as a one-time learner interaction with the question: The question is presented, an answer is given, the score is computed. QTI calls this exam format non-adaptive items. In another situation, you want to teach something to the learner and use questions as the mechanism for doing so. In this case, the learner usually has more than one opportunity to answer the question. More detailed feedback might be given ("This particular answer is wrong," "I have a tip for you: …"), and you can adjust the score based on the number of attempts.

For this situation, QTI defines adaptive items. An adaptive item is identified by setting the adaptive attribute on the root element to true(), as in Listing 11.

Listing 11. Making an item adaptive by setting the root element's adaptive attribute
<assessmentItem adaptive="true" … >
 … 
</assessmentItem>

In an adaptive item, the interaction with the learner no longer ends by default after the answer is provided. Instead, it can end in two ways:

  • By something defined in the rendering engine (an explicit Stop button, a time-out, and so on)
  • The response processing explicitly setting a built-in variable called completionStatus

The variable completionStatus has four possible values of type identifier: completed, incomplete, not_attempted, and unknown. If you set it to completed, the interaction will end.

Another built-in variable that is handy for adaptive items is numAttempts . As you might expect, this variable simply counts the number of attempts the user has made to answer the item. For instance, Listing 12 shows a response processing fragment that ends the interaction after the user has made more than four attempts.

Listing 12. Using numAttempts to set the completion status
<responseIf> 
   <gt> 
      <variable identifier="numAttempts"/> 
      <baseValue baseType="integer">4</baseValue> 
   </gt> 
   <setOutcomeValue identifier="completionStatus"> 
      <baseValue baseType="identifier">completed</baseValue>
   </setOutcomeValue> 
</responseIf>

Conclusion

This article touched on the way a QTI rendering engine performs (or should perform: good rendering engines are rare) response and result processing—what should happen after the learner answers a question. In most cases, a score is set, but you can also use it for other purposes.

If you don't want to or don't have to implement the full QTI response and response processing, you can still stay within the standard and fall back on comparing the given correct responses or use the built-in response processing templates.

Resources

Learn

Get products and technologies

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into XML on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=XML, Industries
ArticleID=785444
ArticleTitle=Explore QTI in depth
publish-date=01172012