HOW TO WRITE TEST QUESTIONS
MULTIPLE CHOICE QUESTIONS
Writing Multiple Choice Items
Multiple choice (MC) items are widely used, and have many advantages. They also have disadvantages, some of which can be reduced through careful attention to good item-writing and item analysis practices.
Advantages of Multiple Choice Items
Versatility — MC items are adaptable to the measurement of a wide variety of learning outcomes including reasoning, making inferences, solving problems, exercising judgement and demonstrating knowledge of facts through interpretation and analysis of information.
Efficiency — Because of the large number of items that can be posed in a given length of time, MC items permit wide sampling and broad coverage of the content domain.
Scoring accuracy and economy — Expert agreement on the correct answer to MC items is easy to obtain, and scan-score machinery can economically apply scoring keys.
Reliability — Consistency in scoring and wide sampling of content provide test results that can be generalized to the domain of interest.
Diagnosis — Patterns of incorrect responses can provide diagnostic information about the learning of individual test-takers or groups.
Control of difficulty — The level of difficulty of a test can be increased or decreased by adjusting the degree of similarity among the options for the items.
Reduction of guessing — In comparison with two-choice (e.g., true-false) tests, the consequences of guessing are reduced with MC items.
Freedom from response sets — MC items are relatively uninfluenced by response sets, such as a tendency to answer "true."
Amenable to item analysis — Item difficulty (the percentage of test-takers who select the correct response) and item discrimination (a correlation coefficient that indicates how well the item separates test-takers who know the material well from those who do not) can be used to improve MC items and inform instruction.
Disadvantages of Multiple Choice Items
- Multiple choice tests can be difficult and time consuming to write. The construction of plausible distracters can be especially difficult. The quality of the test is therefore dependent on the item-writing skill of the instructor.
- There is a tendency to write items requiring only factual knowledge rather than higher-level skills and understandings.
- Performance on MC items can be influenced by test-taker characteristics unrelated to the subject matter, such as reading ability, deductive reasoning, the use of context clues and risk-taking.
- MC items are subject to clueing (that is, the item provides a clue to the correct answer, increasing the risk that test-takers will guess).
- MC items do not measure ability to organize and express ideas.
General MC Item Guidelines
- Write items that deal with significant facts or concepts, not trivial questions or overly specific details.
- Write items that have a definite answer. Test-takers may be asked to select either the correct answer or the best answer. The former instruction is usually more suitable for items dealing with factual knowledge, where the correct answer is a matter of record. For items dealing with interpretation, understanding, or inference, instruction to select the best answer is usually preferred.
- Communicate clearly. The wording and presentation of the items should not present obstacles to the test-takers’ ability to demonstrate what they know. The item should be written in clear language with vocabulary other than the vocabulary being tested, that is as simple and precise as possible.
- Don't give away the answer by including irrelevant clues in the item.
- Don't write items that require skills or knowledge irrelevant to what you are trying to measure.
- Avoid language that may be offensive to some groups.
- Have items reviewed by knowledgeable persons other than the composer of the question if possible.
Item Stem Guidelines
- Write an item as either a direct question (Who was the first President of the United States?) or an incomplete statement (The first President of the United States was ________). Often one form or the other will produce simpler and clearer wording.
- Present a single, complete problem or question in the stem. Most of the reading should be in the stem.
- Eliminate excess wording; include only what is necessary to present the problem or question.
- Include in the stem all the information needed to arrive at an unambiguous answer to the item.
- Include in the stem any words that would be repeated in each option.
- Use an introductory sentence for the item if it seems useful. Two sentences may express the problem more clearly than one.
- Write completion items with the blank at the end rather than the beginning or middle of the sentence.
- Avoid the use of negative wording in items. If negatives are necessary, emphasize them with bolding, underlining or capitalization.
- Do not write items that require a series of true-false answers, i.e., questions of the form: "Which of the following is true?".
- Make sure that items are independent. The information in one item should not supply the answer to another.
- To test understanding and interpretation rather than factual knowledge, ask the questions "How?" and "Why?" rather than "Who?" and "When".
Consider variations on the simple MC format:
- Present material to be interpreted--such as a reading passage, a table, a graph, and a map--and base several items on it.
- Use the same set of response options for several items, presenting the options first, in effect creating a small matching task.
- Present items in analogy form: A is to B as 1 is to _____.
Item Options Guidelines
- Be sure there is one best response to the item. Options must be mutually exclusive and not overlap.
- Make the length of the options comparable. Avoid over qualifying the keyed response.
- Make the options parallel in form.
- Make all options grammatically consistent with the stem.
- Avoid using absolute language such as "never" and "always" as a means of making options incorrect.
- Make the distracters plausible and equally attractive to test-takers who do not know the correct response.
- Use 3-5 options. Four or five options are desirable to reduce guessing, but a good item with three options can be useful. Do not discard an item with only three good options or add implausible options, but make the number of options consistent.
- List the options in a logical order if there is one.
- Present the options in a list format rather than in a paragraph with the stem.
- Distribute the correct option randomly among the option positions.
- Don't use "All of the above" as an option. "None of the above" should not be used as an option with "best answer" items but can be used effectively with computational items.
- It can be helpful to define the class of things to which the correct answer belongs, and then write distracters based on members of that class.
- Consider as distracters responses that are correct but do not answer the question posed by the stem.
- Obtain distracters from responses of test-takers to items administered in completion or short answer form on prior tests.
TRUE & FALSE ITEMS
Advantages of True-False Items
True-false items share most of the advantages of multiple choice items:
Versatility — True-false items are adaptable to the measurement of a wide variety of learning outcomes.
Efficiency — Because of the large number of items that can be posed in a given length of time, permits wide sampling and broad coverage of the content domain.
Scoring accuracy and economy — Scoring keys can be economically applied.
Reliability — True-false tests that are highly reliable can be constructed.
Amenable to item analysis — Item difficulty (the percentage of test-takers who select the correct response) and item discrimination (a correlation coefficient that indicates how well the item separates test-takers who know the material well from those who do not) can be used to improve true-false items and inform instruction.
In comparison with other selected-response items, such as multiple choice, true-false items have several additional advantages:
Content efficiency — More test responses can be obtained from a given amount of written material and in a given amount of time from true-false items than from other forms. Three true false items can be answered for every two multiple choice items. Consequently, true-false items permit the widest sampling of content.
True-false items are especially useful for questions where there are only two reasonable answers.
True-false items are especially useful in testing misconceptions.
True-false items can be expressed in few words, making them easy to understand and less dependent on reading ability.
Disadvantages of True-False Items
- True-false items are especially subject to guessing. Half the items would be expected to be correct by chance. (But the likelihood of obtaining a substantially higher score by guessing alone is very small).
- In general, individual true-false items are less discriminating than individual multiple choice items.
- There is a tendency to write trivial true-false items, which lead test-takers to verbatim memorization.
- No diagnostic information is available from incorrect responses to true-false items.
- True-false items are not amenable to questions that cannot be formulated as propositions.
The wide range of items that can be used in a true-false test minimize the effects of guessing and lower item discrimination. Therefore, true-false tests can be equal in quality to multiple choice tests and can be constructed for administration in a given testing time. The key is to write sound test items for either form.
True-False Item Guidelines
- Write items that test significant material, not trivial details. The correct answer should require the specialized knowledge being tested, not common sense.
- Focus each item on a single idea so that test-takers do not have to deal with the possible truth or falsity of two or more propositions at once. Compound statements can sometimes be divided into two separate items.
- Express each item in simple and clear language.
- Provide sufficient information in the item to allow its truth to be judged. Include attribution of opinion or other context where appropriate.
- Word items precisely so that they can be unequivocally determined to be true or false. Use quantitative rather than qualitative language or make comparative rather than absolute statements. The keyed response should be clearly defensible and agreed on by experts in the field.
- Reformulate principles or use examples, rather than use the same language as the text or reference materials, especially stereotyped phrases, to avoid encouraging reliance on rote memorization.
- Avoid negatively worded statements in general and particularly double negatives. If the statement cannot be formulated positively, be sure to emphasize negative terms with underlining or bolding.
- Do not use specific determiners, such as "always", "never", "every", just to make a proposition false; conversely, do not use qualifiers such as "usually", "often", and "seldom" only in true statements. Such terms can serve as extraneous cues to savvy test-takers unless the pattern of their use is varied.
- If a proposition expresses a relationship, such as cause and effect or premise and conclusion, present the correct part of the statement first and vary the truth or falsity of the second part.
- Use somewhat more false than true statements because false statements discriminate better between high-ability and low-ability test-takers. (Test-takers tend to mark true more often than false when guessing blindly.)
- Make true and false items of approximately equal average length throughout the test.
- Randomize the sequence of true and false statements.
- Make use of popular misconceptions or beliefs as false statements.
- Try to word items so that the incorrect response is more plausible or attractive to those without the specialized knowledge being tested.
- Make use, where appropriate, of multiple true-false items, in which a single stem is followed by several statements or phrases based on the stem, each of which requires a true-false response.
- Consider the use of introductory material on which to base true-false items such as maps, graphs or written passages.