OH 1
Major Points from Last Week

1. Your goal is valid, reliable, useful assessment
2. Which requires:
1. Determining what is to be measured
2. Defining it precisely
3. Minimizing measurement of irrelevancies
3. And is promoted by following good procedures

1. Deciding its purpose
2. Developing test specifications
3. Selecting best item types
4. Preparing items

What location in instruction?

1. pre-testing

   • readiness
     1. limited in scope
     2. low difficulty level
     3. serve as basis of remedial work, adapting instruction

   • pretest (placement)
   1. items similar to outcome measure
   2. but not the same (like an alternative form)

2. during instruction
   • formative
   1. monitor learning progress
   2. detect learning errors
   3. feedback for teacher and students
   4. limited sample of learning outcomes
   5. must assure that mix and difficulty of items sufficient
   6. try to use to make correction prescriptions (e.g., review for whole group, practice exercises for a few)
   • diagnostic
   1. enough items needed in each specific area
   2. items in one area should have slight variations

3. end of instruction
   • mostly summative –broad coverage of objectives
   • can be formative too

OH 4
Step 2: Develop Test Specifications

• Why? Need good sample!

• How? Table of specifications (2-way chart, "blueprint")
  1. Prepare list of learning objectives
  2. outline instructional content
  3. prepare 2-way chart
  4. or, use alternative to 2-way chart when more appropriate
  5. doublecheck sampling

OH 5
Sample of a Content Domain (For this course)

1. trends/controversies in assessment
2. interdependence of teaching, learning, and assessment
3. purposes and forms of classroom assessment
4. planning a classroom assessment (item types, table of specs)
5. item types (advantages and limitations)
6. strategies for writing good items
7. compiling and administering classroom assessments
8. evaluating and improving classroom assessments
9. grading and reporting systems
10. uses of standardized tests
11. interpreting standardized test scores

OH 6
Sample Table of Specifications (For chapters 6 and 7 of this course)

OH 7a
Spot the Poor Specific Learning Outcomes (For use with previous table of specifications)

1. Knowledge
1. Knows correct definitions
2. Able to list major limitations of different types of items
2. Comprehension
1. Selects correct item type for learning outcome
2. Understands limitations of true-false items
3. Distinguishes poor true-false items from good ones
3. Application
1. Applies construction guidelines to a new content area
2. Creates a table of specifications
4. Analysis
1. Identifies flaws in poor items
2. Lists general and specific learning outcomes
5. Synthesis
1. Lists general and specific content areas
2. Provides weights for areas in table of specifications
6. Evaluation
1. Judges quality of procedure/product
2. Justifies product
3. Improves a product

OH 7b
Why are These Better Specific Learning Outcomes?

1. Knowledge
1. Selects correct definitions
2. Lists major limitations of different item types
2. Comprehension
1. Selects proper procedures for assessment purpose
2. Distinguishes poor procedures from good ones
3. Distinguishes poor decisions/products from good ones
3. Application
1. Applies construction guidelines to a new content area
4. Analysis
1. Identifies flaws in procedure/product
2. Lists major and specific content areas
3. Lists general and specific learning outcomes
5. Synthesis
1. Creates a component of the test
2. Provides weights for cells in table of specifications
6. Evaluation
1. Judges quality of procedure/product
2. Justifies product
3. Improves a product

OH 8
Step 3: Select the Best Types of Items/Tasks

What types to choose from? Many!

1. objective--supply-type
1. short answer
2. completion
2. objective--selection-type
1. true-false
2. matching
3. multiple choice
3. essays
1. extended response
2. restricted response
4. performance-based
1. extended response
2. restricted response

1. most directly measures learning outcome
2. where not clear, use selection-type (more objective)
1. multiple choice best (less guessing, fewer clues)
2. matching only if items homogeneous
3. true-false only if only two possibilities

• Strengths

  • Can have many items
  • Highly structured
  • Scoring quick, easy, accurate

• Limitations
  • Cannot assess higher level skills (problem formulation, organization, creativity)

• Strengths
  • Can assess higher level skills
  • More realistic

• Limitations

  • Inefficient for measuring knowledge
  • Few items (poorer sampling)
  • Time consuming
  • Scoring difficult, unreliable

OH 10
Step 4: Prepare Items/Tasks

1. specifying more precise learning outcomes leads to better-fitting items
2. use 2-way table to assure good sampling of complex skills
3. use enough items for reliable measurement of each objective
• number depends on purpose, task type, age
• if performance-based tasks, use fewer but test more often
4. keep in mind how good assessment can improve (not just measure) learning
• signals learning priorities to students
• clarifies teaching goals for teacher
• if perceived as fair and useful

Strategies to Avoid Contamination

1. eliminate barriers that lead good students to get the item wrong
2. don’t provide clues that help poor students get the item correct

1. use table of specifications as guide
2. write more items than needed
3. write well in advance of testing date
4. task to be performed is clear, unambiguous, unbiased, and calls forth the intended outcome
5. use appropriate reading level (don’t be testing for ancillary skills)
6. write so that items provide no clues (minimize value of "test-taking skills")
1. a/an
2. avoid specific determiners (always, never, etc.)
3. don’t use more detailed, longer, or textbook language for correct answers
4. don’t have answers in an identifiable pattern
7. write so that item provides no clues to other items
8. seeming clues should lead away from the correct answer
9. experts would agree on the answer
10. if item revised, recheck its relevance