Selected Response Exams

The exam is immediately auto-graded upon completion. The exam is graded by adding up all of the part option scores for the raw score and then by using the score settings set for the exam, the scaled score is calculated. The system takes the raw score, subtracts the “passing raw score” number from it.  At that point, the score will be >= 0 for passing exams and failing exams will have a score of < 0.  Then it multiplies the score by either the multiplier for passed exams (if >= 0) or for failed exams (if < 0).  Next it adds in the “passing scaled score”.  Next it forces the number to be between minimum and maximum scaled score by clamping and that’s how it arrives at the final scaled score. Once auto-graded the exam is moved into “Final” status and candidates will be able to see their score. At this point they will also see the option to sent their results to an organization that they are a member of.

Constructed Response Exams

By using the result schemas the system is able to determine how many vignette scorings per vignettes are needed and then collects them per vignette. Once all vignettes in the exam have been rated by the specified amount of raters and the score is available for each vignette, the rest of the exam is then auto-graded (see above description for Selected Response Exam). Once auto-graded the exam is moved into “Final” status and candidates will be able to see their score. At this point they will also see the option to sent their results to an organization that they are a member of.

Viewing Exam Responses

CASLI Administrators have the ability to view exam responses and scores to exams that have “Final” status. There are two ways to do this:

To see responses from a particular candidate
  1. Go to “Accounts” from the menu
  2. Select the blue highlighted ID of the candidate
  3. Select the pencil (edit) icon next to “Exams”
  4. Select the blue highlighted ID of the exam you would like to see responses to
  5. Select “View responses”
To see responses to a particular exam
  1. Go to “Reports” from the menu
  2. Select “Finalized exams”
  3. Select the blue highlighted ID of the exam you would like to view responses of
  4. Select “View Responses”

Vignette Scoring: 

Raters that are qualified to evaluate responses to a vignette can be assigned a vignette scoring. Raters will only see the list of vignette response waiting to be scored that they are qualified to rate. CASLI administrators will be responsible for maintaining vignette scoring assignments to qualified raters.

Creating a New Vignette Scoring
  1. Go to Vignette Scoring from the menu
  2. Select “Create a new vignette scoring”
  3. Enter in the name of the vignette scoring
  4. Select “Create”
  5. Select the pencil (edit) icon next to Instructions, Levels, and Criteria to modify those (description of each below)
  6. Select “Update”
Instructions

Text that will display to your raters as instructions for the vignette scoring

Levels

Sets a result based on the minimum number of primary and secondary scoring criteria that must be met

Criteria

Descriptive statements that provide information about the qualities, characteristics, and aspects expected, that the rater will compare to the candidate’s performance to determine their evaluation.

  • Scoring types (categories that each criteria can be assigned to) – includes Primary, Secondary, Mandatory

Note: To adjust the order or step of a scoring criteria select the arrows as needed from the vignette scoring criteria catalog

Assigning a Vignette Scoring to a Rater
  1. Go to Accounts from the menu
  2. Select the blue highlighted ID of the rater account you would like to assign the vignette scoring to
  3. Select the pencil (edit) icon next to “Vignette scorings” 
  4. Select “Add a vignette scoring” 
  5. Choose which vignette scoring you would like to add from the drop down list
  6. Select “Add”

Vignette Feedback

CASLI Administrators can create vignette feedback for a rater to check off after the scoring criteria for the candidate’s response. This feedback will go to candidate once the rater has submitted the score. For more information see Vignettes.

Domains

Domains are categories that exam parts are assigned, in order to score by the category of parts. A scoring breakdown by domain is then shown in emails to candidates regarding their result.

Creating a New Domain
  1. Go to Domains from the menu
  2. Select “Create a new domain”
  3. Enter in the name of the domain
  4. Select “Create”
  5. Select the pencil (edit) icon next to Description, Maximum raw score, Feedback, and Overrides to modify those (description of each below)
  6. Select “Update”
Description

Text that will display as the description of the domain

Maximum raw score

This maximum raw score will be used to calculate the score percentage for the domain, to be shown on the scoring breakdown per domain to candidates.

Feedback

Sets feedback to be included in the scoring breakdown for the domain, based on the minimum score and maximum score that the feedback applies to.

Overrides

Domain overrides allow bigger value to be placed on a particular domain, to the point of overriding the result of the entire exam if the score from a particular domain is at or between the minimum score and maximum score for the domain override.

Result Schemas

Result schemas are used to group results together to create a framework for scoring. Result schemas can then be assigned to a Record video or Play vignette and record video part. For more information see Vignettes. To create a new result schema:

  1. Go to Result Schemas from the menu
  2. Select “Create a new result schema”
  3. Enter in the name of the result schema
  4. Select “Create”
  5. Select the pencil (edit) icon next to “Schema”
  6. Select “Create a result”
  7. Fill out the Pass minimum, Borderline pass minimum, Borderline fail minimum, Fail minimum, and the Score
  8. Select “Update”

Result Routes

Result routes are used to send exam results to organization(s) that the candidate is a member of. Once the candidate has completed the exam they will see an option to “Send exam result.” This action will send an email to the organization(s) that the candidate is a member of with the results of the exam. To create a new result route:

  1. Go to Result Routes from the menu
  2. Select “Create a new result route”
  3. Enter in the name of the organization
  4. Select “Create”
  5. Select the pencil (edit) icon next to “Contact name”
  6. Enter in the contact name
  7. Select “Update”
  8. Follow steps 5-7 for “Contact email address” and “Reference ID label”