East Tennessee State University
Assessment Strategies
Compiled by Dr. Norma MacRae,
Vice Provost for Public Service,
Dean, School of Continuing Studies
§ Standardized
§ Professional (licensure, certification)
§ Commercial
§ From the literature
§ Home-made
§ Naïve, ad hoc
§ Item analysis, test bank
§ Custom-made tests
§ Structured / unstructured (“open-ended”)
§ with/without standardized probe & reinforcement procedures
§ Formal / informal
§ Face to face or small group (focus group)
§ Telephone, on-line chat, videoconference
Comparison to Standards, Protocols, or Norms
§ National, regional, state, local
§ Professional groups
§ Accrediting bodies
§ Surveys of literature
§ Key decision makers
§ Site visits to similar programs, agencies, or institutions
§ Self—past performance
§ Formal – informal/anecdotal
§ Staff/unit member or client or person who is neither
§ Shadowing
§ Trained raters
§ Steering committee or supervisors as observers
§ Professional evaluation staff
§ Systematic fieldwork techniques
§ Individual or team
§ External to institution
§ Internal to institution & external to unit/function
§ Professional evaluators
§ Role players to test system
§ Expert panel
§ Expert judgment techniques
§ Delphi
§ Devil’s Advocate
§ Nominal Group
§ Dialectical Inquiry
§ Commercial, home made, custom made
§ Client satisfaction
§ Paper & pencil
§ Electronic forms
§ Email, regular mail
§ Representative cases
§ Random cases
§ Outliers
Unobtrusive, non-reactive, or incidental measures
§ Collections or surveys of local media products
§ Demographic data
§ Analysis of existing or incidental written documents
§ Agency records
§ Memos
§ Minutes
§ Official records and actions
§ Letters, etc.
§ Data collected in conjunction with routine processes such as greeting clients, intake and registration, payments made, courses taught, etc.
§ Archival data
§ Counts of behaviors or phenomena
§ Electronic or mechanical
§ Web page hits
§ Observer/rater
§ Art works, crafts
§ Material culture—tools, implements, functional systems
§ Written documents, projects, proposals
§ Portfolios
§ Complaints, comments, questions, errors
§ Students
§ Entering
§ Non-returning
§ Transfers
§ Exiting
§ Graduates
§ Alumni
§ Faculty/staff/administrators
§ Clients, project participants, or direct recipients of services
§ Stakeholders (parents, employers, personnel directors, community sponsors, etc.)
§ Key decision makers (elected officials, funding agencies, governance board and staff, etc.)
§ General public—community perceptions or needs
§ Representative members
§ Opponents or skeptics
§ Supporters/donors
§ Opinion leaders
§ Members of organizations involved:
§ Project staff members
§ Community volunteers
§ Administrators and supervisors
§ Professional peers and peer institutions
Recording Processes and Outcomes
§ Written records
§ Notes, memos, letters, reports
§ Forms
§ Activity logs (staff, clients)
§ Timesheets, mileage sheets
§ Contact summary forms
§ Journals (staff, clients, or stakeholders)
§ Flowcharts and timelines
§ Biographies & autobiographies
§ Financial records
§ Budgets
§ Receipts and expenditures
§ Formal audits
§ Audiotapes/videotapes, etc.
§ Meetings, focus groups, performances, services, examples
§ Oral histories
§ Journals
§ Photographs, sketches, etc.
Measures must be reasonably reliable and valid.
¨ Reliable = will give about the same result each time it is used unless there has been a real change.
¨ Valid = measures what it is supposed to measure.
¨ Construct validity = extent to which a measure can be shown to assess the construct or conceptualization it supposedly measures.
¨ Content validity = degree to which the measure represents or covers the total domain being assessed.
¨ Predictive validity = degree to which the measure predicts later behavior or outcomes.
¨ Concurrent validity = extent to which one measure of a phenomenon correlates with another measure of the same phenomenon.
Consequential validity = extent to which the values and social consequences embedded in the measure are legitimate.
¨ A Good Program for assessing institutional effectiveness will
¨ Make strategic use of targeted ad hoc assessments (e.g., reports from Continuous Improvement Teams, special task forces, internal reviews).
¨ Include provisions for regularly compiling and analyzing data and presenting them in comprehensible formats, i.e., “pictures” with meaningful comparison data.
¨ Include provisions for regularly disseminating data to key groups and individuals and encouraging them to read and then evaluate the data.
¨ Include provisions for regularly involving key groups and individuals in discussion and evaluation of results, in identifying where results are less than desirable, and in proposing corrective strategies,
¨ Include provisions for regularly developing and implementing strategies to address identified weaknesses,
¨ Include provisions for regularly evaluating the implementation of strategies and their effectives and for feeding the information back into the evaluation and strategic problem-solving process.
¨ Link the outcomes of assessment processes to the institutional processes for strategic planning, budgeting, and resource allocation.