11 Conclusion and Discussion
The Dynamic Learning Maps® (DLM®) Alternate Assessment System is based on the core belief that all students should have access to challenging, grade-level academic content. The DLM assessments provide students with the most significant cognitive disabilities the opportunity to demonstrate what they know and can do. It is designed to map students’ learning after a full year of instruction.
The DLM system completed its fifth operational administration year in 2019–2020. This technical manual update provides updated evidence from the 2019–2020 year. Due to the COVID-19 pandemic, only limited results were provided For a summary of results provided in 2019–2020, see Chapter 7 of this manual. that were appropriate and technically defensible. Because of this, the intended uses of the results were different than what would be intended under a normal administration year. Thus, the contents of this manual are meant to describe changes to the DLM Alternate Assessment System in 2019–2020 and the limited results that were provided, rather than contribute to the assessment validity argument to support the original intended uses of results. For evidence supporting these original intended uses, see the 2015–2016 Technical Manual—Science (Dynamic Learning Maps Consortium, 2017) and the subsequent technical manual updates (Dynamic Learning Maps Consortium, 2017, 2018a, 2018b, 2019a). The contents of this manual are summarized in Table 11.1.
| Chapter | Contents |
|---|---|
| 1 | Provides an overview of information updated for the 2019–2020 year |
| 2 | Not updated for 2019–2020 |
| 3, 10 | Provides evidence collected during 2019–2020 of test content development and administration |
| 4 | Describes the new instruction and assessment planner |
| 5 | Describes the statistical model used to produce results based on student responses, along with a summary of item parameters |
| 6 | Not updated for 2019–2020 |
| 7, 8 | Describes results and analyses from the fifth operational administration |
| 9 | Provides additional studies from 2019–2020 focused on specific topics related to validity |
This chapter describes future research studies as part of ongoing and iterative processes of program responsiveness, validation, and evaluation.
11.1 Operational Assessment
As noted previously in this manual, 2019–2020 was the fifth year the DLM Alternate Assessment System was operational. The DLM Consortium is committed to continual improvement of assessments, teacher and student experiences, and technological delivery of the assessment system. Through formal research and evaluation as well as informal feedback, some improvements have already been implemented for 2020–2021. This section describes significant changes, if applicable, from the fourth to fifth year of operational administration, as well as examples of improvements to be made during subsequent years.
Overall, there were no significant changes to the item-writing procedures, item flagging outcomes, or the modeling procedure used to calibrate and score assessments from previous years to 2019–2020.
Based on an ongoing effort to improve Kite® system functionality, several changes were implemented during 2019–2020. The new Instruction and Assessment Planner was introduced to assist teachers in the selection of EEs and linkage levels for their students during the optional fall instructionally embedded window. A new writing sample upload process was also created in the system to support teachers in uploading student writing samples associated with student records for evaluation of interrater agreement, rather than having local designees submit the samples outside the system. Additional updates to the writing sample upload process are planned for 2020–2021 to tie samples to specific testlets.
Studies planned for 2020–2021 to provide additional validity evidence are summarized in the following section.
11.1.1 Future Research
The continuous improvement process also leads to future directions for research to inform and improve the DLM Alternate Assessment System in 2020–2021 and beyond. The manual identifies some areas for further investigation.
In 2020–2021, the test development team will begin a review of existing Essential Element Concept Maps (EECMs) and supplemental item writing resources used to develop DLM testlets in science. Using principles of evidence-centered design and Universal Design for Learning, teams will focus on revising the existing EECMs to incorporate additional descriptions of knowledge, skills, and understandings associated with the linkage levels for EEs, revise existing vocabulary suggestions, and integrate additional supplemental materials, as needed. While the structure and function of EECMs will remain the same, this revision will allow staff to improve the quality and utility of this tool.
DLM staff members are planning several studies for spring 2021 to collect data from teachers in the DLM Consortium states. The teacher survey will be repurposed to evaluate the impact of COVID-19 on instruction and assessment during the 2020–2021 year. However, some sections of the survey that have been included for multiple years (e.g., opportunity to learn) will continue to be included to examine any changes to longitudinal trends. State partners will continue to collaborate with additional data collection as needed.
Research is also underway to improve the model used to score DLM assessments. This includes the evaluation of a Bayesian estimation approach to improve on the current linkage-level scoring model and evaluation of item-level model misfit. Additional research is also planned to evaluate various methods for estimating reliability. The research agenda is guided by a modeling subcommittee of DLM Technical Advisory Committee (TAC) members.
Other ongoing operational research is also anticipated to grow as more data become available. All future studies will be guided by advice from the DLM TAC and the state partners, using established processes.