Educators have widely accepted the importance of using data to inform their treatment of students’ needs. This is a good thing, as research touts the benefits of effective data use. Thus many educators worldwide have embraced data use with gusto. Unfortunately, educators’ widespread data use is not quite a good thing.
A significant portion – and some research claims most – of educators analyzing and using data are doing so incorrectly. For example, a study conducted by the U.S. Department of Education (2009) at districts with reputations for strong data use found only 48% of teachers correctly interpreted given data. Much research supports this trend: educators are using data to inform decisions, but they do not always understand the data they are using. Since their data-misinformed decisions impact students, this is a significant problem. Fortunately, edtech can do much to help if it improves in key areas.
Consider an Analogy
Which offers you better medical care: 1) treatment or medicine administered via a personal visit with a doctor, or 2) over-the-counter medicine administered by yourself? The answer is easy: a personal visit with a doctor, as he or she can ensure the treatment or medicine is used appropriately. However, this does not supplant the need for over-the-counter medicine, as it is not always possible to visit a doctor for every cough or sniffle. While a doctor is not present to explain the over-the-counter medication’s purpose, ingredients, dosage instructions, and dangers, any remedy you buy in a store comes with a detailed label and other supports outlining these matters. Just as it would be wrong to board up pharmaceutical aisles in stores and withhold basic medicine from people who are not receiving it directly from doctors’ hands, it would be equally ludicrous to rip the labels off over-the-counter medications, leaving people with no way to use them wisely.
So, how does this relate to schools and edtech? Educators use data to treat students, and they use edtech data systems (or reports generated from those systems) to analyze this data. However, these educators are usually operating without the data-equivalent to over-the-counter medicine. They are essentially opening data bottles with no labels, and swallowing the contents nonetheless. This is because supports that commonly accompany over-the-counter medicine to ensure it is used properly (thorough labeling, supplemental documentation, online access to a help system, effective package/design, and appropriate contents) are not utilized in most of the edtech data systems or reports educators use to analyze data. Thus this highly utilized sector of edtech typically denies educators the analysis supports that could significantly improve educators’ use of the data.
How Has Edtech’s Data Role Progressed?
Before the rise of technology and user-friendly data systems, school staff did not have site-wide access to data like they typically do today. Staff and students could not benefit from extensive, timely data such as classroom assessment data that could be used to closely monitor student progress and customize instruction accordingly. Perhaps staff received blocks of data by the government, a consultant, or a district-based trainer once or twice per year, but this could not help their students on an ongoing formative basis, and it only offered educators isolated points in time during which students’ needs could be assessed and remedies determined. With this arrangement, educators were like patients with limited access to doctors and no access to over-the-counter medication.
Now most schools are using a data system with an intelligible interface that puts data at the fingertips of most staff members. However, the data reports contained therein are generally not accompanied by guidance on their purpose, how to properly read them, how to properly interpret the data, and analysis misconceptions to avoid. Essentially, educators are like patients using medication in unmarked – or marginally marked –containers. A data system might label a report (“Growth Report,” “Year-to-Year Comparison,” etc.), but relying on mere titles is akin to taking pills in a bottle marked only as “Cold” or “Flu.” Anyone using such data to inform decisions is assuming as much risk as someone opening a pill bottle that reads “Flu” and chancing ingestion of the wrong number of pills, at the wrong time, for the wrong type of flu, and in direct opposition to dangers. Imagine the ramifications for students and staff affected by the decisions made in such a way.
With educators making so many data analysis errors even at districts known for their data prowess, current data analysis supports are clearly not enough. Professional development (PD) is needed and effective, as are good leadership and staff to assist with data use, but we further these supports’ impact when we prescribe an Over-the-Counter Data (OTCD) format for the data we are using.
What Is Over-the-Counter Data?
There is no price tag tied to OTCD. Rather, it is a design approach any edtech data system or report is free to utilize to help educators make more accurate analyses of data. While most data systems are simply “showing the data,” OTCD addresses key ways in which they can better help educators use that data correctly. Inspired by the varied ways in which over-the-counter medication supports the successful use of its contents, OTCD involves embedding data analysis supports directly within reporting environments and adhering to best practices within 5 component categories:
Label – The label functions as an appropriate report title (clear, concise, consistent with logic of report suite titling) and appears as a footer and/or annotation on each report to provide the most important information needed for analysis.
Supplemental Documentation – Since you should not cram all information needed for analysis onto the report itself, explanatory information accompanies each report via links to an abstract and/or interpretation guide providing more analysis guidance specific to the report and its data.
Help System – An online help system built within the product and accessible via an easy-to-spot link offers lessons on using the system (tech lessons), which is common, but also on data analysis (specific to the data and assessments used in the users’ region), which is uncommon.
Package/Display – Report format (how the data is displayed) assists proper analysis by maintaining credibility, offering key features, and adhering to principles of good design that assist analysis ease and accuracy. Reports are accessible (e.g., it’s easy to find a particular report) in a manner that offers easy navigation.
Contents – Reports operate with input controls that specifically facilitate recommended data investigation practices, feature expiration- and audience-sensitive contents, and comprise a suite of reports covering all key needs without overwhelming users.
Specific attributes contained within these OTCD components are detailed at www.overthecounterdata.com/otcd. These recommendations summarized by OTCD are based on research in education and edtech, as well as research in a variety of other fields (e.g., behavioral economics, design, business analytics, technology, etc.). OTCD assumes an edtech data system or data component already meets basic requirements widely accepted and discussed for at least a decade (students tied to unique identifiers, online access for all staff, longitudinal reporting capabilities, etc.), which so many data systems already include, and summarizes what their next milestones should be.
What Can We Do?
OTCD holds great potential to help educators and – most importantly – students, but only if it is implemented in areas of edtech and reporting that communicate data to education stakeholders. Everyone involved in education and edtech can play a role in this endeavor:
Educators – You are the edtech customer and thus edtechs respond to your requests and want to meet your needs. Communicate your need for OTCD to your edtech vendor and work to see your OTCD requests implemented in any area where data is being communicated. Knowing most educators are misusing data in its absence, OTCD is an urgent request.
Edtech Staff – If your system includes a component that communicates data, adjust your system to adhere to OTCD. This has a significant impact on students and thus is a matter of moral responsibility. Fortunately, OTCD also creates more cost-effective efficiency in your product and improves your product’s success and marketability.
Edtech Investors – When determining which edtech startups or projects in which to invest, check edtech candidates for OTCD. Many edtech products contain a feedback component where they are communicating data, and poor communication is not cutting edge. Rather, a firm understanding and implementation of OTCD allows edtechs to perform better and thus compete better in the market, all while being more efficient with design and resources. Products adhering to OTCD are more worthy of your investment than those failing to implement research-based practice.
OTCD is the next frontier for data in edtech. It is time for conversations about edtechs that display data to hold higher standards and reflect OTCD best practices ascertained by the research community. When Over-the-Counter Data is realized in products educators use to view, understand, and use data, it will be a great victory for students and all roles seeking to help them.