A few months ago, a colleague sent me a link to an article describing the difference between “data-driven” vs. “data-informed” decisions. It’s real nerdy…I love it, but it’s real nerdy. The author even wrote a separate, round-up style article highlighting the specificity of these terms!
[Reader, I clicked every link in that roundup. It was a deep dive.]
Ultimately, I don’t think it matters much whether you call your process "data-driven", "data-based", or "data-informed". What matters most is that you have a process for using data to improve the way your school supports its community. In one study, teams who had a defined process for incorporating data in their decisions were more likely to define their problems with greater precision and they were more likely to define the changes they wanted to see.[1]
Consider the way you make decisions as a team. If someone asked you whether you used data in your process, what would you say?
Researchers Gina Schuyler Ikemoto and Julie A. Marsh asked school teams a similar question. They wanted to know more about how educators used data to steer the decisions they made.[2] More often than not, the sentiment they heard was something like: “We are completely data-driven. We base all our decisions on data.”
Pretty great, right?
These researchers probed a little further and asked participants “When you say ‘completely data driven’ what exactly do you mean?” They heard a wide range of responses. What they didn’t hear was a common set of steps or even a shared language to describe the processes these teams used. However, what they did learn was: the data teams used and every process they implemented landed somewhere along two continuums from simple to complex. Each of the 36 teams fell into one of these four decision-making models depending on how complex their data and process got.
Now, when I saw this graphic, I thought, “That lower right-hand corner must be the best model for making decisions.” Well, yes and no. Each of these four models is a great option depending on the decisions you’re trying to make. For example, a straightforward decision doesn’t require multiple data sources and an iterative process. Going with a gut decision doesn’t make sense when you’re changing school-wide systems and policies.
And…
While all of that is true, according to this particular study, “the literature tends to emphasize the value of engaging in inquiry-focused” data-driven decision making…or that bottom right-hand corner. What does that mean? I’m glad you asked.
What is Inquiry-focused Decision Making?
Schools engaging in inquiry-focused decision making “utilize the process as a means of continuous improvement and organizational learning”[3] The data you use in inquiry-focused decision making cover multiple aspects of the problem you’re trying to solve; they can be disaggregated, and they allow you to view trends over time. With a model like this, your decisions are collaborative and iterative — meaning you’ll monitor the decisions you make and revisit your solutions as needed. Teams following this model often recruit help from an expert – like a coach – to help them navigate the process.
When you think about using behavior data to inform the decisions you make related to your systems and practices, an inquiry-focused decision-making practice makes a lot of sense. Student behavior is contextual. That means the data you collect need to be multi-dimensional. The process you use needs to help you create solutions tailored to the precise problem you’ve identified. You’re also going to evaluate the impact your decision has overtime. One inquiry-focused approach lots of PBIS teams use is called Team Initiated Problem Solving (TIPS). Using the TIPS process, school-based teams implement "the foundations needed to run more effective meetings, a process for using data to identify school needs and goals for change...and a process for using, monitoring, and adapting solutions."[4] The approach makes a big difference! Teams are able to identify problems early, develop practical, culturally responsive solutions, implement those solutions with precision, and document the benefits for students.
If your team is ready to test drive an inquiry-focused process, here are a few factors researchers say can help take your data-driven, data-based, or data-informed decision making to the next level.
Accessibility & Timeliness of Data
Across the board, educators were more likely to use data when they had access to them online. Think about it. When you make a data request to someone outside of your school, by the time those data get back to you they’re likely out of date and you’ve moved on to other decisions. So, remove that obstacle and be sure you have direct access to the data you need, when you need it.
You also want to know the data you’re using are up to date. If the data included in the report you’re looking at are current as of two months ago, the decisions you make solve the problems you had in February instead of your April ones. One way to ensure your data are up to date is to give teachers the ability to enter their own referrals. By spreading out the data entry workload, you make that process more efficient and keep the information current.
Making decisions while you watch a clock is stressful. You want to be sure you allow enough time for analysis and discussion, but not so much time that you slip into admiring the problem instead of solving it.
Perceived Validity of Data
When you’re about to make a decision to change something about a system or a practice in your school, don’t you want to know the data you use to inform that decision are reliable? The answer to that question is yes! You have to be able to trust your data. One way to do that with your behavior data is to make sure you have a defined referral process and definitions for every problem behavior. When you define these two pieces, you make sure everyone knows how unwanted behaviors are handled and they know what those behaviors look like.
Staff Capacity & Support
After surveying educators who participated in their study, Gina Schuyler Ikemoto and Julie Chen found that only 23% of teachers who responded said they felt moderately or very prepared to interpret and use data. These folks didn’t feel like they had the skills necessary to use data in an inquiry-focused type of way. The piece that made a big difference was the addition of a data analyst. A data analyst is someone who takes a first pass at looking at the data. They filter out the noise and dial a team into where they need to pay closer attention.
I bet there is at least one person on your team who likes to look at graphs and tables. They get excited to look for trends and drill down to the root of an issue. The next time you project a graph in your meeting, look around and see whose face lights up. That’s your data analyst. Assign them the task of looking at the reports ahead of your next meeting. They can put together the Cliff Notes version of your data pile — something that highlights the key takeaways and focuses your attention on the problems to start solving.
Time
Making decisions while you watch a clock is stressful. You want to be sure you allow enough time for analysis and discussion, but not so much time that you slip into admiring the problem instead of solving it. That sweet spot is something you’ll need to find for yourself. Lately, one tactic we’ve heard from teams especially at this point in the year is to split their action planning meeting into two meetings. During that first meeting, focus specifically on the analysis. Dive deeply into the fidelity, feedback, and outcome data at your disposal. Make connections and get specific about what was successful and the challenges you want to solve. Then, use the second meeting to create that action plan. Develop solutions as a team, decide who will do what and by when, and identify the data you’ll need to continue to monitor your progress over time.
Tools
The tools you use to collect, store, and report the data you need to inform the decisions you make are critical components to any decision-making process you implement. The tools you use to enter referral data must capture several dimensions related to the behavior and the context in which it occurred. Knowing each of the pieces helps you answer a specific question about the behaviors happening in your school. (If you use SWIS, these dimensions might look pretty familiar…)
- Student Name: Who did the behavior?
- Staff Name: Who noticed the behavior?
- Time: When did the behavior happen?
- Location: Where did the behavior happen?
- Behavior: What did they do?
- Motivation: Why do you think they did it?
- Others Involved: Who was part of the situation?
- Admin Decision: What did we decide to do about it?
Your tools must allow you not only to collect this information, they have to generate the type of graphs that make your data easier to analyze. Tables alone aren’t going to cut it; you need those visuals. SWIS gives you exactly what you need to collect the multi-dimensional data an inquiry-focused process requires, and it generates those graphs you need to make more efficient decisions.
Inquiry-focused decision-making models also require you to disaggregate your data by groups. The SWIS Equity Report is one example of how disaggregated behavior data help you see the way your systems and practices affect specific student groups differently. It reveals deeper levels of understanding you would otherwise miss if you stuck strictly to interpreting aggregate school-wide referral data.
Your school collects a growing number of data points related to your students and the way your school works to support them. You need a way to incorporate these complex data sources into a collaborative, iterative, continuously improving process that takes your decision making from basic to inquiry-focused. No matter what process you use, you need access to timely, trustworthy data. Those data need to be collected in systems that give you multi-dimensional information about the behaviors happening in your building and the contexts where they occur. Your tools should make your analysis more efficient and allow you to disaggregate your data in order to uncover the ways groups of students may experience your systems and practices differently. But, most of all, your team needs someone (or two) with the knowledge, interest, and time available to coach you through the process and guide your discussions toward finding solutions to the most pressing problems ahead of you.