This is a guest post: a lightly edited letter that was emailed to the Board of Education of the Urbana School District 116 on 3.8.22 by Ruth
In several meetings, members of the Board and the Equity task force discussed the importance of qualitative data in the equity audits. The members talked about how information from the focus groups can give context and will be used to collect more data to help find causes of the educational disparities in our district.
I agree – qualitative data is important!
However, it seems that the Equity Task Force and members of the Board are under the false impression that the consulting firms that conduct focus groups, such as Dr Dubiel’s firm, will use the information from the qualitative data to collect more data and/or analyze the qualitative data to find patterns that can point to the causes of the disparities.
Unfortunately, neither Dr Dubiel’s firm nor the Equity Imperative group collect more quantitative data or qualitative data (such as classroom observations) after conducting the focus groups! No additional data at all is collected after the focus groups. Also, contrary to what you may have been led to believe, these groups do not perform data analysis.
You can see what Dr Dubiel is actually offering when she described the 5-phase process of the equity audit in her presentations to the Board and to the Equity Task Force. It is also explained and demonstrated in her audit reports for other districts.
To understand what you are really getting from Dr Dubiel’s audit, let us look at the equity audit report for McLean County Unit 5 (which I am attaching to this email). This district is similar to Champaign Unit 4 in size and has a diverse student population. This is a particularly relevant example since their District Equity Leadership Team (DELT) was highly concerned about their IEP students, as is our own Equity Task Force.
The “5-phase process” is described in pages 14-16 as follows:
- Pre-Phase 1: DELT is formed: 25-30 staff members (no parents members. This was explained at the ETF presentation).
- Phase 1: DELT members fill out a questionnaire to determine “needs assessment” (page 18-22) and decide what quantitative data to collect, and what questions to ask the focus groups (6 is the base number. You pay more if you want more questions – this was explained to the Equity Task Force).
- Phase 2: the quantitative data is prepared by the district.
- Phase 3: focus groups (qualitative data); description on pages 15-16.
- Phase 4: data analysis.
- Phase 5: findings and recommendations (the report!).
Did you see in the report any mention of collecting more data after the focus groups (Phase 3)? Any diagram showing a feedback loop?? No, it is not there! It is not part of the process!
The Board and the ETF discussions were unintentionally misinforming the public about the equity audit process offered by these consulting firms.
Another misconception held by some Board members is that Dr Dubiel’s equity audit is external and will use “fresh eyes” (this was the argument against collaboration with the ‘University Team’ – which may have associations with people who have children in the district and is therefore “biased”). If you read the details of Phase 1, you see that the DELT, which consists of staff members and the superintendent, self assesses the district’s need and then the DELT decides on the quantitative data to collect and which (6) questions to ask the various focus groups (with the help of Dr Dubiel). So much for unbiased “fresh eyes”…
What we see is that, instead of following a procedure that has been recommended for school districts by the National Academy of Science for a comprehensive data collection for equity audits (as was proposed by the University Team, to be done for free), the DELT is going to make its own choices, based on Dr Dubiel’s (assumed) expertise (which is presumably unbiased by her conflict of interests, as her firm also provides PD services (on unconscious bias)…).
Moving on to Phase 3: the focus groups! You can see the 6 questions McLean County’s DELT chose for the various group categories on pages 15-16. You may wonder why the students’ group were not asked directly about barriers for taking honors courses or participating in extracurricular activities, given that the district was especially concerned with the lack of participation of minorities in honors courses and extracurricular activities. Instead, 2 questions out of the 6 were about “connecting with adults”.
There is a limit to the number of focus groups (and limited amount of time – an hour per group). As expected, the focus groups do not represent the population and are skewed toward staff members. For McLean County Unit 5, only ~0.3% of the students (42 students) and less than 1% of families (63 people) participated in the focus groups, while 8% of the staff (73 people) participated in the “staff focus group”.
Just for comparison, in the Board discussion of the Panorama Survey on January 4, where 13% of the district’s families responded (and over 80% of them said that there are “barriers to engagement” and felt that “the school is not welcoming to parents”), the Board president, Paul Pouloski, said: “my biggest concern… with a 13 participation rate from families,… we are not going to get anywhere near representative data from that group”
If, when all were welcomed to participate, 13% participation rate was not considered by the Board a representative data, how can a DELT-selected 1% of families be considered representative data of the district’s highly diverse families?
On to phase 4: data analysis! Section 2 of the report for McLean County Unit 5, pages 23-75, is the Quantitative Data analysis. It consists of tables and charts that you can get directly from your 5-labs. Seriously, ask your IT person! There is NO data analysis there! (do you see any regression analysis?) In other words, the consulting firm simply reported the data back to the district, without doing the data analysis! WOW! (How much did Unit 5 pay for this?!?)
Section 3 of the report is the Qualitative data analysis. The quotes from the various focus groups (Staff, Students, and Families) are sorted out into the 5 Strands: Systems, Teaching and Learning, Student Voice, Climate and Culture, Professional Learning, and Family and Community as Agency.
After sorting the responses (this part can be done by a middle-schooler), the content for each Strand is summarized in a table with 4 columns: Theme, Stakeholder, Areas of Strength, and Areas of Needed Attention/Improvement. No further analysis was performed. This is not “data analysis” by any standards! Which of these stories are isolated conflicts between particular individuals (such as one problematic teacher or student) and which point to a problem in the system? Which are rumors (‘I heard from a friend’) and which are personal experiences, and why are they given the same weight? Etc.
At this point, one would expect additional data collections, such as classroom observations, surveys, or more interviews. Unfortunately, this is not what Dr Dubiel audits offer. Instead, you are immediately given the final Findings and Recommendations.
Finally, after all this hard sorting, we arrive to Phase 5, Findings and Recommendations, and to the last section of the report (Section 4).
Here you would expect to find a revelation and insightful recommendations on how to fix the district. Instead, you are informed that:
“It is the district’s responsibility to determine next steps, and continuously progress monitor and improve toward systemic equity.”
The recommendations are the same across the various audit reports done by Dr Dubiel. Send all the staff to (Dr Dubiel’s) PD courses on unconscious bias, diversify the instructional material, hire more minority teachers, etc. None of which will directly improve the gaps our district wrote into the Statement of Need in its Educational Equity Policy document.
On the plus side, one important theme that repeats itself in all Dr. Dubiel’s reports is to listen to “student voice” and have “School-community collaboration!” (This is based on the literature quoted on page 6, “Build leadership at all levels” and “Cultivate district wide engagement” (Fullan, 2015), where it is also recommended to “Use transparent data to improve practice for innovation and improvement.”)
Maybe it’s worth it to pay $40,000 to have Dr. Dubiel tell the district to collaborate with members of the community?!
Does the district really need to pay Dr Dubiel $40,000 to be told by her that accepting the University Team’s offer to conduct a comprehensive equity audit for free is a good idea??