The NNERPP Blog
A day in the life: Lessons from shadowing a district leader
I come from a strictly researcher background when it comes to the practice of K-12 education: while I have had the opportunity to visit schools on a couple of occasions to observe teachers, I have never been at the head of a K-12 classroom, nor have I any experience working in leadership roles at an education agency. In fact, up until Monday, February 6, I had never set foot in a district office.
Sitting in this position then, as director of a national network focused exclusively on research-practice partnerships, I have felt a little bit like a fraud at times. I have read plenty of excellent articles that describe the practitioner side of partnerships (see here, for example), but as we learned in 10th grade while working on our history fair projects, books, articles, and other written documents are second-hand information. If you really want to capture the first-place blue ribbon, you need to include at least one source of first-hand information (read: interviews).
I understood the importance of this distinction at the time (although I definitely grumbled about trying to find someone locally who could speak about their experiences on ancient Egypt), as much as I do now. Being able to see things with your own eyes is invaluable; a close second is being able to talk with someone who has experienced it for themselves. While the written word does contain value (cough, cough…like this blogpost…cough), spoken word is interactive, dynamic, animated.
Fortunately in the present case, I was able to do something about it. Last week, I had the absolute pleasure of shadowing Carla Stevens, Assistant Superintendent at the Houston Independent School District (HISD) and key practitioner-liaison for the Houston Education Research Consortium (an RPP between HISD and Rice University, and member of NNERPP). This is not a novel idea by any means but I hoped to see first-hand what it was like to work in a district office of one of the largest school districts in the U.S. (And by the way, yes, there are cubicles. Lots of them.).
In one word, it was illuminating. I cannot stress enough how eye-opening it was. If you are a researcher and there is any possibility of hanging out with a friendly district leader, seize on this opportunity. (Kudos to those history fair folks for trying to emphasize this…!).
Below I share my key takeaways from the experience.
(1) There is no.time.
From what I could tell, every minute of the workday is accounted for by projects that relate directly to their job descriptions. In other words, they have NO TIME to do anything that is not directly assigned to them. Schedules are packed with meetings and activities that are exclusively tied to tasks related to a particular job role. What this means for researchers hoping to work with a practitioner is something I’ve read in various other places: short, digestible pieces of information that they can quickly turn into knowledge is best. It is highly unlikely they will be able to dedicate much time to reading a traditional piece of research that you float their way, and even more unlikely they will be able to take a leisurely stroll through the literature to see “what’s new” (unless, of course, it’s in their job description).
(2) Research use and organizational structures / cultures may be intimately related.
Research-practice partnerships are often described as a promising way to increase the use of research evidence in practice (e.g., here). While there are multiple efforts underway to measure the veracity of this hypothesis, based on my observations, it is not as simple as: 1. Make a practitioner or researcher friend. 2. Start an RPP. 3. Increase the use of research evidence in practice? Done. (But honestly, nobody thought it would be that easy…right?).
Step three is a very complicated endeavor, for a few reasons: While your new practitioner friend is obviously very open to incorporating research into his or her life, they may be a very small piece working within a much larger ecosystem. Consider that HISD has approximately 30,000 employees across multiple levels of management; organizational processes that enable it to function have been developed over time and are not likely to change quickly (if at all).
Second, the culture itself may not be especially open to change or research use, in general. Directives in this particular district come from the top-down (and this is likely not uncommon). If the top level of administration is either not interested in research or too busy to consult with academic research, it’s not clear that cultural changes initiated from the bottom-up will have much impact.
Lastly, from what I could tell, the current use of research is very sporadic and ad hoc — if there is a current policy that needs to be developed or a program that needs to be evaluated, then research may be consulted in some way.
(3) Understanding how to incorporate each other’s expertise takes effort and time.
This observation (or something very close to it) is certainly yet another one that has been noted in the literature before (see here, here, or here). Typically, though, we hear about the importance of developing trust and building relationships. One thing Carla and I talked about, however, is that her team doesn’t necessarily know how to make good use of their research partners. While many folks within the district have research experience (e.g., PhDs), it is not a given that they will be able to clearly articulate how an external research partner could help them. For one thing, as I learned during my shadowing, you just don’t know what someone else does all day until you see it, nor are you necessarily seeking to change current processes (see point 2 above).
How can the research side of the RPP help?
Rather than overhauling their current procedures (which may happen at some point given sufficient time), Carla and I talked about finding places where the processes they have in place could be “tweaked”:
Her team already does lit reviews for reports they have to produce, but Carla mentioned that they are often too long and require a larger time commitment than what she’d like. I suggested having the research arm of the RPP do this for them (i.e., a short list of the most recent, most rigorous, or best quality papers). They really appreciated this idea and further noted that even a short annotated bibliography from which to pull from would be extremely helpful. I think this is one of the rare cases where RPP work is, in fact, mutually beneficial (and not just in service to the other partner). First, it is part of any research paper so it has to get done anyway. Second, it would help the practice side of the team directly. Third, it would enable researchers to “plug themselves” into current work and become more engaged in the district’s processes. Fourth, it could lead to follow-up conversations later, where questions such as “have you answered this question?” or “is this of interest to you?” or “have you thought about doing this?” may pop-up.
In furthering this idea of tweaking the process, I asked Carla’s team if they had considered having a researcher from the partnership sit in on meetings pertaining to data collection initiatives, such as surveys or other similar items. In the very least, the researchers would then be aware of the data collection efforts underway. They (the researchers) might even be able to offer some suggestions of which questions to include, which may then facilitate a follow-up research project. We chewed on this for a bit and then Carla’s team suggested that actually, it would be really helpful if the researchers could help develop the survey itself! Again, this allows the researchers to engage with the practitioners on something that they are already planning on doing, but with the added benefit of helping to shape a future potential research project that would already align with the district’s goals.
This one came from Carla directly: they have a mountain of reports that get completed, along with program evaluations that they themselves conduct. At the end of these, they typically offer a line that says “The purpose of this project was X. Future research should examine Y.” But rarely, if ever, are they able to complete the future research proposed. Thus, she thought it might be very helpful to hand this off to their research partners to explore. First, it provides a direct benefit to the district (they will have the future research to cite at some point). Second, it includes the researchers in the process and encourages more frequent discussion of the research agenda. Third, it benefits researchers since a project that is relevant and useful to the district is already defined — perhaps suggestions on future data collection would also be well-received. One could imagine that a follow-up project like this could end up being a dissertation chapter or a master’s thesis, or even an undergrad honors paper, depending on the scope of the question.
I am immensely grateful to Carla and her fabulous team for allowing me the invaluable opportunity to follow them around for a day. If you are currently working in an RPP or thinking about working in one, I strongly encourage you to find out if this type of (very fun!) field trip is available to you. And if you have additional thoughts on how researchers can be of better use to practitioners, please let us know. Onwards!
Developing a Joint Research Agenda: What questions remain?
I recently had the privilege to attend a convening of NNERPP’s newest member, the Tennessee Education Research Alliance (TERA), to discuss, develop, and refine a research agenda specifically tied to one of TERA’s priority research topics: “Reimagining State Support for Professional Learning.” TERA is one of the first research-practice partnerships of its kind, bringing together a state education agency (in this case, the Tennessee Department of Education) and a research institution (Vanderbilt University in Nashville, Tennessee). While this progression in the RPP world is uniquely exciting, the event itself was thrilling for me personally as it afforded the opportunity to witness the development of a joint research agenda in action. Although I have heard about the procedures used by some of our partnerships to lay out upcoming plans (see, for example, here), as with many things in life, there is no substitute for first-hand experience. Here I share some of my key takeaways from the meeting, along with several remaining unanswered questions for us to collectively think about as we continue to navigate through these domains.
(1) It’s important to have small, targeted discussions that include multiple viewpoints and a variety of stakeholders.
Approximately 50 people attended the convening, consisting of internal (to Vanderbilt) and external researchers, teachers and practitioners from around the state, and policymakers from the Tennessee Department of Education. This larger group was then broken up into smaller teams of around 8 people featuring folks from each of these units, which were then led through brainstorming, discussions, and decisions.
I had the chance to serve as a facilitator in a few of the breakouts, and can speak first-hand about the importance of having these smaller, diverse working groups tackle specific deliverables. For example, our first task during the meeting was to identify problems of practice around professional learning in Tennessee. Each member of the team brought very specific expertise to the exercise: teachers and practitioners spoke to what was happening in classrooms, policymakers shared intentions or goals behind related policies, and researchers described findings or theories to shape the conversation further. Additionally, because there was diversity in viewpoints (i.e., teachers came from multiple grade levels and practitioners included principals), the complexity of problems were more fully represented. I was amazed at the depth of conversation that took place with folks that had only known each other for about 30 minutes!
(2) Did I mention multiple viewpoints? Specifically, teachers are key.
I think the importance of having teachers actively participate in the conversation cannot be underscored enough. They bring much insight to the discussion, especially because they are closest to the learning (with the exception of the students, which raises questions about what their roles might be in this as well…). Teachers will additionally be instrumental in any post-research implementation that occurs. Having regular, open lines of communications with them will most likely improve support for these efforts.
(3) What happens after the meeting is over?
After such a fantastically executed meeting leading to a virtual treasure trove of research questions and projects, TERA will have their hands full prioritizing and completing the work. In the meantime, I offer the following questions for us to consider.
–How do RPPs keep practitioners engaged long after the meeting?
I heard many teachers/practitioners in attendance express gratitude and enjoyment at being included in the research agenda development process. They additionally worried, though, that this inclusion was a “one-time” deal.
How can RPPs maintain meaningful connections with practitioners, when the group of practitioners includes the entire state, in this case? How often should communications go out to them? What channels are best to not fall off their radars? Is the answer different for teachers versus principals?
–What is the role of external researchers that are not specifically a part of the RPP and when should their collaboration begin?
What is the best way for RPPs to take full advantage of these extra teammates? Can anyone join the external team or should it be limited to those participating in current networks? At what point in the existence of the RPP should they be included? What resources are needed to facilitate this type of collaboration? How should alignment with the research agenda be maintained?
–What communications around the research agenda are important and to whom is it important?
This is a question, but perhaps it really falls under a recommendation. I find that posting a research agenda on the partnership website is very useful, especially for those interested in learning more about the research being conducted at the RPP (see here, here, and here, for example). Not only does it help organize the publications and policy briefs that will surely follow the research, it can potentially spark connections with others interested in doing work in similar areas. Furthermore, it provides a small level of accountability for the partnership to demonstrate their progress to external stakeholders.
I had a wonderful time at the TERA convening, making new friends but also being granted a tremendous learning opportunity to better understand how developing a joint research agenda within an RPP works in practice. Going forward, my hope is that I am able to participate in more of these — learning by doing is an irreplaceable teacher and I would love to be able share the knowledge with all of you. Onwards!
New year, new goals? RPPs and planning for success
Happy new year, everyone! I hope the ushering in of 2017 went smoothly for you and brought with it much joy. As is typical during these exciting early days of a new year, many of us set out a list of resolutions, goals, or intentions to guide our actions for the coming year. (Given that it is already January 6, hopefully you haven’t fallen off the track just yet!). NNERPP is no exception, as I’ve spent the last few weeks reflecting on all of our accomplishments in our first full year of existence, and enthusiastically planning for the many projects we have in store for year 2 (stay tuned…).
This exercise is particularly useful for assessing the previous year’s activities to determine what degree of “success” has been accomplished; on the other hand, it can also help clarify where new goals should lie. Because NNERPP has four main objectives that guide its work, this is a relatively straightforward task. In the case of research-practice partnerships, however, how should we go about this opportunity? What activities should be assessed to gauge “success”? What types of goals are appropriate for a new year?
The question of “success” or how to measure “effectiveness” regarding RPPs has been raised in the literature recently, and we’ve seen it be an ongoing topic of conversation throughout 2016. While answers remain elusive, it is becoming increasingly important for partnerships to consider this aspect of their operations, as it directly affects the health of the organization. We explored this topic at the 2016 NNERPP Annual Forum back in August and talked about some of the challenges.
One of the key concerns that surfaced was noting the multiple objectives that guide the work of an RPP, given the different roles of those working within the partnership (i.e., researchers, practitioners, policymakers, postdocs, and so forth). For example, during our discussions, practitioners were more likely to suggest that positively impacting student outcomes should be a top priority for the RPP. The researcher group, though, shared that a top priority for them was ensuring that research was actually used to make decisions.
Should both objectives be considered in an overall schematic intended to measure RPP success? Perhaps, although both suffer from measurement challenges albeit for slightly different reasons. Impacting student outcomes, while a highly valuable and important goal for a partnership, can be difficult to connect to partnership work directly, given the large number of parameters at play influencing education policy in general (see here, for example). In terms of measuring research use in practice, research suggests that districts’ use of evidence is “complex and at times messy.” Liz Farley-Ripple, co-director of the IES-funded Center for Research Use in Education, further details four challenges to measuring the use of research evidence in practice.
In addition to measurement challenges, both of these objectives likely require a longer time frame in which to assess progress. So what might an RPP focus on in the meantime? In a survey administered to participants prior to the 2016 NNERPP Annual Forum respondents indicated that building trust and cultivating relationships were the most important factors for ensuring a productive partnership. These goals, while not specifically indicative of whether or not the RPP is itself successful, are imperative for “greasing the wheel,” so to speak. In the absence of efforts to build trust or relationships across the different roles of the partnership, the work itself will most likely be adversely affected and thus, larger objectives will not be met.
How should one go about improving trust and relationships, then? From our conversations with NNERPP members, we hear several mention the importance of opportunities for repeated interactions and further, the development of new systems or routines to engage multiple stakeholders. A shared respect for all partners is also key, as is the recognition of the varied expertise brought by all of those working in the partnership. We’d also like to share the following resource for those looking to refine their facilitation skills (a special thanks to Laura Wentworth of the Stanford-SFUSD Partnership for passing it along!): Facilitator’s Guide to Participatory Decision-Making, by Sam Kaner.
We will continue to work together to develop measures of effectiveness going forward; in the meantime, we hope these will give you a starting point from which to launch your reflection of 2016’s RPP activities and your upcoming partnership goals for 2017. Onwards!
Conversations with NCRPP: How can Research-Practice Partnerships Support Educational Leaders’ Use of Research?
Note: We’re cross-blogging in September with NCRPP and NNERPP! Read below for how partnerships can play an important role in research use in education. Head over to the NCRPP website for Paula’s report on the recent NNERPP annual forum.
The National Center for Research in Policy and Practice studies how educational leaders—including school district leaders and principals—use research when making decisions and what can be done to make research findings more useful and relevant for those leaders. Earlier this year, we released findings from our survey of 733 educational leaders in the nation’s mid- and large-sized school districts. Together, the findings create a portrait of district work where research-practice partnerships can play a critical role.
District leaders viewed research as valuable, credible, and relevant. Respondents reported very positive attitudes about the value of educational research, with nearly all endorsing the ideas that research can address practical problems and that researchers provide a valuable service to educational practitioners. Leaders endorsed the idea that research is relevant to practice, but indicated that the time lag between conducting and publishing research can decrease its usefulness.
District leaders used research to inform decisions, expand understanding, and persuade others. Respondents most commonly reported instrumental use of research. Common forms of instrumental use were to design professional development for teachers and administrators and direct resources to programs. With respect to conceptual use, 71% of respondents indicated that the research they encountered had expanded their understanding of an issue. Among the symbolic uses presented to respondents, leaders most frequently reported using research to convince others or using research selectively to support a decision.
Networks matter for access to research. We asked leaders to select from 14 different sources where they obtained research relevant to their work. Leaders were most likely to access research through professional associations and professional conferences. Leaders were less likely to access research through individual researchers or from stand-alone websites, like the What Works Clearinghouse or the National Center for Education Statistics. Opportunities to interact with other professionals and/or researchers around research-based ideas seem to be quite important.
Across survey results, we see an appetite for research from educational leaders, particularly research that is timely and relevant to the district context. There are a range of different activities where research can be brought to bear within the flow of district work. Access to useful research seems to be a social phenomenon.
If this is the state of affairs for educational leaders, we see a promising role for partnerships between researchers and practitioners. Research partnerships engage in joint research and development activities with practitioners. Rooted in districts’ identified needs, the research efforts may be seen as more usable and actionable. It may also be more credible to local practitioners and policymakers because it was done with their students and their local conditions. The research developed out of the partnership can fit the local needs, whether it be related to curriculum selection, design of professional development, or creation of new policies. The partnership may also provide a needed infrastructure to support the social side of research use, enabling research ideas to flow between trusted partners.
Caitlin Farrell is Director of the National Center of Research in Policy and Practice (NCRPP). See here for more on NCRPP or here for additional survey findings.
#NNERPP16: Re-Cap of the NNERPP Annual Forum, August 3-5 in New Orleans
I can hardly believe a full week has gone by as I reflect on the amazing time we spent convening last week. What a fantastic few days! This year, which marked the first official Annual Forum of NNERPP as an organization (see 2014 Agenda hereand 2015 Agenda here, where the meetings were hosted by the Houston Education Research Consortium), was the largest one yet, with over 75 folks representing state and local agencies that administer education, research institutions, foundations, and policy advocacy groups from across the U.S.
Hosted in association with the one of our members, the Education Research Alliance for New Orleans, this convening featured a broad range of topics aimed at improving the productivity of research-practice partnerships in education. Although several participants were attending for the first time, friendships were formed quickly, and folks were eager to dive deep into how to make this type of work better. While the full agenda can be viewed here, some of the highlights included job-alike huddles that grouped participants among three job roles: the directors of research-practice partnerships, the leaders of RPPs that work in education agencies, and associates in the partnerships (such as post-docs or graduate students).
Here we discussed the types of skills required to work in an education research-practice partnership (note: fostering trusting relationships built on mutual respect is a top demand) and we also thought more carefully about how we might assess whether RPPs are in fact “effective” (it was fascinating to hear the diversity in suggestions given the different job roles represented within the partnership!).
We also benefitted from hearing Kim DuMont of the William T. Grant Foundation share knowledge around how research evidence is (or is not) used in practice. This session featured two member partnerships in our network, the Houston Education Research Consortium and the UChicago Consortium on School Research. Carla Stevens and Ruth López Turley (HERC) and Sarah Dickson and Kylie Klein (UChicago Consortium) shared their experiences on how research projects conducted through the partnership impacted decision making within their respective school districts once it was completed.
We also reserved time to facilitate several breakout sessions. Here, participants were encouraged to bring ideas, share experiences, and learn from each other on important RPP-related topics such as supporting implementation of research findings, how to communicate with an external audience, issues to consider when navigating highly political environments, and tips for those interested in building a new partnership. We made use of several post-it boards and markers throughout the sessions – interactive participation among attendees was a key priority.
We even had time to connect while dining together! Several of our participants graciously agreed to host informal “table talks” on additional topics that Forum attendees indicated high interest in exploring via our pre-Forum survey. These included conversations about Improvement Science, Design Based Implementation Research, how to address leadership turnover when working in a partnership, ESSA implementation, and strategies to secure funding for those in middle to later phases of partnership work.
Speaking of ESSA implementation, we also had the pleasure of hearing the varied perspectives of several organizations that are interacting with ESSA as it moves forward. Our next steps are to think about how research-practice partnerships fit into the newly adopted bill…stay tuned!
Our final “team building” activity of the Forum was to squeeze in for a group photo on one of the many staircases of the gorgeous Le Pavillon hotel, where our meeting was held.
Pulling off such a fantastic meeting involved the work of many folks, which I’d like to acknowledge in this final space: immense gratitude for the thoughtful participation of the attendees, Jim Kohlmoos (Senior Advisor to NNERPP), the NNERPP Steering Committee members, Margaret De Sosa and Bill Fulton of the Kinder Institute, the Kinder Institute for Urban Research, note-taking powerhouses from the Houston Education Research Consortium and ERA New Orleans, and of course, our funders (William T. Grant Foundation, Spencer Foundation, Laura and John Arnold Foundation, Wallace Foundation, and Annie E. Casey Foundation), without which this meeting could not have taken place.
As those in attendance know, NNERPP: Phase 2 has officially launched. I could not be more thrilled!