is supposed to happen<\/strong>, and the implementation analysis goes to see if it actually happened on time and as planned. <\/p>\nIn the case of Family STAR, the critical points identified in the drafting phase of the logical framework concerned the weight of the “human variable” and, in particular, the possibility of poor adherence<\/strong> of the different actors (schools, families, teachers) to a very innovative design and evaluation approach-a risk that materialized and was addressed within the project (see next section).<\/p>\nGiven these critical issues, the most suitable tools<\/strong> for analyzing specific project phases were identified. The development of these tools took several months. Specifically, these were: <\/p>\n\n- an effects estimation questionnaire to be administered to students, for which the paper machine-readable mode was chosen: a more expensive mode, but also easier for schools;<\/li>\n
- A satisfaction questionnaire, to be administered to all participants in the RFPs, again paper-based and machine-readable;<\/li>\n
- An online questionnaire for training participants;<\/li>\n
- An online questionnaire to class coordinating teachers;<\/li>\n
- A logbook, compiled by facilitators and spokespersons throughout the journey;<\/li>\n
- several tracks for interviews and focus groups with local project coordinators, trainers, school leaders, school project liaisons, facilitators and spokespersons.<\/li>\n<\/ul>\n
The main difficulty encountered in the development phase of data collection instruments, particularly with regard to questionnaires, is finding the right balance<\/strong> between the comprehensiveness of the questions and the streamlining of the instrument itself: on the one hand, in fact, it is in the interest of researchers to collect as much information as possible; on the other hand, however, it is necessary to use parsimony so as not to overburden the respondent.<\/p>\nSuch a trade off is always valid, and in the Family STAR project it required several reflections: the research team opted to survey in greater detail<\/strong> the characteristics of the RFDs through the tool of the logbook, filled in by facilitators and spokespersons, who-as project workers-could count the time spent filling in in the total amount of time devoted to preparing and conducting each RFD.<\/p>\nOn the other hand, with regard to the instruments aimed at parents and-to a lesser extent-teachers, special attention was paid to ease of response<\/strong>, both in terms of content (avoiding asking the respondent questions that involved particularly difficult reflections) and in terms of the length of the questionnaire and the time required to complete it.<\/p>\nRealization vs Evaluation: opposing needs?<\/h2>\n
The design of the evaluation framework and development of the tools took about a year<\/strong> to complete, during which the implementers began the phase of engaging schools.<\/p>\nThe second year<\/strong> of the project saw the start of the actual evaluation and project activities, immediately encountering a low adherence of schools to the project and the application of a small number of students by teachers, such that the achievement of the project objectives (number of participants in the RFPs and the control group) and the significance of the achievable estimates were jeopardized.<\/p>\nThe implementation analysis investigated the reasons underlying the low uptake of the project by schools and individual teachers. Project activities were reshaped<\/strong>, through: <\/p>\n\n- a lengthening of the implementation phase of the RoFs over two years and the engagement of new local organisations (and therefore new schools), to widen the pool of participants and compensate for the low take-up;<\/li>\n
- the suggestion to teachers to report “twin cases,” i.e., students who-according to their responses to the initial questionnaire-appeared to be facing similar difficulties as the students nominated for RoFs. Once the list of twin cases was received from the research team, the decision on whether to nominate them for a RoF remained with the teachers, subject to the need to draw them into the treaty or control group; <\/li>\n
- the possibility for teachers to indicate cases for which a DRF was urgent and which were therefore included in the treatment group without being randomized and thus effectively exiting the evaluation.<\/li>\n<\/ul>\n
Thus, an attempt was made to resolve the conflict between the needs of schools<\/strong> and those of evaluation<\/strong>. The needs of assessment have not always been well received by school leaders, who have said that they feel they have to devote more attention to teaching rather than research. Such divergence of interests is typical in the world of evaluation, where opposing needs<\/strong> meet (and sometimes clash). The evaluator is therefore called upon to carefully consider whether to make pressing demands of the stakeholders, yet without losing the scientific rigor required to provide answers to scientifically relevant questions. <\/p>\nA deep understanding of the context in which one operates, as well as a phase of shared involvement and planning certainly help to identify appropriate timeframes and tools that can meet the opposing needs.<\/p>\n
Relevant-though not always taken into account-is the aspect of future prospects: while the evaluator’s task ends with the presentation of the results of the analysis and policy directions, the actors involved in service design certainly have broader perspectives, and it is useful to take them into account to ensure that the proposed innovations can be followed up even after the research activities are completed.<\/p>\n
Lessons learned: assessment outcomes<\/h2>\n
Despite the difficulties encountered, the effects analysis<\/strong> was able to show that the intervention was effective in improving students’ perceived parental support and reducing their conflicts with teachers. However, it also showed that DRFs did not improve students’ self-efficacy or make them more optimistic about their future educational paths. The effects seem more positive when considering students from families with higher cultural capital and when the relationship between parents and teachers is not conflictual. <\/p>\nThe implementation analysis<\/strong> revealed lights and shadows of the implementation process of the activities: there were the aforementioned obstacles to achieving full involvement of schools in the project, mainly due to lack of resources and skepticism towards innovative models; however, in specific circumstances, the data show a general appreciation for participatory and family empowerment models by all actors. Interesting information related to gender and father figure involvement was also collected. <\/p>\nIn conclusion, the counterfactual approach adopted in the research design developed for the Family STAR project is obviously not replicable<\/strong> to all European projects, as it requires dedicated resources, a structured approach, a large volume of potential beneficiaries, and the ability to build a control group with certain characteristics. In addition, the consistent adoption of a counterfactual approach made it necessary to overcome obstacles and reshape the project activities themselves. <\/p>\nHowever, the combination of two different evaluation logics<\/strong> (effects evaluation and implementation analysis) allowed the project to produce solidevidence<\/strong> with respect to both its effects and the processes activated. The results of the evaluation made it possible to draw firm conclusions with respect to the effectiveness of the approach tested and can guide the implementation and progressive improvement of future interventions. <\/p>\n","protected":false},"excerpt":{"rendered":"Let’s delve into what it means to design and implement an evaluation system in a European project with a concrete case.<\/p>\n","protected":false},"author":10,"featured_media":41231,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[731,2189],"tags":[2314,2325],"class_list":["post-41401","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-archive-archive","category-archive-stories","tag-experiences","tag-how-to-methods-for"],"acf":[],"_links":{"self":[{"href":"https:\/\/euknow.it\/en\/wp-json\/wp\/v2\/posts\/41401","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/euknow.it\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/euknow.it\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/euknow.it\/en\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/euknow.it\/en\/wp-json\/wp\/v2\/comments?post=41401"}],"version-history":[{"count":0,"href":"https:\/\/euknow.it\/en\/wp-json\/wp\/v2\/posts\/41401\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/euknow.it\/en\/wp-json\/wp\/v2\/media\/41231"}],"wp:attachment":[{"href":"https:\/\/euknow.it\/en\/wp-json\/wp\/v2\/media?parent=41401"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/euknow.it\/en\/wp-json\/wp\/v2\/categories?post=41401"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/euknow.it\/en\/wp-json\/wp\/v2\/tags?post=41401"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}