The Role of Monitoring and Evaluation in Assessing Progress of Operational Research in the EAPHLNP Study Sites in Kenya

Abstract

Introduction: Studies on performance of Operational Research (OR) projects have outlined the various enabling factors leading to  favorable research outcomes. OR plays a key role in filling the gap between what we know from research and what we do with that  knowledge. This has been expressed over time, based on monitoring (progress indicators) and evaluation (performance indicators) results.
Objective: To document the performance of OR activities based on Monitoring and Evaluation (M&E) indicators as well as highlighting lessons learned.
Methodology:  M&E framework was developed for the OR studies in three thematic areas; Tuberculosis (TB), malaria enterics as well as components of capacity strengthening and administrative. That was done by KEMRI OR Secretariat in consultation with the East Central Southern African – Health Community (ECSA-HC) Secretariat and the principal investigators of each thematic area from East Africa Community partner states (namely Burundi, Rwanda, Tanzania and Uganda). The framework included outcome indicators for each study, target values defined in accordance with approved protocols. Reporting interval was set at quarterly per year. TB studies had 8 reporting indicators, Enterics had 5 reporting indicators, malaria had 11 indicators, while administrative and capacity building had 15 reporting indicators. The framework was then adopted by the region. In Kenya, the initial roll-out of the research in all three thematic areas was
done in February 2013. The first quarter of M&E was conducted in the study sites, as defined in the editorial of this journal, in June 2013 while the second “quarter” was carried out in June 2014.
Findings: Between February 2013 and June 2014, there was little progress in all the three thematic areas. During the first evaluation, the number of enrolled respondents presumed to have TB at the satellite facilities were 185(6.2%) of the expected target number 3,000 persons. Non-satellite sites enrolled 124(8.2%) of the expected 1,520 persons presumed to have TB as well. In the second evaluation, enrollment at the satellite sites was at 13.3% compared to 1.2% in the non-satellite sites. That represented a two-fold percentage increase in the satellite sites compared to nonsatellite sites. Using ZN outcome indicator, there were differences in the number of actual TB cases detected in both sites compared with the target values. Number of TB – cases detected using ZN at satellite sites rose from 7.8% (target of 784 cases) to 11.0% (target of 1725 cases). In the non-satellite sites, there was a decline from 3.1% (target of 508 cases) to 1.1% (target of 1118 cases). In the Enteric Study, there was a marginal decline in the number of patients recruited from 21.6% (target of 1440 patients) as at the first evaluation to 17.2% (target of 1800 patients) in the second evaluation. For malaria study, a total of 333 patients had been enrolled against a targeted of 300 patients into the study representing an over enrollment of 111% from one site. For the administrative indicators, the OR Secretariat had over attained in three target areas namely publication and sharing of OR findings in country and regional bulletins and held OR-Technical working group meetings.
Lesson Learnt: Low performance in achieving indicators in both TB and enteric studies was resulted by including; high staff turnover particularly in the non-satellite sites, high workload and breakdown in communication among sites' personnel in regard to participation in research activities. In the satellite sites where study interventions were provided, better performance in achieving indicators was attributed to improved capacity in personnel and other non-financial motivational aspects, such as site exchange visits, refresher courses and frequent attendance of project workshops and meetings. Scientists from KEMRI assisting in patient recruitment, specimen collection and shipment alongside with study site staff resulted to over-attainment of performance indicators. Such was demonstrated in the malaria study. The observed lag time between the various M&E field visits by OR team could have partially contributed to the missed opportunities of identification and correction of any deviations from the project protocols. Changes in leadership especially at the project top management at the Ministry of Health affected the overall performance of OR activitie due to delays in disbursement of funds and delays in obtaining no-objection to incur expenditure on essential activities that were not originally in the approved annual workplans.

Conclusion and Recommentdation: Itt was established that several factors, some of which could have been augmented if the M&E exercise was conducted in accordance with the framework, affected the achievement of study indicators. M&E component is a crucial activity especially in tracking research progress and should be conducted consistently within the stipulated timeline. This will subsequently provide opportunities of early identification and correction of any deviations from the protocol.

Keywords: Operational Research, Monitoring & Evaluation, indicators, targets, performance, factors