Assessment is required to know whether or not a project is achieving its broad goals. Monitoring is the process of collecting information about project implementation as it is happening to allow for changes if things are not working. To measure the impact of a project, it is important to assess the situation both before, and after the project intervention. The Assessment and Monitoring Team (AMT) monitors the effectiveness, results and impact of the project on the communities served. This work is integrated with the work of other groups participating in the project (Content and Outreach) to ensure projects are responsive to observed impacts, however, where independent assessment is required, a third party research partner is engaged.

   Assessment & Monitoring
   Quantitative & Qualitative Assessment
   Ethnographic Action Research (EAR)
   Using Stories to Illustrate Impact
   Interactive Feedback Loop

• Assessment and Monitoring for EA projects in Nepal

Assessment & Monitoring Team (AMT)
The Assessment and Monitoring Team (AMT) is composed of in-house Monitoring and Assessment staff (including staff who are either based in the field or who spend significant time traveling to field sites) and international and local assessment partners. Partner organizations are chosen on the basis of their experience and skills in establishing baseline data, undertaking ongoing monitoring and assessments and conducting evaluations of community based projects and projects involving media and community awareness. The AMT collaborates to establish measurements for the impact of the project, thus permitting effective reporting to donors, the Advisory Board and project management, as well as for use by Equal Access in mobilizing resources.

The rigorous assessment process carried out by the AMT helps us to obtain critical information on key development indicators, such as literacy, women's empowerment, income and employment in connection with our project work. Careful and consistent monitoring of the communities' responses to the broadcasts are fed back to the content group, facilitating program improvements both in regards to the specific project, and also for similar development initiatives.

Quantitative & Qualitative Assessment
The AMT uses both quantitative and qualitative means to assess the impact of projects on community participants by assessing changes in knowledge, attitudes and behaviors (KAB) in the program areas being addressed (e.g. HIV/AIDS Prevention and Women's Empowerment). Quantitative tools are used to assess the project impact in terms of numbers and statistics (e.g. what percent of people can accurately identify modes of HIV/AIDS transmission?). Quantitative tools allow for comparisons to be made across different groups and measure the extent of change brought about by a particular intervention. Qualitative tools are used to assess more subjective and intangible project impacts - generally extremely important in the context of our communications projects, (e.g. people's feelings about discussing sensitive subjects like HIV/AIDS, changes the program brought into their lives etc.). These tools are useful to understand why particular effects occur as a result of an intervention.

A comprehensive quantitative questionnaire is produced to assess the KAB at three intervals:

1. Before the program intervention (i.e. before the audio broadcasts commence)
2. Midterm through the project, and
3. At the end of the project

This allows us to measure the changes brought about as a result of the program interventions.

In addition to quantitative techniques such as audience sampling and Knowledge, Attitude and Behavior change surveys, Equal Access is using a variety of approaches to obtain qualitative impact information including Ethnographic Action Research (EAR), the use of stories from beneficiaries to indicate impact and the extensive use of letters from listeners as a source of feedback.

Ethnographic Action Research (EAR)
EAR is developed by UNESCO and Queensland University of Technology, Australia, and is being implemented by UNESCO in its different Information and Communication Technology (ICT) projects throughout South Asia. The integration of the EAR methodology within each of the projects leads to a 'research culture' that ensures research findings are continuously fed back into the development of projects. Training through workshops, through field visits and through ongoing support and communication will provide local researchers with a range of tools to ensure that a rich understanding of local contexts is used to guide and inform a projects ongoing development.

Additional qualitative tools such as interviews with community members, anecdotal information, focus group discussions held with members of the listening groups will integrate into the EAR approach to give us a deeper understanding of the quantitative information. It is also provided to the Content & Outreach Teams to ensure that programs resonate with the needs of community members and are tailored to maximize project effectiveness.

Using Stories to Illustrate Impact
We are cognizant of significant program impact; whenever we visit beneficiary communities there are many specific examples of program impact, however, we have difficulty measuring this impact against traditional indicators using quantitative means of data collection as life changing events often do not fit into neat frameworks of quantitative surveys. Using and analyzing stories to illustrate impact offers a method to define and extrapolate impact from qualitative data - primarily "stories" (which are really examples of change).

We are currently investigating the use of the Most Significant Change (MSC) approach. The quantitative or structured means which MSC uses to classify, filter, verify and draw conclusions from stories of positive and negative changes that emerge from a project. Feedback letters from listeners, numerous stories and other observations from the field can all be filtered to draw out consistent themes and solid examples of impact, evidenced by single stories, but indicative of broader trends. As the filtering and classification process is quite similar to the processes we currently use with listeners feedback letters, and the Content Advisory Group, we are well placed for implementing this methodology.

We can also use stories to provide a real connection between our supporters and the communities we serve. Our site www.myvillagelink.org offers more information and the opportunity to learn about the lives of those we work for.

Interactive Feedback Loop
The AMT establishes an interactive feedback loop between the Content and Outreach Teams to ensure that the content formats and information resonates with the needs of the participants and achieves impact. Listening group facilitators record the demographics of group attendees by gender, age, and marital status. They also record the comments and suggestions provided by the group members and forward these to the content team at regular intervals. Letters from listeners (we receive more than 800 per month) also provide feedback. The Content Team modifies the programming to address these comments and suggestions. Our programming is responsive to community needs because of the quick turnaround time that the satellite technology provides. Answers to listeners letters can be broadcast in a timely way and new program themes and refinements can be introduced within a 2-8 week timeframe.

Challenge Programs Stories Methodology Technology About Us Donate