Qualitative Evaluations: The easier route??

Recap: “We soon discovered that we did not have enough data to do a quantitative evaluation and thus had to focus on a more qualitative one. The next few days involved writing up questionnaires and contacting as many stakeholders as we possibly could in order to set up interviews to ask them questions about the organization and their relationship with it.”

There are a number of different ways to do evaluations. Utilization evaluation is an approach that constructs evaluations based on its usefulness to its intended users. This is what we aimed to do with our evaluation of RealStart – we wanted to ensure that our evaluation proved useful to the organization.  In order to do this, we focused on RealStart’s Program Theory: is the organization and operation of RealStart conducive to achieving its intended mission and vision?  To do this, we created a series of surveys designed to target this central question.

We assembled the contact information for everyone involved with RealStart: donors, board members, staff, former students and current students.  While we administered similar survey questions to everyone, we also tailored certain questions designed to measure a specific aspect of the program for the different surveyed parties.  In order to assess RealStart’s Program Theory we needed to construct survey questions that could “compare” responses among the different groups to determine the cohesiveness of the organization.  Through a comparative analysis we also sought to determine whether or not the program was having the intended effect on its recipients that its mission articulated.  To do this, we asked staff, former and current students similar questions regarding specific aspects of the program.  We then compared staff responses with student responses as well as analyzing the difference in responses among former and current students.

Qualitative Analysis taught us a lot to say the least.  Given the lack of feasibility of mass communication for some of the former students, we literally had to physically track them down in the city.  While interviewing some of these students, we also had to jump over a few communication hurdles.  The students we interviewed came from either Xhosa or Afrikaans backgrounds.  While they spoke fluent English, we found that certain ideas weren’t communicated through our survey questions.  As these survey questions were administered through oral interviews, we had the opportunity of re-wording our questions when they didn’t understand what we were asking.  We realized, though, that produced bias in our interviews.  If we didn’t ask all of our interviewees the same questions verbatim, could we expect to generate unbiased responses?  

A journey of a thousand miles begins with a single step...

Hello out there!  Our names are Julianne and Ally and we spent six weeks of our summer performing a program evaluation for a non profit (RealStart) in George, South Africa.  George is a picturesque seaside town located about 4 hours east of Cape Town in a region called the Western Cape, and sits right on the coast of the Indian ocean.

George is a relatively small town by South Africa standards, with a population of 120,000. Like the rest of South Africa, it is incredibly diverse and unequal with large affluent golf resorts backing onto poverty stricken townships.

We arrived in George on Thursday, May 16^th, after 48 hours of traveling, two upgraded first class tickets, one ripped pair of jeans and one coke stained pair of leggings…and one lost bag. All three of our suitcases managed to make it across three continents, yet somehow the journey from Johannesburg to George was too much of a stretch. Here, finding a lost bag is like trying to track a tadpole in a large pond.  All of the bags have labels on them, but no one knows where they are.  After numerous phone calls with the typical response, “um, we’ll look for it” or “let me ask my friend who works in the airport” (we know what that really means), and a day of sharing clothes, we finally decided to head out to the airport and wait there until they located the bag. They told us they had heard nothing of the bag since their last employee had called earlier that morning.  Their suggestion was wait for the next flight from Jo-burg landing in George in 15 minutes and see if it had appeared with the luggage.  We waited for the flight and then watched all of the luggage filter through and no bag.  Then, Julianne looked across the room and low and behold her bag was sitting in the corner of the room...just chillin’.

A little bit about RealStart – RealStart is a South African based non-profit which focuses on the holistic development of impoverished young people. They currently run a year-long youth development program across two campuses, with the hope of expanding the program throughout the country. They also have a number of other exciting projects in the pipeline. I (Alison) have been affiliated with RealStart for a number of years and have known its founder for five years. This was Julianne’s first interaction with the organization.

The first week of our evaluation consisted of us finding out a little more about RealStart and what it was they wanted us to achieve with our evaluation. As non-experts in the field, we spent copious hours sitting outside in the sun reading a program evaluation textbook one of our professors had given us.

We soon discovered that we did not have enough data (RealStart is only 3 years old) to do a quantitative evaluation and thus had to focus on a more qualitative one. The next few days involved writing up questionnaires and contacting as many stakeholders as we possibly could in order to set up interviews to ask them questions about the organization and their relationship with it.

Our next blog post will be coming soon...check in for the next installment, which may contain a crazy adventure or two!