Despite early success, their first evaluation was not considered rigorous or scientific enough, in particular by the public health and academic communities, and so they worked with experts and peers who had criticised the first evaluation to do another larger scale evaluation which combined qualitative and quantitative approaches over a three year period. They were keen to assess impacts both at the individual and social level and to capture some of the complexity of social change.
The evaluation showed that the programme had lead to changes in the areas of gender and equality norms, stigma reduction and personalisation of risk.The TV show was just one element of the programme, but with the widest reach: 6 out of 10 people have regularly watched at least 2 out of the 3 seasons produced. This equates to 729,784 young people.
Despite the impacts shown, many people found the results disappointing, due the lack of large scale direct impact. However, the evaluation showed evidence of promoting debate around taboo subjects amongst the public which was another objective, and the interpersonal communication that the programme encouraged – in particular communication with a partner – had an important indirect role in promoting condom use and encouraging more gender equitable social norms.
There were other complexities that Puntos were able to draw from the evaluation results, such as the fact that change was promoted at each end of the scale – with those who already had progressive views, these were re-affirmed, and for those with more conservative opinions, they also showed small changes in attitude – small but significant in the context of promoting wider social change.
One of the challenges for the evaluation was that rather than promote clear cut messages, with characters in the TV drama being simply good or bad, they created more complex characters that would encourage people to engage with and discuss the issues themselves. It was aired at Sunday dinner time in a widely catholic country so it created discussion and conflicting views/confusion within individuals. These complex changes are hard to measure, especially as the process is long-term.
Understanding the process is often as important as the outcomes, and yet traditional evaluation techniques do not always help with this understanding, or address the fact that people’s behaviour and the way they learn is messy, not linear. Even while the donors were quite happy with the results of the evaluation, Puntos wanted to understand more about the process of change. It was mainly Puntos themselves who still had questions – in addition to the public health community and academics who were interested to see that the evaluation met their standards.
Rather than experts designing messages, Puntos were keen to find ways to work with other organisations to develop a sense of key issues and areas that needed discussion and to work on processes to promote such discussion at many levels together. They had worked in this way with the original story-lines for the Television show and wanted to extend this in future.
