Treatment service users (TSU) project: phase two

5.2 Evaluation findings

Page last updated: March 2011

Evaluation interviews were conducted with staff and consumers of the five participating sites in the months following the baseline interviews. The primary aim of these interviews was to explore and record the impact of the demonstration projects, from both consumer and staff perspectives. The following section details the findings under a number of organising themes: awareness, understanding, interest and valuing in consumer participation; stability; who is a consumer; sustainability; expectations — project, consumer representative; power and empowerment.

Prior to addressing the findings proper, it is worth briefly revisiting the breakdown by service type of the five participating agencies. A brief overview of each demonstration project has also been included. These overviews have been written using primarily data from the staff interviews augmented with that from consumer interviews. To ensure the independence of the evaluation and the confidentiality of participating services and service consumers, the names and identifying details of the services have been withheld.


  • A metropolitan, government-operated pharmacotherapy service:

    The aim of the demonstration project was to recruit, train and notionally employ two services users as consumer representatives. The role of the consumer representative was envisaged to provide both support and advice to peers on the pharmacotherapy program, particularly those new to the service. As part of the recruitment process, the local drug user organisation was invited to run training and education sessions for both staff and consumers; subsequently the two representative positions were filled. It was at this juncture — just prior to the consumer representative's deployment on 'the floor' — that the project stalled. The primary explanation offered subsequently by staff interviewees was the high turnover of staff, particularly the key contact staff member. The protracted delay that followed included the late remuneration of the two representatives. At the point of evaluation, the key contact had returned to the service and efforts were underway to revitalise the project and, most especially, the flagging enthusiasm of the consumer representatives.

  • An inner-metropolitan, non-government, primary health-care service model:

    Here the aim of the demonstration project was two-fold: Firstly, to garner feedback from consumers concerning the service's recent move into new premises, especially the co-location of the needle syringe program with the primary health-care unit; and secondly, to organise a consumer-driven review of the existing 'Client Rights and Responsibilities Charter'. A well-known, long-term service user was approached directly by key staff and appointed (discretely) as the consumer representative. A questionnaire was drafted by the project's key staff member, with input from the local drug user organisation, and reviewed widely by staff. This instrument was then administered by the consumer representative to those consumers willing to participate. At the point of evaluation a short report had been tabled by the key staff member using de-identified material from the questionnaire. The second part of the project was yet to be completed.

  • A regional, non-government, community-based service model:

    There were three aims of the demonstration project: Appointing a consumer representative; reviewing policies and procedures; and running a series of regular focus groups with consumers. It is important to note, however, that the consumer consultant did not identify as having any current drug treatment experience but did have other 'consumer' experience. At the point of evaluation, the service had only recently appointed the consumer consultant and the other outcomes had not been achieved. The manager of the drug and alcohol team, however, remained committed to the project and it was intended that the consumer consultant would continue reviewing policies and procedures and, with the help of a staff member, commence running the focus groups. At this juncture, concerns were noted regarding the apparent disengagement of both consumers and staff from the project. It was hoped that the appointment of the consumer consultant would begin to address at least the former. Top of page
  • An outer-metropolitan, government-operated, multi-disciplinary service model:

    The principal aim of the demonstration project was to 'legitimise the voice of the consumer' via the establishment of a 'consumer participation council'. The 'amazing reluctance' to get involved, even among supportive staff, coupled with misunderstanding (even resistance) from other staff, resulted in little progress at the point of evaluation. The notion of a 'consumer council' was already a concession to those staff concerned about higher levels of consumer participation (such as the involvement of consumers in staff recruitment etc.). Nonetheless, the service director remained committed to the concept of consumer participation and a long-term goal of having a consumer representative on the payroll. In practical terms, a steering committee, comprising partner agencies (including the local drug user organisation), had been established and the terms of reference and job description for the 'lead' consumer councillor had been completed.
Residential rehabilitation:
  • Based on a therapeutic community model:

    The broad aim of the demonstration project was to better meet the needs of service consumers via increasing their levels of participation. At the point of evaluation, staff fed back three key achievements: Increased levels of consumer feedback sought (and received) across all stages of the program; the establishment of a 'formal' meeting held once a week between management and senior residents; and lastly, the establishment of a 'consumer reference group' that met fortnightly and was drafting a questionnaire regarding staff training needs. Staff of the therapeutic community reported very positively on the progress and outcomes of the demonstration project. This was commonly at odds with data from consumer interviews — a point of discussion addressed later in the report.
5.2.1 Awareness, understanding, interest and valuing in consumer participation
5.2.2 Stability
5.2.3 Who is a consumer?
5.2.4 Who should represent treatment consumers?
5.2.5 Sustainability
5.2.6 Expectations - project, consumer representative
5.2.7 Power and empowerment

5.2.1 Awareness, understanding, interest and waluing in consumer participation

Baseline interviews with service consumers noted a limited awareness of the term 'consumer participation'. The introduction of the demonstration projects made limited, if negligible, impact on general levels of consumer awareness — both in terms of their knowledge of the actual demonstration projects or the concept of consumer participation generally. The clear exception was in those instances where consumer representative positions had been created as part of the demonstration project and the consumers fulfilling these roles were able to be interviewed. Those services that engaged drug user organisations to deliver tailored education and training appeared to foster a better understanding of consumer participation among service users than those that did not.

As noted in the baseline findings, staff were more likely than consumers to be familiar with the term 'consumer participation', even if they had had no practical experience of consumer participation in a drug treatment setting. With regards to an awareness of the demonstration project itself, staff interviewed in the evaluation stage exhibited a range of understanding and interest. While individual staff differences — such as their employment status as casual or part-time — may be partially responsible for the disparity in levels of knowledge, a further explanation may lie with the developmental histories of the demonstration projects. In some instances, projects had been developed in relative isolation, with only a manager or several interested staff involved, and with little communication more broadly among staff. It is worth noting that the model of consumer participation promoted in the TSU Project: Phase Two shows the importance of involving staff and consumers from the outset. However, in practice, none of the services consulted or included consumers in the development or writing of the project proposals, or in later activities such as writing the position description of the consumer representative or, even when consumers were actively involved, reviewing and revising the project plan. While the lack of pre-existing consumer participation may have understandably precluded consumers from early involvement in projects, it is less clear why staff should also have had such limited engagement.
Top of page
The general high level of support for the concept of consumer participation among staff noted at baseline remained so during the evaluation interviews. As both the following examples illustrate, for some staff the notion of consumer participation was a matter of principle:

'It is actually part of our organisational ethos that we're a community health centre so you can't call yourself "community" if you don't have community involvement.' (Male staff member)
'Our clients are the experts in their own lives and, you know, we've gotta listen to them... And it's not a matter of we're the experts sitting here on our high and lofty mountain looking down on people, you know. It's about people. And they're actually the experts in their own life. And if we don't work with them then we're, I feel, then we're not, we're not doing what we're supposed to do.' (Male staff member)
Nonetheless, staff support for consumer participation was neither uniform nor unproblematic. As one senior staff member explained, there was still an attitude of 'I don't like this', 'I don't want this' (male staff) among some staff. While another service manager noted that he anticipated some staff to 'whiteant' the demonstration process and had put in place contingency plans for strong staff opposition to the consumer participation project (male staff). The attitude of staff towards the demonstration project — specifically the notion of consumer participation — was an issue of concern for some of the staff interviewed. As one staff member explained, when introducing the concept of consumer participation, one must 'tread very carefully', particularly when dealing with a large cross-section of staff (male staff). Indeed, this same staff member went on to suggest that the greatest barrier to the project succeeding was 'us — the staff'. So while none of the staff interviewed actually expressed any opposition to the concept or implementation of consumer participation, they were nonetheless aware of antipathy among members of their team.

Levels of awareness and/or understanding of consumer participation remained universally low among consumers. Moreover, and more alarmingly, knowledge of their services' demonstration projects was equally poor. Consumer interviewees were frequently unfamiliar with the subject matter of their interview and commonly required prompting by staff prior to interview. The type of service, the state of project progress, and the particular consumers interviewed were all influential in shaping responses to questions of interest and belief in consumer participation. The role of service type will be addressed in detail elsewhere in the report but, generally speaking, consumers of the residential rehabilitation and detoxification services appeared to exhibit relatively less belief in their capacity to participate — that, indeed, having input into the running of a service was typically considered a 'staff job'. As with staff, the level of project progression clearly contributed to levels of awareness and understanding of consumer participation among service users.

There appeared to be a correlation between the extent of project progress and the level of awareness and knowledge demonstrated among consumers at interview. As noted earlier, this was also evident where drug user organisations had provided education and training sessions as part of the demonstration projects' implementation. Finally, there was a clear relationship between levels of consumer understanding and their level of involvement (or otherwise) in the demonstration project. Unfortunately, interviews with 'appropriate' (i.e. involved) consumers could not always be arranged, suggesting a limited involvement of consumers in the demonstration project and/or a limited relationship between services and consumers.

Having remained unobserved or unremarked upon during baseline interviews but evident during evaluation, was varying degrees of consumer cynicism regarding staff motives for promoting consumer participation. Some service users indicated the involvement of consumers in service evaluations appeared tokenistic at times — that staff would seek out what consumers wanted but not put it into action. Some consumers were critical of what they considered to be bias in the manner in which consumers were recruited to provide service feedback, suggesting services' attempts to tailor responses. The following examples are illustrative:

'They are very picky in this place sometimes, like who they pull out... that's a lot of what goes on here, it's just fuckin' feathering of the nest in the extreme.' (Male consumer)
'They are very careful about who gets to say what.' (Male consumer)
'They tend to, with things like this, they let certain people know... who would be good for it.' (Male consumer)
What remained a strong theme during evaluation interviews, as during the baseline interviews, was a conviction among consumers in the value of 'lived experience'. The latter was invariably coupled with and judged superior to what might be euphemistically termed 'textbook learning'. The following are illustrative:

'You can't learn that stuff from a textbook.' (Male consumer)
'My experience tells me that only an addict knows an addict.' (Male consumer)
This seems to suggest that while considerable skepticism, if not explicit cynicism, exists among consumers towards a service's commitment to genuine consumer participation, there was nonetheless a privileging of consumers' knowledge (as 'lived experience') over that of a textbook variety commonly attributed to staff.Top of page

5.2.2 Stability

Among both staff and consumers interviewed at baseline, the subject of 'stability' was interpreted exclusively as a question of consumers' suitability for roles as representatives or advocates. While at baseline the definition of 'stability' was somewhat fluid — in so far as its definition changed across service type, invariably reflecting the service's treatment ethos (typically 'abstinence' or 'harm reduction') — it was nonetheless used and understood as an attribution belonging to individuals. During baseline interviews the range of views regarding stability was similar among both staff and consumers, including among those individuals identifying as current 'users'.

In evaluation interviews the term 'stability' was again employed to describe an individual's status, with a number of consumers stating that many consumers were simply too 'unstable' to assume responsibility within the organisation. However, what also became apparent during evaluation was the question of service stability: Was the service itself fit for duty? The lack of stability of staff and the concomitant lack of organisational memory was a predominant feature of evaluation interviews, particularly among service staff. All five participating sites experienced considerable delays and disruptions with their projects. One service's project that had initially exhibited promise was effectively left 'feel[ing] like a rudderless ship' in the aftermath of continual staff changes; or as the interviewees remarked:

'And I have to be perfectly honest here, it's not gone smoothly... We've had a succession of people being the prime mover and as people leave, [are]seconded, lose interest, that sort of thing... We've had many, many hands being, being the prime person.' (Male staff member)
'I think a handover would have saved us five months of stuffing around.' (Female staff member)
While there was divided opinion as to the need for a paid, dedicated staff position solely responsible for the promotion of consumer participation, there was universal support for the need for stability of key staff during the project, including — as a minimum — a designated and recognised 'contact person'. One service advocated having few staff directly involved in the demonstration:

'Easier to have less people, keep it simple.' (Male staff member)
Nonetheless, opportunities were afforded all staff to contribute. This same service emphasised the importance of having not only a 'stable' management team during the demonstration but a 'stable and experienced' staff, ideally one with a history of, and familiarity with, consumer participation.

The term 'core' was one commonly deployed by staff interviewees to describe service 'business' that was deemed essential as opposed to matters considered marginal or 'non-essential'. In at least one service the demonstration project was considered to have suffered because of its 'non-core' status. As one staff member explained:

'And I guess at times the, just the workload... in the whole centre meant that the consumer rep project wasn't getting the, the time and input that it required.' (Male staff member)
In this instance it was felt that the absence of 'core' status meant that while the 'lofty ambitions' of the demonstration was laudable, it was neither well-planned nor adequately resourced. Furthermore, it was stressed by this interviewee that this lack of attention to the project was compounded by other 'non-core' projects taken on by the service at the same time. This participant suggested that, in future, such projects be introduced one at a time, ensuring adequate resourcing, including a dedicated staff member to promote sufficient and sustained focus.

An analysis of the broader, public health and socio-economic context within which these individual services operate, and within which the demonstrations were introduced, lie outside this report. Nonetheless, it is worth noting that several senior staff emphasised the pressures of a 'system' that placed ever greater demands on services and staff with little or no provision of additional resources; or, indeed, the role of 'decision makers' who are not always aware of the particular constraints operating within individual workplaces.

5.2.3 Who is a consumer?

Not unlike the concept of 'stability', the term 'consumer' was somewhat free-floating in its application, tending to reflect both individual and contextual variations. The term took on different meanings within different types of treatment. Many users of residential rehabilitation or detoxification services considered all ex-users to be 'consumers'. Alternatively, some consumers receiving opiate substitution therapy believed ex-users to be too far removed from the drug-using community and therefore unrepresentative of 'consumers'. In the context of consumer participation they believed such ex-users had the potential to do more harm than good as 'they think they are better than us' (consumer).
Top of page
It was noted during interviews and visits to services that, in some instances, a high proportion of staff identified as past drug users. In one service, for example, it was reported that anecdotally up to 70% of staff were 'ex-users'. The vexing question of when a 'consumer' becomes an 'ex-user' remains an important, if unresolved, one. It is of central importance precisely because the TSU Project has been founded on a division between those who provide and those who consume services. Are, for example, some staff of the aforementioned service to be considered 'consumers' — and, by extension, others not?

In another service the team leader mooted possible concerns about staff 'allegiances' when discussing several young team members he identified as having their own drug- using histories. These staff were also considered by their team leader to be the most vocal and committed advocates of consumer participation. Such staff have invariably been employed on the basis of their professional qualifications and not as a result of their personal history, yet are sometimes still identified on the basis of the latter. This raises important conceptual and practical questions concerning self-other identity making: blurring what has been constructed as an absolute distinction between staff and consumer.

5.2.4 Who should represent treatment consumers?

One unexpected issue that arose in interviews with consumers in both residential rehabilitation and detoxification settings was the matter of discrimination between people within the same services. The TSU Project: Phase One excluded services (and, by extension, the consumers of those services) that provided treatment for alcohol use only. As the TSU Project: Phase Two was focused on evaluation of a series of demonstration projects in a realist framework, it was not feasible or appropriate to exclude consumers on the basis of whether they were being treated for alcohol or illicit drug use issues. A number of the services involved in the demonstration projects provided a 'mix' of services for both licit and illicit drug use issues and therefore projects were open to all service consumers. Although the majority of consumers interviewed at baseline and evaluation were illicit drug treatment consumers, others were accessing the service for either licit or a combination of licit and illicit drug use issues. For other services, such as pharmacotherapy, it should be noted that some consumers are managing issues associated with the use of multiple substances.

At baseline and evaluation a number of consumers talked of the differences between their own drug use and that of others, the primary differences being 'drug choice'. On separate occasions, service consumers receiving treatment for alcohol use expressed a very clear distinction between the use of alcohol and what they deemed as 'drug' use, which for them was illicit drug use only. In one instance a consumer stated he didn't like 'chemical users' for a number of reasons including negative perceptions of illicit drug users being nasty, selfish, untrustworthy, and unable to stick to the rules, such as bringing drugs into the service, unable to socialise with others, etc. Following these comments, the interviewer questioned how such views (that 'alcohol' users were perceived to be of a better calibre or more suitable for certain roles than those in treatment for illicit 'drug' use) might be managed in relation to consumer participation in a service that provides treatment for both alcohol and other drug dependence? In particular, could one person represent both groups if there were different needs for different consumers? The consumer stated in response that he thought it would be best for different groups of consumers (based on drug of choice) to be represented separately.

5.2.5 Sustainability

The question of sustainability, with regards to consumer participation, is a complex and multi-faceted one. What follows is necessarily only an overview of several key issues.

Higher/external management:

The role of higher management, such as area health services or executives of community health boards, were cited as crucial for both the short- and long-term success of consumer participation. Without their imprimatur, service provider participants considered current or future consumer initiatives were unlikely to be introduced or sustained. The presence of external structures required careful consideration and negotiation, along with a recognition of potential limitations. Policies such as those governing the payment of consumer representatives, mandatory police checks for employees, and the placement of volunteers were three examples cited. Conversely, the presence of supportive senior management acted as a facilitator, as the following example illustrates:

'... having a supportive management structure was really important — really important. And it gave us the confidence to do this.' (Male staff member)
Top of page Experience:

Feedback from evaluation interviews suggested that the small successes experienced during demonstration projects were instrumental in building the confidence and the belief to attempt further initiatives. The following description from a residential rehabilitation staff member captures the enthusiasm and momentum that can follow consumers' participation:

'It's magic really... just incredible seeing people's confidence build as they're given more and more responsibility. And they are actually stepping up to the mark.' (Female staff member)
The benefits of consumer participation were not limited to consumers but were also noted among service providers, particularly those who had previous positive experiences of consumer participation. Some staff with backgrounds in mental health brought with them valuable training and experience in consumer participation. For some staff these earlier experiences had proven formative in their ongoing approach to service provision and consumer work.

5.2.6 Expectations - project, consumer representative

Given the low levels of awareness among consumers of either consumer participation as a concept or its application via the demonstration projects, it is difficult to comment generally on the expectations of consumers. Of those consumers who were interviewed at evaluation most had little to no expectations. A notable exception was among those consumers who became consumer representatives during the course of their service's demonstration project. Among this small cohort the expectations were high at baseline, as the following example illustrates:

'... what I'm hoping it will achieve is a common ground for staff and consumers to work out things, like you know, how do you put it in to words? It's like an even plane so everyone's on the same level, everyone's the same, like no-one's better than anyone, no-one's — even though this person works here and you're a consumer, you both got the same rights and you're both equal. You don't have to sit there, you know put your head down and think "Oh they're the staff I can't say anything what if they take my kids? What if they do this? What if they do that?" you know.' (Consumer)
In this instance, despite a promising start — including the active utilisation of the local drug user organisation and a carefully crafted consumer representative selection process — the demonstration stalled at the point of real possibility. The reasons for this were varied — principally the high turnover of key staff — but one consequence for the consumer representatives was the service's protracted delay in remunerating them. This failure, itself a symptom of broader failures, understandably resulted in the diminished enthusiasm, motivation and trust of the two consumer representatives affected.

One senior staff member expressed his 'annoyance' at what he considered to be the lack of adequate support or resources made available to undertake the service's demonstration properly. This had resulted in what he saw as the 'unfair' treatment of consumer representatives — 'playing with people's emotions', their sense of self-worth and trust in the organisation:

'I'm thinking of one consumer rep in particular [who] was taken on board as a life-changing experience... to have responsibility, to have input, to be able to give back to a service...' (Male staff member)
Alongside the limited outcomes described above were also positive reports of some services' reinvigorated commitment to the principle and practice of consumer participation as a result of the demonstration project; of high expectations for the future. One staff member described what she believed to be the effect of the project:

'It's reinforced what we do but I think, more importantly, the "why" we do it. It's not just habitual. But it's made us revisit the, the rationale. It's been very, very useful; professional practice-wise it's been very useful.' (Female staff member)
The same optimism is also present in the staff member's description of consumers' participation:

'The absolute goodwill and enthusiasm, dedication and commitment from all the consumers... consumers have been the stars of this project.' (Male staff member)
Similarly, a senior staff member of another service suggested the outcome of the demonstration project had been to reinforce the value of consumer participation and the service's confidence in it, auguring well for its future development and expansion.

Finally, the position of consumer representative, introduced as the key constituent of several demonstration projects, warrants mention within this discussion of expectations. The position of consumer representative was challenging, if not also rewarding, for the individuals appointed. The position of consumer representative was subject to not only occasions of jealousy and mistrust from fellow service users, but burdened too by their inappropriate and sometimes onerous expectations. The risk for consumer representatives was exposure to a level and type of demand that was often unrealistic and inappropriate. As this consumer representative foresaw at baseline:

' [Service users] want [methadone takeaways] and they won't be able to get them because it's just government policy, and we can't change that, even as consumer reps. Even as staff they can't change that, so...' (Consumer)
It was as a result of such demands that services focused considerable attention on publicly clarifying the role of the consumer representative and enacting appropriate forms of support and supervision for the position. In several cases the latter involved ensuring external supervision was available from the local drug user organisation in addition to internal staff support.Top of page

5.2.7 Power and empowerment

The themes of power and empowerment remain a defining issue if consumer participation is to be successfully developed in the drug treatment context. The following brief discussion of power and empowerment will focus on the position of consumer representative, drawing on comments and observations made in the course of the evaluation interviews. While obviously other expressions of power were at play during the demonstration projects, such as the higher management of area health authorities discussed earlier, these will be explored in a more general and in-depth exploration of power and empowerment canvassed later in the report.

The position of consumer representative provides an elucidating illustration of the complex and, at times, blurry nature of power and empowerment within the context of consumer participation in drug treatment. On one hand the consumer representative appears to occupy a position of increased power (and empowerment) relative to their peers. Indeed, it can be understood as a promotion, even attracting jealousy from others. On the other hand, what power can be said to accrue to the position of consumer representative is only so if permitted by staff; the reigns must be loosened by those ordinarily holding them. This 'letting go' of control was, in turn, described by one senior staff as a 'challenge' that was 'complex and layered'.

In one service, staff reported witnessing the personal empowerment (growth in confidence etc) that accompanied a consumer's appointment as a consumer representative, while within another service a staff member bemoaned the 'playing with people's emotions' (the 'disempowerment') that resulted when the consumer representative positions were not appropriately remunerated nor properly exercised. Paradoxically, the position of consumer representative can only be effective if the individual is still recognised as a 'peer' by the consumer group. Coming to be identified as 'one of them' (a staff member) effectively robs the position of credibility, and therefore power, among its constituents. (A hypothetical variation on this theme was mooted by interviewees and might be termed the 'yes man': effectively an overly compliant peer chosen by staff to act as a token consumer representative.) In both instances it seems that the 'voice' of the representative must remain identifiably 'ours' to consumers in order to remain authentic and empowered — as 'us' and not 'them'. Yet this in turn appears to draw upon and reinforce the very dichotomy (of staff versus consumer) that consumer participation purports to challenge. Consumer participation may work best when these distinctions are appreciated in positive and collaborative terms.