Methods of Evaluating Helplines and Hotlines

There are a number of methods in place to evaluate helplines and hotlines for the purposes of obtaining grants or other funding as well as demonstrating their effectiveness to their stakeholders (volunteers, clients/callers and staff). Some of these approaches are discussed here, along with their benefits and drawbacks.

Most organizations will not find one method suitable for every situation; for instance, while a qualitative method of evaluation may allow staff to determine which volunteers are ready to work the helpline on their own, a cost-per-call approach may be needed for a governmental funder, while key performance indicators may be necessary for clients or strategic organizational improvement.

Technical and Quantitative Methods of Assessing Helplines

• General Program Evaluation Approach
• Response Time
• Information Collected
• AAS Accreditation Standard
• Fowler Technical Effectiveness Scale
• Helper’s Response List
• Cost-Benefit Calculation

Clinical and Qualitative Methods of Assessing Helplines

• Carkhuff & Truax Scale
• Crisis Call Interaction Form

Each of these methods are briefly reviewed below; links to the scales (where available have been provided in their descriptions.)

Technical and Quantitative Methods of Assessing Helplines

General Program Evaluation Approach

The program evaluation approach focuses on the goals and outcomes of the individual organization and determining if the calls that are taken are adequately meeting those goals. Often this requires self-reports on the part of the volunteer and judgement calls about the content.

Advantages

• Versatile technique, can be easily adapted
• Can be given to a variety of stakeholders, and used to justify funding
• Easy for volunteers to understand what is expected of them

Disadvantages

• Requires significant planning to define and monitor data collection requirements
• Can be hard to operationalize outcomes

Response Time

The response time is simply the amount of time from a caller initiating a phone call to the helpline. In services that simply route calls to a phone line that must be answered. Services that use an answering service to take calls that are not immediately answered will obviously have a slower response time. Services that take no direct phone calls, routing all calls to an answering service typically have the slowest response.

Advantages

• Easy to record initial data (computerized system such as Chronicall or manually)
• Easy to see data trends

Disadvantages

• Doesn’t describe skill of volunteer
• Data vulnerable to manipulation if self-reported
• Of limited value to external stakeholders

Information Collected

Similar to response time, this approach simply uses a checklist to ensure that the helpline collects the information required. For instance, in helplines that collect gender, age, type of problem, and so on, a simple check to ensure that this information is being collected for every call can help establish if volunteers are in fact taking calls and asking the required questions.

Advantages

• Easy to record data, low tech solutions (can be done with pencil and paper)
• Easy to analyze data in Excel or similar program

Disadvantages

• Vulnerable to manipulation or fabrication of information
• Issues of confidentiality are raised
• Of limited value to external stakeholders

AAS Accreditation Standard

The American Association of Suicidology has a set of criteria or standards that they use to accredit crisis centres, primarily in the United States but to a lesser extent in Canada. These standards are organized across a number of dimensions:

• Administration and Organizational Structure
• Screening, Training and Monitoring Crisis Workers
• General Service Delivery
• Services in Life-Threatening Crisis
• Ethical Standards and Practice
• Community Integration
• Program Evaluation

In each of these areas there are components, and those components have standards that have to be adhered to, most of which use a 3-level structure.

For instance, under the area “Administration and Organizational Structure”, the first component is “Governance”, and the rating scale is presented below:

Rating Scale

Level I

• The legally constituted Board meets at least four times a year.

OR

• The program is a part of a for profit organization that is dedicated to serving the general community.
• There is community input into governance.

Level II

• Level I requirements are met.
• There are policies and procedures that detail the Board’s responsibilities and role.
• There is broad community representation on the board or there is a highly involved advisory board providing input to the program.

Level III

• Level I and II requirements are met.
• The Board has a succession plan for itself and Executive Director, a board orientation procedure and a strategic plan.
• The strategic plan is reviewed at least quarterly.
• The board arranges an annual audit or review of finances by an independent auditor.

Advantages

• Extremely comprehensive and detailed
• Easy to tell if requirements have been met
• Standards are directly relevant to crisis line work

Disadvantages

• Standards may not be relevant to centres outside the United States
• Can be difficult to achieve the minimum standard in organizations with limited budgets
• Leaves the “on the phone” evaluation up to the Centre

Helper’s Response List

The Helper’s Response List, described in Mishara and Daigle (1997) is a list of behaviours that helpline works can do, both good and bad, while on the helpline. The purpose of this tool in the context of Mishara’s study was to determine which of these behaviours was associated with better outcomes. The 20 items have been reproduced below:

1. Sustained silences by the helper between two caller’s statements.
2. Incomplete thought. Pauses in the middle of a helper’s sentence to force the caller to go on.
3. Orientation/investigation. All kind of techniques, such as questions, aimed at orienting the call in a particular direction or at investigating a problematic aspect.
4. Short and nondirective statements (“Hum-hum,” “I see,” “Go on”) showing acceptance to the caller.
5. Statements to lower the caller’s anxiety.
6. Selective approval of caller’s behaviors.
7. Intentional misinterpretation. Intentional misinterpretations of the caller’s statements to have him/her improve the communication.
8. Referring directly to moral or ethical explanations.
9. Rejecting false, negative, or unproductive caller’s statements.
10. Classic reflections of feelings but also simple or selective restatements.
11. Information/suggestion/advice. Giving specific information, suggesting possible alternatives or directly advising.
12. Very imperative statements to stop the course of an action (“Stop taking this drug or we will give you no more help”).
13. Information/suggestion/advice (P). Same as in 11 but within the procedure (P) of continuing the communication process.
14. Threat (P). Same as in 12 but within the procedure (P) of continuing the communication process.
15. Clarification/interpretation. Trying to clarify or to give more meanings to what was said by the caller.
16. In-depth interpretation. Psychodynamic explanations.
17. Personal experience. Referring to the personal experiences of the helper.
18. Third-party experience. Referring to others’ experiences.
19. Having the caller take the point of view of someone else or of him/herself at a different time.
20. Information about helper. Giving specific information about the helper (name, age, interests, and so forth).

These behaviours can be used to identify the activities that your helpline workers are engaging in to evaluate their skills on the helpline.

Advantages

• Versatile; can be tailored to helpline’s needs
• Learn exactly what helpline workers are doing on the line
• Allows you to monitor changes over time

Disadvantages

• Difficult to disclose to external stakeholders; can’t really be used for fundraising
• Technology required to listen to phone calls raises ethical concerns
• Can be labour intensive to code calls

Fowler Technical Effectiveness Scale

Like the Helper’s Response List, the Fowler Technical Effectiveness Scale (Fowler and McGee, 1973) evaluates worker behaviour with a series of simple yes or no questions.

Advantages:

• Quick to administer
• Simple standards; no debate about whether they were met
• Includes suicide questions in every call

Disadvantages

• One specific disadvantage of this scale is that it is very action-oriented, which may not be helpful for helplines following the Samaritans model

Cost Benefit Calculation

A cost-benefit calculation is a simple statistic that allows you to calculate how much money each call your centre receives. The simplest way to calculate this is to add up all your costs (data that can be found in your audited financial statements for the current year), and dividing it by the number of calls you’ve taken.

For instance, if you have a budget of $500,000 (small for many centres but quite large for others) and took 15,000 calls, you would divide 500,000 / 15,000 for a rate of $33.3 per call, perhaps a prohibitive amount given the source of the funding. If instead you take 50,000 calls with the same amount, the cost drops to $10 per call.

Of course, this doesn’t take into account how long each call is.

A more complicated pair of formulas is below, based on ones used to perform a cost-benefit analysis of a helpline in England (Hunt, 1993):

1. Number of Minutes on Phone / (Hours Available x 60) = Helpline Usage Percentage
2. Take the variable costs of staffing the service (the fees incurred based on the number of calls, for instance the phone bill) and divide them by the number of minutes on the phone
3. The number that results is the average cost per minute on the line.

Advantages

• Useful for fundraising and grant purposes
• Easy to see changes from cost-cutting measures
• Simple to calculate

Disadvantages

• Low utilization rate may affect funding
• Cost reductions may damage service
• Doesn’t give any indication of volunteer effectiveness, only efficiency

Clinical and Qualitative Methods of Assessing Helplines

The clinical scales discussed below are all designed to evaluate individual worker and caller interactions. For that reason they all have similar advantages:

• They can be used to track improvement in workers over time
• They provide direct feedback about caller’s experiences on the line
• They are versatile and can be used for many types of distress or crisis calls

In the same way, they also have similar disadvantages:

• Technology is required to record or listen to the calls for the purpose of rating
• Inter-rater reliability is not perfect so there can be variance in the ratings of the calls

Carkhuff & Truax Scale

The Carkhuff and Truax scale is a qualitative scale that rates the empathy of the responses that the worker gives to a statement or statements by the caller. Every time the worker speaks, their interaction is rated on the scale.

The original scale given by Carkhuff and Truax in their ground-breaking 1967 book used a nine point scale but (perhaps because it was more difficult to differentiate between so many levels, as Carkhuff himself indicates during his original volume), it was later reduced to five volumes (VanCleave, 2007); a version of this five point scale is the one provided.

We have the full scale on a dedicated page in our site.

Crisis Call Outcome Form

The Crisis Call Outcome Form was designed by Echterling and Hartsough (1989) and identifies nine behaviours found in effective helpline calls. It’s important to note that not every behaviour is necessary to occur in every call. An X or checkmark is placed in each call where the behaviour occurs.

There are other tools that may be used for rating the skill of volunteers either before they begin or during.

Additional Tools for Evaluating Potential Volunteers:

• Crisis Center Discrimination Index
• Interreactivity Index (IRI)

Bibliography

Echterling, L.G., Hartsough, D.M. (1989). Phases of helping in successful crisis telephone calls. Journal of Community Psychology. (17)249-257

Fowler, D.E., McGee, R.K. (1973) Assessing the performance of telephone crisis workers: The development of a Technical Effectiveness Scale. In D. Lester & G.W. Brockopp (Eds.). Crisis Intervention and Counseling by Telephone. Springfield: Charles C. Thomas, 1973.

Hunt, S. (1993) Relateline: An evaluation of a telephone helpline counselling service for marital problems. British Journal of Guidance & Counselling. (21)3

Mishara, B. L., & Daigle, M. (1997). Effects of different telephone intervention styles with suicidal callers at two suicide prevention centers: An empirical investigation. American Journal of Community Psychology, 25, 861–885.

Truax, C., & Carkhuff, R. (1967). Toward effective counseling and psychotherapy: Training and practice

VanCleave, D.S. (2007) Empathy Training for Master’s Level Social Work Students Facilitating Advanced Empathy Responding. ProQuest.