The following excellent questions were asked during the recent APCO training Webcast on Public Safety Quality Assurance Best Practices and Tools. Lori VanGilder, Manager of Quality Assurance Services at Replay Systems, APCO Institute Adjunct Instructor and prior Communications Supervisor in Florida, provided the following answers below for your review.
Our QA program is viewed by the dispatchers negatively. We offer several training steps to assist the dispatcher in improving. What advice do you have to make the QA process a more positive experience?
First of all, it is great that you do have a program in place. If there are negative perceptions of the program, start by asking if you utilize the evaluations on the employees annual review? If so, this provides documentation of all the times they did great, and not just a supervisory file of the occasional negative situations that occur. I would also ask if everyone on the staff has been trained on the program to understand its process and objectives. Lastly, perhaps you want to try ensuring that you are including motivational feedback and not just the documented deficiencies. Are you sending out positive feedback on great performance? And is it timely? Also, are you sending out informational responses with your QA regarding stress management? Try to make the program "feel" more empathetic to the Telecommunicators.
Our agency has had QA established for quite some time; however, the scores were always very high and rarely pointed out errors that needed to be addressed. Now, there is a new supervisor completing the QA (previous dispatcher/trainer) and she is fair grading all the same across the board. Many dispatchers/call takers are offended and upset with the supervisor because errors are now being brought to their attention. She also uses positive feedback and constructive criticism when necessary. Is this negativity a typical phase that centers go through and what can we do to help the morale.
Unfortunately, what you have described is very common occurrence. If the program has been in place and not fairly administered, the new person completing the evaluations faces some challenges. I would recommend starting out with a training session for the entire staff regarding the policy that governs your quality assurance program. In that training make sure everyone is aware of the grading standards and what is viewed as compliant/non-compliant/ and exemplary for scoring. There are going to be some hurt feelings, however stepping forward and addressing the issue should also garner her some respect from the team.
Once the criteria is well understood, we would recommend that you introduce some elements of call taker / dispatcher participation into your QA process. For example, you can have them flag some of their calls for evaluation – such as those where they did exceptionally well. Be quick to recognize them for the work well done as soon as such calls are reviewed. You can also allow the call takers/dispatchers evaluate some of their own calls, following the same process as the supervisor. People typically rate themselves more strictly then when third party does it – listening to their calls after the fact provides a different experience than when they are involved in the call for the first time. These evaluations would be subject to supervisory review and finalization. We also recommend that call takers/dispatchers receive a copy of their evaluation with embedded call/radio recording for review and specific, constructive comments that can be entered into the evaluation if they believe that their rating was unjust. The evaluator may still revise the ratings (if the employee feedback is valid) and finalize the evaluation thereafter.
What statistical data suggests that "many agents are under-motivated, and see QA as only punitive"? We have had QA in place for years - and that is not the case here.
You are indeed in a very fortunate situation. This data is published by a variety of organizations to include:
- National Academies of Emergency Dispatch
- 911Trainer.com
- 9-1-1 Magazine whose team conducted an extensive research to assess many aspects of today’s QI/QA at 9-1-1 centers and published results in their magazine. They queried dispatchers, supervisors, QI/QA personnel, managers and EMD system providers – in all states around the nation.
- Individual sharing posts from dispatchers can be found on social media sites (LinkedIn and Facebook) and in blogs
- Apart from this information, multiple attendees of our Webinar meeting posted comments about their centers experiencing this same situation – their dispatchers/call takers view the QA process very negatively
Do you suggest one person do all QA work or divide it up by Shift/Supervisor? And if it is divided up, would it be a good idea to have supervisors only evaluate those not on their respective shift?
The answer would depend upon your agency staffing. In some cases there are too many calls to be reviewed by one person for this to be practical, and in others one single point of contact is an excellent idea. The most important thing is to ensure that any and all evaluators are grading the same way- that is considered calibration of the gradings. Also, it is important that those who conduct the evaluations understand the need for confidentiality, otherwise a Telecommunicator could become embarrassed and end up with a negative perception of the program as a whole.
In some states, medical QA is protected from legal discovery (i.e., it is not available for use by attorneys in lawsuits. Is there any similar protection for dispatch QI? If not, another approach is to not maintain QI info as a permanent record - meaning use it for improvement then delete.
Unfortunately, we are not able to advise an answer for the legal implications as each state has their own statutes. Please turn to your own agency’s legal division.
Do you recommend Q&A if you are not a 911 center or the main emergency answering point. You get the occasional emergency call that you transfer to the 911 center but for the most part you only answer non-emergency or after the call type calls. If you do, what type do you recommend?
Yes. Quality Assurance can be conducted in any type of call center. Any personnel who are coming in contact with the public on telephones can/should be evaluated to ensure that the contact is positive for everyone. Your evaluation forms would follow the requirements and criteria that are relevant to your organization and its objectives.
I am a working Supervisor, how much time is dedicated to the QA program?
To conduct a complete evaluation, document it and apply some coaching/learning (if needed) can take approximately 20 min per call. If the call is exceptional, obviously that time is greatly reduced. Assignments of coaching/e-learning can be automated, so that it is automatically triggered based on results of you QA evaluation.
What is the business model for using ReplayQA? Per call evaluated, monthly fees, etc.?
ReplayQA pricing is based upon call volume and system needs.
Is there anything in CALEA, APCO or NENA certifications that require QA programs in Communications centers?
Yes. All of them have standards that require Quality Assurance. APCO Mini Telecommunicator Training Standards, APCO P33 certification standard, Calea does have a QA standard, NENA call processing standard 56-6 and NFPA standards 1710 and 1221..
How important is calibration of QA? We have 4 people review the same call for monthly random QA. Then calibrate any scores with a 10 point variance? It's a lengthy process.
Calibration is excellent and it is great that you are including it in your quality assurance program. It sounds like there is not a lot of effort invested into the objectivity of your evaluation process. You should be able to reduce the number of calibrations needed by clearly outlining what specific conduct or performance equates to each grade. These internal standards can be conveyed in a meeting format and reinforced via electronic training where the standards are reiterated.
What is the legal liability associated with disposing of QA recordings and coaching documentation? Is ALL QA discoverable?
Unfortunately, we are not able to advise an answer for the legal implications as each state has their own statutes. Please refer to your own agency legal division.
Does RePlay QA hire independant contractors to work from home doing reviews
Yes, Replay hires experienced Telecommunications professionals to perform as Assessors and complete the evaluations from their homes.
Do you see any negative issues with having call takers and dispatchers doing the QA reviews and not just supervisors?
If the grading criteria are firmly established in your policy, then the grading should be consistent, regardless of who performs the evaluation. However, the confidentiality issues may prevent evaluations done by call takers or dispatchers from being a good idea, unless you are referring to agent self-evaluations. We do recommend self-evaluations. In general, the personnel conducting the evaluations need to be capable of keeping the information in the evaluations confidential so as to not embarrass or discredit a peer, which would create a negative view of the entire program.
My agency currently has a QA program and we have an initial evaluator and 2 additional raters per call. Our program takes an average of all three scores for the call-takers final score. Is this overkill to have 3 evaluators. Do most QA programs only utilize one rater?
Your agency clearly has seen the need for calibration of the ratings on the evaluations, which is great. However, we do see having three people evaluate the same call as a redundant, likely unjustified usage of time and resources. Perhaps you want to look at the grading criteria for your evaluations and work to remove all subjectivity so there can be more belief that the correct ratings are issued by a single evaluator. Organizations typically perform one calibration session once or twice a month, which has been shown to suffice as long as grading criteria are clearly defined and the evaluation forms use the proper selection of grading choices to address each evaluation question. For example, 10-point sliding scales invite more variance between evaluators than 3 or 5-point scales. Yes/no answers may be most appropriate with questions that relate to accuracy, and the like.
There are many people who believe that expectations such as the 1221 standard on call processing times (< 60 seconds for 95% calls) is unrealistic when performing EMD on EMS calls. Would you comment on that please?
There are many recent discussions on this topic on APCO PS Connect. Yes, the standards are very high. However, you have to keep in mind that many agencies who have a separate call taker and dispatcher, as well as smaller agencies where your partner can dispatch while you continue talking to the caller, are meeting the standard by sending the call to the dispatcher as soon as basic information is obtained and then updating the screen and the responders after the initial dispatch. That helps a great deal with the call processing time.
Currently, we cannot buy software so how do we best start a QA program?
You can conduct a QA program completely without the software. Create a paper form or use MS Excel and define your grading criteria there. Then randomly select calls from your recorder to evaluate and tabulate the results. If budget is an issue, you may also want to consider leasing the VPI quality assurance software for an affordable monthly payment.


Just like a home that sometimes needs to be refreshed, traditional quality assurance has reached a fork in the road of its 30-year life. Now's the perfect time to re-evaluate the way we monitor and measure our call center agents and “cleaning house” in the Quality Assurance (QA) department to make room for analytics-oriented QA tools and best practices. By implementing this makeover in your contact center, you can radically reduce manual steps required by most QA applications by more than 60 percent. And, who among us can’t use 60% less manual work?




In an already competitive and crowded marketplace, organizations today are continuously seeking ways to lower operating costs and increase revenues. The landscape has changed dramatically in the past decade. Organizations are closely scrutinized by regulators and consumers who have high expectations from outsourced partners and 3rd party vendors. In the wake of the global financial crisis that has negatively impacted some of the world’s leading economies, organizations are starting to revisit their strategy, in terms of outsourcing contact center contracts, and demanding more than just cost savings. This mindset is now creating concerns among outsourcers and 3rd party vendors, who can no longer attract and maintain their customer base just on the basis of their ability to provide a cost savings advantage.
Life has been very exciting around VPI lately. We recently launched our VPI Empower Suite 5.2 and several new capabilities, especially our Smart Evaluations and have received rave reviews from Ventana Research, DMG Consulting and our customers.
Credit unions and community banks are increasingly coping with federal and local legislation, interest rate hikes and volatility in capital markets, all of which entails placing greater emphasis on local and regional customer retention, quality assurance, internal controls, as well as more careful compliance and risk management. Although these smaller banks share many common challenges with large contact centers, they have unique needs that can be addressed using today’s modular, highly customizable call quality monitoring and workforce optimization solutions. 



The days of monitoring contact center agents randomly to find out what they have been doing wrong on calls is history. In these days of enlightened leadership and sophisticated intelligent routing technology, call quality monitoring has also evolved from the days of internal surveillance to performance improvement and skill development. 
In today’s challenging economic climate, analytics is changing the way companies do business and dramatically improving contact center operations, managers want quick and accurate insights into the effectiveness of their contact center operations in order to make prudent, timely decisions, but they don't have the time or resources to listen to and review the vast amount of customer interactions handled by the agents every day. 




