Strong surveys and quality questionnaires: Tips for success
By: Blair Bigham, Ian Blanchard, Jan Jensen
Surveys are a common tool used both in research studies and other projects, such as quality improvement initiatives. Surveys may appear to be a simple undertaking – just ask a few questions, and get some answers, right? One only needs to administer one survey to realize this is not the case. In this edition of Spotlight on Science, we’ll give you some tips on things to think about when developing a survey. As well, read on for a recently conducted pilot survey study by Paramedic Leeann Douglas and her colleagues from Toronto. Find out about why Leeann is involved in research in the Meet Paramedic Researcher feature.
Tip #1. Be specific. What do you want to learn from your survey? The more precise you can be, the better (1). A survey on how paramedics feel about their practice is much less likely to be well designed and yield useful responses than a survey on paramedics’ confidence of their ability to perform advanced procedures. Are you interested in how much your participants know about a topic? What their attitudes are towards something? Their experiences?
Tip #2. Understand what research questions surveys answer best. Surveys can take a variety of formats, including multiple-choice questions, such as:There is an adequate amount of continuing education offered to paramedics in my service.
1. I strongly disagree
2. I disagree
3. I’m not sure
4. I agree
5. I strongly agree
Questions may also be fill-in-the-blank:
Please state how many years you have practiced as a paramedic: ___
or open-ended:
Describe a time when …
If you want to survey many people, stick with multiple-choice questions as much as possible. If your questions can’t be answered well with multiple-choice questions, and you really want to allow your respondents to describe their thoughts or experiences, consider a qualitative study where you would conduct interview or focus groups with a smaller number of people.
Tip #3. Search for existing surveys on the same topic. Chances are you’re not the first person to think of your survey idea. It is preferable to use a survey that has been used in another project with your sample, rather than create a new one from scratch. There is an electronic database for just this purpose! The Health and Psychosocial Instruments database contains thousands of survey tools (you may need to go to a university or hospital library to access it). By typing in “paramedic,” I quickly uncovered several surveys, including ones on stress in medical personnel (2), seat belt use in trauma patients (3), and cardiovascular risk factors in paramedics and emergency nurses (4).
Tip #4. Critically think about each question. You may understand what you are trying to ask – but will everyone who is completing your survey? Face validity means that a question means what it is supposed to mean. Content validity means a question or collection of questions ask about all-important aspects of an issue. Reliability means that two people who feel the same way about a topic will answer the question similarly, and that a respondent will answer a question the same way today and again in a week. It also refers to slightly different questions that are assessing the same thing will be answered similarly (1).
Tip #5. Give it a test drive! Ask a small group of people to pilot test the survey. They are to complete it as they would as a participant, and also provide you with feedback on the flow of the survey, if the survey is too long or short, if the instructions make sense and are in the right spot in the survey, if the questions make sense. Also, analyze the data received from the pilot as you would for the real deal. Make sure the data you receive makes sense, and if it doesn’t, edit the survey.
Tip #6. Seek help with analysis. Analyzing survey data is not as simple as stating what percentage of respondents answered a question with option ‘b’. Statistical analysis can help you make more sophisticated inferences from your data – such as PCPs with less than five years experience are more likely to have positive attitudes towards paramedics providing health prevention initiatives than other paramedics. Consulting a biostatistician or a researcher with experience in survey research when you are designing your survey can make a huge difference in the information you have at the end. However, don’t wait until after you have your survey responses back – your questions may not have been set up in a way to give data that is useful for advanced analysis.
Tip #7. Think strategically about the best way to deliver the survey. Surveys can be administered as paper with mail-out surveys, or electronically, via e-mail or a website. While one may assume that electronic surveys will yield higher response rates, this has not been clearly established (5). Think about your sample. Are they paramedics with work e-mail? Then, a link to an electronic survey may be best. Or, maybe you will give the survey to paramedics while they are at a meeting or education session – a hard paper copy would be best for this situation, so they can complete it right away. Are they members of the public, who may be of various ages and have various levels of computer savvy? Well, maybe it would be better to mail out paper surveys. Survey response rates are often low, so think of the distribution method that will be most convenient for most of your sample.
Tip #8. Think ethically! Do you think you may like to publish the results of your survey in a journal or present them at a conference? Make sure you obtain research ethics approval before you distribute your survey. The Tri-Council Policy Statement, a federal document that guides the need for research ethics review, states that any research that involves a human participant (as surveys do) requires ethics review (6). Research ethics boards will review the study plan to make sure the survey is truly voluntary, that responses will be anonymous and the researcher will keep the data secure. If you’re not sure if you need ethics review, contact your local ethics board and ask!
Surveys can be an extremely valuable tool to glean information from a group of people, and the results can be very meaningful if the survey is well executed. However, there are a few mistakes that can easily happen – such as trying to get too much information from one survey, or not testing the survey out. These are easily prevented with good planning and editing of the survey before distribution.
References
1. Bordens KS, Abbott BB. Research design and methods. A process approach. 5th ed. Boston: McGraw-Hill; 2002.2. Hammer JS, Mathews JJ, Lyons JS, Johnson NJ, Hromco JG, Lyons JS, et al. Medical personnel stress survey--revised--job dissatisfaction and organizational stress subscales. Community Ment Health J. 1995 01/01;31:111-25.
3. Passman C, McGwin G, J., Taylor AJ, Rue,L.W,,III, Passman C, McGwin G, J., et al. Seat belt interview schedule. J Trauma. 2001 01/01;51:105-9.
4. Stanford University MC, Barrett TW, Norton VC, Busam M, Boyd J, Maron DJ, et al. HEAR2T cardiovascular risk factor assessment survey. Prehosp Disaster Med. 2000 01/01;15:86-9.
5. Shih TH, Xitao F. Comparing response rates from web and mail surveys: A meta-analysis. Field Methods. 2008;20:249-71.
6. Tri-council policy statement: Ethical conduct for research involving humans. version 2 [Internet].; 2011 [updated April. Available from: http://www.pre.ethics.gc.ca/eng/policy-politique/initiatives/tcps2-eptc2/Default/.
Death Notification In The Field: A pilot survey study
LeeAnne Douglas BSc MScCH (c), Savithiri Ratnapalan MBBS Med MRCP(UK) FRCPC FAAP, Sheldon Cheskes MD CCFP(EM) FCFP, Michael Feldman PhD MD FRCPC
Background
Out-of-hospital cardiac arrest (OHCA) is a common occurrence. (Statistics Canada, 2010) During the chaos of an OHCA, paramedics are expected to manage the patient and support the family. (Paramedic Association of Canada, 2001) This includes communicating unexpected news to family members, most commonly in the form of death notification. Paramedics feel uncomfortable delivering death notifications under these circumstances. (Norton et al., 1992) These experiences can be very stressful, especially to inexperienced providers. (Norton et al., 1992, Smith-Cumberland, 2006) As such, we sought to explore paramedics’ experiences with this significant event.Method
This survey was completed as a pilot study by a convenience sample of paramedics who were recruited at a meeting of a prehospital journal club. The SurveyMonkey website was used to develop a 12-question online survey to evaluate paramedics’ confidence and experiences with communicating death notifications. Close-ended questions incorporated respondent demographics such as age, qualification level, years of experience, previous training in communicating death notifications and the preferred format for this type of training. A seven-point Likert rating scale was used to rate paramedics' confidence with communicating death notifications. An open-ended narrative allowed respondents to discuss a personal experience with communicating a death notification and the impact of the experience on their practice. Face validity of the survey was ensured by encouraging respondents to comment on the relevance and clarity of the questions and the ease of use of the online survey format. Six paramedics completed the online survey. Five narratives were collected. The narratives were analysed by the first and second authors using the method of meaning condensation. (Merriam, 2009). The analysis was reviewed by the other two authors. This project was approved by the University of Toronto Research Ethics Board.
Results
Two Primary Care Paramedics and four Advanced Care Paramedics completed the survey. The mean ages and years of experience of the respondents were 28.3 years (range 20-47 years) and 6.7 years (range 1-20 years), respectively. The respondents reported varying experience with communicating death notifications. One third of respondents reported communicating a death notification less than 25 times, while another third reported communicating a death notification over 100 times. The paramedics reported their confidence with communicating death notifications to be moderate to high. Most respondents received education in communicating death notifications through lectures, by observing others and through continuing education (CE). College paramedic programs and CE sessions are the primary environments in which paramedics learn about death notification. Reference cards, role-play and small group discussion are the most helpful methods to learn about communicating death notifications.
Sudden, traumatic deaths of younger individuals such as a result of suicide appear to increase the stress of communicating a death notification. “While another medic was patching for a field pronouncement on a 35-year-old male who had hung himself I approached the patient’s wife to gather more information and notify her that we were going to pronounce.” The stress of communicating unexpected news is also increased when there is a delay in discovering the deceased, such as when “Communicating to a family that a pt.[patient] was code 5 as a result of the pt. [patient] committing suicide.” Code 5 is a term used to describe the signs of obvious death. These include: decapitation, transection, grossly charred body, gross rigor mortis, visible decomposition or putrefaction. (Ontario Ministry of Health and Long-term Care, 2007)
Paramedics also find balancing patient care and delivering calm, compassionate support challenging when communicating death notifications. This is exemplified by one paramedic’s response to the reaction of a family member after communicating a death notification.
“While another medic was patching for a field pronouncement on a 35-year-old male who had hung himself I approached the patient's wife to gather more information and notify her that we were going to pronounce. She asked “is he alive.” I told her that he was not and apologized that there was nothing more we could do. She fainted and fell to the ground. I assessed her vitals and left her for a time with a family member and a police officer.”
After a death in the field, the bereaved want the opportunity to see their deceased family member as they were prior to their death” The family wanted the medical equipment removed [from the patient] which was not allowed under policy.” In this situation, the bereaved may not be able to perform rituals that will allow them to begin grieving on their own terms.
Discussion
Paramedics value the concept of working as a team. This includes communicating a death notification in pairs, especially when the death is sudden, traumatic and there is a delay between the death and discovering the deceased. Paramedics also report continuity of care to be important for the patient and the bereaved.
The bereaved are at risk for complicated grief reactions when death occurs in the field. (Smith et al., 1999) and it can be unhelpful and alienating when the bereaved are required to change their cultural and socialized reactions to death in this situation. (Rosenblatt, 2007) This may impact the grieving process. However, paramedics are able to develop a relationship with the family when death occurs in the field. This can reduce the stress of communicating a death notification on paramedics and can help facilitate the grieving process for the bereaved. (Timmermans, 1999)
It is evident that paramedics value a family-centered approach to resuscitation. This approach focuses on caring for the patient and incorporating the family as participants in patient care. (Timmermans, 1999) This approach can create additional stress for paramedics as they attempt to balance patient care and support the bereaved. Health care providers find shifting between these two approaches challenging during a resuscitation. (Schmidt & Tolle, 1990).
Conclusion
Our pilot data suggests that communicating a death notification alone and lack of resources to support bereaved family increases the stress of communicating death notifications for paramedics. A larger, more representative sample is required in order to identify the stressors of communicating death notifications in the field and strategies that can be employed to reduce the stress and support paramedics.
References
Emergency Health Services Branch. Ontario Ministry of Health and Long-Term Care. (2007). Basic life support patient care standards. Toronto ON: Queen's Printer for Ontario.Merriam, S.B. (2009). Qualitative Research. A Guide to Design and Implementation. San Francisco, California: Jossey-Bass.
Norton, R.L., Barkus, E.A., Schmidt, T.A., Paquette, J.D., Moorhead, J.D., Hedges, J.R. (1992). Survey of emergency medical technicians' ability to cope with deaths of patients during prehospital care. Prehospital Disaster Medicine, 7, 235.
Paramedic Association of Canada. (2001). National occupational competency profiles for paramedic practitioners
Rosenblatt, P. (2007). Culture, socialization, and loss, grief and mourning. In D. Balk (Ed.), Handbook of thanatology: The essential body of knowledge for the study of death, dying and bereavement (pp. 115). Florence, Kentucky: Association for Death Education and Counseling, The Thanatology Association.
Schmidt, TA., & Tolle, S.W. (1990). Emergency physicians' responses to families following patient death. Annals of Emergency Medicine, 19, 125.
Smith, T., Walz, B., Smith, R. (1999). A death education curriculum for emergency physicians, paramedics and other emergency personnel. Prehospital Emergency Care, 3, 37.
Smith-Cumberland, T. (2006). Evaluation of two death education programs for emergency medical technicians using theory of planned behaviour. Death Studies, 30(7), 639.
Statistics Canada. (2010). Canada's population estimates.
Timmermans, S. (1999). Sudden death and the myth of CPR. Philadelphia PA: Temple University Press.
Meet Paramedic Researcher Leeann Douglas, PCP
My interest in EMS research was sparked by reflecting upon the nuances of a skill that paramedics perform frequently: Communicating bad news. As a Primary Care Paramedic for an urban EMS provider, I learned to communicate with bereaved families by observing my peers and through trial and error. Seeking a better way to learn this skill, I began conducting research that will explore paramedics' attitudes towards and preferences for education in communicating bad news. This information will be used to develop a workshop for paramedics on this important topic. In the future, I hope to explore paramedics' and families' attitudes towards family witnessed resuscitation and the cultural influences on these attitudes.Please feel free to contact me at leeanne.douglas@utoronto.ca
