Article Text
PDF
Abstract
Background In healthcare, many service evaluation questionnaires use free-text boxes without formal mechanisms for analysis. Patients and carers spend time documenting concerns that are often ignored or managed locally in an ad hoc manner. Currently, palliative care experiences of patients and carers in Wales are measured using a service evaluation questionnaire, comprising both closed and open-ended questions. Previous research, exploring free-text responses from this questionnaire, suggests that questionnaire refinement should accommodate service users’ expressed priorities and concerns, and highlights the need to incorporate free-text data analysis strategies during study design.
Methods Results from a previous analysis of 596 free-text responses provided the basis for an expert consensus day, where the current service evaluation questionnaire was refined. The refined version was tested during cognitive interviews with patients (n=10) and carers (n=7) receiving palliative care from 1 of 2 UK hospices. Data were analysed thematically.
Results Interviews highlighted minor areas for change within the questionnaire and provided broader insight into patients’ experiences of palliative care services. Patients and carers place an emphasis on simplifying language, decreasing the numeric response range and reducing written instructions; relying instead on visual cues, including formatting and layout. Findings highlighted the importance patients attached to providing meaningful free-text contributions.
Conclusions Questionnaire refinement should use the patient perspective to effectively facilitate respondent understanding, pertinence and usability. The importance of employing data analysis strategies during questionnaire design may reduce research waste, thus enabling a better interrogation of service provision.
- Qualitative Research
- Cognitive Interviewing
- Patient-Perspective
- Research Waste
- Palliative Care
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Introduction
Measuring the quality of palliative care
Assessment of healthcare quality is becoming increasingly important, as outcomes are being increasingly used to gauge services’ value against healthcare costs.1 Traditionally, care quality was measured through comprehensive but general data, including morbidity and mortality of the population served.2 Today, demands for specificity and accountability are higher,1 and quality of healthcare services is evaluated through quantifiable and explicit data to inform improvement strategies and health policies.3 ,4
The importance of the patient perspective is widely acknowledged, as are the discrepancies between patient and professional priorities in care practice.5–7 Nonetheless, outcome measures are often distinct from patients’ priorities in care, instead focusing on objective measures including disease severity and cost.8 Patient experience surveys as an outcome measure of quality in healthcare settings are becoming increasingly popular.9 These surveys enable quantifiable assessment of care quality through domains including service efficiency,10 service effectiveness and safety.3 Although these domains are universally essential to care quality, it has been argued that palliative care-specific questionnaires11 ,12 need to better reflect the multifaceted nature of end-of-life care, and refocus to enable a range of care elements to be assessed.
This paper details results from the AFTER study (Ascertaining Feedback To iWantGreatCarE questionnaiRe): a qualitative study that follows previous research.5 AFTER aimed to refine a palliative care feedback questionnaire, making it more usable, understandable and pertinent to patients and carers accessing palliative care services. This paper focuses on questionnaire design, specific outcomes assessed, and effective use of free-text data within this questionnaire, and how this maybe generalised to similar questionnaires.
Use of questionnaires in healthcare
Questionnaires objectively measure subjective experiences, including preferences, attitudes and beliefs.13 Closed questions facilitate efficient aggregation of data for funders and key decision-makers, as a means for reporting quality outcomes.9 However, many problems have been identified with questionnaires as a research tool in general and healthcare settings.14 Unlike randomised controlled trials, questionnaire methodologies are not subject to any reporting framework making it difficult to eliminate common methodological errors.9 Closed questions limit richness of potential data, as respondents are restricted to domains addressed rather than raising their own concerns; by standardising the patient experience, patient-satisfaction levels may be overestimated.10 The data is easily managed, but lacks deeper insights, becomes repetitive and can frustrate respondents.15
To overcome these problems, closed questions need to be aligned with domains that reflect patients’ and families’ needs.7 ,12 Opportunities for free-text feedback also overcomes closed question limitations by identifying overlooked care elements.16 Although patients willingly provide qualitative survey feedback,10 there is little evidence of routine analysis of this qualitative data in mixed-methods questionnaires, resulting in waste of valuable data.
Currently, palliative care provision in Wales, UK, is evaluated using a mixed-methods service evaluation questionnaire.5 This questionnaire was commissioned regionally to better understand patient and carer priorities to inform palliative care provision.2 Previous research suggests that this questionnaire could better reflect care experiences by using free-text feedback to inform questionnaire content.5
This study aimed to use free-text feedback, relating to patient and carer priorities, to refine a new iteration of the service evaluation questionnaire, and subsequently test its acceptability and pertinence with patients and carers.
Methods
Study design and data analysis
A consensus meeting was held with NHS and third-sector healthcare professionals, researchers and a lay representative (N=12) to refine the iteration of the questionnaire currently in use.i Results from the previous analysis of a data set of 596 free-text responses to the original questionnaire by Sampson et al 5 were presented as a basis for discussion. The service users’ priorities and identified concerns over questionnaire wording, structure and pertinence were addressed before refining the questionnaire.
The refined questionnaire was then tested with patients and carers accessing community and inpatient palliative care services, through cognitive interviewing. Unsolicited comments were also analysed, using a thematic approach.
Cognitive interviewing has been validated for testing refined questionnaires in general healthcare17–19 and palliative care settings.20 ,21 Cognitive interviewing can identify difficulties with content, format and understanding of questions using a small number of carefully selected participants who provide in-depth information about the question set and their responses to it.22 ,23
Cognitive interviews were conducted in participants’ homes or in the hospice at which they received care, according to preference. Predetermined coding categories18 were used to assess:
specific and general probes—respondent experience of questionnaire completion
comprehension—understanding of questionnaire wording, structure and content
recall—memory for the time period referred to in the questionnaire
confidence—ability to make a judgment about an appropriate response and response format
paraphrasing—interpretation of question.
An interview guide, developed specifically for this study, outlined preselected probes aligned with coding categories to test interview questions. Participants completed the questionnaire systematically. The interviewer used concurrent probing, stopping incrementally to question participants regarding their experience in completing the questionnaire. Additional flexible probes were used as required. Noteworthy or unexpected occurrences were recorded using field notes. Interviews were audio-recorded and transcribed verbatim. Transcripts were anonymised and imported into NVivo V.10 (QSR)24 qualitative analysis software for coding.
Data collection occurred in two phases. In phase 1, the questionnaire refined during the consensus day was tested with participants. Thematic analysis was applied deductively to search for specific items in the interview transcripts, and inductively to explore unsolicited comments. Each transcript was analysed against predetermined coding categories to assess issues relating to questionnaire interpretation and completion. A sample of transcripts was dual coded with two members of the study management group to ensure appropriateness of codes. Other comments, including difficulties encountered with questionnaire completion, solutions suggested and perceived question irrelevancy, were also considered. The questionnaire and interview schedule were adapted accordingly. In phase 2, the refined questionnaire was tested again using cognitive interviews. A comprehensive thematic analysis, using deductive and inductive approaches was conducted at the end of the study, which led to final revisions in the questionnaire.
Participants
Seventeen participants were recruited from two hospices in South Wales (table 1); 21 people declined participating. All participants approached were eligible to partake and were identified, contacted and given a Participant Information Sheet and invitation letter by hospice staff. Participants used the return slip on the invitation letter to express interest, and were then contacted by the researcher after at least 24 h, to arrange an interview. Each participant was assigned a pseudonym to maintain confidentiality. Participants (n=17) were mostly women, aged over 55 years, from the socioeconomically deprived area of Blaenau Gwent, which is served by Hospice of the Valleys (table 1). No specific demographics were collected from individuals (n=21) who declined to partake in the study. Two overarching reasons given for declining to participate were:
No time to participate, for example, due to caring duties.
Individual had reached saturation with number of requests to participate in questionnaire studies (from various organisations).
Demographics of participants in the AFTER study
Results
Respondents comfortably understood most questions in the proposed iteration of the questionnaire (figures 1 and 2). All participants were keen to provide feedback on their care, resulting in unsolicited comments of their experiences, reported below. Findings from the analysis provided practical suggestions for questionnaire refinement to enable a more efficient feedback process. Suggested adjustments to the wording, response format and aesthetics provided overarching themes. Dominant suggestions are used to present the principal issues and solutions generated during interviews. Unsolicited data is then reported, to evidence the value of free-text data in service interrogation and development.
Revised and proposed iteration of feedback questionnaire (front).
Revised and proposed iteration of feedback questionnaire (back).
Questionnaire refinement
Changes to questionnaire wording
Most participants found the use of the word ‘palliative’ confusing, and viewed the focus of questions as relating interchangeably with generic care staff rather than being restricted to the specialist palliative care team. When participants overtly stated understanding of ‘palliative’, true understanding was not necessarily present. Although some participants recommended clarifying ‘palliative’ with examples, most advised omitting ‘palliative’ from the questionnaire to make it more user friendly:I couldn't understand palliative team… care something like that so you can understand it more… That makes me feel like I'm not very intelligent by not understanding that word. (Paul, Carer)
Changes to response format
Participants, particularly those with advanced disease, and carers under considerable stress, felt that the numeric 1–10 range for tick-box responses was unhelpful and confusing. The majority of phase 2 participants suggested a reduction to a 1–5 scale. This was implemented during final questionnaire refinement:I'm not very keen on tick-boxes… what is the difference between eight and nine… they're either good or they're not… you could cut down the numbers. (Russell, Carer)
Changes to questionnaire layout
The original questionnaire conformed to a two-step format, with questions 1–9 listed in ‘step-one’, and question 10 alone under ‘step-two’. Participants felt that ‘step-one’ and ‘step-two’ were unnecessary, preferring to have all questions in a single list. The research team felt that the removal of the two-step format would also de-clutter the questionnaire:I don't quite understand (.) why it needs to be called step two… it doesn't need to be called anything. (Peter, Patient)
Unsolicited data
An inductive thematic analysis was conducted on unsolicited data gathered during interviews. Although this data was not directly relevant to the aim of the study, it was spontaneously triggered by cognitive interviewing. These data are coded into three overarching themes: care experience in the hospice; cancer conversations; and care support networks. These data evidence the importance of free-text comments to interrogate service provision and guide development; they are reported here to demonstrate participants’ perceptions of service development. The extensive data from content analysis will be reported separately.
Consequences for service development at national level
Unsolicited responses showed questionnaires to be an accepted means of service evaluation and guide to patient-focused service development. Offering feedback was viewed as routine in palliative care, and the ability to describe individual experiences was appreciated:Everybody should be entitled to their they say, no matter what… you should always have room to write a bit extra… if it's something you feel passionate about… you need it wrote down. (Danielle, Patient)
The unsolicited data highlighted patient priorities for care. Local initiatives were discussed and valued, for example, lifeline cards with link worker contact numbers. These initiatives offer scope for national implementation, patient-informed strategic planning and future research:You come out and you feel like all on your own… you keep the little card… it's like your lifeline… you want to be able to ring up and whatever problem you would know that there was a nice, kind, happy voice… who is not gonna’ say oh that's stupid go away. (Lydia, Carer)
Consequences for service development at local level
Prior to each interview, respondents were explicitly told that the questionnaire was particular to the healthcare setting providing their care. Despite this, unsolicited data offered unique accounts of various services within their Health Board locality that were involved with meeting their palliative care needs. Respondents’ experiences highlighted examples of best practice across and between individual services, as well as individual and local shortcomings that were not reflected by tick-box scores:For the night times… we could have done with some people… I don't know wh-who's responsible for that as I said if it's the borough councils you don't know… whether they can afford or we say is there a small charge would that be any help. (June, Carer)
The free-text feedback option enabled respondents to identify specific local concerns including access to care in semirural areas and the benefits of receiving care in a cohesive community. This demonstrates how feedback redundant at a national level, and thus inappropriate for quantitative measurement, is best suited to qualitative feedback and analysis at a local level:I was born and bred in (place name)… when I go to clinic in (place name) there was people there who I haven't seen for years you know… so it's nice to go down there… it's like a little club. (Paul, Carer)
Consequences for service development at an organisational level
Unsolicited comments also highlighted information essential to service development of individual care providers. Patients and carers identified particular aspects of their respective service which could provide evidence to support funding bids by non-NHS providers:You can have your feet massaged… that makes you all relaxed and if you're feeling under the weather you just… lay there and you almost go to sleep… I would say that's the wellbeing bit when you have all that done. (Gaynor, Patient)
Unlike scored answers, the free-text box enabled respondents to name individual members of staff and to thank their palliative care service, wishing their praises to be used as a platform for direct feedback:They're very aware of all their patients… how they're looking, how they're feeling—particularly (names nursing chief)… if she sees that you're looking slightly pasty… she'll have you out and give you a third degree (laughs) as to what the problem is but in a lovely way… that's incredibly important. (Peter, Patient)
Equally, areas for improvement can be linked to particular staff, both for staff development and to avoid adverse events:She covered the aspect of what would happen when it got to end-of-life… where (names husband) would prefer to be… she handled it exceptionally well… it must be a dreadful thing to try and talk about to people… we got a little bit upset, but she just said, ‘well I'll leave you now and sorry if I've upset you’, which was a nice person to person thing… she didn't want to upset us, but there are times when questions have to be asked. (Emily, Carer)
Discussion
The study findings reinforce the importance of questions targeted at the patient and carer experience of palliative care to shape service provision. The unique patient and carer insights revealed during interviewing, allowed the questionnaire to become more understandable, usable and relevant to those completing it.
This study confirmed concerns frequently experienced by patients and carers when completing healthcare questionnaires,14 ,25 ,26 including confusing wording, aesthetic and answer format. It has provided practical, patient-generated solutions to overcome commonly occurring issues, facilitating lessons for future healthcare questionnaire design.
Respondents identified specific strengths in the original questionnaire, particularly the inclusion of both qualitative and quantitative data. Respondents appreciated choice within the questionnaire; using tick-boxes for efficiency and free-text to elaborate. A quantitative design alone measures aspects of healthcare quality efficiently, providing score ratings that funders and key decision-makers often demand. However, qualitative data allows respondents to supplement and contextualise their closed-question responses by subjectively elaborating their experiences27 in a way that tick-box questions prevent;9 ,10 while providing an opportunity to record negative experiences not addressed by closed questions.27
This study enhanced the understanding of patients’ and carers’ motivations to add free-text responses and how this may benefit service providers, confirming that reciprocity and the opportunity to influence change were key motivators to actively contribute free-text comments.28 Respondents placed greater value on the opportunity to reciprocate care received through free-text, than closed questions, as it allowed for a more ardent recognition of hospice practice. By reciprocating care, respondents were able to restore a sense of equilibrium and equity to their previously one-sided relationship with the hospice. It has previously been suggested that measurement tools that appraise subjective experiences of end-of-life care instil a sense of control in disempowered patients and carers.12 The subtheme of ‘Reciprocating Care’ supported this, demonstrating the positive impacts of access to feedback opportunities on patient and carer well-being, with participants implicitly reporting feedback eliciting feelings of influence and cathartic release. Most often this was to commend positive care practice by staff members, but some participants emphasised the value of providing feedback whether their experience of care was positive or negative, satisfied in the knowledge that their contributions would be fed back and used in a meaningful way.
The interviews evidenced the benefits of the routine collection and effective analysis of free-text data, both on an individual basis per questionnaire, and as data sets per setting. Primarily, free-text data enables closed-question responses to be qualified, often resulting in a more accurate reflection of the reality of palliative care in practice.10 Free-text data allows respondents to express concerns and share experiences not directly addressed by the closed questions, by identifying service pitfalls, recognising and commending patient-valued best practice, and acknowledging staff efforts. There is also the potential to use data for wider purposes, including performance management of staff. Qualitative feedback enables patients to report on particular domains significant to patient-centred care, elaborate their scorings, and enables the patient's and carer's voices to be considered during service development.
All participants in the current study expressed interest in providing subjective feedback via free-text contributions; this could be attributed to levels of patient and carer satisfaction with the care received.10 Although respondents commonly believe that their free-text comments will be used in shaping services, many health surveys fail to routinely or rigorously analyse these data.29 As previously discussed, many participants in this study felt the addition of comment and ‘not applicable’ boxes against each question, would be useful for service users and providers alike. This study also confirmed that the addition of qualitative comments can enrich the data.9 It highlights questions of particular importance; is more likely to gather spontaneous thoughts specific to individual questions, which may be forgotten on reaching the free-text box; and can indicate why some questions are unanswered. Before valid conclusions can be drawn from such data, it must be analysed by an individual with qualitative expertise; evidencing the need for a planned approach to analysis.9
Strengths and limitations
The relatively small sample size is usual for this methodology,17 but may not be fully representative of the breadth of experience and perceptions of potential questionnaire responders. However, the sample size reflects a saturation of the data in terms of testing the face value of the revised questionnaire. A range of patient and carer groups were interviewed during the study, including community and inpatients, current and bereaved carers. In retrospect, it would have been beneficial to meet the recommended sample for cognitive interviewing per participant group, ensuring clarity of level of questionnaire pertinence and understanding for individuals in each group.21
The study design enabled flexible iterative revision of the topic guide to accommodate individual differences and unforeseen issues arising from interviews. Often, methodological limitations relating to informing questionnaire design emphasised the methodological benefits of collecting free-text data. Similar to previous research, it was noted that overt statements of participant understanding did not directly relate to true understanding.30 The repetitive nature of deductive probing resulted in practice effects over the course of each interview. Consequently, answers to recall, paraphrasing and confidence probes became progressively shallow,28 while offerings of patient and carer experiences became more frequent and elaborate, but again supported the patient-perceived importance of sharing subjective experiences.
Conclusions and future directions
The use of qualitative data in surveys provides a comprehensive reflection of patient experience and improved interrogation of healthcare quality. Despite the abundance of questionnaires containing free-text comments, these are rarely formally analysed as entire data sets or, individually, for specific patient feedback. This study has shown the value of routinely analysing free-text feedback to contextualise data gathered by traditional quantitative methods. As qualitative analysis requires much time and skill to interpret the data correctly, alternative means of handling free-text data systematically should be sought for healthcare settings. It is important that the opportunity for free-text comments is only included if the intention is to analyse and use them in a structured way; failing to do this misleads respondents.
Acknowledgments
The authors would like to thank all participants who took part in the study. They would also like to thank our recruiting hospices, Hospice of the Valleys, Blaenau Gwent and Marie Curie Hospice, Cardiff and Vale.
References
Footnotes
Contributors IF, AN and CS were responsible for conception and design; AN, AVB and JB for analytic input; and AVB for article writing. All authors were responsible for article revision and approval of the final version.
Funding This study and the post of Anna Bowyer were funded by the Tenovus Innovation Grant 2011 and supported by Marie Curie Cancer Care core grant funding to the Marie Curie Palliative Care Research Centre, Cardiff TIG2012-31. AN's and ABs posts are fully and partially, respectively, supported by Marie Curie Cancer Care core grant funding, grant MCCC-FCO-14-C.
Competing interests None declared.
Patient consent Obtained.
Ethics approval The study was approved by proportionate review by the NRES Committees—North of Scotland 2 (REC number: 13/NS/0096). A research site agreement letter was obtained from both research sites before participants were recruited.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Additional unpublished material may be made available on request, if in agreement with all authors and chief investigator.
↵i Correct at the time of writing this paper.