DESIGNING RESPONDENT COMMUNICATIONS
FOR ENTERPRISE SURVEYS: TRIALS AND TALES
FROM AUSTRALIA
Spurti D’souza, Caitlin Huppert, Jodie Kline
Data Collection Design Centre
Australian Bureau of Statistics
June 2023
PAPER PRESENTED AT
UNITED NATIONS ECONOMIC COMMISSION FOR EUROPE
CONFERENCE OF EUROPEAN STATISTICIANS
Expert Meeting on Statistical Data Collection – Rethinking Data Collection
12-14 June 2023
DESIGNING RESPONDENT COMMUNICATIONS FOR ENTERPRISE SURVEYS
Trials and Tales from Australia
June 2023
Page 2 of 12 - www.abs.gov.au
1.0 ABSTRACT
Behavioural insights and human-centred design underpin respondent communications the
Australian Bureau of Statistics (ABS) designs and develops. This approach is guided by the ABS’
enterprise-level strategic priorities, including to enhance organisational capability, resilience, and
adaptability, and to reduce burden on respondents. In doing so, our service offering improves
respondent experience and supports respondent engagement with our survey program.
This submission showcases modes and features of communications trialled in enterprise surveys
during 2022-23, including the drivers for delivery, the design process, and the outcomes of initial
implementation.
Specifically, we share learnings associated with:
1. Authoritative messaging in survey invitation materials for new-to-sample respondents.
2. Proactive reminder card to promote self-initiated response.
3. Late reminder SMS.
4. ‘We missed you’ email sent following an unanswered outbound call.
5. ‘Thank you’ paper letter.
The trials demonstrate benefits of placing the respondent at the core of service design, as well as
raise critical questions about how to balance respondent preference and statistical integrity.
2.0 INTRODUCTION
Respondents are central to the success of the Australian Bureau of Statistics’ (ABS) survey program.
Annually, the ABS runs more than thirty statistical collections. Without the willing cooperation of
businesses and organisations, generation of our national statistics would be compromised. We
couple this knowledge with recognition that our surveys need to be prioritised alongside other
business activities, in a context where labour and supply pressures are acute (ABS, 2022, June). With
this in mind, we strive to support respondent engagement through delivery of a very good
respondent experience.
Traditionally, we focused our energies on growing internal understanding of human behaviour
broadly, and theories of survey participation more specifically. Through a team of experts in the
fields of behavioural economics, sociology, psychology, and data science we applied behavioural
insights to the design of our services and products. Our knowledge of social exchange theory and
cognitive heuristics underpinned the development of respondent engagement strategies. Designs or
prototypes were often presented to respondents for feedback, via consultation late in the
development process, but respondents themselves were not routinely central to the design process.
More recently we have shifted away from solely relying on internal expertise, to also involve our
respondents in a respondent-centred development approach. Specifically, our design work now
draws primarily on human-centred design tools, coupled with application of behavioural insights.
Respondent perspective is now at the heart of the process.
DESIGNING RESPONDENT COMMUNICATIONS FOR ENTERPRISE SURVEYS
Trials and Tales from Australia
June 2023
Page 3 of 12 - www.abs.gov.au
Human-centred design (HCD) recognises the benefit associated with co-designing with users (in our
context, the survey respondents), encourages rapid prototyping and iterative testing, and obliges
designers to view problems from the perspective of users (Digital Victoria, 2022). HCD invites user
involvement in the design of products and services that meet their needs and deliver business value
(Smith, 2021; Digital Victoria, 2022). By using HCD, we have invested in research to engage with and
better understand the situation of the respondents.
Our application of HCD to enhance statistical data collection is demonstrated through case studies
and trials of five innovations associated with a large-scale project to redesign all economic survey
correspondence (Meld Studios, 2022 April; Data Collection Design Centre, 2023 May):
1. Authoritative messaging in survey invitation materials for new-to-sample respondents.
2. Proactive reminder card to promote self-initiated response.
3. Late reminder SMS.
4. ‘We missed you’ email sent following an unanswered outbound call.
5. ‘Thank you’ paper letter.
Development and implementation of these innovations delivered benefit to the survey program, as
well as exposed opportunities to mature ABS’ use of HCD and highlighted the complexities
associated with balancing respondent preference and statistical integrity.
3.0 INTERVENTIONS
3.1 Authoritative messaging in survey invitation materials for new-to-sample
respondents
Research question: Can increased authoritative messaging improve new-to-sample respondents’
survey participation?
Most ABS surveys are compulsory to complete; however, new-to-sample respondents often
comment that this is unclear, in turn, delaying their survey participation. This motivated us to
investigate whether increased authoritative language improves the effectiveness of the data
acquisition process with new respondents.
We conducted concept generation and user testing with respondents to explore variations in firmer
and authoritative language. The research concluded that respondents expected correspondence to
include a clear statement about whether completing an ABS survey is compulsory and they
preferred a plain English answer to this question over a detailed legal explanation. These HCD
techniques helped us understand the respondents’ cognitive processes, and understand that clear
and authoritative messaging can inform and influence their survey participation.
To determine how the strengthened authoritative messaging impacts respondents’ behaviours, in
2022 two small-scale randomised control trials were conducted over two cycles of a quarterly survey
(Data Collection Design Centre, 2022 October; Data Collection Design Centre, 2022 December). The
experimental and control groups contained 100-200 respondents each. The compliance model of
survey response as articulated by Snijkers, Berkenbosch and Luppes (2007), including Groves,
DESIGNING RESPONDENT COMMUNICATIONS FOR ENTERPRISE SURVEYS
Trials and Tales from Australia
June 2023
Page 4 of 12 - www.abs.gov.au
Cialdini and Couper’s (1992) authority compliance principles was referenced and applied. Messaging
in approach materials for new-to-sample units was modified to emphasise the legal authority for
statistical data collection, and compulsion messaging was strengthened. Changes included removing
the word ‘survey’ and using language such as ‘obligation’. For example, ‘Complete your survey online’
was replaced with ‘Complete your obligation online’. Additionally, ‘Notice of Selection’ was the
header used for the experimental approach materials. In sum, only the text (no other design
elements, such as formatting) was changed between the control and experimental conditions.
The strengthened authoritative messaging in approach letters and emails did not make a statistically
significant difference to response rates or respondent engagement, as measured in the trials.
Findings suggest that the experimental compliance messaging had neither a detrimental impact on
timely survey participation, nor did it appear to deliver benefit compared to existing materials. There
are several ways to interpret the results:
• The compliance messaging tested was not strong enough or clear enough to impact
response and engagement by due date.
• The compliance principle of ‘liking’ coupled with soft use of the ‘authority’ compliance
principle embedded in the existing materials is as effective at driving response as firm
compliance messaging alone.
• Optimisation of other elements in correspondence are more likely to drive improvements in
response and engagement.
Subsequent prototype and user testing has moved away from focussing on the compulsion
messaging alone, and a shift towards a holistic review of the whole suite of correspondence and
escalating the authoritative tone in correspondence sent after the due date (see Willimack &
Snijkers, 2013 for discussion of escalation). This has tested well with respondents in usability testing,
and we continue to work closely with respondents and our policy and legal team to develop and test
variations of authoritative messaging as part of a project aiming to enhance the end-to-end
respondent journey.
3.2 Proactive reminder card to promote self-initiated response
Research question: Can a proactive reminder tool, a ‘reminder card’, improve survey participation
including timely response?
Obtaining timely response, a complete and submitted survey form by the respondents’ due date, is
an ongoing and growing challenge we face across many collections. Concurrently, from the
respondents’ perspective, they have competing priorities of which an ABS survey is simply one of
many. Respondents requested tailored tools to help them submit their survey on time. Goal-setting
theory (Locke & Latham, 2002), the INSPIRE framework (Faulkner et al, 2018), and a study to
enhance influenza vaccination rates (Milkman et al, 2011) posited that a goal needs to be set by the
acting individual, and their intention needs to be clear and specific.
Inspired by these researchers and by positive reaction from respondents to proactive reminder
materials, we randomly selected 500 respondents in a biennial enterprise survey to receive a
DESIGNING RESPONDENT COMMUNICATIONS FOR ENTERPRISE SURVEYS
Trials and Tales from Australia
June 2023
Page 5 of 12 - www.abs.gov.au
reminder postcard in the same envelope as their invitation letter (Data Collection Design Centre,
2023 January). The reminder card contained key pieces of information the respondent needed to
complete their survey online: business-as-usual (BAU) due date, blank space for them to handwrite
their intended submission date, alphanumeric code to access the web form, URL to complete the
survey, URL for frequently asked questions, and help line phone number (see Image 1).
Image 1. Reminder card
The treatment group (25.2%, n = 126) experienced a significantly greater response rate, by 4.71
percentage points (p = 0.0158, Power = 94.09%), by the BAU due date than units in the control group
(20.49%, n = 320). The significantly different response rate held through for the two weeks that
followed the due date. We saw 4.74 percentage points greater response rates one week after the
due date (p = 0.0215, Power = 92.13%), and 5.39 percentage points greater response rates two
weeks after the due date (p = 0.0127, Power = 95.08%).
Of the timely responders, most were returning units in both the control group (73.75%, n = 236) and
in the treatment group (77.78%, n = 98). This could indicate that:
i. familiarity with the survey is beneficial to the respondent’s ability and willingness to
complete the survey by the due date, and
ii. the Reminder 1 email and SMS, sent on the day before the BAU due date, continues
to significantly boost response rates across a wide range of ABS business surveys.
DESIGNING RESPONDENT COMMUNICATIONS FOR ENTERPRISE SURVEYS
Trials and Tales from Australia
June 2023
Page 6 of 12 - www.abs.gov.au
Of the units that responded by the due date, significantly more of the treatment group responded
without any contact attempts than those in the control group (+ 7.58 percentage points, p = 0.0252,
Power = 88.78%). That is, the reminder card significantly improved true self-initiated response,
which directly reduced our follow-up outbound call costs and efforts.
Further trials will help us understand which surveys or types of respondents benefit from the
reminder card. For example, this biennial survey contacts individuals who are rarely in other ABS
surveys. The infrequent contact with the ABS coupled with the design principles in the reminder card
may have contributed to its success. Additionally, this survey sample has a high proportion of
returning units (73.79%, n = 3,043). Perhaps familiarity with the survey and the novelty of the
reminder card helped shape the favourable outcome. It could even be a combination of these and
other factors, that only further trials will help us unpick.
3.3 Late reminder SMS
Research question: Can a late reminder via SMS, after the business-as-usual due date, influence
survey response rates?
The ABS is exploring whether a late SMS reminder is an efficient method to improve the response
process for enterprise collections. Sending an SMS is identified as an effective form of behavioural
intervention, such as improving reporting behaviours among Australian welfare recipients (Copley et
al, 2017: Dillon et al, 2021). Much of the existing literature has focused on social surveys as opposed
to enterprise surveys. Respondents involved in the review of our correspondence suite appreciated
the convenience of SMS communications, but noted that mobile phones are, in the main, personal
devices, and advised us to tread cautiously when it came to use of SMS (Data Collection Design
Centre, 2023 May). This is consistent with current research into mobile device use within Australia,
which suggests that employees with personal mobile devices have at some point used their device
for business related purposes (Deloitte Access Economics, 2019).
Much of the existing literature has focused on individual based surveys as opposed to business-
based surveys. As such, through a concurrent project we sought respondent perspectives on use of
SMS. We coupled this work with a randomised control trial (RCT) and several live trials with the goal
of contributing to the literature, building an understanding of how to use SMS communication
between a national statistical organisation and survey respondents, specifically in the enterprise
survey space (see Image 2). The RCT was conducted within a survey of the Australian building
industry, while the live trials were conducted across three cycles of two quarterly Australian
employment and wages surveys.
DESIGNING RESPONDENT COMMUNICATIONS FOR ENTERPRISE SURVEYS
Trials and Tales from Australia
June 2023
Page 7 of 12 - www.abs.gov.au
Image 2. Late reminder SMS
The RCT found that SMS as a reminder is an effective means to communicate with businesses once
the due date has passed. We observed statistically significant increases in both survey response rate
and inbound contacts made by the respondents who received the SMS. Including a 13% response
rate for the treatment group and a 5% response rate for the control group. Additionally, in the
inbound calls received to the ABS call centre, there was a 5% call rate for the treatment group and a
2% call rate for the control group. Of the calls received from the treatment group, 82% resulted in
their data being collected over the phone.
In the live trials, however, we could not directly correlate the response rate increase with the SMS
due to it not being conducted under an experimental design. The results of these trials support the
need for further investigation.
3.4 ‘We missed you’ email sent following an unanswered outbound call
Research question: Can a ‘We missed you’ email help convert unsuccessful outbound calls into
successful contacts?
The December quarter is riddled with seasonal impacts due to the summer and holiday season in
Australia. Our outbound calls are answered less frequently, and more voicemail messages are left.
However, it is unclear whether respondents know the missed calls are from the ABS. We explored
DESIGNING RESPONDENT COMMUNICATIONS FOR ENTERPRISE SURVEYS
Trials and Tales from Australia
June 2023
Page 8 of 12 - www.abs.gov.au
whether an email tailored to this situation would improve respondent contact efficiency during this
season.
To help bridge the gap between the missed call and the successful contact desired, we developed
and tested a prototype, and then trialled a ‘We missed you’ email (see Image 3). Our call centre staff
send it to a respondent following an outbound missed call, an unsuccessful contact. Respondents
suggested the design concept during idea generation sessions and additional respondents enhanced
the design during prototype testing (Data Collection Design Centre, 2023 May).
Image 3. ‘We missed you’ email
For easy adoption by our call centre staff, it was trialled with all business surveys in the field from
December 2022 to January 2023. Although business survey respondents prefer electronic
communications, there are challenges with emails, telephone calls and text messages in the current
environment due to spam, scams and phishing experiences. Additionally, we experienced
operational challenges whereby the ‘We missed you’ emails were applied inconsistently between
call centre staff as it added to their administrative burden.
Although these emails yielded no significant impacts on our usual metrics (e.g. response rates,
successful calls to contact), they helped generate a small volume of returned calls and online
responses. Furthermore, the call centre staff who did use it provided positive anecdotal feedback. It
has since been rolled out for wider and more consistent use, and continued monitoring. We expect
to see its impact in less tangible aspects, such as the respondent-ABS relationship.
DESIGNING RESPONDENT COMMUNICATIONS FOR ENTERPRISE SURVEYS
Trials and Tales from Australia
June 2023
Page 9 of 12 - www.abs.gov.au
3.5 ‘Thank you’ paper letter
Research question: Will a ‘Thank you’ paper letter sent in between cycles help maintain a longitudinal
panel?
Many of our cyclical enterprise surveys face longitudinal panel maintenance issues. We are currently
exploring whether a ‘Thank you’ paper letter in between cycles will help us maintain the panel, and
in turn improve their future survey participation.
Past ABS research suggests that survey respondents who submit in their first cycle are likely to
continue submitting regularly in subsequent cycles (Data Collection Design Centre, 2022, August).
However, our analysis across multiple quarterly surveys for multiple cycles, shows that this pattern
becomes disrupted after the third cycle of the survey, with respondents’ submission behaviours
becoming irregular (Data Collection Design Centre, 2022, August).
Literature suggests that positive reinforcement of desired respondent behaviour will lead to
respondents repeating the behaviour in future survey cycles (Greenberg & Dillman, 2021). Past
cognitive testing indicated that respondents feel their relationship with the ABS is very one-sided.
This reduced their motivation to engage with the ABS or to spend time ensuring the data that they
submitted was accurate.
In a large HCD project (Meld Studios, 2022), respondents indicated they wanted their efforts to be
acknowledged. Additionally, a small ABS study conducted in 2022 suggested that respondents who
receive a ‘Thank you’ letter after submitting a survey in one cycle are more likely to submit again
during the next cycle, compared with those who did not receive the ‘Thank you’ letter (Data
Collection Design Centre, 2022). We are now extending this work with a quarterly survey.
The current longitudinal study follows new respondents split into treatment and control groups for
five cycles, analysing their submission habits. This study is in the second of five phases, and as
expected at this point, there is currently no statistically significant difference in response rates
between the groups. At this stage there is no evidence of a detrimental impact of the ‘Thank You’
letter on the respondent. However, we will continue to monitor this and will have further
information in late 2024.
4.0 CONCLUSION
In the context of declining response rates (Australian Bureau of Statistics, 2022 April 27; Luiten, Hox
& de Leeuw, 2020; The World Bank, 2020), actively engaging with respondents as assets is accepted
as the way forward for ABS data collection. As indicated in the Information Notice of this expert
meeting (UNECE, 17 January 2023), there has been a ‘paradigm shift that puts the respondents at
the centre of the data collection design and management’.
DESIGNING RESPONDENT COMMUNICATIONS FOR ENTERPRISE SURVEYS
Trials and Tales from Australia
June 2023
Page 10 of 12 - www.abs.gov.au
In the past we focused on the symptoms of barriers and looked to external literature, theory, and
operational data to help us understand behaviour. Respondent feedback received via various
pathways was considered and valued and respondents were consulted and involved in prototype
testing, but they were not central to the design of solutions. There was not a two-way conversation,
whereby respondents were not invited to help define the problem or involved in idea generation
and concept testing. Instead, they were often engaged late in the design process once prototypes
had been developed.
The five interventions discussed demonstrate the benefits of placing the respondent at the core of
service and product design. By working with them we can understand barriers to response as
perceived by respondents. The iterative approach enabled us to evolve and test ideas, ahead of
scaled implementation. As such, by the time deliverables land in the field – which not all did – they
are more likely to connect with respondent need and therefore yield the desired outcomes.
Recent respondent-focused design and evaluation work has also revealed the benefits of HCD as an
approach that enables us to balance respondent preference and statistical integrity, by involving
both respondents and ABS staff in the design process. Through this, our requirements are
communicated more effectively, and respondents are clearer about their role in the construction of
national statistics. We introduced several of the ‘friendlier’ and pro-active communications that
respondents requested, and successfully tested the more immediate and urgent messaging to
promote compliance late in the survey cycle. Each intervention has the potential to boost data
integrity and does so by adhering to respondent-specified modes, content, and timing contact.
5.0 REFERENCES
Australian Bureau of Statistics. (2022, April 27). Raising survey response rates by using machine
learning to predict gold providers. www.abs.gov.au/statistics/research/raising-survey-response-
rates-using-machine-learning-predict-gold-providers#abstract/
Australian Bureau of Statistics. (2022, June). Business conditions and sentiments.
https://www.abs.gov.au/statistics/economy/business-indicators/business-conditions-and-
sentiments/latest-release
Copley, S., Brewer, J., Turenko, A., Wilson, J., & Hiscox, M. (2017). Effective use of SMS: Timely
reminders to report on time. Australian Government, Canberra.
Data Collection Design Centre. (2022, August). Analysis of response behaviour in new respondents.
Australian Bureau of Statistics National Data Acquisition Division, Melbourne.
Data Collection Design Centre. (2022, June). Tailoring early contact for late and non-responders:
Monitoring the impact of previous cycle interventions. Australian Bureau of Statistics National Data
Acquisition Division, Geelong.
DESIGNING RESPONDENT COMMUNICATIONS FOR ENTERPRISE SURVEYS
Trials and Tales from Australia
June 2023
Page 11 of 12 - www.abs.gov.au
Data Collection Design Centre. (2022, October). Can reminding business survey respondents that
completion is compulsory improve survey engagement? Perhaps, but only by a bit. Australian Bureau
of Statistics National Data Acquisition Division, Geelong.
Data Collection Design Centre. (2022, December). Trial of compliance messaging in approach
materials, Pilot study no.2. Australian Bureau of Statistics National Data Acquisition Division,
Geelong.
Data Collection Design Centre (2023, May). Designing survey correspondence to improve
engagement with businesses. Australian Bureau of Statistics National Data Acquisition Division,
Melbourne.
Data Collection Design Centre (2023, January). Reminder card mailed with approach letter to
encourage timely response. Australian Bureau of Statistics National Data Acquisition Division,
Melbourne.
Deloitte Access Economics (2019). Mobile Nation 2019: The 5G future.
https://www2.deloitte.com/content/dam/Deloitte/au/Documents/Economics/deloitte-au-
economics-mobile-nation-2019-080419.pdf
Digital Victoria (2022, August). Human-centred design playbook. Department of Premier and Cabinet
Service Design Team, Melbourne.
Dillon, A., Glazerman, S., & Rosenbaum, M. (2021). Messaging to improve response rates:
Effectiveness of pre-survey SMS messages. Global Poverty Research Lab Working Paper, (21-106).
Faulkner, N., Borg, K., Bragge, P., Curtis, J., Ghafoori, E., Goodwin, D., Jorgensen, B. S., Jungbluth, L.,
et al. (2018). The INSPIRE framework: How public administrators can increase compliance with
written requests using behavioural techniques. Public Administration Review, 79(1), 125-135.
Greenberg, P., & Dillman, D. (2023). Mail communications and survey response: A test of social
exchange versus pre-suasion theory for improving response rates and data quality. Journal of Survey
Statistics and Methodology, 11(1), 1-22.
Groves, R. M., Cialdini, R. B., & Couper, M. P. (1992). Understanding the decision to participate in a
survey. Public opinion quarterly, 56(4), 475-495.
Locke, L. A., & Latham, G. P. (2002). Building a practically useful theory of goal setting and task
motivation. American Psychologist, 57(9), 705-717.
Luiten, A., Hox, J., & de Leeuw, E. (2020). Survey nonresponse trends and fieldwork effort in the 21st
century: Results of an international study across countries and surveys. Journal of Official Statistics,
36(3), 469-487.
Meld Studios (2022, April). ABS client experience review insights report. For the Australian Bureau of
Statistics National Data Acquisition Division, Melbourne.
DESIGNING RESPONDENT COMMUNICATIONS FOR ENTERPRISE SURVEYS
Trials and Tales from Australia
June 2023
Page 12 of 12 - www.abs.gov.au
Milkman, K. L., Beshears, J., Choi, J. J., Laibson, D., & Madrian, B. C. (2011). Using implementation
intentions prompts to enhance influenza vaccination rates. Proceedings of the National Academy of
Sciences, 108(26), 10415-10420.
Smith, K. (2021). Human-centred design online workshop 23-24 November 2021. Meld Studios.
Snijkers, G., Berkenbosch, B., & Luppes, M. (2007, June 18-21). Understanding the decision to
participate in a business survey. 3rd ICES-III, Montreal, Quebec.
The World Bank. (2020, July 6). Survey of National Statistical Offices (NSOs) during COVID-19.
www.worldbank.org/en/research/brief/survey-of-international-statistical-offices-nsos-during-covid-
19/
United Nations Economic Commission for Europe (UNECE), Statistical Division (2023, January 17).
Notice No.1, email correspondence. UNECE conference of European statisticians expert meeting on
statistical data collection: Rethinking data collection Information.
Willimack, D., & Snijkers. G. (2013). The business context and its implications for the survey response
process. In Designing and Conducting Business Surveys (pp. 39 – 82). Wiley.