PC holder survey 2019 – State of the Nation, technical report
To ensure we remain relevant and accessible to our changing membership base, an online approach to surveying members was tested using the annual Practising Certificate (PC) holder survey.
This survey has traditionally taken the form of 20-minute telephone interviews with a random sample of 1,500 PC holders from private practice, government and the in-house sector.
Why take an online approach?
An online approach to surveys could improve the convenience and experience of taking part in research.
We appreciate that taking 20 minutes out of an already busy day to take a call is not always possible. Response rates to telephone surveys have been steadily declining. An online approach, with mobile enabled software, means you can complete a survey at your own convenience, in the office, at home or on the train.
An online approach could also improve cost effectiveness. In the long term, taking either a partial or totally online approach will reduce the cost of surveying members, allowing us to:
- reallocate resources, to commission reporting and other outputs
- free up time for further analysis
- reach out to a wider group of members (overseas, those on the roll without PCs)
What are we testing for?
We wanted to be sure that taking an online approach would not bias the findings of the survey, by changing either the profile of the sample or how people respond to different question types.
How did we trial an online approach?
Half of the 2019 PC holder survey sample was sent an email, introducing the survey and providing a link to the survey.
A hard copy letter was sent out to the remaining sample, who were then contacted via telephone to either complete the survey at that time, or to arrange a more convenient time.
What did we find?
Overall, the response rate for the telephone approach was higher, at 18% compared to 8% online.
The telephone approach achieved a sample that’s closest to the population data overall in terms of gender and years since admission.
In the telephone approach, response rates were better amongst private practitioners (24%) compared to 15% of those working in government and 15% of those in other in-house roles.
Using the online approach, response rates were better amongst government (10%) and other in-house solicitors (10%), compared to private practitioners (7%).
However, the telephone approach took over two months, compared to four weeks for the online approach.
Responses to questions were often not significantly different between the telephone and online approaches, particularly in relation to more ‘fact-based’ responses (organisation, salary, demographics, etc.).
However, more sensitive or attitudinal questions appear to reveal more honest answers online (for example, perceptions of employers). There also appears to be greater social desirability bias on the telephone approach, with respondents more likely to claim they are aware of certain technologies.
Topic and timescales available are liable to determine the approach of future research. We would aim to make both online and telephone surveys shorter to reduce time burden.
We would encourage solicitors to provide a direct dial telephone number on My Law Society.
The next step might be to openly provide access to both methods for members to choose their favoured approach.
For further information about the PC holder survey or the technical report, contact firstname.lastname@example.org.