As previously discussed online focus groups can play a major role in whether or not you will win your case. Of course, attorney skill and the strength of a case are of great importance, jury focus group can help drive new attorneys and even weak cases to a win if used correctly. Focus groups, both quantitative and qualitative, provide insights into not just the case but also the general perceptions that potential jurors have about the attorneys, experts, evidence, and voir dire questions. Yet, many focus groups result in mediocre findings leading to costs outweighing the benefits. There are many reasons for these types of results, but the most common issues with focus groups are a lack of structure, lack of science-based techniques, and poor quality samples. The quality of the focus group sample is the most important factor as to whether or not your focus group facilities will be beneficial. Regardless of whether the focus group is well-structured and organized by an expert or is a DIY focus group conducted by a novice, they will both end up with lackluster results if the sample that was gathered is not high quality. All research, regardless of industry or topic, is completely reliant upon the quality of the sample.
Common Mistakes Made During Recruitment
The quality of a sample is highly supervised recruiting and qualification process used in gathering the sample. The criteria for a high-quality sample differ depending on the goal of the market research. For example, if you are wanting to learn what iPhone users think about the new iPhone, you must not only recruit iPhone users but make sure you recruit enough of them and ensure that they match the estimated demographics that you are targeting. Another example would be a new treatment for diabetes. With this research, they would want to, of course, make sure the respondents all have diabetes, but also ensure that they match the common demographics of those who suffer from diabetes. This same logic should be used for research in the field of law. Below are five common mistakes that firms make when recruiting for focus groups.
Using the Wrong Sampling Strategy
An important factor that is often overlooked in DIY focus groups conducted by law firms is the type of sampling needed to answer your questions. There are dozens of sampling methods1 but they can be categorized into three main sampling strategies: convenience, probability, and purposive. Convenience samples are the easiest to gather but are by far the worst type of sample. Like the name says, they are convenient. They involve simply getting anyone you can to participate. They have severe bias and usually a lack of diversity. This is never good. Even in small focus groups with only a handful of people, you should avoid convenience sampling. Probability, or random, sampling usually does not fit focus groups unless you are looking for a large sample size that is generalizable to an entire state or country. If generalizability is important, like in political science research, then random sampling methods such as telephone and mail should be used. These samples are great but are quite a hassle and costly to gather. Purposive sampling is the most balanced method. This sampling method involves having a targeted audience or specific criteria that the respondent pool must meet. It is more difficult to conduct than convenience sampling but has far more robust results and allows you to focus on specific jurisdictions without being fully probabilistic. With structure and demographic quotas, purposive sampling can be very robust and effective for law focus group research.2 Mixed methods for sampling are also a great way to gather respondents. This means that you use multiple types of sampling to get your pool of respondents resulting in a well-rounded and robust sample.
Focus Group Size – Gathering the Correct Number of Participants
There is a big difference between conducting quantitative and qualitative focus groups. Your quantitative focus groups need large sample sizes, usually over 100. The purpose of this is to make sure that the error, such as outliers and bad participants, is low. The higher the number of participants, the stronger the results of the focus group. Yet, you do not need to overdo it. Recruiting 500 people for a focus group is far too many unless you are in a very large metro where sampling can be cheap and easy. There is a science behind quantitative focus group sampling methods. You need to have strong confidence in your results, however, there is a small difference between 100 respondents with an 88% confidence interval and 300 with a 93% confidence interval. Finding the right balance is key.
Qualitative focus groups, those involving small numbers of participants, differ immensely from quantitative focus groups. You simply cannot use Craigslist for this!3 Using sources like Craigslist and other online platforms take advanced recruitment methods and filtering to ensure you get high-quality respondents.4 Though the results of qualitative samples are not generalizable or representative like in a quantitative focus group, this group still needs to be of high quality.5 They need to be gathered from reputable sources, such as using a mixed-method approach. The problem with sources like Craigslist is that the demographic sample is very homogenous unless filtered correctly. They are often lower education, lower-income individuals looking for quick money. You want someone who will actually take your questions seriously and Craigslist has historically led to problematic respondents and poor results.
Poor Sample Sources
This leads us into the most common problem found in DIY focus groups conducted by law firms, the source of the sample. Samples cannot just be randomly gathered from anywhere. Every location has its own bias associated with them. If you gather participants at a University, you will likely end up with a majority of your sample being young, educated females that are likely liberal. If you recruit participants at a gun show in a rural county, you will likely end up with a majority of your sample being conservative, white males. Even if you evenly used both of these venues for recruitment for your focus groups, that would still not be enough diversity because you are missing the middle ground between the two profiles. Even going to ten different venues, you still may end up with a sample that is heavily skewed in a certain direction. This is because your source is an in-person recruitment method, which is quite archaic on its own. Again, a mixed-method approach is best.6 The best source for recruiting a sample is to use multiple methods within your target jurisdiction. For example, if you are looking for potential jurors in Los Angeles, California, you can use online ads, phone recruitment methods, as well as mail recruitment, even in-person recruitment. Make sure you mix it up and get different resources. There are even panel companies that will provide you with participants, however, each of these companies is biased as well. So make sure you use Census data for that county to find your target demographics. You can then “control” your sampling by setting quotas. A common problem with online recruiting is the male to female ratio being 70/30. You can control this by capping your females at a certain amount, say 55% of your sample, to allow the rest to fill with males only.
Lastly, if you plan to recruit your target through online participants, choosing the right panel is also critical. It’s already been discussed that Craigslist is very unlikely to be a source of high-quality data due to the fact that the potential respondents found on there are very homogenous unless filtered correctly, and are often lower education, lower-income individuals looking for quick money. However, this also applies to other platforms even when conducting quantitative research with a hundred or more respondents. In particular, the popular Amazon Mechanical Turk (MTurk) platform is frequently used by researchers. MTurk has several advantages over using in-person recruiting and/or Craigslist, in that it has a large and diverse participant pool, it is easy to access and quick to collect a large amount of data, it is reasonably cost-effective, and it allows for flexibility regarding study design. However, in recent years it has been found that the quality of data collected through MTurk has been relatively poor. Several reasons have been attributed to this: first, starting in 2018, concerns were raised about the rise of “bots” (computer programs that automatically complete surveys) and “farmers” (individuals using server farms to bypass location restrictions). By comparing data collected from the same measures in consecutive years from 2016 to 2019, researchers did indeed confirm that the proportion of low-quality responses, defined as responses that were statistically improbable (such having more than 4 children of the same age), inconsistent (e.g., reporting a different age on different pages of the survey), and/or unusual responses to open-ended questions had markedly increased over this time period, from 10.4% to as high as 62%. As such, filtering the sample – discussed in the next section – is very important.
Second, MTurk is known for participants having non-naivete, where they are aware of what is being examined in the study, likely due to either respondents spending a large amount of their time taking surveys (it has been estimated that 40% of all MTurk responses are given by 10% of the participants) and/or they belong to MTurk communities, where members share information about studies in order to maximize how much they get paid and avoid studies from researchers that they think do not pay well. Lastly, related to participants aiming to maximize their earnings, inattention, self-misrepresentation, self-selection bias, and socially desirable responding are all prevalent issues as well. In short, MTurk participants often just want to get paid and are not always answering honestly, which inevitably leads to poor-quality data.
Not Filtering the Sample
Like with quotas, you need to filter the sample. This can be done in two phases, initial qualifications, meaning they must fill out a recruitment survey in a certain way to qualify for your study, or actively filtering, meaning that you filter out poor respondents while the online research is being conducted. The latter can only be done with online quantitative research as qualitative focus group las Vegas would not allow you to remove poor responding participants. This active filtering can be done by adding validity checks in your research where you ask simply questions that they must answer correctly to go on. These can be simple, such as “Please select ‘Maybe’ for this question.” This removes those who are putting in a poor effort, or bots that are automatically completing surveys with random responses.
Not Recruiting Replacement Respondents
In many cases, it is a good idea to oversample for your focus groups. Many firms tend to try and get a certain amount of people and fall short once they show up for the focus group or they get removed as poor quality responses later if they are doing an online questionnaire. There are multiple sampling methods that can compensate for these missing or poor quality respondents, however, the most simple solution is to gather extra participants to ensure you meet your quota.7 A good rule of thumb for quantitative research is to gather 10% more respondents than you initially wanted assuming that you are using a high-quality source of data. So, if you want 150 high-quality respondents, make sure you gather 15 extra, a total of 165, to make up for some poor quality respondents that will inevitably be removed later. Though often ignored, sampling sizes for qualitative focus groups are quite salient.8 For qualitative focus groups, you should always do 150% of your planned number of participants. So if you want to get 12 potential jurors to do your mock trial, you should recruit 18 individuals. This allows you to make up for the no-shows, lack of diversity, or people you would immediately remove during voir dire. You can provide these extra individuals with a smaller amount of compensation for their time if they are not used.
Final Thoughts
Though likely the most salient and mishandled aspect of focus groups, recruitment and sampling are just a single cog in the complex machinery that is a focus group. The above list contains just a handful of the dozens of mistakes DIY focus-groups often have. There are a plethora of videos and walkthroughs discussing how easy it is to conduct focus groups in my area, but they could not be further from the truth. Focus groups are not only a science but also an art, much like trials. They take specific expertise. They involve intricate and detailed sampling methods, organized and structured questionnaires and procedures, as well as advanced data collection and analytics. Unless your firm has a focus group expert that is familiar with both quantitative and qualitative research methods and social/behavioral science, you are not getting the best results possible.9 However, this does not mean you cannot conduct focus groups in house, it simply means that you need some guidance and education on the best methods and practices to get robust results. We at Jury Analyst can provide you with both quantitative and qualitative focus group services as well as educational services to help you better understand your limitations with your current methods and how you can drastically improve the outcome of your focus groups and studies.
2https://headachejournal.onlinelibrary.wiley.com/doi/10.1111/head.13707
5https://www.tandfonline.com/doi/pdf/10.1080/13814788.2017.1375091
7https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3903025/
8https://onlinelibrary.wiley.com/doi/abs/10.1002/nur.4770180211