Easy as A/B Testing?

In addition to economic and housing studies, NAR Research utilizes surveys as a way to connect to and gather information about the opinions, priorities, and experiences of REALTOR® members. Unlike the data analyzed for housing statistics, reports such as the Member Profile and the Profile of Home Buyers & Sellers use self-reported data from random samples of members and consumers. These surveys are typically sent to groups of anywhere between 50,000 to 150,000 members, and a rigorous methodology is used in order to ensure that this information is representative of all REALTORS®.

Response rate, or the number of people who take a survey after it is sent to them, is one of the most important factors when calculating the accuracy of data that are collected. In order to boost response rates, and therefore make the data more representative of the larger REALTOR® population, the survey invitations members receive are designed to be eye-catching, easy-to-read, and interesting.

During the past year, the NAR Research team has worked on improving survey invitations through multiple strategies, such as adding personalized elements, adopting a format that is more modern and interactive, and including links to relevant pages that could be of interest to members. To explore whether these changes made a difference to members who received the survey invitations, A/B testing was conducted, which means that when a survey is sent out, half of the random sample of members gets one survey invitation format (in this case the traditional format), and the other half of the sample would get an email with the new format. Statistics are then collected regarding the number of people who opened and clicked the survey link in each format group and are then compared for differences. This is called A/B testing.

Before any changes were added to the survey invitations, a median of 29% of members who received a survey invitation opened it, and 9.70% percent clicked the survey link. Now, with the new format, those statistics are 31% and 14%, respectively. This may not seem like a large change, but it is the difference between 4,850 versus 7,000 people taking a survey, which leads to a much higher response rate and therefore, more accurate information.

New Format

Old Format