Conducting Surveys

If you do your homework before you start the survey process, you’ll find that surveys can save you time and money, and lead to the answers you’re seeking with regard to your members and the association’s strategic direction. View a sample survey: Florida REALTORS® membership survey (DOC: 863 KB)

Before you draft survey questions:

1. Ask yourself:
   – What specific business decisions do we need the results of this study to guide?
   – What are we going to do with this information? Again, be specific.
   – What is our objective for this survey? Phrase it, “To determine/measure X in order to Y.”
   – What questions are we trying to answer?

2. Determine your target audience. This will help you select the best way to design and distribute your survey. If you have more than one demographic, you may need more than one survey – or you may need to enlist a research professional to help you capture information you can analyze by all the necessary demographics.

3. Calculate how many people must respond to ensure the data you collect is statistically valid. Validity is based generally on response rate and final sample size, and assumes you have invited participation from either all of your members or a randomly-selected sample of them. The higher the response rate and the larger the sample size, the more reliable the data. A solid final sample size will ensure that any data analyses will provide a strong basis for good decision-making. A valid response level depends on the size of your membership. The smaller your membership, the greater the proportion of your members you need to hear from. For instance, if you have only 400 members, you would need to hear from at least 196 of them to achieve a reasonable margin of error (+/-5%). However, if you have 10,000 members, you only need to hear from 400 to get the same margin of error. Assuming you’ve surveyed all of your members, or a randomly-selected sample of them, a good rule of thumb is you need at least 400 responses for 1,000 members or over, or at least 150 for less than 1,000.

4. Plan your timing. The timing of your survey launch can impact the response rate. A poorly chosen launch time can decrease the response percentage. Survey launch timing can also change the answers you get, and ultimately the decisions you make. Keep your survey open for at least a week to ensure that you pull in a broad range of people and opinions. And plan to send at least one reminder at a different time of day than you sent the original invitation before you close the survey.

5. Be careful how you introduce the survey. Disclosing the exact subject matter may bias your results by drawing in those most interested in (or least interested in) the subject matter. Be vague and general about the content of the survey.

6. Consider an incentive to increase responses. Include the people who return completed surveys in a drawing for a prize. Or offer a free copy of the (non-confidential) result highlights to those who complete the survey.

Surveying Methods

 Next, decide on your data collection method. Below, we discuss the unique advantages and disadvantages of several popular methods.

Personal Interviews

Advantages:

• Lets the interviewee see, feel, and/or taste a product
• Seeks the target population
• Longer personal interviews are sometimes tolerated over the phone or other methods

Disadvantages:

• Usually costs more per interview than other methods (travel costs, etc.)
• Depending on the location, interviewees may not represent the target population, which may create a non-representative sample

Telephone Surveys

Advantages:

• People are usually contacted faster over the telephone than with other methods
• If using computer-assisted telephone interviewing (CATI), the results can be available minutes after completing the last interview
• Skilled interviewers can often elicit longer or more complete answers, and ask for clarification of unclear responses
• Some software can combine survey answers with pre-existing information

Disadvantages:

• Many telemarketers give legitimate research a bad name by claiming to do research when they start a sales call; consequently, many people screen calls or decline to participate
• Few people are home during the day; this limits calling time to a window of about 6–9 p.m. (when dinner or a favorite TV program is interrupted)
• You cannot show or sample products by phone
• Requires many dials to complete a single interview, so cost can be higher than other methods

Mail Surveys

Advantages:

• Can be less expensive than personal or telephone interviews
• The questionnaire can include pictures (not possible over the phone)
• Respondents can answer at their leisure, so the survey seems less intrusive

Disadvantages:

• Takes longer to receive and process
• In populations of lower education/literacy, response rates are often too small to be useful; even in well-educated populations, response rates vary from 3% to 90%
• Printing and postage makes up fully half the cost of a mail survey

Computer Direct Interviews

Advantages:

• Virtually eliminate data entry and editing costs
• Yield more accurate answers to sensitive questions
• Eliminate interviewer bias
• Ensure skip patterns are accurately followed
• Response rates are usually higher

Disadvantages:

• Interviewees must have access to a computer, or one must be provided for them
• May have low response rates in low-education and low-literacy populations
• Generally need to be conducted in high-traffic areas like trade shows, which incurs bias, since you can’t be sure you are reaching your entire population

Email Surveys

Advantages:

• Gather several thousand responses within a day or two
• Practically no cost involved once the setup is completed
• You can attach pictures and sound files

Disadvantages:

• You must possess (or purchase) email address lists
• Some people will respond multiple times or pass questionnaires along to friends to answer
• You cannot use email surveys to generalize findings to whole populations
• Email surveys cannot automatically skip or randomize question/answer choices
• Results still need to be gathered and added to a central database for analysis, since respondents are replying to an email with their completed surveys
• Responses can be garbled and easily misunderstood by the data entry person, since there are no forced formatting capabilities

Internet/Intranet (Web Page) Surveys

Advantages:

• Extremely fast and can gather several thousand responses within a few hours
• If no analysis is needed, there is practically no cost involved once the setup is completed
• You can show pictures; some Web survey software can also show video and play sound
• Can use complex question-skipping logic, randomizations, and other features
• Can use colors, fonts, and other formatting options
• Many people will give more honest answers to questions about sensitive topics when giving their answers to a computer
• On average, people give longer answers to open-ended questions on Web page questionnaires than on other self-administered surveys
• Some Web survey software can combine the answers with pre-existing information about individuals taking a survey
• Virtually all REALTORS® now have web access and use computers, so response bias is minimal

Disadvantages:

• May not reflect the population as a whole unless you are sending to the entire population of interest or to a randomly-selected sample
• People can easily quit in the middle of a lengthy questionnaire, or be dropped if they lose their internet connection
• If your survey pops up on a web page, you often have no control over who replies
• Depending on your software, there is often no control over people responding multiple times to bias the results
• Your survey invitation may bias the results if you give too much information about the nature of the survey.

Scanning Questionnaires

Advantages:

• Can be the fastest method of data entry for paper questionnaires
• A scanner is more accurate than a person in reading a properly completed questionnaire
• Can be a good way to get information from attendees at a trade show or other event where you’d like them to fill out a quick form and return it onsite

Disadvantages:

• Best suited to "check the box"-type surveys and bar codes; scanning programs have various methods to deal with text responses, but all require additional data entry time
• A scanner is less accurate than a person in reading a poorly marked questionnaire
• Requires investment in additional hardware to do the actual scanning
• As with mail surveys, requires investment in printing questionnaires

Summary of Survey Methods

Your choice of survey method will depend on several factors. These include:

• Speed: Email and web page surveys are the fastest methods, followed by telephone interviewing. Mail surveys are the slowest.
• Cost: Personal interviews are the most expensive, followed by telephone and then mail. Email and web page surveys are the least expensive for large samples.
• Internet usage: Web page surveys offer significant advantages, but you may not be able to generalize their results to the population as a whole unless you receive an adequate number of responses and are certain the survey has been sent to your entire population.
• Literacy levels: Illiterate and less-educated people rarely respond to mail surveys.
• Sensitive questions: People are more likely to answer sensitive questions when interviewed directly by a computer.
• Video, sound, graphics: A need to get reactions to video, music, or a picture limits your options. You can play a video on a web page, in a computer-direct interview, or in person. You can play music on a web page, in a computer-direct interview, in person, or over a telephone. You can show pictures on a web page, in a computer-direct interview, in person, and in a mail survey.