Week 2 Surveys and Survey Experiments with Qualtrics
This section is drafted based on Dr. Mark Hoffman’s previous SOC 10 lab notes.
2.1 Overview
In this week, we will go from using data that someone else collected to collecting data of our own using Qualtrics. Qualtrics makes collecting survey data online easy. If you have ever used something like Google Forms, it is, in many respects, quite similar, except that it is much more powerful. As you will see, we can quickly and easily build a survey experiment, administer the experiment, download the results, and import them into R. Let’s get started!
2.2 Creating a Qualtrics account
Stanford is kind enough to provide all of us with access to Qualtrics. Click here to get started.
Click on the “Set up a Qualtrics account” button and it will tell you to go to https://stanforduniversity.qualtrics.com. Go to that link!
There, it will ask if you have a preexisting Qualtrics account or not. Click the button that pertains to you. If you don’t have a preexisting account, it will either prompt you to log-in to your Stanford account via WebAuth, or else it will automatically log-in for you and set up your account if you are already logged in.
Now you should be into Qualtrics! Since you haven’t made a project previously, you only have one option: Create a new project (bottom left). Everytime you want to make a new survey you will click this button. We will make an example survey to show you the ropes.
It will bring you to a page with a bunch of options for creating a new survey. You can start from these pre-designed surveys if they fall in line with the kind of survey you want to run. There are even pre-built surveys for academics. For learning purposes though, we will start from scratch by clicking “From Scratch–Survey”.
Now it asks us to name our project. I called it First Survey.
## Building a survey
With a project created, we have been ported to Qualtric’s main user interface, where we can build a survey. Have a look around at all of the different options. Qualtrics is full of tools for creating your ideal survey, but at times that functionality can be a bit overwhelming.
We will start simple. Let’s start by learning how to add a question to our survey. Click the menue under “Question type” on the left and select “Text entry”.
There are a bunch of options and I recommend experimenting/reading up on all of them. The most essential are Text Entry, Multiple Choice, and Matrix Table. Our first question will use Text Entry. It allows us to ask a question and then have the user put in any value that they like. Multiple Choice on the other hand would let us specify the answers they can provide. For a question like – How old are you? – there are too many potential responses for multiple choice to work well, so text entry will do.
We can change the text at the top of the question box to specify the question the respondent should answer:
Let’s for example ask “How old are you?”
Next let’s make a more complicated multiple choice question using the matrix table. It basically allows you to ask multiple multiple choice questions at once. Let’s start by clicking the blue button “Add new question” and selecting the option “Matrix table”.
For example, we can copy a question from the GSS about trust in institutions to see the extent to which our respondents trust different groups, like their family or the government.
Cool! Maybe trust of this sort varies by income - we might expect rich people to support institutions more than the poor, for example, because they have historically benefitted from them.
If we want to put the income question after age, we can simply drag it upwards so that it comes before the question on trust.
We don’t have to come up with all of the questions ourselves - Qualtrics has a question bank that you can use to find pre-written questions. Just click on the “Import from library” button at the bottom of the page.
I here import one on education level from the US demographics library.
2.3 Survey options
At the bottom, we could add more question blocks (we will skip for now) and edit your end of survey message. You can customize it by scrolling down the options under “Messaging” on the left.
Now let’s preview our min-survey by clicking “Preview” button on the top!
2.4 A quick survey experiment
To set up a survey experiment, we have to set up a block of questions from which respondents will only get one of the N questions in the block. This selection is randomized, so that we can see how the randomization affects their responses. Here is a quick example.
First add a new block. Now we have two blocks. Now add the two different versions of the survey prompt to the block.
The last step is to make sure that the questions from this block are sampled randomly for each respondent, and that only one of these questions will be selected. We can achieve this by editing the block’s options. Click the block and select the Question Randomization option one the left.
This should pull up the Question Randomization menu. We can choose: no randomization, to randomize the order of questions, or to randomly select N questions from the block to present to the respondent. Select the last optin and set the number to 1 and we will have our first survey experiment.
2.5 Publish it!
Now we can either preview again to double check or publish the survey! Once it is published, we can distribute it to respondents. We could do this on Mechanical Turk, as we will learn in a later class, or we can just send it to our friends. To do that click on the “Distributions” tab at the top of the survey. It will ask how you want to distribute your survey.
You can do it by email, or by my preferred way, which is to “Get a single reusable link” which you can send to anyone who you want to take your survey. You can also generate a trackable link for each respondent if you want to keep track of who responded.
When someone responds to your survey, inside the Data and Analysis pane in Qualtrics, you should see a response!
We can export it using the “Export & Import” data button and then clicking Export Data. Export it as a .csv because those files are super easy to load into R. It will download a .zip file to your computer. Once you unzip it, you will be able to load the resulting file into R. First, I would rename it, so that it is easy to input into R. I called the file trial_survey.csv. Drag it to your R project and then you can load it into R using the read.csv() function. Simple!