How (not) to do academic surveys

As you know, me and my co-author were running a survey recently that lots of you took part in (thanks!). We had 959 responses by the time it closed. Most of the data is numerical so I'll be working on the analysis for a while yet, relearning how to do all that stuff, but from a first eyeball it looks like we've got some really clear results. Turns out people's judgements on most of these things is very clear! I thought I'd reflect a little bit on the specific way we set up this survey and the methodological lessons we learnt.

We had a few questions at the end to find out a bit more about the respondents. We asked what their language background was, whether they considered themself to be a native English speaker, how much they socialise online, and whether there was anything else that they felt was relevant. We asked these questions - and these questions only - for reasons. 

The 'native speaker' question was because in our corpus study, we'd found lots of examples of the construction that we'd have considered ungrammatical, and that seemed to be written by non-native speakers. However, if this is an 'internet language' phenomenon, and internet language is global, we need to consider those varieties too. Some people were surprised that we had 'other' as an option alongside yes and no. This is because people's language situations are complicated! It's not always easy to define what counts as a native speaker. We wanted people to be able to say 'well technically no, not on the narrow definition, but I think of myself as such' or whatever. And some people did. 

We also asked about language background. This was because we thought it might make a difference for the reasons above, and also because we found a lot of examples of some specific types that, again, we found ungrammatical, in tweets written in Indian English, so we wanted to try to capture some of this information. I also wanted to allow people to state this in their own words. 

One person (and I hope they don't mind me talking about it here) mentioned in their comment that the survey didn't take into account other varieties such as AAVE (their example). AAVE is also known as African American English or African American Language, so-called because its speakers are mostly Black Americans. It's true, we didn't explicitly ask about this, just as we didn't explicitly ask about Indian English (which we knew might behave differently) or Multicultural London English (the rough equivalent of AAL relevant to our UK context) or French (which we know also has a because-X construction). The 'language background' and 'any other information' boxes were there for people to provide any information that might be relevant there, such as 'I'm bilingual in Mainstream American English and AAL and here are the differences in my judgements' or 'This is exactly the same in French by the way'. No one did this, but they could have done. I kind of wished they would, though, and maybe I should have explicitly asked about it. I'd be interested to know if this person knows that the because-X construction is similar or different in AAL, and I would very much like to read that study, but this survey wasn't about comparing those two varieties so it was beyond our scope. We were happy to accept responses from any variety of English because we want to know how because-X works in general, and we know that it spans a number of varieties. 

We asked about how much people socialise online because we think this phenomenon is more widespread, or at least familiar, to people who are more used to 'internet English'. Lots of the comments we got confirmed that other people also think this. This was a vague nod to Gretchen McCulloch's 'internet age', and in fact many people used her scale to give their answer here. This is about not how old you are in years, but in how immersed in the internet you are. It's a complex scale because there are the kiddos who've never known anything but the internet as it is now, late adopters, early adopters... age doesn't match how long you've been online. It's an interesting typology and you should read her book 'Because Internet' to find out more about it. 

We didn't ask about people's age - for exactly the same reasons. Some people were surprised we didn't ask this, and gave their age in the 'any other information' question. But also I had asked about age in the survey I did in 2014, and I had no reason to ask for this information again. You should only ask about the personal information that you actually need. For those who are interested, here's the data from that survey, which was a bit sloppy so don't judge me. I hope that if you click this image you can expand it so that it's readable. It shows the acceptability ratings for 22 sentences for each of three age groups (I didn't include the oldest and youngest because there weren't many respondents in those groups), arranged by the overall rating of the youngest age group (blue bar on the left of each group) from lowest to highest. 

On the right hand side are the sentences that should be fully acceptable for everyone, like I was late because I forgot to set my alarm. Not much difference in the age groups here. On the left are the ones that everyone hates, like I was late because that I got lost. Here the millennials seem most accepting, with unusually high ratings for verb phrases and full noun phrases with articles. Gen X were particularly happy with I'm edgy because if I left the oven on and with the ones with prepositional phrases like I'll be late because at the doctor's. These are ones that could be elliptical: short for I'll be late because I'm at the doctor's. This is a much, much, older and more established construction than because-X, so it makes sense they'd interpret the sentences that way and be happier with them (still not that happy, mind). 

From about halfway along the graph until the highest rated ones, there's a clear set where the youngest respondents, aged 18-25, were the most accepting of the sentences. These are the ones that are characteristic of because-X, so they can't really be ellipsis like the prepositional phrases above, and they have a noun or an exclamation after the because, like I'm here because the internet, Studying because school or I can't believe she did that because honestly. So yes, we think age does make a difference, but now we know this we didn't need to ask it again. We also only need to know that if a whole group of people simply doesn't accept this construction at all, maybe we need to factor that into the analysis; we're not interested in tracking the change in the construction via an apparent time study, for instance, which is one good reason for asking about age. This is not a sociolinguistic study, so sociolinguistic variables are only relevant to the extent they'll affect our results. 

Another thing we didn't ask about was gender. I didn't ask about that last time, either, because I had no reason to think it would make any difference. Even more so than with age, we have no reason to believe that a whole gender of people simply don't use this construction. If there is a bit of difference among genders, that's fine - our analysis can cope with that. We want to know, for the people who use the construction, how does it behave syntactically? The gender of the language users, as long as there's not a total categorical difference, therefore isn't that informative here. Again, we aren't trying to find out who the speakers are; that's a study for another linguist. In terms of sampling it might be a problem if, for instance, we only had men taking our study. That would mean we couldn't generalise to all language users, and if we didn't know who the participants were we wouldn't know that and might generalise wrongly. We took the view that this was very unlikely to be the case with nearly a thousand responses. We know that at least some people of various genders took part because they told us in the comments. 

So those are the things that I think we're happy with in terms of how we set up the survey. We got a lot of data that will be pretty hard to work with, because it's all free text, but it's also very rich so we'll see what we can do with it. 

(cw: discussion of fatphobia)

But we also did some things that weren't quite right. The biggest one of these was one of our examples, which several people pointed out was fatphobic. The way we created our sentences was to take them from our corpus if a sentence of the right form existed, and then modify them (replacing words) to prevent them being searchable and therefore identifiable (thanks to Mercedes Durham for this tip). For ones that didn't exist, we took similar constructions that did, and modified them to be the right syntactic form. This meant, we hoped, that they were all realistic examples. In doing this, we also thought we had avoided using any that were potentially offensive or harmful (obvious examples being offensive language). Clearly, we messed this one up. I can't speak for my co-author on this but I come from a position of my personal relationship with weight being basically the default/stereotypical societal one, and therefore I have to work harder to remember not everyone's experience is the same - just like as a white person I have to remember that I might miss instances of racism and be more aware. I'm aware of campaigns like Health At Every Size, but I just wasn't aware enough here to catch this. Sorry to anyone who we triggered or upset with that sentence; lesson learnt and thanks for pointing it out to us in the survey comments. 

Less harmfully, but annoyingly, we ended up using some wording that didn't chime with everyone. I thought again I'd removed anything that was region-specific (like I asked people about the verbs call vs ring), but some people mentioned that 'club together' is a British phrase (at least, they thought so). So that might affect that particular item, which is not what we wanted. Similarly, on mobile was a bit unidiomatic for some respondents. 

One type of comment that really interested me was the ones that took issue with the wording of the survey. Some were just along the lines of 'None of these sentences make any sense to me', which is fine, we knew for some people that would be the case. But some said things like 'These are not sentences', and they didn't mean exactly that they're not grammatical, but that they don't meet the definition of a sentences and they're something else. We used the word 'sentence' in the survey because that's what normally seems familiar to people. Linguists typically don't use it in any technical sense, precisely because it doesn't have a good definition. We might use 'utterance' instead, which would have probably been more accurate for these commenters as it doesn't imply a certain form, but that's not a familiar term for everyone. I'm guessing these commenters feel that a sentence must be grammatical, and otherwise it's not a sentence, which is a position similar to the people who say that something is or is not a real word. It's a perfectly acceptable definition of a sentence for someone whose goal is grammatical writing, but it's a circular definition for a linguist so it's no good if you're studying utterances that are grammatical for some people and not others, as we were here. I'm not sure what we should have done here instead; you can make explicit that some of them might not be full grammatical sentences but we really wanted to get away from priming people to give the 'right' answer. 

One last thing that I hope doesn't affect our results too much is that some people missed a part of the instructions. We had a 'fill in the blank' question. We wanted to allow for people to say nothing was missing, but we didn't want to make the questions optional as that wouldn't tell us if they thought the sentence was fine as it was or if they'd just skipped it. So we made the questions required, but said 'put an x in the box if you think it's fine as it is'. Quite a few people didn't see that part of the instructions, which we could tell because they wrote something else like 'This is fine'. So I hope that not too many people wanted to leave it blank but felt obliged to fill it in. If they did, it's OK, because we really just wanted to know what people filled in there, but still, it makes the survey annoying for them to do. 

Long post, sorry! But reflecting on this was a really useful experience for me and I hope that it's interesting to you as well. You didn't have to read this far, so thanks for doing so! 

Gwynne, and racism, classism, sexism and all the isms

A couple of weeks ago (because I am slow to get round to blogging), that Neville Gwynne person was on the BBC again talking his usual nonsense. In case you've forgotten, he's a retired teacher who wrote a grammar book and occasionally pops up in the media with odd opinions and crazy thoughts (he is a big fan of Michael Gove, which really tells you all you need to know). I wrote about Michael Rosen's response to him and his antics back in March.

This time, I wanted to talk about correcting people's grammar, and whether it's a good thing or not.

First, let's clear something up. Rosen says (in the article cited in the previous post I linked to)

In fact, we would neither be able to speak nor understand if we didn't know [grammar].

Gwynne says (in his book, on page 5)

For genuine thinking we need words... If we do not use words rightly, we shall not think rightly.

(Note his use of the hideous adverb 'rightly' because of his obsession with not using adjectives as adverbs!)

These are two very different positions, and I agree wholeheartedly with Rosen and disagree wholeheartedly with Gwynne. Rosen is right because language is, basically, grammar. 'Grammar', when not used in the school sense, is simply the way language works. Without it, you just have a bunch of sounds and maybe not even that. But you cannot go from that to saying that thinking requires words. This is a complex philosophical issue and one that I'm not going to get into here. It is possible that thinking does require language (definitely not just words), and if it does, OK, fine, we need to know grammar (not just words) to think, or at least think clearly and in some sophisticated manner which goes beyond what all animals do. But it is absolute fact that we do not need 'correct' grammar to think, and that is the dangerous message that Gwynne peddles.

I can see his argument, but he's got it wrong. He sees well-argued, cogent writing with correct grammar and poorly-argued, badly-written writing and thinks that one leads to the other. No. The two may well be related, but one is not a cause of the other. I see it myself: the most able students are often those who can also write well. The students less able to put together a clear, logical argument also tend to write less well. But this is far more likely to be due to general ability level, which affects both variables independently.

Furthermore, take this video of Boris Johnson and Russell Brand on Question Time (it's a BBC thing on YouTube so if it disappears, just look for any other footage of these two people):



Russell Brand has a persona, and part of that persona is his dialect, which includes many non-standard forms that Gwynne would consider to be wrong. However, Brand is also intelligent and is often booked for these things because people love to see him in opposition to the posh morons in power. He is highly articulate (he also writes well, and has a newspaper column). He sometimes says stupid things, but he says them all quite well and certainly makes a good, insightful argument. Boris, on the other hand, is educated, probably got taught all this 'proper grammar' stuff, but can barely string a sentence together. His spoken grammar is not 'correct' because he never utters one single 'grammatically correct' sentence. Ever. (I haven't checked everything he's ever said, but his hit rate is a lot lower than Brand's, for sure. Scripted speeches don't count.)

Gwynne teaches mostly non-native speakers of English, as far as I can tell, via the internet. This is admirable. I should think that his students, many of them in developing countries, will benefit greatly from having a good standard of written and spoken English. It is important to teach the standard, because that is what learners will be expected to learn by those who test them and those who judge them for entry to university courses and employment, and it's what they themselves will expect to learn. Some people can go too far with their linguist-centred descriptivist outlook and claim that it's oppressive to do this, but in fact we are failing language students if we don't teach them the standard. Similarly, when I mark essays, I correct them if they don't use standard English. I do think, though, that students should be taught what people really say and what the acceptable alternatives are, alongside what is 'correct'. Sometimes this is beyond the time constraints of the course or the abilities of the student.

Likewise, children should be taught how to use the standard dialect of their language (in their country, as standard US is different from standard British and so on). This will help them to get on in life. But they should not be told that any other way they speak is wrong: if you are a normally developing human, you speak your own language perfectly. There simply isn't any other way to do it. It's like telling a person they walk wrong. Maybe they walk in a way that displeases some people (dragging their heels, or too slowly) but they still walk perfectly fine.

It might not seem that important, but there are serious consequences of views like Gwynne's. In the recent Trayvon Martin case, Rachel Jeantel's testimony given largely in AAVE (African American Vernacular English) caused a lot of debate. A common example used in sociolinguistics classes is the Oakland school board case from the 1990s: in a county in California, it was decided that something needed to be done to help the black children, who were systematically performing worse than the white children, even when all else was equal. It was suggested that they might be at a disadvantage because they spoke AAVE at home, which is significantly different from standard English, and needed to be taught how to 'translate' it into standard English. This stirred up an enormous amount of racism and eventually the scheme came to nothing. People would rather black children continue to underperform and start their academic life at a significant disadvantage than risk their own children coming into contact with this dialect in the classroom. But the point is, there is no inherent reason why AAVE is not the standard dialect: it's simply an accident of history. What is now the standard is the variety spoken by the people in charge. A good education should teach children how to use the standard as well as their native dialect, but you can't judge a person's intelligence by the dialect they speak.

In the UK, there is no equivalent to AAVE. We have Multicultural London English, which is similarly stigmatised, but the two are very different. We do have a lot of regional dialects, though, and these are generally looked down upon and their speakers thought to be stupid, lazy or simply wrong. This means that working-class people, who are more likely to use regional dialect features, have to make a far greater effort than those who grew up with the standard as their native dialect. They may therefore do less well at school, and they may get less good jobs. Given that they are already at a societal disadvantage simply by being working class rather than middle or upper class, this only serves to compound the problem.

Gwynne no doubt considers people who use non-standard forms to be either stupid or lazy (because why else wouldn't they have learnt the correct forms?). Some of them are. But some people who use standard English are also stupid and lazy, and their language doesn't give you an easy way to judge them. Meanwhile, the dialect-users who are intelligent and hard-working are unfairly discriminated against. Young women are routinely judged to be less intelligent, because they tend to use linguistic features that often become mainstream later on. All of this perpetuates an unequal society in which rich white men have greater advantages and poor people, women and ethnic minorities have to work harder to achieve equal or lesser status.