Answering Some Questions About Our Survey

Screen Shot 2014-08-21 at 10.20.01 PM

By R.L. Stollar, HA Community Coordinator

On Monday we released the first-ever survey created by alumni of Christian homeschooling for alumni of Christian homeschooling. The 2014 Survey of Adult Alumni of the Modern Christian Homeschool Movement, facilitated by HA’s parent non-profit organization Homeschool Alumni Reaching Out (HARO) in consultation with the Coalition for Responsible Home Education (CRHE), aims to investigate the life experiences of Christian homeschool alumni by collecting information that past surveys of homeschool alumni have not.

The response thus far has been amazing. In just four days over 2,200 individuals have completed the survey. Individuals from every single one of the United States have taken it, as well as individuals from the District of Columbia and Puerto Rico. We have also received international responses, including individuals from Australia, Canada, Guatemala, Mexico, New Zealand, Portugal, Russia, Singapore, Spain, South Africa, Thailand, and the United Kingdom.

The survey remains open until Monday, September 15, 2014 at 11:59 pm Pacific time. To qualify to take it, you must be 18 years or older and have been homeschooled for at least 7 years in an environment classifiable as Christian. If you haven’t taken it yet, please do! And share with anyone you know who qualifies!

As the survey has picked up steam, a number of questions seem to be commonly popping up. So I wanted to answer the most common questions here.

Q: Why are you doing the survey?

The most significant alumni survey was from over a decade ago, commissioned by HSLDA and conducted by Brian Ray in 2003. As CRHE has pointed out, it involved a highly selective sample population and has been repeatedly presented in a disingenuous and inaccurate manner. Our goal is to (hopefully) get a more diverse, nuanced, and current look at the Christian homeschool alumni population. We also are interested in data points that previous surveys have never researched.

Q: Why is the survey limited to Christian homeschool alumni with 7 or more years of homeschooling?

The survey is limited in a number of ways simply because we need some basic parameters. We in no way believe that you must be homeschooled for at least 7 years to be an “alumni,” or to be impacted significantly (whether positively or negatively) by homeschooling. The last large survey of homeschool alumni (the aforementioned 2003 survey) was limited to alumni with at least 7 years’ experiences. Since we want our survey to provide a more up-to-date counter-balance to the 2003 survey, we decided to limit ourselves to the same experiential time length.

Q: Why is the survey limited to alumni of Christian homeschooling?

Only because it’s our area of experiential expertise as individuals and our focus as an organization! As the author of the survey, I was homeschooled K-12 in the Christian homeschool movement and I have limited firsthand knowledge of non-religious homeschool subcultures. I wanted to keep the survey as focused and accurate as possible — and to do that, I had to limit the survey to experiences and groups I know. That said, HARO would be 100% interested in doing a survey for alumni of non-religious homeschooling. So if you are such an alum, and would be interested in consulting with us and sharing your experiential expertise, please feel free to email us at homeschoolersanonymous@gmail.com. We would be happy to pursue the possibility of such a project.

Q: THIS SURVEY IS NOT REPRESENTATIVE.

So this isn’t a question, really. It’s more just a statement that a number of people seem to throw at the survey in an attempt to “discredit” it.

My response is: The very first page of the survey says, “As we are not randomly sampling the population, our results will be descriptive rather than representative.” So we state this fact upfront. For any survey to be representative of any given population, you need to have a random sample of that population. That is nearly impossible to obtain with the homeschooling population. Every survey conducted by Brian Ray’s NHERI and HSLDA are just as non-representative as ours. The difference is that, unlike Ray and HSLDA (usually), we will not pretend our data is representative. Hence our being upfront on the very first page that our survey’s results will be descriptive.

Q: Why do you ask “what gender were you assigned at birth?” rather than “what is your gender?” Or to put this in the remarkable language of one respondent, “F*ckin’ seriously? Why phrase it like that, p*ssies? I’m a male.”

Numerous individuals do not identify with the gender assigned to them at birth. You can take that as some sort of ideology and/or you can take it as the simple recognition that intersex and transgender individuals exist. We are not interested in erasing either of those populations — both because we object to such erasure inherently and because erasure will lead to less accurate data.

Intersex individuals, according to the Intersex Society of North America, are “born with a reproductive or sexual anatomy that doesn’t seem to fit the typical definitions of female or male.” Thus some are literally assigned a gender that does not necessarily correlate to stereotypes about physical anatomy. And yes, there are intersex homeschool alumni. Several of them have taken our survey. There might even be intersex students in your homeschool community right now. (Does your community daily erase their existence? Have ever you thought about that?)

Transgender individuals are, according to GLAAD, “people whose gender identity differs from what is typically associated with the sex they were assigned at birth.” Now, we desire to respect individuals’ gender identities. You might not. But the statistical reason why we ask the question the way we do is because simply asking a transgender person, “What is your gender?” (or as one respondent suggested, “the gender God created me as”) will not give us the data we are looking for. A transgender individual will answer that question with the gender they identify with, and not necessarily the gender they were assigned at birth. Our survey has a number of goals, and one of those goals is to analyze homeschool alumni experiences based on gender roles placed on children growing up. So regardless of what gender people now identify with, we need to ask this particular question in a way that (1) respects the existence of intersex and transgender individuals and (2) gives us the specific answers we need to do our analysis accurately.

Q: Aren’t the creators of the survey just angry ex-homeschoolers?

Well, I am the author of the survey. Let me introduce myself: My name is Ryan Stollar. I was homeschooled my entire life. I had a generally positive experience. I was a national award-winning high school debater. I got to tour the United States throughout high school and make friends all over the country. I have a B.A. and an M.A. I love my family. They have shown me that unconditional love is a reality. My family is also very supportive of HA and HARO: my dad is proud of what I do, my mom has contributed a post to HAmy older brother has contributed a post to HA, and my younger sister has promoted this survey. So no, the creator of the survey is not an angry ex-homeschooler. Get your facts right.

Q: What’s with the question about BDSM/kink?

Sorry to burst anyone’s bubble, but the answer to this question is quite simple: In some of our homeschool alumni communities, numerous conversations about BDSM/kink have arisen. A decent number of people seem attracted to or interested in such lifestyles and activities. I was merely curious to get data about it and see if there are any trends. Plus, it is a question that is never on surveys like these. Questions about frequency of porn use are rather popular on surveys of evangelicals, for example. Those questions have been done so many times and in so many ways. We would not be examining anything new. But I have never seen a survey address BDSM/kink. So that’s what’s up.

Q: What are you going to do with the data?

HARO as an organization is interested in using the data for educational purposes. For example: There are questions about if you struggle with mental health issues. What we are not going to do is make arguments like, “Homeschooling leads to _____.” Rather, we are interested in using the data to help educate and improve homeschooling communities. The data gives us information to say things like, “Out of this group of x many alumni who responded to the survey, z many have dealt with mental illness in their lives. One of the most common mental health conditions was q.” Such statistics can help tailor what sort of resources we focus on for HARO’s website, focus our efforts on educating homeschooling communities about the most common mental illnesses homeschool alumni deal with, etc.

Or take the abuse section, as another example: We’re not trying to — and honestly, we can’t without a representative sample — say how common abuse is in homeschooling. But we can say, “Out of this group, x many people have dealt with abuse in their homes, or even outside their homes, or knew people who were abused.” This data can help us communicate the importance of homeschooling communities creating homeschool co-op child abuse policies, educating people about the fact that abuse happens to homeschoolers (regardless of if it’s related to homeschooling), etc.

Q: Are you going to cast evil spells on the data so that it says things it doesn’t say?

No.

Q: ARE YOU TRYING TO BAN HOMESCHOOLING?

No.

If you have any other questions about the survey, feel free to email us at homeschoolersanonymous@gmail.com! And don’t forget to take and share the survey!

When Precision is a Red Pen

Heather Doney is the Executive Director of the Coalition for Responsible Home Education and blogs at Becoming Worldly.

As someone who has been studying and working on homeschooling issues from an academic as well as personal angle and who recently co-founded the Coalition for Responsible Home Education (CRHE), I want to say that Benjamin Keil makes some good points in “A Call for Precision”. He makes good points about the plural of anecdote not being data and also how we want to avoid confusing different types of arguments or reasoning for one another. I also think that Sarah Henderson made some good points, too.

We are talking about, within, and to a group of people who often suffered educational neglect. I know I did. Some people have been able to largely overcome it. I too have a masters degree today. Some have not. We want to be very careful not to intimidate anyone or make them feel like their story or perspective is not “correct” or “educated” enough to be told.

This is a place for people to tell their stories as they see them through their own eyes and for others to provide feedback and support, not judgement or academic critique.

So while I am working with a team of former homeschoolers who are trying to do our best to get the quantitative data we have (which is slim) all in one place and collect and share the qualitative data (which is just coming together), fact is we (and by this I mean all of us) do not have the kind of data to know how abuse and neglect in homeschooling compares to that occurring in other educational settings. It is a question to be answered, a known unknown. We just know that it happens and that there are some really bad cases and the watered down or nonexistent laws on homeschooling in many states don’t pass a basic common sense smell test.

I also think it is instinct for people to use the info they have and generalize based on their social milieu. It happens a lot, annoying social science researchers everywhere, since we want to measure and quantify.

But is a natural human tendency.

So I think Keil’s points would have been stronger if he had noted that homeschool parents who keep saying “these stories are rare” and “most homeschoolers are ______” really need to notice that they do this way too much, that it isn’t helping, and they need to knock it off. A really good example of exactly what we don’t need any more of: this post in Christianity Today.

“Anecdote passed off as data” doesn’t make for an airtight case if anyone does it and frankly so many of us have had to sit by and have our experiences silenced and dismissed while homeschool parents and leaders got a pass for this sort of nonsense for years. The “data” collected by Brian Ray’s NHERI was spread around in the media and the homeschool community as proof of homeschooling’s excellence across the board.

As a matter of fact, Ray’s “Strengths of Their Own” study isn’t proof of anything except that self-selected participants in a survey (with just under a 30% response rate, I might add) of white, middle and upper middle class Christian homeschool families usually do pretty good. I could do a voluntary study of prep school kids, say they represented American students as a whole, and it would be much the same kind of result.

Which is to say it is not an accurate depiction of the population at all.

My initial thoughts from combing through the quantitative and qualitative data available and also running a support group are that it seems that homeschools aren’t too different from public school in terms of us having “haves” and “have-nots.” The difference is we pretend our have nots just don’t exist because we don’t measure them. There are generally no mechanisms in place to shut down failing homeschools or fire failing or abusive homeschooling teachers.

Because there seems to be this huge socio-economic status/class difference in homeschool student experiences and outcomes, we will need to pay a lot more attention to that gap before any of us do any more generalizing about what homeschooling as a whole is and isn’t. We also need to make sure we leave wide open spaces where people can safely tell their stories without worrying that the rest of us will be judgy perfectionists or parse it apart harshly.

Even if we are well-meaning in taking the red pen to someone else’s story and perspective, that can be very intimidating and used as a means to quiet their voice.

Too many of us have already had more than enough of that happen in our lives and don’t need any more. So I want to say that while I want solid arguments and good data as much as the next person, even more than that I want people to feel free to tell their own story and share where they see it fitting into the whole. After all, it is because a growing group of people are telling their first-person stories that we are even discussing the need for data in the first place.

Stories are powerful things.

How Bad Homeschool Research Hurts Homeschoolers

HA note: The following is reprinted with permission from Kathryn Brightbill’s blog The Life and Opinions of Kathryn Elizabeth, Person. It was originally published on July 24, 2013.

When I was in college, I was one of the participants who answered the NHERI/HSLDA/Brian Ray survey of homeschooling graduates. I don’t remember how I got the survey, probably via email forward from my mom, but what I do remember is sending an email to my family after taking it in which I said something about how I wouldn’t believe a thing from the eventual study results. Aside from the fact that an email forwarded among homeschool groups asking people to take an internet survey is a lousy way of getting a representative sample (especially when we’re talking more than a decade ago, when we were nowhere as close to ubiquitous computing as we are now and computer access was still largely along socioeconomic lines), the survey itself was rife with methodological problems.

My memory is of a survey where I could tell exactly what answer they were looking to find, based on both the questions asked and the possible answers given for those questions. It was a survey that, even as a college student who had a positive experience and didn’t have the criticisms of the system that I do now, was so bad that I wondered why anyone would design it the way they did unless the goal was not usable data but a certain set of predetermined results.

If I got anything out of being homeschooled for twelve years by a math teacher, it’s a healthy appreciation for numbers. Numbers explain the world, or at least they can explain the world if they’re used correctly. Without good numbers, you might as well be stumbling around in the dark bumping your shins into the furniture. Bad numbers are even worse than no numbers; they’re like spreading Legos on the carpet while stumbling around in the dark with bare feet.

The HSLDA/NHERI numbers on homeschooling are like stumbling around on that Lego-strewn carpet. Aside from the aforementioned homeschool graduate survey, they also have test scores that they trot out to prove that homeschoolers do better on standardized tests than other students.

Except that the numbers don’t show that. What they actually show is the results of those homeschooled children whose parents not only use achievement tests (which isn’t all of them), who were aware of the NHERI survey (again, not all of them), and who agreed to submit their children’s scores for a study that they were already told was intended to make homeschooling look good. We have no idea how representative that group is of homeschoolers as a whole because it was self-selected and we don’t know whether that self-selected group in any way mirrors homeschoolers as a whole.

The modern homeschool movement is more than 30 years old and we still don’t have a good idea of homeschooling demographics. We know that they’re predominantly white and might be mostly religious (but even the religious makeup is hard to guess because the religious and the secular homeschoolers seldom run in the same circles). We don’t know what the socioeconomic makeup of the homeschool population is, and we don’t know the average education level of parents or a breakdown of those parents’ degrees. Without that, we can only guess at whether the socioeconomic levels of homeschoolers in NHERI’s test score data in any way represent homeschoolers as a whole.

Homeschoolers are fiercely independent even without taking into consideration that the religious ones don’t trust outside researchers because they’ve been told by HSLDA for years that outsiders want to hurt homeschooling. As for the secular unschoolers, they aren’t exactly fans of the establishment, which makes it hard to pin them down as well. Compounding the problem, the people the religious ones do trust (aka, HSLDA and NHERI) aren’t interested in giving good data, they’re interested in giving a good sales pitch for homeschooling. And so, we’re stumbling around in the dark knocking our shins into furniture and stepping on stray Legos.

There are a handful of small studies about adult former homeschoolers, but the rest is guesswork. We don’t have a good picture of the socioeconomic status of homeschooling families. We don’t really know how homeschoolers as a population do on achievement tests because we don’t have any surveys that aren’t self-selected. We don’t even have a good idea of how many people are homeschooling. It might be a million and a half, it might be twice that. We need good data and we simply don’t have it.

Are the kids I knew in high school who were all upper middle class children, mostly of parents with degrees in STEM fields, an accurate representation? What about the kids I knew when I was younger whose blue collar parents had no college and were passing their non-standard English dialect on to the kids rather than standard English grammar? Was my dad right when he always said that math was a weakness that homeschoolers needed to watch out for because the kids were being taught primarily by mothers who had been put on the consumer math track in school? Was I right when I used to tell people that the reason they thought homeschoolers were weird was because the normal ones who are in the majority didn’t advertise that they were homeschooled?

I don’t know because no one does.

And yet the Brian Ray/NHERI research keeps getting repeated in conservative and mainstream media even though it doesn’t actually tell us anything.

I understand why homeschool parents want to like the NHERI numbers; they help reassure them that they’re on the right path and not totally screwing up their kids with the homeschool choice. That’s why what NHERI’s doing with self-selected, unscientific research methods is so messed up. Parents, good parents, are relying on data that’s given a veneer of science but that may well be leading them to make educational choices that aren’t the best possible for their children. If math education, for example, really is the weakness for homeschoolers that my dad thought it was, homeschoolers need to know that so they can compensate for the weakness. If more than a fringe number of homeschool kids are having trouble fitting in with mainstream society because of the homeschool bubble, parents need to know that to correct for it.

With nothing but bad data to rely on, parents are left stumbling around in the dark hoping that they aren’t going to bash their shin on the corner of the coffee table or step on a stray Lego. That’s not serving homeschool parents well, and it’s certainly not doing anything for the kids.