Junk science 101

[unixronin]

Yes, it’s more junk science out of the UPenn School of Medicine.  We’ve become accustomed to seeing some horrendously bad junk science out of the medical academic sector where firearms are concerned, but this is worse than most.  I honestly think this one is, methodologically speaking, possibly even worse than the Kellermann study.

News release here, American Journal of Public Health abstract here.

Let’s briefly summarize the methodology here.

First, Dr. Branas compiled a list of 677 people shot in the course of assaults in Philadelphia between 2003 and 2006, and determined that 6% of them were reported to have been in possession of a gun in a holster, pocket, waistband or vehicle when they were shot.  (Note:  By elementary arithmetic, that means 94% of those shot were unarmed.)

Next, Dr. Branas and his intrepid law students picked up the phone book, randomly called 684 people in Philadelphia who had not been assaulted, soon after reported shootings, and asked them whether at the time of the shooting they had a gun in their possession.

From these two sets of data, Dr. Branas somehow derived the conclusion that possessing a gun makes you 4.5 times more likely to be shot in an assault.

... No, I don’t see a logical path from data to conclusion either.  But I have a pretty good idea the conclusion was predecided and the study designed to fit it.

So!  Just for fun, let’s play a game.  Let’s see if we can think of ALL of the logical and methodological problems with this study.

Have at it!

Comments

[lebo-superman.livejournal.com]

I'll play. Let's see, how about asking the 684 people who were NOT assaulted about their geographic location at the time of the shooting, as contrasted with the location of the reported assaults.

Let's also put the number of people who were shot OUTSIDE of their vehicle, who reported that their gun was INSIDE their vehicle, into the "unarmed at the time of the shooting" category.

How is that for a start?

[unixronin.livejournal.com]

Even aside from the question of whether people called by a random stranger in a gun-unfriendly city with a high crime rate are going to admit to owning firearms, I think the entire methodology of choosing the control set in this "study" is flawed to the core. It's like trying to evaluate cancer survival rates under different care regimens, using a control set of people who don't have cancer.

[jhetley.livejournal.com]

Well, how about checking how many of the original (assaulted) group were *legally* in possession of a weapon? Drop those who were not from the statistical pool. Eliminates at least some of those who may have been engaged in criminal behavior, increasing all kinds of risk factors . . .

Also compensates for reporting bias in the second sample, as people in illegal possession of a weapon are not likely to admit that to a random caller . . .

[unixronin.livejournal.com]

Personally, in Philadelphia, a very gun-unfriendly city with a high crime rate, I suspect very few people are going to admit to an unknown phone caller that they own a gun — legally or not — any more than they'd admit to possession of any other stealable valuables to someone who called them out of the blue claiming to be a canvasser. If some stranger called me on the phone and asked whether I owned guns or other valuables, I'd most likely hang up on them and consider filing a police report of suspicious activity. "Let your fingers do the casing"?

[jordan179.livejournal.com]

The most obvious point is that some of those shot may have been shot in the course of committing violent crimes. Another point is that some of those shot may have been law enforcement personnel, who are required to keep guns about them.

[unixronin.livejournal.com]

Yup, that's another good question. I couldn't help but notice how the study (or at least the news release) is conspicuously silent on the issue of how many of those shot were victims and how many were perps.

[argonel.livejournal.com]

You know for simplicities sake, I think we can safely assume that nearly all of the people who were shot were in the vicinity of a firearm, and therefor being near firearms make you much more likely to be shot. It is just as misleading, but more truthful.

[lblanchard.livejournal.com]

We are Philadelphia gun owners in one of the top five "homicide hotspot" police districts, and my husband has a concealed carry permit. We have NRA stickers on our doors out of sheer cussedness and are known/respected/feared for our general level of "craziness".

But tell a canvasser we have a weapon? In a pig's eye. In fact, we usually hang up on canvassers anyhow, before they get a chance to ask us their questions. So perhaps we're listed as "don't have a gun" in this study.

Small world...Dr. Branas contributed an essay to a volume on academic uses of GIS that we were putting together for the Council on Library and Information Resources. The volume was never published, though.

[theonetruejim.livejournal.com]

There's also a statistical fallacy in deriving any relationship, based on their sampling methodology. They called a "random sampling" of folks who happened to in the phone book, and used that as their control group.

Let's posit for a moment that everyone who answered was absolutely truthful in all their answers, and the number of personal firearms was accurately represented by the survey. It's still a breathtaking leap of illogic to form the conclusion that having a firearm in your possession is more likely to result in an assault in which you are shot, because they picked the wrong control group. The proper control group would be those involved in a reported assault with a deadly weapon where nobody got shot.

There's a vast gulf between "not assaulted" and "shot", and the implicit assumption that firearms possession is the only variable is very badly flawed - it's proving that 1 = 2 by dividing by zero wrong.

Ding ding! Give that man a diploma.

[unixronin.livejournal.com]

Indeed, that's the biggest flaw of the lot. The choice of a wrong (and, frankly, meaningless in context) control group makes the entire study "all sound and fury, signifying nothing". Garbage in, garbage out.

One more thing...

[theonetruejim.livejournal.com]

They actually had a built-in control group (assuming they were careful enough to get the police reports for the assaults that made up the selection criteria for their study group - an assumption that is by no means certain). That would be the other parties in the assault. Unless the gunshots were self-inflicted, there was at least one other person in every case who was in possession of a firearm at the time of the assault.

Now, there's not enough data to be sure about any conclusions, but that didn't stop the original reporters, so I'll make up some data and form my own conclusion. I'm going to assume there was one and only one other person in each assault, none of the wounds were self-inflicted, and only a single party in each assault was actually shot.

With my "data" (which is probably more reliable than the study's control group), there were 677 people in possession of a firearm at the time of an assault resulting in a shooting that did not get shot. In contrast, there were 38-44 people in the study group who both were in possession of a firearm and got shot, in contrast to the 633-639 who were unarmed and got shot.

Based on the study methodology, I'd have to conclude you're about 16-19 times as likely to be shot in an assault if you're unarmed.

Re: One more thing...

[unixronin.livejournal.com]

Good point. :)

[wyrdling.livejournal.com]

this just makes my head hurt. what passes for standards with these people? was it actually published anywhere reputable, or just stuffed through the fax machines of a bunch of bored press agents?

[wyrdling.livejournal.com]

i guess the american journal of public health gets low marks for standards...

[lwj2.livejournal.com]

Correct."F" would be the applicable grade, for accepting a paper which was done with such obviously flawed methodology.

That it was released under the aegis of the University of Pennsylvania, at one time a reputable institution, says much of academia these days.

[unixronin.livejournal.com]

Honestly, that's not news. The AJPH, just like the JAMA, has earned itself a reputation for being willing to publish, with very little or no examination, any paper or study that claims to support a finding that guns are bad.

[wyrdling.livejournal.com]

no wonder people don't trust "science," when "research" comes up with this kind of BS...

Another study

[theonetruejim.livejournal.com]

Using the methodology of this study, I performed an experiment of my own. First, I found a group of 677 people who were being treated for asthma with Advair (an inhaled medication that contains both a steroid and a bronchodilator). Of that 677, 44 had experienced an episode of shortness of breath within the last week.

Next, I called people at random from the phone book until I found 684 who'd never been treated for asthma for my control group. In my control group, only 3 had experienced an episode of shortness of breath.

In other words, Advair makes you 16 times more likely to experience shortness of breath than not doing anything at all. Why, oh why! Does the FDA let Big Pharma poison us like this?