Mike Godwin <mnemonic () eff org> on Rimm for Hot Wired part 3 of 3

[David Farber <farber () central cis upenn edu>]

this piece, and your subsequent dishonesty in defending it, I don't even
want to know you. That I have defended you in the past, and that I
defended this piece before I saw it ("Philip will try to balance the
story, I'm sure," I told people) is now an embarrassment to me that I'm
going to spend a long time living down.




media.1029.125: Avant Garde A Clue (mnemonic)  Mon 26 Jun 95 22:00




ped writes:


"The story was careful to keep the usenet stats separate from the adult
BBS stats."


The TIME story states:


"THERE'S AN AWFUL LOT OF PORN ONLINE. In an 18-month study, the team
surveyed 917,410 sexually explicit pictures, descriptions, short stories
and film clips. On those Usenet newsgroups where digitized images are
stored, 83.5 percent of the pictures were pornographic."


The Rimm study states:


"In addition, the team obtained descriptive listings from sixty-eight
commercial 'adult' BBS containing 450,620 pornographic images, animations,
and text files that had been downloaded by consumers 6.4 million times;
six 'adult' BBS with approximately 75,000 files for which only partial
download information was available; and another twenty-seven 'adult' BBS
containing 391,790 files for which no consumer download information was
available. Thus, a total of 917,410 descriptive listings were analyzed by
the  research team.'


Hmmm. Looks like the number 917,410 (450,620+75,000+391,790) has to do
with '"adult" BBS' -- and yet that number appears in the very  same graf
as some statistics on Usenet newsgroups.




Yet I could have sworn Philip just said he kept the stats separate!










media.1029.136: Avant Garde A Clue (mnemonic)  Mon 26 Jun 95 22:21




Philip writes:


"I was satisfied that your cricicism was based on the wording of the
executive summary you had read, and was not a flaw in the study."


An abstract is not an "executive summary." Abstracts discuss methodology
in detail.


"The fact is, that if you look at .binaries on Usenet, most of them come
from adult BBSs."


That you think this somehow supports applying your characterization of the
contents of Usenet from the contents of commercial porn BBSs shows that a
good remedial methods course is in order. Hint: you *do not know* whether
the sample of porn-BBS content that appears on Usenet has the same
distribution of content as the porn-BBS content surveyed in the Rimm
study. Rimm doesn't know either. There's no part of the study that
supports this leap.


Want another blunder?


You never mention in your story that the CMU researchers didn't look at
the images themselves -- they based their study on the *description* of
the images that appeared on the porn BBSs.


You never mention that the way they justified this was by checking 10,000
*Usenet* images against *their* descriptions -- that was taken to be the
measure of the accuracy of the *porn-BBS descriptions*.


You never mention that the CMU researchers probably didn't download the
images they purported to be surveying (that costs money, and you have to
do it over phone lines, not the Internet, and that takes a lot of time),
so *you don't even know*  if there actually *are* 917,000+ separate and
independent images, etc. -- many of the items that appear with different
descriptions may in fact be the same item.


"[If we are to beliee the study, and I still have no reason not to.]"


And you have *every* reason *to* believe the study, don't you, Philip?
Because if you *disbelieved*, it would undermine the value of your
exclusive.


"And yelling at me is not going to make that go away."


Why don't you buy a clue and respond to the real reasons I'm yelling at
you, Philip? If you think I'm upset because Philip the upright  journalist
has reported some unpleasant facts, you're even more self-delusional than
I thought you were.


Hint: "sex with dogs"/Internet. How to explain that little comment, when
the only conclusions about the availability of bestiality images concerned
porn BBSs, not the Internet?


Note: I'm not making any statement at all about whether bestiality images
are available on the Internet -- but the Rimm study can't be read as
saying there are, *since the Rimm study is not about images on the
Internet at all*. In fact, it's *not about images* -- it's only about
*descriptions of images*.


Of course, we needn't bore Time's readers with these little nuances.


To read the Rimm study about porn BBSs as saying something about Usenet is
like surveying Times Square adult bookstores and using the results to
characterize *all* bookstores in San Francisco.






media.1029.144: Avant Garde A Clue (mnemonic)  Mon 26 Jun 95 22:39




ped writes:


"Mike, thse are two separate sentences and two separate statements."


Well, now, Philip, let's look at how those two separate sentences and two
separate statements are presented:


"THERE'S AN AWFUL LOT OF PORN ONLINE. In an 18-month study, the team
surveyed 917,410 sexually explicit pictures, descriptions, short stories
and film clips. On those Usenet newsgroups where digitized images are
stored, 83.5 percent of the pictures were pornographic."


Why, those statements don't look separate to me. In fact, they appear to
be right next to each other. And they are adduced to support the same
conclusion: THERE'S AN AWFUL LOT OF PORN ONLINE.


Guess what, Philip -- you're still screwed on this one. Because *either*
one must read the sentences together (in which case you do mix BBS and
Usenet stats) *or* one must read the sentences separately, *in which case
the second sentence does not in itself support the conclusion that THERE'S
AN AWFUL LOT OF PORN ONLINE.* (Basic stat hint: percentages by themselves
tell us nothing about quantity. Most people learn this basic fact about
percentages in high school.)


But if you are seriously arguing that those two sentences, right next to
each other and used to support the same point, won't be read as  being
about the same thing, you are seriously cracked.






 media.1029.170: Avant Garde A Clue (mnemonic)  Tue 27 Jun 95 06:13




ped writes:


"When I posted above that I hadn't heard any criticism of the study's
methodology beyond the fact that it was conducted by an undergraduate, I
meant I had hadn't heard any *here." I had, of course, heard Mike and prof
criticize it."


Even if we grant that you were trying to say, in response to my comment
about Rimm's status as an undergraduate, that you had not heard other
criticisms of the study from me *here*, you *had* heard other criticisms
of the study from me elsewhere. So the implication that I and others are
simply taking cheap shots and not making substantive criticisms is just
wrong.


But I think it's important to acknowledge the extent to which you were
unwilling to accept those criticisms of the study. Yes, I believe you
reviewed the study with my criticisms in mind -- you needed to find a way
to explain them away or dismiss them, or else they might have undermined
the whole raison d'etre of the piece.


So you reminded yourself that we hadn't seen the study (I offered to
review it for you with the promise not to leak it -- Donna would surely
have done the same). You relied on Sirbu's endorsement of Rimm (although
Sirbu seems to have had no prior experience in conducting this sort of
study either). You told yourself that, regardless of what (implicitly
minor) methodological flaws there might be in the study, or whether the
conclusions were supportable by the data they did gather, *this is an
important study*.


In this, you remind me of the infamous last line of Newsweek's cover story
on the Hitler diaries [I paraphrase Newsweek here]: "In the end, this has
been such a big story that it doesn't really matter whether the diaries
are genuine or not."




----------------




I was venting at this point. But it took a single epiphanic moment to
convince me that the thing to do was not merely criticize Philip, but
instead to do the kind of reporting he and Time had failed to do. What was
that moment? I recounted it in a WELL posting reproduced below:




-----------------




media.1029.197: Avant Garde A Clue (mnemonic)  Tue 27 Jun 95 15:33




Well, I'm going to be on Nightline tonight, debating Ralph Reed of the
Christian Coalition. It has already been taped -- we mostly debated
whether the Exon legislation was a better fix for protecting your children
than the software tools and filters that I advocated.


But the taped lead-in focuses on the Rimm study, and stresses how the Rimm
study shows how easily pornography is available to children on the Net.
And not just any old pornography, but the hebephilia, urination, etc.,
that the Rimm study shows there is so much demand for.


Before we taped, I mentioned to one of ABC's reporters, Richard Harris,
that there were a number of methodological criticisms of the Rimm study.


So, afterwards, Harris and his researcher arranged to have me in a
conference call with Philip, Marty, and apparently one or more other
people who were involved in the study. I was given a chance to raise my
concerns about the study's methodology, with mixed results, so  that the
ABC people could hear, at least, some of the reasons for  believing that
the study focused not on cyberspace as a whole, but on a nonrandomly
selected subset of commercial BBSs that focus on selling porn, that the
study was based on descriptions, not images,  and that the conclusory
links between Rimm's sample and the "information highway" as a whole were
not supported methodologically.


I doubt it made much difference -- ABC guys aren't terribly interested in
hearing nerds talk about statistical inferences. But they were kind enough
to give me a hearing on the methodological and ethical problems I have
with how this done.


During the call, Philip noted that I'm an advocate, so it follows that I
feel compelled to argue against a study that reports inconvenient  facts.
(I later pointed out to the ABC guys that the Levy piece has inconvenient
facts, but I'm not outraged about that one, and that before I was a lawyer
I was a researcher.) The implicit dishonesty of his casting doubt on my
motives, of course, lies in the suggestion that Philip and Martin Rimm
don't have far stronger motives of their own to pitch the study as
something groundbreaking and compelling and reliable.


Rimm answered a lot of the question I raised, some adeptly and others with
dodges. He claimed to have no agenda.


We wound down, although a few voices (mostly mine) were raised. But before
we lost the connection, I heard this:


Philip: "Marty, you there?"


Rimm: "Yes, I'm here."


Philip: <slight pause> "Good job!"




--------------------------




To a journalist, Philip's "good job" was a revelation. At that point, I
turned to Harris and mouthed (with regard to Philip): "He's on the team!"
It was stunningly clear that Philip had so identified himself with the
story that he believed his and Rimm's interests were essentially the same.




There was only one fix, I thought -- do everything I could to make sure
that the truth about the Rimm study, and about Time's collusive arrangment
that prevented it from being properly criticized, be made as public as
possible, as widely as possible. I had already fedexed a copy of the study
to Donna Hoffman -- working with EFF's legal interns, I made several
copies of the article and fedexed them to people who knew enough to
criticize the piece the way it should have been criticized at the
beginning.


I later learned that Philip characterized my labors as an "orchestrated
campaign" to discredit him, conducted by a "professional lobbyist." Since
I've never been a lobbyist in my life, that comment did sting, but of
course in a sense he was right that I was conducting an "orchestrated
campaign." It was orchestrated largely from my PowerBook, and the campaign
consisted of putting copies of the study in front of independent reporters
and other commentators who were capable of reading it and seeing the
obvious. That's what led to Elizabeth Corcoran's insightful piece in the
Washington Post, and to Peter Lewis's thorough reporting in the Times.
It's what led to the critiques of the study that you see on Donna
Hoffman's Web page and here at Hotwired. It's what has led to the
revelation that Rimm's own faculty adviser, named in the study's
biographical footnote, doesn't think Rimm's data support his conclusions
about Usenet.


And all the time I was getting other people to read the study, I was doing
my own reading. Perhaps the single most damning discovery I made appears
in this posting:




---------------------


media.1029.511: Avant Garde A Clue (mnemonic)  Sat 1 Jul 95 18:34




Let's come back to the Footnote Quiz, which Philip declined to answer. I
had written this:


*********************


"9. As a result of federal legal action against a few well known 'adult'
BBS operators, including Robert and Carleen Thomas (Amateur Action) and
Robert Copella (Pequena Panacha), some systems have removed their
paraphilic, pedophilic, and hebephilic imagery from public display. This
has created a thriving underground market for 'private collections' and
anonymous ftp sites on the Internent, which cannot be studied
systematically. Thus, it may be difficult for researchers to repeat this
study, as much valuable data is no longer publicly available. See infra
notes 89-95 and accompanying text."


Now, Philip, try answering this quiz:


Of the many unsupported assertions in this single footnote, which one
would raise the *biggest* red flag for a reader/editor working for a
peer-reviewed journal?


*********************




Now, this footnote is rife with candidates for "red flag status." The
"some systems have removed" claim is undocumented and unsupported, as is
the "thriving underground market" and the "anonymous ftp sites [market
created for such sites because of porn crackdown]" comment. So's the claim
that anonymous ftp sites can't be studied "systematically." (Note: it may
well be true, but *it is not supported by the study*.)


But the single biggest red flag is the penultimate sentence--"it may be
difficult for researchers to repeat this study...."


It is, in my opinion, designed to make the study unkillable, so that
anti-porn activists will be able to use it forever, *regardless of
subsequent studies that seem to disprove it*.


Think about what would happen if subsequent studies seemed to  support
Rimm's conclusions:


(Rimm: "See? I was right!")


Now think about what would happen if subsequent studies seemed to
*disprove* Rimm's conclusions.


(Rimm: "See? I was right!")


You begin to see why the author might have felt compelled to  sidestep
peer review if at all possible.


------------------------




The process of exposing the Rimm study, which I increasingly believe may
have been a deliberately political ploy in the guise of "research," is
ongoing. You can see the results here and elsewhere on the Net. And the
same is true of the process of epxosing the extent to which Time and
Philip Elmer-DeWitt traded their responsibility to the American public in
return for questionable exclusive, written and packaged to maximize panic
about the Net -- it's ongoing.


But there's one thing that's not ongoing, and that's any "orchestrated
campaign" to discredit Philip Elmer-DeWitt. My response to that particular
charge is best expressed in one last posting that I'll share with you
here:




-----------------


media.1029.423: Avant Garde A Clue (mnemonic)  Fri 30 Jun 95 13:19




Philip writes:


"I can understand why you feel obliged to discredit the Rimm study."


I don't think you do. Based on what you have said up to now, you think
it's because it comes to some conclusions that are inconvenient for my
work. You think I'm just playing out some role as an advocate for
net.freedom, and therefore feel compelled to challenge the study out of a
sense of loyalty to my cause. I'm sure that's what you told yourself when
you decided to dismiss my comments out of hand.


So long as you labor under this self-delusion as to my motives, you won't
have a clue about why I'm doing what I'm doing now.


"But I'm having a hard time understanding why you felt obliged to
discredit me at the same time."


Philip, you should be clear on this:


I have never, ever had it in my power to discredit you, nor have I ever
thought I did.


You discredited yourself.






            -30 -