Journal publication of ESP paper prompts "outrage"

Moderators: Elvis, DrVolin, Jeff

Re: Journal publication of ESP paper prompts "outrage"

Postby slomo » Thu Jan 06, 2011 9:18 pm

DrVolin wrote:In each of the smaller sets of observations, you expect more variance than in the larger, overall set from which they are drawn. Imagine that you run telekinesis experiements on the ability of subjects to affect the outcome of a coin toss. You run 20 tosses, 5 for each of 4 subjects. Overall, you find close to 10 tails and 10 heads. No effect. Now you analyse the tosses for each subject individually. You find that one subject has 4 heads, and one subject has four tails. Are you going to claim that those two are telekinetic while the other two aren't? If you had each subject perform 3 tosses one day, and 2 tosses on the next, are you going to find that they are all telekinetic at some times if you analyse their coin toss sessions individually?

You would be able to quantify such heterogeneity using a generalized linear mixed model.

I have not yet read the paper, so I cannot say whether they analyzed the statistics properly. I would, in fact, classify myself as a "top-flight" statistician, given my credentials (though not my personal inclinations). When I'm less tired, I'll take a crack at it.
User avatar
slomo
 
Posts: 1781
Joined: Tue Dec 06, 2005 8:42 pm
Blog: View Blog (0)

Re: Journal publication of ESP paper prompts "outrage"

Postby slomo » Thu Jan 06, 2011 9:57 pm

OK, I will try to critique Experiment 1.

There are potentially numerous methodological issues in the description of the experiment on pages 7-8, but I am not qualified to discuss them, as they are specific to psychology. Notably, the required "settle-down" period between erotic images could introduce some kind of confounding, although again, I can't say.

As for the statistics described on page 9:
Across all 100 sessions, participants correctly identified the future position of the erotic pictures significantly more frequently than the 50% hit rate expected by chance: 53.1%, t(99) = 2.51, p = .01, d = 0.25.3 In contrast, their hit rate on the nonerotic pictures did not differ significantly from chance: 49.8%, t(99) = -0.15, p = .56. This was true across all types of nonerotic pictures: neutral pictures, 49.6%; negative pictures, 51.3%; positive pictures, 49.4%; and romantic but nonerotic pictures, 50.2%. (All t values < 1.) The difference between erotic and nonerotic trials was itself significant, tdiff(99) = 1.85, p = .031, d = 0.19. Because erotic and nonerotic trials were randomly interspersed in the trial sequence, this significant difference also serves to rule out the possibility that the significant hit rate on erotic pictures was an artifact of inadequate randomization of their left/right positions.

3 Unless otherwise indicated, all significance levels reported in this article are based on one-tailed tests and d is used as the index of effect size.

It appears from this description that the unit of observation is a "session", with n=100 independent observations. Given that they report 99-degree-of-freedom test statistics, I gather that the quantitative outcome being assessed is the number of correct guesses (out of 36 trials, see page 6), treated as an approximately-normal continuous variable. Given the Central Limit Theorem (CLT), it may be appropriate to approximate a binomial variable with a large number of trials as continuous, but I worry that 36 trials (and as few as 18 when aggregating by picture type) are insufficient for the CLT to apply. I would probably have used a generalized linear mixed model (GLMM) with logistic regression, to address the heterogeneity concern that Dr. Volin raised. However, there are issues with this approach as well, given that there is (possibly) between-subject heterogeneity as well as (possibly) temporal autocorrelation, the latter introducing a fair bit of computational complexity; in any case, it would have been a good sensitivity check to at least try a random-intercept GLMM, and/or a generalized estimating equation (GEE) with a random intercept and an autoregressive working correlation assumption. There is a multiple testing problem here (3 tests), that requires multiplying each p-value by 3 (for a crude Bonferroni correction), or else something potentially more sophisticated that could improve power. However, I am not impressed by p=0.03, which is the best p-value after Bonferroni-adjustment. I am also concerned that the authors used one-tailed tests. If you convert to 2-tailed tests with Bonferroni correction, the best p-value is 0.06, i.e. insignificant at the conventional alpha=0.05. In any case, the effect sizes are very small.

In other words, this is not particularly impressive.

And this is a critique issued by a statistician/epdemiologist who believes in woo.
User avatar
slomo
 
Posts: 1781
Joined: Tue Dec 06, 2005 8:42 pm
Blog: View Blog (0)

Re: Journal publication of ESP paper prompts "outrage"

Postby ultramegagenius » Sat Jan 08, 2011 6:26 pm

i suppose many of you have heard of Dean Radin. here's 90 minutes of him insisting that a cross-section of psi experimental data definitively proves that the incidence of ESP is higher than pure chance, particularly with regard to pre-cognition and the Gansfield experiment:
User avatar
ultramegagenius
 
Posts: 262
Joined: Tue Aug 05, 2008 11:15 am
Blog: View Blog (0)

Re: Journal publication of ESP paper prompts "outrage"

Postby Avalon » Sat Jan 08, 2011 8:49 pm

In a 2003 tribute to his good friend Marcello Truzzi, Jerome Clark had this to say about the idea that "extraordinary claims demand extraordinary evidence":

I might note here that it was Marcello, not Carl Sagan, who
coined the often-misattributed maxim "Extraordinary claims
demand extraordinary evidence." In recent years Marcello had
come to conclude that the phrase was a non sequitur --
meaningless and question-begging -- and he intended to write a
debunking of his own words. Sad to say, he never got around to
it.

http://ufoupdateslist.com/2003/feb/m03-014.shtml

Wikipedia's page on Truzzi gives these two preceding iterations of the concept that I haven't run into before:

Truzzi is credited with originating the oft-used phrase "Extraordinary claims require extraordinary proof," which Carl Sagan then popularized as "Extraordinary claims require extraordinary evidence."[12] However, this is a rewording of a quote by Laplace which goes, "The weight of evidence for an extraordinary claim must be proportioned to its strangeness."[13] This, in turn, may have been based on the statement "A wise man, therefore, proportions his belief to the evidence" by David Hume.

http://en.wikipedia.org/wiki/Marcello_Truzzi

Jerry Clark amplified Tuzzi's rejection of the phrase in another posting a year later:

Here is the late sociologist of science Marcello Truzzi, the man
who coined the phrase (which in his last years he rejected as
effectively without meaning) "extraordinary claims demand
extraordinary evidence." (No, Carl Sagan, usually credited, did
not invent the maxim. Basically, Truzzi concluded that the
twice-used "extraordinary" is largely a question-begging,
subjective judgment and that, moreover, the maxim has serious
flaws as a description of how science actually judges truth
claims. I know this from my many personal conversations with
Marcello, a close friend, who died, sad to say, before he could
write a paper or book elucidating his revised thinking.) Anyway:

"If a critic asserts that there is evidence for disproof, that
he has a negative hypothesis... he is making a claim and
therefore also has to bear a burden of proof. Sometimes, such
negative claims by critics are also quite extraordinary... in
which case the negative claimant also may have to bear a heavier
burden of proof than might normally be expected.

"Critics who assert negative claims but who mistakenly call
themselves 'skeptics' often act as though they... have no burden
of proof placed on them at all... [But] if a critic asserts
that the result was due to artifact X, that critic then has the
burden of proof to demonstrate the artifact X can and probably
did produce such results under such circumstances.... Alas, most
critics seem happy to sit in their armchairs producing post hoc
counterexplanations."

http://ufoupdateslist.com/2004/dec/m09-032.shtml

Whenever the "Extraordinary claims demand extraordinary evidence" idea is thrown around, I'd suggest it is a legitimate response to quote Clark's account of Truzzi's rejection of his phrase, and to point out that while Truzzi was a founding member of CSICOP, he later left the organization because of its move toward debunking rather than true skepticism.
User avatar
Avalon
 
Posts: 1529
Joined: Tue Jun 21, 2005 2:53 pm
Blog: View Blog (0)

Re: Journal publication of ESP paper prompts "outrage"

Postby Canadian_watcher » Sat Jan 08, 2011 8:55 pm

So far, at least three efforts to replicate the experiments have failed


I knew that would happen. :ohno: ;)
Satire is a sort of glass, wherein beholders do generally discover everybody's face but their own.-- Jonathan Swift

When a true genius appears, you can know him by this sign: that all the dunces are in a confederacy against him. -- Jonathan Swift
User avatar
Canadian_watcher
 
Posts: 3706
Joined: Thu Dec 07, 2006 6:30 pm
Blog: View Blog (0)

Re: Journal publication of ESP paper prompts "outrage"

Postby Avalon » Sat Jan 08, 2011 9:02 pm

Jonah Lehrer on the Decline Effect and the scientific method:

"Many results that are rigorously proved and accepted start shrinking in later studies."

http://www.newyorker.com/reporting/2010 ... ntPage=all
User avatar
Avalon
 
Posts: 1529
Joined: Tue Jun 21, 2005 2:53 pm
Blog: View Blog (0)

Re: Journal publication of ESP paper prompts "outrage"

Postby Searcher08 » Sat Jan 08, 2011 9:03 pm

Avalon wrote:In a 2003 tribute to his good friend Marcello Truzzi, Jerome Clark had this to say about the idea that "extraordinary claims demand extraordinary evidence":

I might note here that it was Marcello, not Carl Sagan, who
coined the often-misattributed maxim "Extraordinary claims
demand extraordinary evidence." In recent years Marcello had
come to conclude that the phrase was a non sequitur --
meaningless and question-begging -- and he intended to write a
debunking of his own words. Sad to say, he never got around to
it.

http://ufoupdateslist.com/2003/feb/m03-014.shtml

Wikipedia's page on Truzzi gives these two preceding iterations of the concept that I haven't run into before:

Truzzi is credited with originating the oft-used phrase "Extraordinary claims require extraordinary proof," which Carl Sagan then popularized as "Extraordinary claims require extraordinary evidence."[12] However, this is a rewording of a quote by Laplace which goes, "The weight of evidence for an extraordinary claim must be proportioned to its strangeness."[13] This, in turn, may have been based on the statement "A wise man, therefore, proportions his belief to the evidence" by David Hume.

http://en.wikipedia.org/wiki/Marcello_Truzzi

Jerry Clark amplified Tuzzi's rejection of the phrase in another posting a year later:

Here is the late sociologist of science Marcello Truzzi, the man
who coined the phrase (which in his last years he rejected as
effectively without meaning) "extraordinary claims demand
extraordinary evidence." (No, Carl Sagan, usually credited, did
not invent the maxim. Basically, Truzzi concluded that the
twice-used "extraordinary" is largely a question-begging,
subjective judgment and that, moreover, the maxim has serious
flaws as a description of how science actually judges truth
claims. I know this from my many personal conversations with
Marcello, a close friend, who died, sad to say, before he could
write a paper or book elucidating his revised thinking.) Anyway:

"If a critic asserts that there is evidence for disproof, that
he has a negative hypothesis... he is making a claim and
therefore also has to bear a burden of proof. Sometimes, such
negative claims by critics are also quite extraordinary... in
which case the negative claimant also may have to bear a heavier
burden of proof than might normally be expected.

"Critics who assert negative claims but who mistakenly call
themselves 'skeptics' often act as though they... have no burden
of proof placed on them at all... [But] if a critic asserts
that the result was due to artifact X, that critic then has the
burden of proof to demonstrate the artifact X can and probably
did produce such results under such circumstances.... Alas, most
critics seem happy to sit in their armchairs producing post hoc
counterexplanations."

http://ufoupdateslist.com/2004/dec/m09-032.shtml

Whenever the "Extraordinary claims demand extraordinary evidence" idea is thrown around, I'd suggest it is a legitimate response to quote Clark's account of Truzzi's rejection of his phrase, and to point out that while Truzzi was a founding member of CSICOP, he later left the organization because of its move toward debunking rather than true skepticism.



Avalon, this beautiful post is one of the reasons I feel so lucky to know about (and spend time in) this place
Thank you!
User avatar
Searcher08
 
Posts: 5887
Joined: Thu Dec 20, 2007 10:21 am
Blog: View Blog (0)

Re: Journal publication of ESP paper prompts "outrage"

Postby Simulist » Sat Jan 08, 2011 9:23 pm

I'm not sure which inspires the greater degree of skepticism in me: (A) extraordinary claims without sufficient proof and sometimes no proof at all, or (B) rigid, dogmatic beliefs that cannot permit at all any interpretation of a given question, other than ones own.

Probably "B" though — and I wish this tendency were a trait of the strictly religious only, but it isn't.
"The most strongly enforced of all known taboos is the taboo against knowing who or what you really are behind the mask of your apparently separate, independent, and isolated ego."
    — Alan Watts
User avatar
Simulist
 
Posts: 4713
Joined: Thu Dec 31, 2009 10:13 pm
Location: Here, and now.
Blog: View Blog (0)

Re: Journal publication of ESP paper prompts "outrage"

Postby Avalon » Sat Jan 08, 2011 11:46 pm

Extraordinary claims demand extraordinary investigation!
User avatar
Avalon
 
Posts: 1529
Joined: Tue Jun 21, 2005 2:53 pm
Blog: View Blog (0)

Re: Journal publication of ESP paper prompts "outrage"

Postby barracuda » Sun Jan 09, 2011 12:05 am

More and more it seems as if proof of anything is extraordinary.
The most dangerous traps are the ones you set for yourself. - Phillip Marlowe
User avatar
barracuda
 
Posts: 12890
Joined: Thu Sep 06, 2007 5:58 pm
Location: Niles, California
Blog: View Blog (0)

Re: Journal publication of ESP paper prompts "outrage"

Postby slimmouse » Sun Jan 09, 2011 12:16 am

I have difficulty proving anything I don't believe. But then again that could be just we.
slimmouse
 
Posts: 6129
Joined: Fri May 20, 2005 7:41 am
Location: Just outside of you.
Blog: View Blog (3)

Re: Journal publication of ESP paper prompts "outrage"

Postby 8bitagent » Sun Jan 09, 2011 1:39 am

What do you folks think of the long running Princeton "EGG" global consciousness project? I never fully understood how it works, but I find it mighty interesting to say the least
"Do you know who I am? I am the arm, and I sound like this..."-man from another place, twin peaks fire walk with me
User avatar
8bitagent
 
Posts: 12244
Joined: Fri Aug 24, 2007 6:49 am
Blog: View Blog (0)

Re: Journal publication of ESP paper prompts "outrage"

Postby Sounder » Thu Jan 20, 2011 10:54 am

8bitagent wrote...
What do you folks think of the long running Princeton "EGG" global consciousness project? I never fully understood how it works, but I find it mighty interesting to say the least


It would seem to be more robust because it deals with a larger and less personal data set.

The ‘outrage’ shown toward Daryl Bem refutes the assertion of objectivity that science claims as its foundation. The current iteration of ‘science’ has since 1666 been a non-objective materialistic conditioning system. It is not rational at all for ‘science’ to get its panties up in a bunch over the notion that there may exist a subtle underlying and connecting field. I mean, who put these materialists in charge of science anyway? HMW perhaps? Fortunately science can be bigger than this, it can, theoretically, examine the things that don’t fit at the same time it provides the conceptual structures that make things appear to fit. The lost ideal within science of looking at the anomalous needs a boost. It will be a new day for science when it can examine how conceptual structuring itself (consciousness) impacts the flow of energy within the matrix of being.

Until then though we can take comfort in seeing rigidity of science being softened by the experiences of scientists and lay people alike. This article by Mark Morford is a fine example, that sub-consciously at least we all know that there is a connective field that stands behind appearances, yet so many twist themselves into knots denying the possibility, while the brave ones might take a few timid pokes at orthodoxy.

Mark Morford
-
Your science is stuck in my mystical

Extraordinary Knowing: Science, Skepticism, a… (Paperback)
by Elizabeth Lloyd Mayer

http://www.amazon.com/gp/product/055338 ... 0553382233

I'm reminded of a fantastic little book by the late psychoanalyst Elizabeth Lloyd Mayer, called Extraordinary Knowing. In the book, Mayer decides, against her better, experienced scientific judgment, to investigate a rather astonishing phenomenon that happened to her -- the pinpoint locating and return of a stolen harp by means of a psychic dowser.

She investigates. She interviews her peers, experts, various geniuses in their scientific fields, from surgeons to psychoanalysts, researchers to professors, and sure enough begins to find something, well, extraordinary.

One after another, in hushed and private, slightly embarrassed tones, many of her most serious and widely respected colleagues begin to tell her that the real reason they are great at what they do, the explanation for why they harbor some sort of special touch, wisdom, healing power, knowledge and so on, is not merely due to rigorous training, massive education, endless practice. There is... something else…..

…..We know so much, we know nothing. Outside of both these truths lies a third thing, in a space where the first two intersect and dance and leave behind a gleaming, impossible residue that tastes like God, but probably isn't. What do you think it is?


Read more: http://www.sfgate.com/cgi-bin/article.c ... z1BZSKWpGu


I’m going to half to get back to you later on that question Mark.
All these things will continue as long as coercion remains a central element of our mentality.
Sounder
 
Posts: 4054
Joined: Thu Nov 09, 2006 8:49 am
Blog: View Blog (0)

Re: Journal publication of ESP paper prompts "outrage"

Postby Joe Hillshoist » Thu Jan 20, 2011 8:02 pm

slimmouse wrote:I have difficulty proving anything I don't believe. But then again that could be just we.


No I think its you.
Joe Hillshoist
 
Posts: 10616
Joined: Mon Jun 12, 2006 10:45 pm
Blog: View Blog (0)

Re: Journal publication of ESP paper prompts "outrage"

Postby hanshan » Tue Jan 25, 2011 8:46 pm

The Consul wrote:
nathan28 wrote:
But many experts say that is precisely the problem. Claims that defy almost every law of science are by definition extraordinary and thus require extraordinary evidence. Neglecting to take this into account — as conventional social science analyses do — makes many findings look far more significant than they really are, these experts say.


Can someone unpack this language for me? I really don't understand something about it--probably the underlying assumptions, which are unclear to me as well.


Someone has a clinic or institute. From previous case studies a specific model is composed. You might be studying, for example, people who have dreams that seem to forsee the future. Let's say 100 people are qualified to participate in this study. Each person is a known "dreamer" who has exhibited some form of having dreams that seem to portend later events.

The laws of physics do not contain a methodology of measuring dreams against reality. Still, you press on.
Let's say you study the 100 dreamers for 100 days and in that time they report 10,000 dreams. You have funding that allows you to maintain data collection and analysis for 5 years. The dreams are categorized into personal (dreams of family), professional (dreams of work life and processes) and global (dreams of outside work-home events).
Imagine that after 5 years you are able to determine that of the 10,000 dreams nearly 3500 of them came "true."
An outside scientist could probably eliminate all but four or five of them as being coincidence. Which isn't saying it couldn't be true, but there is insubstantial proof that it is actually true.
James Joyce asked the greaet pyschological question in Ulysses: "Coincidence, or intuition?" Verification bends away hard and fast from intuition, yet some of the greatest scientific theories originated out of intuitive thinking. But once again, for every Einstein there are 100,000 Hubbards and should be a matter of no small distrubance that one barely outweighs the other.
In other words, If I put a ball on a string and drop it from a wall that is twenty feet away from another wall and the wall I am on top of is 30 ft highter than the wall and the string is attached to the side of the opposite wall I can make various observations based on experiments of varying string length and wall distance and angle as to the arc and impact of the ball as well as size and composition of the ball and even color of ball for phenomenological purposes.
But....
If I have a dream that Michael Vic throws six touch down passes against the Packers and it happens, I have no way of proving that it is not coincidence. Even if I dream the exact outcome of all the other playoff games I cannot prove that I "saw" what happened, (I would be much better going off to Vegas than trying to convince an emeritus professor from Oregon that I have a third eye, dude, I mean really).
I would have to have EXTRAordinary evidence to prove that the dreams were not just an amazing coincidence. I would have to dream about something that could be proven was completely outside my knowledge and experience. For example, If I were a Quaker who had no television or access to internet and only had access to printed materials produced before 1890 I could dream that Benyamin Netanyahu's attache in Tel Aviv dies suddenly from a bleeding ulcer while eating in the 360 Cafe on the Montparnass in Paris and the cause is uranium oxide poisoning.... this would be extraordinary evidence, but not extraordinary proof unless it were a repeated behavior that could be verified.
The time will come when a means is devised for this without muddying the scientific fields with a bunch of egocentric loonies. This will happen from within science itself from all the professionals who have experienced the inexplicable and just can't hide it anymore.


...

James Joyce asked the greaet pyschological question in Ulysses: "Coincidence, or intuition?" Verification bends away hard and fast from intuition, yet some of the greatest scientific theories originated out of intuitive thinking

un-huh. Try, as e.g., the standard DNA test.

If I have a dream that Michael Vic throws six touch down passes against the Packers and it happens, I have no way of proving that it is not coincidence. Even if I dream the exact outcome of all the other playoff games I cannot prove that I "saw" what happened, (I would be much better going off to Vegas than trying to convince an emeritus professor from Oregon that I have a third eye, dude, I mean really.

Viva Las Vegas


___________________
(bad old embed code originally meant for old windows media player embeds of some sort. now removed from main posting screen.)
here's what was enclosed in old wonky embed tags:
http://www.youtube.com/watch?v=uPuKoqu6kMk
___________________



http://www.youtube.com/watch?v=uPuKoqu6kMk


...

...
hanshan
 
Posts: 1673
Joined: Fri Apr 22, 2005 5:04 pm
Blog: View Blog (0)

Previous

Return to General Discussion

Who is online

Users browsing this forum: No registered users and 170 guests