good, just checking!
Donna Dierker wrote:
No, Doug. I think I was the only one mis-interpreting the FDR steps, and I'm considerably closer to the Caret source than the Freesurfer source. ;-)
On 10/19/2009 11:18 AM, Douglas N Greve wrote:
I believe that the FreeSurfer version is also doing the right thing. Do you have reason to believe otherwise?
thanks
doug
Donna Dierker wrote:
Mike Harms tried to explain that to me, but I was missing how the resampling affected the i term. Thanks, Tom, for spelling it out for me. ;-)
Fortunately, John Harwell has less noise in his neural network than I do, so Caret does the right thing. (Just checked.)
On 10/17/2009 10:29 AM, Doug Greve wrote:
Here's Tom Nichols explanation. He points out that you use the maximum i, not the minimum, so you don't have to get past i=1. Thanks Tom!
---------- Forwarded message ---------- Date: Fri, 16 Oct 2009 23:11:09 +0100 From: Thomas Nichols t.e.nichols@warwick.ac.uk To: Doug Greve greve@nmr.mgh.harvard.edu Cc: Donna Dierker donna@brainvis.wustl.edu Subject: Re: [Freesurfer] FDR correction (fwd)
Dear Doug,
Great discussion! But I think it misses the point. Can you forward the following to the list?
The Benjamini-Hochberg (BH) FDR procedure uses the inequality P(i) <= i / V * q for the ordered P-values P(1)<=P(2)<=...<=P(V) and desired level-q FDR control, which suggests that one will never get significance unless P(1)<q/V, just like Bonferroni. HOWEVER, the key element of BH-FDR is that you get to search over i and find the *largest* P-value that satisfies the inequality and use that P-value as a threshold.
Hence when considering changing resolutions (i.e. increasing V, but increasing resolution/smoothness and roughly keeping the information content of the images the same) the magic of BH-FDR is that if the 5-th %ile of uncorrected P-values is FDR-significant at one resolution, i.e. i' = 0.05*V satisfies P( i' / V ) <= i' / V * q then one would expect that the 5-th %ile of P-values after up-sampling would have a similar value, and thus would also satisfy the inequality even though V has changed.
That is the motivation that FDR is resilient to changes in resolution. However, I should note that in my own experience, leaving V fixed but changing smoothness, often changes the distribution of P-values dramatically, and thus changes the FDR result. But that should be a different beast that up-interpolation.
Hope this helps!
-Tom
On Fri, Oct 16, 2009 at 5:43 PM, Doug Greve greve@nmr.mgh.harvard.eduwrote:
Tom, here's an FDR question for you. It appears that the FDR correction is dependent on the number of voxels (need p < fdr/N just to get past i=1). Meaning that as N grows, the min p-value must also shrink to get past i=1. Any way to get around this?
thanks
doug
---------- Forwarded message ---------- Date: Fri, 16 Oct 2009 11:30:59 -0500 From: Donna Dierker donna@brainvis.wustl.edu To: Michael Harms mharms@conte.wustl.edu Cc: freesurfer@nmr.mgh.harvard.edu, Yulia WORBE yulia.worbe@upmc.fr Subject: Re: [Freesurfer] FDR correction
Regardless: FDR's sensitivity appears resolution-dependent to me.
On 10/16/2009 10:39 AM, Michael Harms wrote:
Interesting post Donna, but my understanding of FDR is that it sets the p-value threshold based on the LARGEST p-value that satisfies the FDR relationship.
That is, steps 3 and 4 in Genovese et al. (2002) are: 3) Let r be the largest i for which p <= i/V*q (assuming c=1) 4) Threshold the image at the p-value p(r).
So, it isn't the case that you require the most significant p-value to satisfy p <= 0.05/V "just to get past i=1" as you put it in your post.
Rather, you pick the largest p-value that satisfies the relationship, meaning that lower (more-significant) p-values may not have necessarily satisfied p <= i/V*q for their particular position in the sorted list of p-values.
cheers, Mike H.
On Fri, 2009-10-16 at 10:13 -0500, Donna Dierker wrote:
I never heard anything on my post here, but it might just be high
> surface resolution: > > > http://www.mail-archive.com/neuro-mult-comp@brainvis.wustl.edu/msg00026.html > > > On 10/16/2009 09:58 AM, Michael Harms wrote: > > Your FDR analysis sounds correct. You probably have a rather > small > >> number of "marginally" significant vertices, which is why none >> survive >> FDR. You could try increasing the "q" value from say 0.05 to >> 0.1, in >> which case 10% of the surviving vertices would be expected to >> be false >> positives. >> >> cheers, >> Mike H. >> >> On Fri, 2009-10-16 at 12:03 +0200, Yulia WORBE wrote: >> >> >> Dear Freesurfer team, >> >>> We are currently doing a cortical thickness studies between a >>> group of >>> psychiatric patients (n=60) and controls (n=30). We tested >>> several >>> smoothing levels (15mm, 20mm, 25mm) >>> >>> When setting an uncorrected threshold (such as p<0.005), we >>> obtained >>> several regions of decreased thickness, which are consistent >>> with the >>> pathology. >>> >>> However, when trying to correct for multiple comparisons using >>> FDR >>> ("Set Using FDR" button in qdec), the computed threshold is >>> very high >>> (e.g. 4.3 for 20mm smoothing) and, obviously, no significant >>> regions >>> are left. >>> >>> Did we do anything wrong in the analysis ? >>> >>> Thank you very much for your help, >>> Yulia >>> >>>