I believe that the FreeSurfer version is also doing the right thing. Do you have reason to believe otherwise?
thanks
doug
Donna Dierker wrote:
Mike Harms tried to explain that to me, but I was missing how the resampling affected the i term. Thanks, Tom, for spelling it out for me. ;-)
Fortunately, John Harwell has less noise in his neural network than I do, so Caret does the right thing. (Just checked.)
On 10/17/2009 10:29 AM, Doug Greve wrote:
Here's Tom Nichols explanation. He points out that you use the maximum i, not the minimum, so you don't have to get past i=1. Thanks Tom!
---------- Forwarded message ---------- Date: Fri, 16 Oct 2009 23:11:09 +0100 From: Thomas Nichols t.e.nichols@warwick.ac.uk To: Doug Greve greve@nmr.mgh.harvard.edu Cc: Donna Dierker donna@brainvis.wustl.edu Subject: Re: [Freesurfer] FDR correction (fwd)
Dear Doug,
Great discussion! But I think it misses the point. Can you forward the following to the list?
The Benjamini-Hochberg (BH) FDR procedure uses the inequality P(i) <= i / V * q for the ordered P-values P(1)<=P(2)<=...<=P(V) and desired level-q FDR control, which suggests that one will never get significance unless P(1)<q/V, just like Bonferroni. HOWEVER, the key element of BH-FDR is that you get to search over i and find the *largest* P-value that satisfies the inequality and use that P-value as a threshold.
Hence when considering changing resolutions (i.e. increasing V, but increasing resolution/smoothness and roughly keeping the information content of the images the same) the magic of BH-FDR is that if the 5-th %ile of uncorrected P-values is FDR-significant at one resolution, i.e. i' = 0.05*V satisfies P( i' / V ) <= i' / V * q then one would expect that the 5-th %ile of P-values after up-sampling would have a similar value, and thus would also satisfy the inequality even though V has changed.
That is the motivation that FDR is resilient to changes in resolution. However, I should note that in my own experience, leaving V fixed but changing smoothness, often changes the distribution of P-values dramatically, and thus changes the FDR result. But that should be a different beast that up-interpolation.
Hope this helps!
-Tom
On Fri, Oct 16, 2009 at 5:43 PM, Doug Greve greve@nmr.mgh.harvard.eduwrote:
Tom, here's an FDR question for you. It appears that the FDR correction is dependent on the number of voxels (need p < fdr/N just to get past i=1). Meaning that as N grows, the min p-value must also shrink to get past i=1. Any way to get around this?
thanks
doug
---------- Forwarded message ---------- Date: Fri, 16 Oct 2009 11:30:59 -0500 From: Donna Dierker donna@brainvis.wustl.edu To: Michael Harms mharms@conte.wustl.edu Cc: freesurfer@nmr.mgh.harvard.edu, Yulia WORBE yulia.worbe@upmc.fr Subject: Re: [Freesurfer] FDR correction
Regardless: FDR's sensitivity appears resolution-dependent to me.
On 10/16/2009 10:39 AM, Michael Harms wrote:
Interesting post Donna, but my understanding of FDR is that it sets the p-value threshold based on the LARGEST p-value that satisfies the FDR relationship.
That is, steps 3 and 4 in Genovese et al. (2002) are: 3) Let r be the largest i for which p <= i/V*q (assuming c=1) 4) Threshold the image at the p-value p(r).
So, it isn't the case that you require the most significant p-value to satisfy p <= 0.05/V "just to get past i=1" as you put it in your post.
Rather, you pick the largest p-value that satisfies the relationship, meaning that lower (more-significant) p-values may not have necessarily satisfied p <= i/V*q for their particular position in the sorted list of p-values.
cheers, Mike H.
On Fri, 2009-10-16 at 10:13 -0500, Donna Dierker wrote:
I never heard anything on my post here, but it might just be high
surface resolution:
http://www.mail-archive.com/neuro-mult-comp@brainvis.wustl.edu/msg00026.html
On 10/16/2009 09:58 AM, Michael Harms wrote:
Your FDR analysis sounds correct. You probably have a rather small
number of "marginally" significant vertices, which is why none survive FDR. You could try increasing the "q" value from say 0.05 to 0.1, in which case 10% of the surviving vertices would be expected to be false positives.
cheers, Mike H.
On Fri, 2009-10-16 at 12:03 +0200, Yulia WORBE wrote:
Dear Freesurfer team,
> We are currently doing a cortical thickness studies between a group of > psychiatric patients (n=60) and controls (n=30). We tested several > smoothing levels (15mm, 20mm, 25mm) > > When setting an uncorrected threshold (such as p<0.005), we obtained > several regions of decreased thickness, which are consistent with the > pathology. > > However, when trying to correct for multiple comparisons using FDR > ("Set Using FDR" button in qdec), the computed threshold is very high > (e.g. 4.3 for 20mm smoothing) and, obviously, no significant regions > are left. > > Did we do anything wrong in the analysis ? > > Thank you very much for your help, > Yulia > > > > _______________________________________________ > Freesurfer mailing list > Freesurfer@nmr.mgh.harvard.edu > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer > > > _______________________________________________ > > Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer