Hi all,
I have a longitudinal dataset that I'd like to calculate test-retest reliability for. However, I am not sure whether to calculate this for the cross-sectionally processed data or longitudinal data? It would seem to make sense to use the cross-sectional data, as the time points are independent, but then it means that the test-retest results would not be applicable in a dataset processed longitudinally. On the other hand, calculating reliability metrics for data processed in the exact same way as will be used in further studies seems to make sense also, but would calculating test-retest on the longitudinally processed data bias the results, as the data points are not fully independent?
Does anyone have any idea of the best way forward?
Thanks,
A
Hi Amanda
it really depends what your goal is. If you want to estimate the noise for a power analysis and you are going to process the data longitudinally, then certainly you should use the longitudinal stream to measure the reliability
cheers Bruce
On Wed, 20 Sep 2017, Worker, Amanda wrote:
Hi all,
I have a longitudinal dataset that I'd like to calculate test-retest reliability for. However, I am not sure whether to calculate this for the cross-sectionally processed data or longitudinal data? It would seem to make sense to use the cross-sectional data, as the time points are independent, but then it means that the test-retest results would not be applicable in a dataset processed longitudinally. On the other hand, calculating reliability metrics for data processed in the exact same way as will be used in further studies seems to make sense also, but would calculating test-retest on the longitudinally processed data bias the results, as the data points are not fully independent?
Does anyone have any idea of the best way forward?
Thanks,
A
Hi Amanda,
I would recommend you to quantify reliability using the same processing pipeline you will use to test for longitudinal effects in your sample. Relevant literature include those listed below. For reliability you would want to evaluate subjects over a time period in which you do not expect the main effects of your study (e.g., disease related or learning related morphometric effects), but rather standard physiological and MRI system variability. Ideally this group also has the same age/gender representation of the group in which you later want to test for longitudinal effects.
Cheers,
Jorge
https://www.ncbi.nlm.nih.gov/pubmed/22430496
https://www.ncbi.nlm.nih.gov/pubmed/23668971
On 20/09/2017 15:00, Worker, Amanda wrote:
Hi all,
I have a longitudinal dataset that I'd like to calculate test-retest reliability for. However, I am not sure whether to calculate this for the cross-sectionally processed data or longitudinal data? It would seem to make sense to use the cross-sectional data, as the time points are independent, but then it means that the test-retest results would not be applicable in a dataset processed longitudinally. On the other hand, calculating reliability metrics for data processed in the exact same way as will be used in further studies seems to make sense also, but would calculating test-retest on the longitudinally processed data bias the results, as the data points are not fully independent?
Does anyone have any idea of the best way forward?
Thanks,
A
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
Hi,
Thank you very much for your fast response. I had thought that it would make sense to use the exact same pipeline, as I would be using in a longitudinal study in future. If I could just be a bit more specific about the details, as I have a publication under review at the moment and one of the reviewers isn't happy about this approach. I just wanted to check that we haven't misunderstood how the algorithm works if that is ok?
Reviewer's comment
“It is unclear to me what is the value of establishing test-retest reliability of applying FreeSurfer subfield segmentation to a within-subject template constructed by the Reuter algorithm. Once this template is constructed and mapped to the different time point images, test-retest measures become meaningless because the segmentations are not independent, but instead, highly dependent on each other.”
Our response
"our understanding is that the use of a within-subject’s template to initialise the segmentation will minimise potential bias or inaccuracy in segmentation in multi-session data. The segmentations are themselves run independently, but with the template being used to initialise (provide a starting point) for the nonlinear parameter fitting. We don’t believe that this would be expected to, and does not appear to, constrain the segmentation to the degree that it would produce “identical” values, as the reviewer suggested in a previous review. The alternative is to risk a biased segmentation which could potentially give a false impression of poor reliability and increased risk of false positive results. "
Any thoughts appreciated.
Thanks,
Amanda
________________________________ From: Jovicich, Jorge jorge.jovicich@unitn.it Sent: 20 September 2017 15:51:40 To: Worker, Amanda Cc: Freesurfer support list Subject: Re: [Freesurfer] Test-retest reliability in longitudinal data
Hi Amanda,
I would recommend you to quantify reliability using the same processing pipeline you will use to test for longitudinal effects in your sample. Relevant literature include those listed below. For reliability you would want to evaluate subjects over a time period in which you do not expect the main effects of your study (e.g., disease related or learning related morphometric effects), but rather standard physiological and MRI system variability. Ideally this group also has the same age/gender representation of the group in which you later want to test for longitudinal effects.
Cheers,
Jorge
https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ncbi....
https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ncbi....
On 20/09/2017 15:00, Worker, Amanda wrote:
Hi all,
I have a longitudinal dataset that I'd like to calculate test-retest reliability for. However, I am not sure whether to calculate this for the cross-sectionally processed data or longitudinal data? It would seem to make sense to use the cross-sectional data, as the time points are independent, but then it means that the test-retest results would not be applicable in a dataset processed longitudinally. On the other hand, calculating reliability metrics for data processed in the exact same way as will be used in further studies seems to make sense also, but would calculating test-retest on the longitudinally processed data bias the results, as the data points are not fully independent?
Does anyone have any idea of the best way forward?
Thanks,
A
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.nmr....
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.partne... . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
-- Jorge Jovicich, Ph.D.
MR Lab Head Center for Mind/Brain Sciences University of Trento, Via delle Regole, 101 38100 Mattarello (TN) Italy
Telephone: +39-0461-28 3064 Fax: +39-0461-28-3066 Email: jorge.jovicich@unitn.it MRI Methods Group: https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fr.unitn.it... CIMEC: https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.cimec....
I'll defer to Martin or Doug on this except to note that this objection applies to *all* longitudinal analyses. I think the response is that certainly longitudinal analysis improves reliability - that is the whole point. The question is does it to so by sacrificing sensitivity. You could for example call every voxel hippocampus in every subject and have perfect reliability.
On Wed, 4 Oct 2017, Worker, Amanda wrote:
Hi,
Thank you very much for your fast response. I had thought that it would make sense to use the exact same pipeline, as I would be using in a longitudinal study in future. If I could just be a bit more specific about the details, as I have a publication under review at the moment and one of the reviewers isn't happy about this approach. I just wanted to check that we haven't misunderstood how the algorithm works if that is ok?
Reviewer's comment
“It is unclear to me what is the value of establishing test-retest reliability of applying FreeSurfer subfield segmentation to a within-subject template constructed by the Reuter algorithm. Once this template is constructed and mapped to the different time point images, test-retest measures become meaningless because the segmentations are not independent, but instead, highly dependent on each other.”
Our response
"our understanding is that the use of a within-subject’s template to initialise the segmentation will minimise potential bias or inaccuracy in segmentation in multi-session data. The segmentations are themselves run independently, but with the template being used to initialise (provide a starting point) for the nonlinear parameter fitting. We don’t believe that this would be expected to, and does not appear to, constrain the segmentation to the degree that it would produce “identical” values, as the reviewer suggested in a previous review. The alternative is to risk a biased segmentation which could potentially give a false impression of poor reliability and increased risk of false positive results. "
Any thoughts appreciated.
Thanks,
Amanda
From: Jovicich, Jorge jorge.jovicich@unitn.it Sent: 20 September 2017 15:51:40 To: Worker, Amanda Cc: Freesurfer support list Subject: Re: [Freesurfer] Test-retest reliability in longitudinal data Hi Amanda,
I would recommend you to quantify reliability using the same processing pipeline you will use to test for longitudinal effects in your sample. Relevant literature include those listed below. For reliability you would want to evaluate subjects over a time period in which you do not expect the main effects of your study (e.g., disease related or learning related morphometric effects), but rather standard physiological and MRI system variability. Ideally this group also has the same age/gender representation of the group in which you later want to test for longitudinal effects.
Cheers,
Jorge
https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ncbi.... 22430496&data=01%7C01%7Camanda.worker%40kcl.ac.uk%7Cb536198d4e48444c920908d5003722cd%7C8370cf1416f3 4c16b83c724071654356%7C0&sdata=5Zna%2FnZpaN62IpZVdWf%2FUE7MTSEjA0ygXQCUIIQP%2BuA%3D&reserved=0
https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ncbi.... 23668971&data=01%7C01%7Camanda.worker%40kcl.ac.uk%7Cb536198d4e48444c920908d5003722cd%7C8370cf1416f3 4c16b83c724071654356%7C0&sdata=yQaldU8xwFyenqpdPlZ%2FLDVu2aH7zpeWPVxfd5YqccQ%3D&reserved=0
On 20/09/2017 15:00, Worker, Amanda wrote:
Hi all,
I have a longitudinal dataset that I'd like to calculate test-retest reliability for. However, I am not sure whether to calculate this for the cross-sectionally processed data or longitudinal data? It would seem to make sense to use the cross-sectional data, as the time points are independent, but then it means that the test-retest results would not be applicable in a dataset processed longitudinally. On the other hand, calculating reliability metrics for data processed in the exact same way as will be used in further studies seems to make sense also, but would calculating test-retest on the longitudinally processed data bias the results, as the data points are not fully independent?
Does anyone have any idea of the best way forward?
Thanks,
A
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.nmr....
an%2Flistinfo%2Ffreesurfer&data=01%7C01%7Camanda.worker%40kcl.ac.uk%7Cb536198d4e48444c920908d500372 2cd%7C8370cf1416f34c16b83c724071654356%7C0&sdata=KvaIAsox9nAxI1YQATw%2FAPoTe71b1QploiDCPR158Ls%3D&r eserved=0
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.partne...
&data=01%7C01%7Camanda.worker%40kcl.ac.uk%7Cb536198d4e48444c920908d5003722cd%7C8370cf1416f34c16b83c 724071654356%7C0&sdata=JoPr2povR4H4TMt2ltd1mYQ8HU3Nx7nExVknM9InqAI%3D&reserved=0 . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly dispose of the e-mail.
-- Jorge Jovicich, Ph.D.
MR Lab Head Center for Mind/Brain Sciences University of Trento, Via delle Regole, 101 38100 Mattarello (TN) Italy
Telephone: +39-0461-28 3064 Fax: +39-0461-28-3066 Email: jorge.jovicich@unitn.it MRI Methods Group:https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fr.unitn.it... a=01%7C01%7Camanda.worker%40kcl.ac.uk%7Cb536198d4e48444c920908d5003722cd%7C8370cf1416f34c16b83c7240 71654356%7C0&sdata=kRpiN3LA5iPcdnDVWjgFIL1aM49MYOXcskjcwr6bNmo%3D&reserved=0 CIMEC:https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.cimec.... 1%7Camanda.worker%40kcl.ac.uk%7Cb536198d4e48444c920908d5003722cd%7C8370cf1416f34c16b83c724071654356 %7C0&sdata=C1lda%2B9ZyDfcuXFDs3DarFdJxreZbfCsGZRe8ki5CTM%3D&reserved=0
Hi Amanda, I’m with Jorge: you should test the same method that you’ll use later on to analyze your longitudinal data. Having said that: the reviewer mentions that you’re applying the longitudinal segmentation to the base subject? What are you exactly trying to do? Cheers, Eugenio
-- Juan Eugenio Iglesias ERC Senior Research Fellow Translational Imaging Group University College London http://www.jeiglesias.com http://cmictig.cs.ucl.ac.uk/
From: freesurfer-bounces@nmr.mgh.harvard.edu on behalf of "Worker, Amanda" amanda.worker@kcl.ac.uk Reply-To: Freesurfer support list freesurfer@nmr.mgh.harvard.edu Date: Wednesday, 4 October 2017 at 13:05 To: "Jovicich, Jorge" jorge.jovicich@unitn.it Cc: Freesurfer support list freesurfer@nmr.mgh.harvard.edu Subject: Re: [Freesurfer] Test-retest reliability in longitudinal data
Hi,
Thank you very much for your fast response. I had thought that it would make sense to use the exact same pipeline, as I would be using in a longitudinal study in future. If I could just be a bit more specific about the details, as I have a publication under review at the moment and one of the reviewers isn't happy about this approach. I just wanted to check that we haven't misunderstood how the algorithm works if that is ok?
Reviewer's comment
“It is unclear to me what is the value of establishing test-retest reliability of applying FreeSurfer subfield segmentation to a within-subject template constructed by the Reuter algorithm. Once this template is constructed and mapped to the different time point images, test-retest measures become meaningless because the segmentations are not independent, but instead, highly dependent on each other.”
Our response
"our understanding is that the use of a within-subject’s template to initialise the segmentation will minimise potential bias or inaccuracy in segmentation in multi-session data. The segmentations are themselves run independently, but with the template being used to initialise (provide a starting point) for the nonlinear parameter fitting. We don’t believe that this would be expected to, and does not appear to, constrain the segmentation to the degree that it would produce “identical” values, as the reviewer suggested in a previous review. The alternative is to risk a biased segmentation which could potentially give a false impression of poor reliability and increased risk of false positive results. "
Any thoughts appreciated.
Thanks,
Amanda
________________________________ From: Jovicich, Jorge jorge.jovicich@unitn.it Sent: 20 September 2017 15:51:40 To: Worker, Amanda Cc: Freesurfer support list Subject: Re: [Freesurfer] Test-retest reliability in longitudinal data
Hi Amanda,
I would recommend you to quantify reliability using the same processing pipeline you will use to test for longitudinal effects in your sample. Relevant literature include those listed below. For reliability you would want to evaluate subjects over a time period in which you do not expect the main effects of your study (e.g., disease related or learning related morphometric effects), but rather standard physiological and MRI system variability. Ideally this group also has the same age/gender representation of the group in which you later want to test for longitudinal effects.
Cheers,
Jorge
https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ncbi....
https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ncbi....
On 20/09/2017 15:00, Worker, Amanda wrote:
Hi all,
I have a longitudinal dataset that I'd like to calculate test-retest reliability for. However, I am not sure whether to calculate this for the cross-sectionally processed data or longitudinal data? It would seem to make sense to use the cross-sectional data, as the time points are independent, but then it means that the test-retest results would not be applicable in a dataset processed longitudinally. On the other hand, calculating reliability metrics for data processed in the exact same way as will be used in further studies seems to make sense also, but would calculating test-retest on the longitudinally processed data bias the results, as the data points are not fully independent?
Does anyone have any idea of the best way forward?
Thanks,
A
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.nmr....
The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.partne... . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
-- Jorge Jovicich, Ph.D.
MR Lab Head Center for Mind/Brain Sciences University of Trento, Via delle Regole, 101 38100 Mattarello (TN) Italy
Telephone: +39-0461-28 3064 Fax: +39-0461-28-3066 Email: jorge.jovicich@unitn.it MRI Methods Group: https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fr.unitn.it... CIMEC: https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.cimec....
Hi Amanda,
if I understand correctly the specific question of the reviewer, a potential response could be, almost quoting Reuter et al., 2012:
"The variability of the test-retest segmentations typically arises from three main sources: variability in the MRI acquisition (head position, head motion, scanner instabilities, etc.), anatomical variations in the subject such as hydration state, and variability intrinsic to the automated segmentation algorithms. The longitudinal Freesurfer analysis stream aims at minimizing this last factor (Reuter et al., 2012). Therefore, the test-retest segmentation variability when using the longitudinal stream represents the experimental error dominated by MRI acquisition and subject state factors."
Best,
Jorge
On 04/10/2017 14:04, Worker, Amanda wrote:
Hi,
Thank you very much for your fast response. I had thought that it would make sense to use the exact same pipeline, as I would be using in a longitudinal study in future. If I could just be a bit more specific about the details, as I have a publication under review at the moment and one of the reviewers isn't happy about this approach. I just wanted to check that we haven't misunderstood how the algorithm works if that is ok?
*Reviewer's comment*
“It is unclear to me what is the value of establishing test-retest reliability of applying FreeSurfer subfield segmentation to a within-subject template constructed by the Reuter algorithm. Once this template is constructed and mapped to the different time point images, test-retest measures become meaningless because the segmentations are not independent, but instead, highly dependent on each other.”
*Our response*
"our understanding is that the use of a within-subject’s template to initialise the segmentation will minimise potential bias or inaccuracy in segmentation in multi-session data. The segmentations are themselves run independently, but with the template being used to initialise (provide a starting point) for the nonlinear parameter fitting. We don’t believe that this would be expected to, and does not appear to, constrain the segmentation to the degree that it would produce “identical” values, as the reviewer suggested in a previous review. The alternative is to risk a biased segmentation which could potentially give a false impression of poor reliability and increased risk of false positive results. "
Any thoughts appreciated.
Thanks,
Amanda
*From:* Jovicich, Jorge jorge.jovicich@unitn.it *Sent:* 20 September 2017 15:51:40 *To:* Worker, Amanda *Cc:* Freesurfer support list *Subject:* Re: [Freesurfer] Test-retest reliability in longitudinal data Hi Amanda,
I would recommend you to quantify reliability using the same processing pipeline you will use to test for longitudinal effects in your sample. Relevant literature include those listed below. For reliability you would want to evaluate subjects over a time period in which you do not expect the main effects of your study (e.g., disease related or learning related morphometric effects), but rather standard physiological and MRI system variability. Ideally this group also has the same age/gender representation of the group in which you later want to test for longitudinal effects.
Cheers,
Jorge
https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ncbi....
https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ncbi....
On 20/09/2017 15:00, Worker, Amanda wrote:
Hi all,
I have a longitudinal dataset that I'd like to calculate test-retest reliability for. However, I am not sure whether to calculate this for the cross-sectionally processed data or longitudinal data? It would seem to make sense to use the cross-sectional data, as the time points are independent, but then it means that the test-retest results would not be applicable in a dataset processed longitudinally. On the other hand, calculating reliability metrics for data processed in the exact same way as will be used in further studies seems to make sense also, but would calculating test-retest on the longitudinally processed data bias the results, as the data points are not fully independent?
Does anyone have any idea of the best way forward?
Thanks,
A
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu
https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.nmr....
The information in this e-mail is intended only for the person to
whom it is
addressed. If you believe this e-mail was sent to you in error and
the e-mail
contains patient information, please contact the Partners Compliance
HelpLine at
https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.partne... . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender
and properly
dispose of the e-mail.
-- Jorge Jovicich, Ph.D.
MR Lab Head Center for Mind/Brain Sciences University of Trento, Via delle Regole, 101 38100 Mattarello (TN) Italy
Telephone: +39-0461-28 3064 Fax: +39-0461-28-3066 Email: jorge.jovicich@unitn.it MRI Methods Group: https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fr.unitn.it... CIMEC: https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.cimec....
freesurfer@nmr.mgh.harvard.edu
-
Bruce Fischl -
Iglesias Gonzalez, Eugenio -
Jovicich, Jorge -
Worker, Amanda