External Email - Use Caution
Hello FreeSurfer Developers,
We are attempting to use recon-all -all on T1w MRI image analysis. We tested it on AWS EC2 in a docker container successfully. The uname -a output the EC2 is Linux ip-10-193-42-20 5.11.0-1019-aws #20~20.04.1-Ubuntu SMP Tue Sep 21 10:40:39 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux. The docker file entrypoint is a job_script.py python code to call a subprocess of bash script, like recon-all using a python library called subprocess.
When we used the same docker image with AWS ECS or batch processing, from the files created inside the processed subject's mri folder,* the recon-all stuck at "-fill" step because the input files (wm.mgz, aseg.auto_noCCseg.mgz, and transforms/talairach.lta) are created, but NO filled.mgz was created as the output file of "-fill " step in recon-all -autorecon2. *
I have searched the list and no similar errors have been reported. Does anyone have any thoughts on how to troubleshoot this? Also I have attached the reon-all.log in cas it is of any use.
1). FreeSurfer version: 7.2.0 2). Platform: AWS EC2 batch or ECS Fargate
Looking forward to your feedback.
Best Regards Xin
External Email - Use Caution
Hi Xin,
If I understand correctly, recon-all runs successfully in your docker container when testing it on an EC2 instance, but when trying to run the same container on AWS batch or fargate, it doesn't make it past the `-fill` step. Is that correct?
If so, the first thing I'd like to confirm is that the container has enough resources (time and memory) when running on AWS batch or fargate. The recon-all.log you attached shows there is only 8.9Gb of free space available (line 31). Can you try increasing that so there is 32Gb of space available? Can you also try doubling the JobTimeout duration?
Thanks,
-Paul
On Sat, Nov 6, 2021 at 12:10 PM Xin Qi xinqi2000@gmail.com wrote:
External Email - Use CautionHello FreeSurfer Developers,
We are attempting to use recon-all -all on T1w MRI image analysis. We tested it on AWS EC2 in a docker container successfully. The uname -a output the EC2 is Linux ip-10-193-42-20 5.11.0-1019-aws #20~20.04.1-Ubuntu SMP Tue Sep 21 10:40:39 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux. The docker file entrypoint is a job_script.py python code to call a subprocess of bash script, like recon-all using a python library called subprocess.
When we used the same docker image with AWS ECS or batch processing, from the files created inside the processed subject's mri folder,* the recon-all stuck at "-fill" step because the input files (wm.mgz, aseg.auto_noCCseg.mgz, and transforms/talairach.lta) are created, but NO filled.mgz was created as the output file of "-fill " step in recon-all -autorecon2. *
I have searched the list and no similar errors have been reported. Does anyone have any thoughts on how to troubleshoot this? Also I have attached the reon-all.log in cas it is of any use.
1). FreeSurfer version: 7.2.0 2). Platform: AWS EC2 batch or ECS Fargate
Looking forward to your feedback.
Best Regards Xin
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://secure-web.cisco.com/1SqeFzz1UKNQJUVqjoOsy-gI1ZvHNCGQk1wsyHVm3QjTWt1...
External Email - Use Caution
Dear Paul:
Thank you so much for your quick response and we will test on your suggestion!
Best regards Xin
On Mon, Nov 8, 2021 at 1:15 PM Paul Wighton paul@corticometrics.com wrote:
External Email - Use CautionHi Xin,
If I understand correctly, recon-all runs successfully in your docker container when testing it on an EC2 instance, but when trying to run the same container on AWS batch or fargate, it doesn't make it past the `-fill` step. Is that correct?
If so, the first thing I'd like to confirm is that the container has enough resources (time and memory) when running on AWS batch or fargate. The recon-all.log you attached shows there is only 8.9Gb of free space available (line 31). Can you try increasing that so there is 32Gb of space available? Can you also try doubling the JobTimeout duration?
Thanks,
-Paul
On Sat, Nov 6, 2021 at 12:10 PM Xin Qi xinqi2000@gmail.com wrote:
External Email - Use CautionHello FreeSurfer Developers,
We are attempting to use recon-all -all on T1w MRI image analysis. We tested it on AWS EC2 in a docker container successfully. The uname -a output the EC2 is Linux ip-10-193-42-20 5.11.0-1019-aws #20~20.04.1-Ubuntu SMP Tue Sep 21 10:40:39 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux. The docker file entrypoint is a job_script.py python code to call a subprocess of bash script, like recon-all using a python library called subprocess.
When we used the same docker image with AWS ECS or batch processing, from the files created inside the processed subject's mri folder,* the recon-all stuck at "-fill" step because the input files (wm.mgz, aseg.auto_noCCseg.mgz, and transforms/talairach.lta) are created, but NO filled.mgz was created as the output file of "-fill " step in recon-all -autorecon2. *
I have searched the list and no similar errors have been reported. Does anyone have any thoughts on how to troubleshoot this? Also I have attached the reon-all.log in cas it is of any use.
1). FreeSurfer version: 7.2.0 2). Platform: AWS EC2 batch or ECS Fargate
Looking forward to your feedback.
Best Regards Xin
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu *MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be* https://secure-web.cisco.com/1Td8FQ5Z5s9j1VyD3HF_Zm2_9rEUvwUfVbzDZoHpDJhG8Hj... https://secure-web.cisco.com/1SqeFzz1UKNQJUVqjoOsy-gI1ZvHNCGQk1wsyHVm3QjTWt1Bc4CrowVsrYwVcctL1Qeb_PlEU4reThHIvkNIwhX9sXbuGiXf5tkq6A0r07-_u7VLDOblNFU-953AL9byAdBrC20a6Xdo_s1NGPi0SJNv7mFmSOoBI_0u9_WigG70m1Bt04CJFIDKaz14KJVOycTzAJ0CnHfoSTPfgMpYgTaiKM41dA0pJDHVg8n25j4NicKi422VXWVCYqjwCUGGPfBLOfgFBWH-GEWR3vEFyEw/https%3A%2F%2Fmail.nmr.mgh.harvard.edu%2Fmailman%2Flistinfo%2Ffreesurfer
Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://secure-web.cisco.com/1Td8FQ5Z5s9j1VyD3HF_Zm2_9rEUvwUfVbzDZoHpDJhG8Hj...
freesurfer@nmr.mgh.harvard.edu
-
Paul Wighton -
Xin Qi