I know this problem has been discussed earlier, but I think in my case the source of the problem might be quite different. I'm running FS 5.3.0 and MATLAB R2011a on Ubuntu machine with 12 GB RAM. selxavg3-sess runs perfectly for both hemis, but I get the following error when trying to run it on subcortical volume:

$Id: fast_selxavg3.m,v 1.100.2.2 2012/11/30 18:40:38 greve Exp $
/home/marcel/freesurfer/fsfast/toolbox/fast_selxavg3.m
/home/marcel/freesurfer/fsfast/toolbox/fast_ldanaflac.m
/home/marcel/freesurfer/matlab/MRIread.m
-------------------------
outtop = /media/2TB/SDT_MRI/FS
Extension format = nii.gz
 1 DC_vs_HC.mat
 2 HI_vs_DC.mat
 3 HI_vs_HC.mat
 4 NAT_vs_PAL.mat
 5 NAT_vs_PAT.mat
 6 PAL_vs_NAT.mat
 7 PAL_vs_PAT.mat
nruns = 1
autostimdur =


outanadir = /media/2TB/SDT_MRI/FS/AC1004/bold/sdt.sm8.mni
Found 52644/537168 ( 9.8) voxels in mask
Creating Design Matrix
 ... creation time =  0.007 sec
DoMCFit = 1
ntptot = 851, nX = 19, DOF = 832
Saving X matrix to /media/2TB/SDT_MRI/FS/AC1004/bold/sdt.sm8.mni/Xtmp.mat
XCond = 31.0456 (normalized)
Computing compensation for resdual AR1 bias
 1  -0.5  -0.500489    (t=0.097569)
 2  -0.25  -0.255326    (t=0.427178)
 3  0  -0.011955    (t=0.516085)
 4  0.25  0.229273    (t=0.839369)
 5  0.5  0.466983    (t=0.928175)
AR1 Correction M: 0.0147777 1.03322
Computing contrast matrices
OLS Beta Pass
  run 1    t= 0.0
    Global Mean  1120.98
Global In-Mask Mean = 1120.98 (983.432)
Rescale Target = 100
RescaleFactor = 0.0892077
OLS Residual Pass
  run 1    t= 0.0
??? Out of memory. Type HELP MEMORY for your options.

Error in ==> fast_selxavg3 at 489
    rsserun = sum(rrun.^2);

>> ------------------------------------------