[Freesurfer] QDEC labels vs annotations

I'm using the labels saved in QDEC to extract cortical thickness values for certain regions. For example, I've used the lh.v1.label to map onto the fsaverage brain and then to map this onto each individual subject. When I load the lh.PALS_B12_Brodmann.annot or the lh.PALS_B12_Visuotopic.annot onto the fasaverage brain, the areas that are named v1 do not quite match up with the actual lh.v1.label. The more inferior part of the lh.v1.label is considered v2 in the the annotation space. Should I be worried about this? Or can I still use the lh.v1.label to extract information for v1 (even though the annotation file would consider it v2).

Thanks!