My Community

I set ISYM=2 to constrain the ionic relaxation of bulk system according to the symmetry group, but the resulted structure has different space group from the initial geometry, indicating the relaxation process do not be constrained by the symmetry. How can I relax the internal coordinates of atoms of an metastable (instable) structure to get its accurate energy?

An _instable_ structure has per definition _no_ groundstate ...
vasp will therefore search for a stable groundstate. If you additionally set ISIF = 3 then the spacegroup may change ...

I agree with Veronika: as soon as you relax (the electronic structure, thegeometry) the system relaxes towards the minimum (which may be a local one, of course)
But even for ISIF=3 the symmetry of the space group of the lattice is not lowered. The symmetry operations are checked at the beginning (according to the symmetry of your input given in POSCAR) and kept fixed all over the run. (eg. hexagonal remains hexagonal, though V and c/a may change).
Therefore I am a little puzzled about that question

I did set opposite initial magnetic moments to symmetry equivalent atoms in primary cell to account for antiferromagnetic. Does this kind of setting of inital magnetic moments makes the VASP fail to determine the right symmetry of the structure?

If it is this case, do I need to double the primary cell and set opposite inital moments to each primary cell respectly to account for the antiferromagnet?

If you set different magnetic moments then of course the symmetry is lowered. In this case vasp does not "fail" but detects the lower symmetry (due to the magnetic moments) correctly.