Spin waves in Na₂Co₂TeO₆ studied by high-frequency/high-field ESR: Successes and failures of the triple-q model

The Kitaev candidate material Na₂Co₂TeO₆ is proposed to be proximate to a quantum spin liquid state but a suitable spin model and the nature of its ground states are still under debate. Our high-frequency/high-field electron spin resonance spectroscopy studies of Na₂Co₂TeO₆ single crystals under in-plane and out-of-plane magnetic fields elucidate the ground state by investigating its low-energy spin wave excitations. Several excitation modes are observed in the low-field phase and in the phases induced by B || a∗. In addition, the spectra exhibit a frequency-independent feature at the phase boundary connected to the putative quantum phase transition. For magnetic fields applied along the c axis, the observation of three distinct spin wave modes in the antiferromagnetic (AFM) ground state reveals a previously unresolved splitting of the zero-field excitation gap into ∆ = 211 GHz and ∆₂ = 237 GHz. The softening of one of these modes evidences a field-induced phase transition at B_c1 = 4.7 T, which is corroborated by a clear anomaly in the isothermal magnetization. Spin wave calculations based on the extended Heisenberg-Kitaev model exclude a zigzag ground state of the AFM phase. A triple-q spin configuration correctly predicts two spin wave modes but fails to reproduce the softening mode. Our analysis shows that the triple-q ground state model of Na₂Co₂TeO₆ is incomplete and suggests the relevance of interlayer interactions.