In this paper the electromagnetic waves generated in a circular geometry free electron laser are investigated. In this model, electromagnetic modes are excited as a result of the interaction of a circulating electron beam in a coaxial waveguide with a wiggler magnetic field. The dispersion equations for the transverse magnetic modes are derived. The analysis of the content of the dispersion equation indicates that the instabilities grow at frequencies corresponding to the crossing points of the waveguide modes and the electron beam modes.