In the work, the intensity and phase modulation methods are compared for different versions of the wavelength division multiplexing (WDM) system. Binary and multilevel code formats are evaluated numerically with OptSim simulation software at the bit rate of 10 and 40 Gbit/s, using long-fiber spans and dispersion compensation technique. The problems to be overcome at exploitation of forty Gbit/s-based multiterabit WDM systems are chromatic dispersion (CD), polarization-mode dispersion (PMD), dispersion slope, and nonlinear effects of the transmission line. We investigated the potentialities of future modulation formats for 40 Gbit/s WDM networks where most of the listed problems are avoided. It is shown that Duobinary and differential phase shift keying (DPSK) modulation formats are perfectly suitable for ultra-high spectral efficient WDM systems and possess high resistance to dispersion and nonlinear optical effects (NOE).