The authors investigate the best parameters of optical filters for dense wavelength division multiplexing (DWDM) systems. The investigation is based on the OptSim 5.0 simulation software using a method for solving a complex set of differential equations taking into account the optical and electrical noise as well as linear and nonlinear effects. It is shown that the bandwidth of optical filter extends with data transmission speed, and that the Supergaussian and Raised Cosine optical band-pass filters are the most efficient. It was also revealed that an ASK-RZ 40 Gbit/s data transmission system is impossible to realize with a channel interval less than 50 GHz because of wide spectral density and lower chromatic dispersion tolerance of this modulation format.