The authors have investigated the impact of self-phase modulation (SPM) induced distortions to optical signal transmission over proposed configuration of spectrally efficient wavelength division multiplexed (WDM) fiber-optic transmission system (FOTS). In chosen model of ultra-dense mixed-WDM system optical signals are transmitted with two different per channel bitrates (i.e., 10 and 40 Gbit/s) and three different optical signal modulation formats are used for the encoding of transmitted data (i.e., non-return to zero encoded on-off keying or NRZ-OOK, orthogonal polarization shift keying or 2-POLSK and NRZ encoded differential phase shift keying or NRZ-DPSK). In addition, “non ITU-T” defined minimum and equal frequency intervals are used for the channels separation. This research revealed that 2-POLSK modulated signals are more susceptible to the impact of SPM in such ultra-dense mixed transmission mode than e.g., NRZ-DPSK signals.