This paper aims at assessing of implementation of the concept of mixed systems in wavelength-division-multiplexing passive optical networks (WDM-PONs). At the first stage, the authors evaluate performance of two equal-channel transmission systems with non-return-to-zero (NRZ) and duobinary (DB) coding in terms of Q-factor, minimal channel spacing and supported data rate. At the second stage, two configurations of mixed transmission systems are proposed. The system with unequal channel spacing and multiple modulation formats (MMFs) as well as the system with mixed line rates (MLRs) and MMFs are investigated. The obtained results reveal that the impact of implementation of both systems is positive and concept of mixed transmission system can be considered as a potential solution for the increase of spectral efficiency (SE) of WDM-PON transmission system.