The paper analyses the solutions of increasing the spectral efficiency (SE) of multiple line rate (MLR) wavelength-division-multiplexing passive optical network (WDM-PON). Paper analyses the increasing the SE of WDM-PON by applying of mixed transmission system with MLR. The MLR approach allows implementing gradual transition to higher data rates within the limited expenses. The main goal of the paper is to establish the most spectral-effective solution for applying the eight channels with 40 Gbit/s line rate in sixteen channel 10 Gbit/s transmission system. Duobinary (DB) modulation format is used for channels with 40 Gbit/s line rate, while the non-return-to-zero (NRZ) is used for 10 Gbit/s channels. For the increasing of SE we apply unequal channel spacing between different channels. The paper concentrates on the determination of minimal channel spacing realizable by two 10 Gbit/s channels, as well as between two 40 Gbit/s channels and between 10 Gbit/s and 40 Gbit/s channels. For ensuring maximal possible SE, different channel topologies are considered to ensure the most effective optimization of channel spacing. The paper also evaluates the main limiting factors of SE and system performance, analysing the impact of chromatic dispersion and non-linear effects.