The exponential growth of Internet data traffic and progress of Information and Communication Technology (ICT) sector pushes hard the telecommunication infrastructure for upgrading the transmission data rate. Wavelength division multiplexed passive optical networks (WDM-PONs) can be the next generation solution for nowadays problems which are related to transmission capacity. Next-generation WDM-PON systems based on mixed wavelength transmitters are expected to become more cost-efficient at high per user data rates, e.g. over 10 Gbit/s per channel. Important advantage of this technology is to set various channel spacing and use different modulation formats to increase spectral efficiency in the same time and provide different transmission speeds for end user, based on pay-as-you-grow approach. Therefore, several modulation formats like non-return to zero (NRZ) also called 2-level pulse-amplitude modulation (PAM-2), four level PAM or PAM-4 and Duobinary (DB) are investigated to understand their limitations, advantages and disadvantages to be further used in next generation PON systems to increase its capacity and spectral efficiency.