Third generation fiber-to-the-x (FTTx) will introduce wavelength division multiplexing (WDM) technology to increase data rates up to 40 Gbit/s, as well as coexist with current communication systems. WDM passive optical network (WDM-PON) based fiber-to-the-home (FTTH) system architecture together with the Next-generation passive optical network (NG- PON) can provide solution related to nowadays problems with transmission capacity. It is expected that NG-PON systems will become more cost-efficient at high per user data rates, providing different transmission speeds for end users, based on pay-as-you-grow approach. In this paper, our introduced system model is used to obtain WDM-PON system effectiveness for different data rates of up to 32 Gbit/s per downstream transmission channel. We created combined simulative-experimental WDM-PON transmission system model in environment of RSoft OptSim simulation software to compare the performance of non-return-to-zero (NRZ) on-off- keying (OOK) modulated PON transmission system versus transmission bitrate and distance. Results showed that proposed transmission system is capable to provide data transmission of up to 32 Gbit/s per channel with BER < 1e-3, if additional forward error correction (FEC) is used. Additionally, to increase the reach of investigated system, fiber Bragg grating-based dispersion compensation module (FBG-DCM) is implemented, therefore enabling full pre-compensation of chromatic dispersion.