According to the CISCO Visual Networking Index (VNI) forecast by 2019 annual global IP traffic will pass a new milestone figure of 2.0 zettabytes per year, or 168.0 exabytes per month. Thus service providers under the strong continued pressure in terms of rapidly growing bandwidth demand can expect new challenges in providing higher data transmission speeds and better bandwidth scalability. Wavelength division multiplexing passive optical network (WDM-PON) solution allows operators to enjoy the benefits of a scalable, low cost, passive and centralized optical architecture today, while extending optical transparency from the core to the access network. The upgrade of fiber optical telecommunication systems to higher bit rates very often requires solving the impact of nonlinear optical effects (NOE) and chromatic dispersion (CD). In this paper we have investigated the performance of different modulation formats in dispersion compensated WDM-PON system. In our research two dispersion compensation modules (DCM), namely, fiber Bragg grating (FBG) DCM and dispersion compensated fiber (DCF) DCM were integrated into an optical transmission link and their performance were observed in terms of BER. Besides the standard modulation schemes like non non-return-to-zero (NRZ) and return-to-zero (RZ), we have also compared the performance of chirped return-to-zero (CRZ), carrier-suppressed return-to-zero (CSRZ) and duobinary (DB) modulation formats in our dispersion compensated optical transmission system. The two most common intensity modulation formats used in optical transmission systems are non-return-to-zero (NRZ) and return-to-zero (RZ). Till lately NRZ format has been widely implemented, mainly because of its signal bandwidth and the simplest configuration and setting of transmitter. But this does not mean that NRZ is the best choice for dense WDM systems, while it doesn’t have the highest tolerance against the chromatic dispersion or cross-phase modulation (XPM). RZ modulation is more suitable for nonlinear regime, but it is less preferable to use for higher data rates above 10 Gbit/s due to its’ wider bandwidth of optical spectrum if compared to NRZ. CRZ modulation falls into RZ modulation, but its’ advantage lies in better tolerance to nonlinear effects than RZ or NRZ modulation formats. Duobinary modulation has also stimulated much interest as a practical alternative for long haul transmission systems alongside the differential phase-shift keying (DPSK) technique, since it has high tolerance to dispersion and is also reasonably simple to implement. In general, the purpose of this research paper is to determine the best performing scheme for a 16-channel dense (channel spacing is 50 GHz) WDM-PON transmission system with different modulation formats, where electro-absorption modulator (EAM) and different dispersion compensation schemes (DCF and FBG) are used.