Considering the growing demand for broadband of access networks, in this paper we are investigating various modulation formats as a way of increasing the performance of optical transmission systems. Non-return-to-zero (NRZ) on-off keying, return-to zero (RZ) OOK, carrier suppressed RZ (CSRZ) OOK, duobinary (DB), NRZ differential phase shift keying (NRZ-DPSK), RZ-DPSK and CSRZ-DPSK formats are compared using the maximal achievable reach with bit error rate less than 10-9 as a criterion. Simulations are performed by using OptSim software tool. It is shown that using transmission system without dispersion compensation the best result are shown by Duobinary and CSRZ-OOK modulation formats, but with the system using dispersion compensating fiber (DCF) the longest transmission distance is achieved by RZ-DPSK modulation format. By investigating the influence of channel spacing for best-performed modulation formats, network reach decrease for transmission systems with DCF fiber has been observed due to channel crosstalk.