With the increasing demand for high-speed and bandwidth-intensive applications, the deployment of wavelength-division multiplexing passive optical network (WDM-PON) systems has gained significant traction. This work addresses the critical aspects of designing and maintaining rare-earth-doped fiber amplifiers (REDFAs) for WDM-PON transmission systems, focusing on optimizing performance, reliability, and ease of maintenance. The design phase involves meticulously exploring rare-earth-doped fibers, emphasising selecting dopants that enhance amplification efficiency and maintain signal integrity across multiple wavelengths. Experimental results and performance evaluations highlight the designed REDFA’s effectiveness in enhancing WDM-PON systems’ transmission capabilities. The findings advance the understanding of rare-earth-doped fiber amplifiers and provide practical guidelines for deploying and maintaining high-performance WDM-PON transmission systems in telecommunications networks. This research lays the foundation for more robust and efficient next-generation (NG) optical access networks capable of meeting the escalating demands of modern communication infrastructures. An experimental model of REDFA is tested in different conditions using a 16-channel non-return-to-zero (NRZ) modulated WDM-PON transmission system operated in C-band, where BER quality below 1×10 −9 is used as a threshold. Designed REDFA is realized with both co-propagation (980 nm) and counter-propagation (1480 nm) pump laser configuration to find optimal pumping laser parameters for the maximum performance of the optical communication system.