Migration paths towards 5G require to increase network capacity and raise typical end-user data rates from 10 to 100 times. Wavelength division multiplexed passive optical network (WDM-PON) based third-generation fiber-to-the-x (FTTx) system architectures will be capable to provide solutions related to nowadays network problems with transmission capacity. Therefore, in our research paper we propose analog radio-over-fiber (A-RoF) system architecture with three hybrid photonic mm-wave generation techniques in WDM-PON network. In our developed model of A-RoF WDM-PON architecture, the baseband units are located on transmitter side in central office, by transporting intermediate frequencies along existing fiber optical distribution network and effectively performing signal up-conversion at the remote radio unit (RRU) located in receiver side. Investigated A-RoF WDM-PON transmission system model, created in the environment of OptSim simulation software, shows possible system types of A-RoF implementation for optimized generation of mm-wave signals that can be further used for data transmission between multiple-input and multiple-output (MIMO) antennas. The feasibility of A-RoF WDM-PON system is verified by the complexity of each mm-wave generation in optical transmitter side and the obtained quality of received signal.