The emergence of Field Programmable System on Chip (FPSoC) technology potentially provides significant advantages for computationally intense tasks. Deep understanding of hardware and software design paradigms is necessary to create successful system’s architecture, this task can become even more challenging if software part adopts operating system with memory virtualization. When exploiting potential advantages of system’s dual nature, an important topic is chiplevel communications. In this article, authors target Altera Cyclone V SoC devices, address memory non-continuity in Linux, develop FPGA master based on-chip communications architecture and benchmark different communication scenarios – direct connection to SDRAM interface, connection via Level- 3 interconnect with and without the utilization of Accelerator Coherency Port. Maximum achieved simultaneous read/write throughput for non-cached memory is 20.08 Gbps and 11.26 Gbps for cached memory. Developed Linux modules for FPGA master control and contiguous memory allocation are provided as open source software. This work represents detailed FPGA master communication’s analysis targeting Cyclone V SoC devices and aspires to accelerate development process of Linux based FPSoC projects.