The invention relates to the field of telecommunications, more specifically to the sector of the optical transmission systems using signal amplification. A measuring system and a method are invented for measuring nonlinearity coefficient in the optical fibers using four-wave mixing initiation in the fiber being tested. The measuring system comprises two lasers for tunable continuous wavelength radiation, one polarization controller per each laser output and one common polarizer selectively letting through only one certain state of polarization. The measuring method realized by the invented system ensures that only matching by the state of polarization optical signals enter the fiber being tested increasing four-wave mixing initiation efficiency in the fiber and improving accuracy of measurements.