A search is presented for the production of a pair of Higgs bosons, where one decays into two photons and the other one into a bottom quark–antiquark pair. The analysis is performed using proton–proton collision data at s=13TeV recorded in 2016 by the CMS detector at the LHC, corresponding to an integrated luminosity of 35.9fb−1. The results are in agreement with standard model (SM) predictions. In a search for resonant production, upper limits are set on the cross section for new spin-0 or spin-2 particles. For the SM-like nonresonant production hypothesis, the data exclude a product of cross section and branching fraction larger than 2.0fb at 95% confidence level (CL), corresponding to about 24 times the SM prediction. Values of the effective Higgs boson self-coupling κλ are constrained to be within the range −11<κλ<17 at 95% CL, assuming all other Higgs boson couplings are at their SM value. The constraints on κλ are the most restrictive to date.