The integration of heat pumps into existing buildings is one option to exploit carbon neutral heat and cool sources. Today, this is in line with planned electrification and paths to achieve nearly zero energy buildings by using renewable energy sources. However, the implementation process is more complex compared to new buildings. Moreover, there is no comprehensive study that depicts and critically discusses the main challenges associated with integrating heat pumps into existing buildings from a very practical point of view. This article presents the main technical and economic challenges related to the application of heat pumps in existing building stock based on critical literature review and discussion. Among others, especially, procedures involving the reduction of the supply temperature of the heating medium, the selection of the bivalent point, heat storage, and buffer tanks of heat pumps were discussed. Several best practices for the design and control of the heat pumps in existing buildings along with a discussion of benefits and limitations (technical, economic, operational) of the analyzed systems are provided. Key findings are: (i) it was indicated that the heat pumps in existing buildings can be applied as a heat source for the entire building, part or single room and for different purposes (space heating and/or DHW and/or cooling), (ii) minimisation of the supply temperature is always desirable, with the limit for DHW being 58 °C, and ultimately the control should be adapted to the building or replaced by forecast-based control, (iii) coverage of heat demand by heat pumps should reach 70–90 % of peak demand at nominal outdoor air conditions and be determined on the basis of the analysis of initial and running costs, (iv) from the point of view of reducing the demand for primary energy self-consumption of electricity from PV should be maximised. Finally, future research and development directions are suggested.