The increasing integration of renewable energy sources (RES), particularly inverter-based resources (IBRs), is transforming the behavior of power systems and exposing the limitations of conventional protection schemes. Traditional relays—such as overcurrent, distance, and directional relays—often fail to operate correctly due to low fault current levels, distorted waveforms, and bidirectional power flows typical of high-RES networks. This paper reviews the major challenges of protection under high RES penetration, presents real-world case studies of misoperation, and analyzes the impact of IBRs on relay performance. The study also surveys recent advances in artificial intelligence (AI)-based approaches for fault classification, adaptive autoreclosing, and protection setting coordination. A hybrid protection framework is proposed, combining deterministic logic with AI models and system-awareness mechanisms. Key implementation challenges—including generalization, explainability, data quality, cybersecurity, and regulatory gaps—are discussed. The paper concludes by outlining research directions to enable reliable and adaptive AI-assisted protection for next-generation grids.