Bone is a composite material made up of inorganic and organic counterparts. Most of the inorganic counterpart accounts for calcium phosphate (CaP) whereas the major organic part is composed of collagen. The interfibrillar mineralization of collagen is an important step in the biomineralization of bone and tooth. Studies have shown that synthetic CaP undergoes auto-transformation to apatite nanocrystals before entering the gap zone of collagen. Also, the synthetic amorphous calcium phosphate/collagen combination alone is not capable of initiating apatite nucleation rapidly. Therefore, it was understood that there is the presence of a nucleation catalyst obstructing the auto-transformation of CaP before entering the collagen gap zone and initiating rapid nucleation after entering the collagen gap zone. Therefore, studies were focused on finding the nucleation catalyst responsible for the regulation of interfibrillar collagen mineralization. Organic macromolecules and low-molecular-weight carboxylic compounds are predominantly present in the bone and tooth. These organic compounds can interact with both apatite and collagen. Adsorption of the organic compounds on the apatite nanocrystal governs the nucleation, crystal growth, lattice orientation, particle size, and distribution. Additionally, they prevent the auto-transformation of CaP into apatite before entering the interfibrillar compartment of the collagen fibril. Therefore, many carboxylic organic compounds have been utilized in developing CaP. In this review, we have covered different carboxylate organic compounds governing collagen interfibrillar mineralization.