There is no sentence in the Bible that specifically says that those who die as babies go to heaven. That being said, I want to share two observations from Scripture that would clearly indicate that they do.
First, consider King David in the Old Testament. When his own baby died, David called out to God about his son, “But now he is dead; why should I fast? Can I bring him back again? I shall go to him, but he shall not return to me” (2 Samuel 12:23). David clearly says that he will reunite with his son. He seems to point to joining the child in heaven, not a tomb. David would not have been excited or found comfort about simply going to the grave. He seems to claim a higher reunion found in the presence of God.
Secondly, the Bible does not mention judgment apart from accountability. Since those who die as babies are below the age of accountability, which varies from child to child, they cannot be held responsible for their sin. Even 1 John 2:2 appears to cover those who die below the age of accountability: “and He Himself is the propitiation for our sins, and not for ours only but also for the whole world.” That truth would cover infants whose lives were cut short by death.
With those strong arguments, we can rest with the comfort of knowing that babies who die before they are able to understand and receive the good news of Christ will nonetheless be in the presence of God forever.