The Sin of Balaam

Here's one that I've never quite understood. Beginning in Numbers 22, the bible tells the story of the prophet Balaam and his very surprising donkey. You can read it for yourself, but from my reading, it seems to me that Balaam was faithful to do only what God told him to do and he gave God the glory for whatever he was able to discern.

That's why I don't understand how it is that poor old Balaam gets slammed so badly in the New Testament:

2Peter 2:15 "They have left the straight way and wandered off to follow the way of Balaam son of Beor, who loved the wages of wickedness."

Jude 1:11 "Woe to them! They have taken the way of Cain; they have rushed for profit into Balaam's error; they have been destroyed in Korah's rebellion."

Revelation 2:14 "Nevertheless, I have a few things against you: You have people there who hold to the teaching of Balaam, who taught Balak to entice the Israelites to sin by eating food sacrificed to idols and by committing sexual immorality."

The only problem I can see for poor old Balaam is in Numbers 22:20-22:

20 And God came to Balaam at night and said to him, “If the men come to call you, rise and go with them; but only the word which I speak to you—that you shall do.” 21 So Balaam rose in the morning, saddled his donkey, and went with the princes of Moab. 22 Then God’s anger was aroused because he went, and the Angel of the LORD took His stand in the way as an adversary against him.

What's the problem? Well, God said that IF the men come for you to go but Balaam got up and went before that happened. Where all of the other stuff mentioned in the three texts above came from, I have no idea!

If anyone reading has any idea where the charges mentioned came from, I'd like to know!