Had many landlords. If the paint is fairly okay, and as you say, you fill in any holes etc, and it's in a neutral shade, then I HIGHLY doubt he will go to the huge trouble of the small claims court.
You were told you could, and they bought the paint! *I* can tell you are truthful, and I can't even see you; a court would know too, and would throw him out. Just keep repeating what you told us.

Bastard landlords!