What species of bird is he? How old is he? Did you adopt him, or is he from a store?
Many birds are wary of humans (and their hands) if they haven't been properly socialized from a young age.
I rescued a green cheek conure, Phaedrus, back in late October. He wouldn't come to anyone and would actually lunge at and bite sometimes. I had to let him come to me on his own terms, and eventually he did. Now he trusts me more and will step up when he feels like it.
I suggest finding treats that he really likes (Phae likes plain popcorn, for example) and kind of bribe him with it. Eventually he'll associate you (and your hands) with yummy food and start to become less anxious around you.
The most important thing to remember is that you have to be patient and respect Charlie's boundaries. He'll come around if you're calm and respectful and patient with him.
Talk to him a lot, and let him come out of his cage on his own. Leave the door open somewhat near you (Maybe a few feet from a couch or chair you spend a lot of time on?) and go about your business as usual. They'll sometimes get curious enough and want to come see what you're up to.
Good luck!