How tame was he before you had him? Shaking and backing away sound like fear.
You might be taking things too quickly. Spend time sitting by his cage and talking to him. Offer him treats through the bars until he is happy to take them from your fingers. It might take weeks to get this far.
I'm glad to hear you're letting him out of his cage even though he isn't hand tame, some people don't, but I think it's very important. Don't try and reach into his cage for him, this is his space. Can he get out on his own when you open the cage? You need to be able to clean it, but could you do this when he's out?
They are lovely birds, but they need a good foundation of basic groundwork. They are known not to be very cuddly birds, but should be fine stepping up (I offer mine my wrist not hand at the moment because he bites fingers!) and being touched to a certain extent and some are really cuddly.