When I bought Alex home he was still formula dependant. He's spent most of the last 3 months living on my shoulder or hiding in my jacket where he feels safe. Because, at that age he would still be piled up safe in the nest with his siblings and parents. Since he had neither of those at his new place (he had his sister at the breeders) I had to become the substitute. Otherwise he would act like your bird, frantic to get out to be near you.
And now that he's older and almost weaned and confident in his abilities to fly and climb and feed himself, he loves to explore and do his own thing. He's happy and content now to be left alone in his cage when I have to go to uni. He's happy to be loaded into a travel cage and go on holidays with me. Most importantly he is happy, healthy and well adjusted with no behaviour problems. And I'd be willing to bet that anxiety when so young could cause issues later.
Like merlin said, "You can't disipline a newborn, worry about that later, just like human parents.". And in fact 'disciplining' a newborn and not giving it the love and attention it wants can cause anxiety and mental issues later in life, life difficulty forming attachments. Similar phenomenon happens throughout many animals where they receive extended parental care, parrots quite possibly being one. Let your bird be a baby bird. You need to be the parents and siblings he would normally be surrounded with in the wild at the moment. Training can come later.