I agree, the grey needs more socialization, but in a way that the grey wont become overstimulated in a negative aspect.
Don't know if it would help, but what I'm reading is that you are with the grey, alone, then someone else shows up in your greys eyesight, you get a bite.
What about turning that around? That is, taking your grey with you to a room with another person? Is the behavior the same? Different?
If he wont take peanuts from you anymore, then find another treat! It's ideal to have 5+ rewards that the bird is willing to work for rather than just one. That way, it keeps things interesting and they wont always know what they are getting! However, I realize this is easier said than done... as I've had a conure that 80-90% of the time dropped *everything* I handed to him by hand. However, if I put the item into a bowl in his cage, he'd walk over and go check the item out. Never could find something he loved over all else, but that was before I knew how to figure out the best treats.
Try real nuts (peanuts are legumes), hulled sunflower seeds, large millet, small millet, dried fruit, dried peppers, etc.
Is there any way you can safely take your grey out in public so he can see other people? If yes, you might try going somewhere quiet with your grey and slowly working up to larger crowds of people.