What do you do when he attacks the clothes and how long has it been going on?
Some birds get genuinely upset over certain colors/patterns, but your reaction can also impact the behavior.
My bird HATED certain types of clothes on certain people-- it has improved over the years, but chunky sweaters were a huge biting trigger, as were long-sleeves in general. They are not so much anymore, but there was a time when I told anyone holding her to roll up their sleeves and remove jewelry lol! I also had a blue shirt that I often wore to work and if I was bitten, I was almost always wearing it. It could be the association between that shirt and me leaving, or that shirt and my stress levels, but either way, IF I was bitten, it was almost always when I had that shirt on. This comes down to knowing your bird, but also, I think it has a bit to do with your bond. My bird, while she has been bonded to me for many years, was definitly weirder about clothing within the first 2 years or so. It sometimes continued after that, but it hasn't happened over the last few years (even with the blue shirt...although I do notice that she seems annoyed when she sees it haha!)
If you are getting bitten with a certain shirt on, wear that shirt less lol or deal with the consequences without giving a reaction that could be viewed as favorable in your bird's eyes. they see thing SO much better than we do, so something highly patterned etc could actually appear alien/frightening to them in some cases-- they see something like double the color spectrum that we do and they survive because of their ability to notice small details (unlike us).