Mine took a LONG time to warm up to me. In fact, up until a month or so ago, he was still skittish...I'd sit in front of the tank for a LONG time, waiting for him to come out. He'd get used to me being there. He would spend all day long in the far corner, w/ the cardinal...if the cardinal swam out in the open (which was rarely), he would follow. So the banggai comes near me when I have food, and Buster would follow him. But he'd swim away when the cardinal swims away.
But once I got the tang and the coral beauty, all of a sudden, Buster is all over the tank, swimming with them. I think it's just a matter of having other fish he feels safe with to gain more confidence.