Pretty much the only think I know of that you for sure dont want to go about touching are bristle worms. If you have fish that are known for biting (such as eels) or other venomous fish (like lionfish) then you need to be careful the them too, they can inflict some serious damage.
I also personally avoid the nems as well, I touched an aiptasia I didnt know I had and ended up with a pretty bad sting. But I think I had more of an allergic reaction to their toxins than most hobbiest do.
Like biff said, use your head. Dont eat the contents of your tank, dont stick your hand in your eye or mouth after playing around in the tank, wash your hands, wear gloves, etc. You will be fine :)