If you hold the left side of the touch pad down, you bring up an interaction menu. In this menu there are options to mute the mics of everyone except friends and crew automatically. This only works in the open world though. You'll still hear everyone in missions and heists normally. I usually just kick the people are are making annoying noises or talking about whatever weed they're smoking.
It's not difficult to get by as a random joining what you need in MP without friends. It will require a little patience on your part to get through the heists with randoms, but it's doable. I just finished grinding money and I'm basically soloing online as well.
As for modders, modded accounts sometimes get a port from PS3 but there is very little they can get away with on PS3. The biggest thing you have to contend with is people randomly killing you in the open world. to that end, you can enable passive mode. While in passive mode, other players can't really damage you. You wont be able to use any weapons though, or participate in random world events.