Just set the Auth Type for the forum to Allowed Member List (hidden) and only members added by you to the list will get to see it (all admins get to see it, just need to add moderators).
I saw that toggle to allow Private Forums, but I am having a little difficulty in finding the individual Forum Options. Maybe it is visible after you toggle that option and create a new Forum?