The easiest way is to set Secure on your channel, as this will only allow identified people who have flag access in your room to be op'd or halfop'd, this must be done via chanserv by typing:
/msg chanserv set #channel Secure ON
however, be advised secure will NOT stop the following commands:
/msg chanserv voice #channel nickname
/mode #channel +v nickname