We describe ethical and procedural aspects of setting up and conducting phishing experiments, drawing on experience gained from being involved in the design and execution of a sequence of phishing experiments (second author), and from being involved in the review of such experiments at the Institutional Review Board (IRB) level (first author). We describe the roles of consent, deception, debriefing, risks and privacy, and how related issues place IRBs in a new situation. We also discuss user reactions to phishing experiments, and possible ways to limit the perceived harm to the subjects.