A regular commenter is getting ready to look for her first non-family part-time babysitter. It seems like there are good resources out there for finding a full-time babysitter/nanny. But I think some of the issues are different with a part-time babysitter/nanny, whether that person is going to do a few daytime shifts for you, or exclusively nighttime work.
For me, the most important thing to consider is how you found this person. I'm far more likely to hire a babysitter that is a friend or family member of someone I know and trust than someone I found through an ad. And that may mean that I pass up someone great from an ad, but I just feel like there's more of a safety factor in knowing the person's network. (Note: Trust. Your. Instincts. If it's your cousin that you've know forever, but you get a bad vibe, don't leave that person alone with your kid.)
Now, once you've found someone to interview, what questions are important to ask? Let's assume that the person will need to care for baby through preschooler age.