Read other letters about this article
When someone uses you for a reference, the prospective employer usually asks 'Would you rehire this person?' This is the secret HR code that avoids lawsuits and bloodshed.
When you say 'yes', that is a positive recommendation. When you say 'no', that is a negative one.
If they don't ask, and you don't want to lie, just say 'I would not rehire that person' and leave it at that.
If someone is rude enough to use you for a reference without asking, you can be factual about their qualifications for the job without directly disparaging them.