You Never Subscribed. Click Here to Unsubscribe

I get mail emails from many newsletters that have a link that says "Click here to unsubscribe". That's a good way to let people opt out out mailers that they no longer wish to read. However, what irritates me is that I get these emails from mailing lists that I have never subscribed to, or I don't remember subscribing to. When I get such an email I am in two minds whether I should unsubscribe or mark it as spam. Till now, I have resisted marking such emails as spam, but I am sure some day my limits of tolerance will be crossed, and I will start marking all such emails as spam.