Informally, a social dilemma is a collective action situation in which there is a conflict between individual and collective interest. It is a situation in which individuals could do better if they either changed their strategies or changed the rules of the game.
Another informal definition is that a social dilemma "is defined by two properties: (a) each individual receives a higher payoff for a socially defecting choice ... than for a socially cooperative choice, no matter what the other individuals do, but (b) all individuals are all better off if all cooperate than if all defect."
Some technical definitions:
More formally, a social dilemma is a collective action situation in which the Nash equilibrium results in outcomes below the Pareto optimal.
A dominating strategy is a strategy that yields the best outcome for an individual regardless of what anyone else does.
Definitions of Nash and Pareto are from Ostrom and Walker (eds.), Trust and reciprocity: interdisciplinary lessons from experimental research, Russell Sage Foundation, 2003.
Definition of Dominating Strategy and first informal definition of social dilemma from Kollock, P., "Social Dilemmas: The Anatomy of Cooperation", Annual Review of Sociology, 24: 183-214, 1998.