Informally, a social dilemma is a collective action situation in which there is a conflict between individual and collective interest. It is a situation in which individuals could do better if they either changed their strategies or changed the rules of the game.
Some technical definitions:
More formally, a social dilemma is a collective action situation in which the Nash equilibrium results in outcomes below the Pareto optimal.
A dominating strategy is a strategy that yields the best outcome for an individual regardless of what anyone else does.
Definitions of Nash and Pareto are from Ostrom and Walker (eds.), Trust and reciprocity: interdisciplinary lessons from experimental research, Russell Sage Foundation, 2003.
Definition of Dominating Strategy from Kollock, P., "Social Dilemmas: The Anatomy of Cooperation", Annual Review of Sociology, 24: 183-214, 1998.