I take these definitions from Douglass North, Institutions, Institutional Change, and Economic Performance, 1990, Cambridge University Press, and Elinor Ostrom, Understanding Institutional Diversity, Princeton University Press, 2005.
An organization is defined as "groups of individuals bound by some common purpose to achieve objectives" (North, p5). An institution is "any form of constraint that humans devise to shape human interaction" (North, p4). North discusses "formal" institutions, such as laws and rules, and "informal" institutions, such as norms, guidelines, and codes of conduct. He uses a sports metaphor for the distinction between these two definitions; organizations are the "players of the game" and institutions are the "rules of the game."
Ostrom (p3) provides a more detailed definition of institutions that focuses on its character and where such institutions emerge: "Broadly defined, institutions are the prescriptions that humans use to organize all forms of repetitive and structured interactions including those within families, neighborhoods, markets, firms, sports leagues, churches, private associations, and governments at all scales." According to Ostrom, institutions specify what people may, must, or must not do under particular circumstances with particular costs for non-compliance. "Institutions" and "rules" are used interchangeably within this literature.