Nearly all statistical models contain constrained parameters. Even the simplest models contain them; for example, in ordinary least squares the estimate of the residual variance is constrained to be positive.
Many methods have been devised to enforce these restrictions; for example, the use of concentrated log-likelihoods, or standard deviations are estimated rather than variances. Other techniques for positivity include estimating the square root or log of a parameter. The hyperbolic cosine function can be used for correlations, and the logistic function for intervals.
Often, however, constraints are ignored, or tested for post hoc, for example, stationarity constraints in time dependent processes. Coefficient matrices in simultaneous equation models with lagged variables require specific constraints to ensure stationarity of the system (Greene, 1990:644), but this constraint is enforced by rejection, i.e., estimates where the constraint fails are rejected and the model is re-specified.
Over the last twenty years or so, statistical models have become more complicated, and the trend seems to be models with more constraints in them. For example, the GARCH model requires a complex set of equality and inequality constraints to ensure stationarity. Models of categorical data require normalizations which amount to constraints on parameters. Limited dependent variables contain variance and correlation restrictions.
In psychology, the covariance structure model (Browne and Arminger, 1995) estimates covariance matrices as well as coefficient matrices. The covariance matrices must be positive definite in addition to the the coefficient matrices being constrained by stationarity requirements.
Transformations of parameters and penalty methods have been customarily used to enforce constraints in statistical models. Convergence to a solution with these methods, however, was not always reliable.
Han (1977) proposed the Sequential Quadratic Programming (SQP) method for the optimization of functions with general equality and inequality constraints. This method was not initially exploited for estimating constrained statistical models because it was mostly known in the Operations Research field where the requirement of the method that the Hessian be positive definite was a serious drawback.
Such a requirement is not a hindrance in statistical estimation problems and finally in 1993 it was applied to a statistical problem (Jamshidian, et al., 1993). Then software began to appear - Matlab's optimization toolbox, SAS's Proc NLP, and Aptech System's CML. CML is the first implementation of the SQP method explicitly for the maximum likelihood estimation of constrained statistical models.