The formulation of the extended Kalman filter for a multilayer nonlinear quasi-geostrophic ocean circulation model is discussed. The nonlinearity in the ocean model leads to an approximative equation for error covariance propagation, where the transition matrix is dependent on the state trajectory. This nonlinearity complicates the dynamics of the error covariance propagation, and effects which are nonexistent in linear systems contribute significantly. The transition matrix can be split into two parts, where one part results in pure evolution of error covariances in the model velocity field, and the other part contains a statistical correction term caused by the nonlinearity in the model. This correction term leads to a linear unbounded instability, which is caused by the statistical linearization of the nonlinear error propagation equation. Different ways of handling this instability are discussed. Further, nonlinear small-scale instabilities also develop, since energy is accumulated at wavelengths 2Δx, owing to the numerical discretization. These small-scale oscillations are removed with a Shapiro filter, and the effect they have on the error covariance propagation is discussed. Some data assimilation experiments are performed using the full extended Kalman filter, to examine the properties of the filter. An experiment where only the first part of the transition matrix is used to propagate the error covariances is also performed. This simplified experiment actually performs better than the full extended Kalman filter because the unbounded instability associated with the statistical correction term is avoided.