In reply to Ian Wakeling.

Thanks for the comment. You are correct: I designed the algorithm for an application in which there are no missing values (to appear in next Monday's post). As stated, the algorithm will treat missing values as "wildcards" that will match any value, which is probably not what most people want. In particular, if the first row in the matrix contains ALL missing values, the algorithm will flag every row as a "duplicate"! I will edit the post to point out that fact explicitly.

If you want to treat the missing value as a unique value (instead of a wildcard), then I think the simplest modification is to form an indicator matrix for the missing values. You then process not only the numeric data but also the indicator matrix, and you flag a row as a "duplicate" only when the sum of squares of the values AND the sum of squares for the indicators are BOTH zero, as follows:

start DupRows_WithMissing(A);
N = nrow(A);
dupRows = j(N, 1, 0); /* binary indicator matrix */
mFlag = (A = .); /* 0/1 matrix that indicates missing entries */
do i = 1 to N-1;
if dupRows[i] = 0 then do; /* skip rows that are known to be duplicates */
r = i+1:N; /* remaining rows */
M = A[r, ]-A[i, ]; /* subtract current row from remaining rows */
F = mFlag[r, ]-mFlag[i, ]; /* subtract missing value flags, too! */
b = M[ ,##] + F[ ,##]; /* sum of squares = 0 iff duplicate row */
idx = loc( b=0 ); /* any duplicate rows for current row? */
if ncol(idx) > 0 then
dupRows[r[idx]] = 1; /* if so, flag them */
end;
end;
return dupRows;
finish;

B = {. . .,
3 . .,
. 2 .,
3 . .}; /* duplicate row */
C = B // A;
DupRows2 = DupRows_WithMissing(C);
print C DupRows2;