/* data and analysis to accompany "The frequency of double-letters in Cryptoquotes" by Rick Wicklin http://blogs.sas.com/content/iml/2014/10/10/frequency-of-double-letters-in-quotes */ /* most quotes from https://secure.dailycryptogram.com/index.html and http://www.brainyquote.com */ data CryptoSolns; length quote $128; input quote $ 1-128; datalines4; He harms himself who does harm to another, and the evil plan is most harmful to the planner. --Hesiod Writing well mean never having to say, 'I guess you had to be there.' --Jef Mallett Strive not to be a success, but rather to be of value. --Albert Einstein A hero is no braver than an ordinary man, but he is braver five minutes longer. --Ralph Waldo Emerson In school, you're taught a lesson and then given a test. In life, you're given a test that teaches you a lesson. --Tom Bodett What is the most important thing one learns in school? Self-esteem, support, and friendship. --Terry Tempest Williams The whole purpose of education is to turn mirrors into windows. --Sydney J. Harris The strength or weakness of our conviction depends more on our courage than on our intelligence. --Luc De Clapiers He who doesn't lose his wits over certain things has no wits to lose. --G. E. Lessing Nobody ever went broke underestimating the taste of the American public. --H. L. Mencken The flower that smells the sweetest is shy and lowly. --William Wordsworth Laughter is the shortest distance between two people. --Victor Borge Things alter for the worse spontaneously, if they be not altered for the better designedly. --Francis Bacon We all have big changes in our lives that are more or less a second chance. --Harrison Ford Whenever you find that you are on the side of the majority, it is time to reform. --Mark Twain When I was kidnapped, my parents snapped into action. They rented out my room. --Woody Allen Where lipstick is concerned, the important thing is not color, but to accept where your lips end. --Jerry Seinfeld Democracy is the theory that the common people know what they want and deserve to get it good and hard. --H. L. Mencken We can't help everyone, but everyone can help someone. --Ronald Reagan I say luck is when an opportunity comes along, and you're prepared for it. --Denzel Washington I've got a new invention. It's a revolving bowl for tired goldfish. --Lefty Gomez Humility is the embarrassment you feel when you tell people how wonderful you are. --Laurence J. Peter I don't have a bank account, because I don't know my mother's maiden name. --Paula Poundstone Bachelors know more about women than married men; if they didn't they'd be married too. --H. L. Mencken Though I am not naturally honest, I am so sometimes by chance. --William Shakespeare Anyone can do any amount of work provided it isn't the work he is supposed to be doing at the moment. --Robert Benchley I like work. It fascinates me. I can sit and look at it for hours. --Jerome K. Jerome It is no coincidence that in no known language does the phrase 'As pretty as an airport' appear. --Douglas Adams A thought is often original though you have uttered it a hundred times. --Oliver Wendell Holmes It is our responsibilities, not ourselves, that we should take seriously. --Peter Ustinov Power tends to corrupt, and absolute power tends to corrupt absolutely --Lord Acton Life is a reciprocal exchange. To move forward, you have to give back. --Oprah Winfrey The art of being wise is the art of knowing what to overlook. --William James The voyage of discovery is not in seeking new landscapes but in having new eyes. --Marcel Proust Anybody who watches three games of football in a row should be declared brain dead. --Erma Bombeck How strange that nature does not knock, and yet does not intrude! --Emily Dickinson We are inclined to believe those whom we do not know because they have never deceived us. --Samuel Johnson No man is an island, entire of itself; every man is a piece of the continent. --John Donne What the world needs is more geniuses with humility, there are so few of us left. --Oscar Levant If fifty million people say a foolish thing, it is still a foolish thing. --Anatole France A certain amount of opposition is a great help to a man. Kites rise against, not with, the wind. --Lewis Mumford The wisdom of the wise and the experience of ages may be preserved by quotation. --Benjamin Disraeli It is difficult to negotiate with those who do not share the same reference. --Nelson Mandela Over every mountain there is a path, although it may not be seen from the valley. --Theodore Roethke There are two ways of spreading light: to be the candle or the mirror that reflects it. --Edith Wharton Genius may have its limitations but stupidity is not thus handicapped. --Elbert Hubbard Health nuts are going to feel stupid someday lying in hospitals dying of nothing. --Redd Foxx You wake me up early in the morning to tell me I am right? Please wait until I am wrong. --Johann von Neumann My definition of an expert in any field is a person who knows enough about what's really going to be scared. --P. J. Plauger The easiest kind of relationship for me is with ten thousand people. The hardest is with one. --Joan Baez By all means marry. If you get a good wife you'll be happy. If you get a bad one you'll become a philosopher. --Socrates The future, according to some scientists, will be exactly like the past, only far more expensive. --John Sladek If we don't believe in freedom of expression for people we despise, we don't believe in it at all. --Noam Chomsky Public speaking is the art of diluting a two-minute idea with a two-hour vocabulary. --Evan Esar For a successful technology, reality must take precedence over public relations, for Nature cannot be fooled. --Richard Feynman Wise sayings often fall on barren ground, but a kind word is never thrown away. --Arthur Helps An ignorant person is one who doesn't know what you have just found out. --Will Rogers Feeling gratitude and not expressing it is like wrapping a present and not giving it. --William Arthur Ward Nature is something outside our body, but the mind is within us. --Bhumibol Adulyadej Trust your instinct to the end though you can render no reason. --Ralph Waldo Emerson The hardest thing in life to learn is which bridge to cross and which to burn. --David Russell Nobody knows the age of the human race, but everybody agrees that it is old enough to know better. --Anonymous Most people would like to be delivered from temptation but would like it to keep in touch. --Robert Orben We are what we pretend to be, so we must be careful what we pretend to be. --Kurt Vonnegut If at first you don't succeed find out if the loser gets anything. --Bill Lyon Wilderness is not a luxury but a necessity of the human spirit. --Edward Abbey Everybody gets so much information all day long that they lose their common sense. --Gertrude Stein I am always doing that which I can not do in order that I may learn how to do it. --Pablo Picasso Man invented language to satisfy his deep need to complain. --Lily Tomlin The lion and the calf shall lie down together, but the calf won't get much sleep. --Woody Allen Water, air, and cleanness are the chief articles in my pharmacy. --Napoleon Bonaparte It is absurd to divide people into good and bad. People are either charming or tedious. --Oscar Wilde Justice is a contract of expediency entered upon to prevent men harming or being harmed. --Epicurus I am extraordinarily patient, provided I get my own way in the end. --Margaret Thatcher To have doubted one's own first principles is the mark of a civilized man. --Oliver Wendell Holmes Jr. The place of the father in the modern suburban family is a very small one, particularly if he plays golf. --Bertrand Russell He could not die when trees were green, for he loved the time too well. --John Clare There is always a well-known solution to every human problem: neat plausible and wrong. --H. L. Mencken Men are like wine: some turn to vinegar, but the best improve with age. --Pope John XXIII Critics search for ages for the wrong word, which, to give them credit, they eventually find. --Peter Ustinov Education makes people easy to lead, but difficult to drive; easy to govern, but impossible to enslave. --Henry Brougham The trouble with wedlock is that there's not enough wed and too much lock. --Christopher Morley The only thing to do with good advice is pass it on. It is never any use to oneself. --Oscar Wilde Vigorous let us be in attaining our ends, and mild in our method of attainment. --Lord Newborough It usually takes more than three weeks to prepare a good impromptu speech. --Mark Twain No great deed, private or public, has ever been undertaken in a bliss of certainty. --Leon Wieseltier Nature is an infinite sphere of which the center is everywhere and the circumference nowhere. --Blaise Pascal Politicians are the same all over. They promise to build bridges even when there are no rivers. --Nikita Khrushchev You cannot depend on your eyes when your imagination is out of focus. --Mark Twain Words ought to be a little wild for they are the assaults of thought on the unthinking. --John Maynard Keynes Charm is a way of getting the answer yes without asking a clear question. --Albert Camus It often takes more courage to change ones opinion than to keep it. --Willy Brandt The truth that makes men free is for the most part the truth which men prefer not to hear. --Herbert Agar It is always the best policy to tell the truth, unless, of course, you are an exceptionally good liar. --Jerome K. Jerome He is indebted to his memory for his jests and to his imagination for his facts. --Richard Brinsley Sheridan Obscurity is a good thing. You can fail in obscurity. It removes the fear of failure. --Jason Fried All of us want to do well. But if we do not do good, too, then doing well will never be enough. --Anna Quindlen Put even the plainest woman into a beautiful dress and unconsciously she will try to live up to it. --Lady Duff-Gordon The wit makes fun of other persons; the satirist makes fun of the world; the humorist makes fun of himself. --James Thurber What is an epigram? A dwarfish whole, its body brevity, and wit its soul. --Samuel Taylor Coleridge A day of worry is more exhausting than a week of work. --John Lubbock It's not what you look at that matters, it's what you see. --Henry David Thoreau The more you like yourself, the less you are like anyone else, which makes you unique. --Walt Disney Do not go where the path may lead, go instead where there is no path and leave a trail. --Ralph Waldo Emerson The pessimist complains about the wind; the optimist expects it to change; the realist adjusts the sails. --William Arthur Ward When it is obvious that the goals cannot be reached, don't adjust the goals, adjust the action steps. --Confucius Everything that irritates us about others can lead us to an understanding of ourselves. --Carl Jung A good head and a good heart are always a formidable combination. --Nelson Mandela Obstacles are those frightful things you see when you take your eyes off your goal. --Henry Ford Experience is not what happens to you; it's what you do with what happens to you. --Aldous Huxley I'd rather regret the things I've done than regret the things I haven't done. --Lucille Ball Wisdom, compassion, and courage are the three universally recognized moral qualities of men. --Confucius In every walk with nature one receives far more than he seeks. --John Muir Being entirely honest with oneself is a good exercise. --Sigmund Freud From the errors of others, a wise man corrects his own. --Publilius Syrus Winning is not everything, but wanting to win is. --Vince Lombardi It has long been an axiom of mine that the little things are infinitely the most important. --Arthur Conan Doyle Whatever you do in life, surround yourself with smart people who'll argue with you. --John Wooden If you only have a hammer, you tend to see every problem as a nail. --Abraham Maslow Even if you're on the right track, you'll get run over if you just sit there. --Will Rogers ;;;; proc iml; /************************************************************/ /* Utility functions for counting double-letter bigrams */ /************************************************************/ /* break a message into an array: each word in on one row; each letter is in a separate column */ start SplitByWords(_msg); msg = upcase(_msg); /* msg is scalar string */ m = substr(msg,1:length(msg),1); /* break into array */ SpaceIdx = loc(m=" "); FirstLetters = 1 || (1 + SpaceIdx); LastLetters = (SpaceIdx - 1) || length(msg); numWords = ncol(SpaceIdx) + 1; words = j(numWords, 28, " "); /* Antidisestablishmentarianism has 28 letters */ do i = 1 to numWords; range = FirstLetters[i]:LastLetters[i]; words[i, 1:ncol(range)] = m[,range]; end; maxLength = max(length(rowcat(words))); /* truncate at max length */ return ( words[,1:maxLength] ); finish; /* return a 1x26 vector of frequency counts for double-letter bigrams */ start DoubleLetters(_msg); Letters = "A":"Z"; freq = j(1, ncol(Letters), 0); w = SplitByWords(_msg); jdx = 1:(ncol(w)-1); d = (w[,jdx] = w[,jdx+1]) & (element(w[,jdx],Letters)); idx = loc(d=1); if ncol(idx)>0 then do; dbls = w[,jdx][idx]; * col vec of double letters; do i = 1 to nrow(dbls); j = loc(Letters=dbls[i]); freq[,j] = freq[,j] + 1; end; end; return ( freq ); finish; /* return the total number of bigrams in a message */ start CountBigrams(_msg); Letters = "A":"Z"; w = SplitByWords(_msg); jdx = 1:(ncol(w)-1); b = element(w[,jdx], Letters) & element(w[,jdx+1],Letters); return( sum(b) ); finish; /*********************************************************/ title; title2; reset linesize=140; /* read quotes into matrix */ use CryptoSolns; read all var _CHAR_ into Q; close; /* Find frequency of double letters for each quote. Also count all bigrams in the quotes. */ Letters = "A":"Z"; N = nrow(Q); Freq = j(N, ncol(Letters), 0); BigramCount = 0; do i = 1 to N; Freq[i,] = DoubleLetters(Q[i,]); BigramCount = BigramCount + CountBigrams(Q[i,]); end; /**************************/ /* descriptive statistics: aggregate double-letter bigrams */ /**************************/ AvgDoubles = mean(Freq[,+]); AvgBigrams = BigramCount / N; Lengths = length(Q); minLength = min(Lengths); maxLength = max(Lengths); AvgLength = mean(Lengths); sdLength = std(Lengths); print AvgDoubles AvgBigrams minLength maxLength AvgLength sdLength; /************************************************************/ /* sort columns of frequency matrix by population statistics */ /* proportions double-letter bigrams in Google corpus. See http://blogs.sas.com/content/iml/2014/10/03/double-letter-bigrams/ */ Prop = { 0.00003 0.00011 0.00083 0.00043 0.00378 0.00146 0.00025 0.00001 0.00023 . . 0.00577 0.00096 0.00073 0.00210 0.00137 . 0.00121 0.00405 0.00171 0.00001 . . 0.00003 . 0.00003 }; call sortndx(jdx, T(Prop), 1, 1); /* sort descending */ F = Freq[, jdx]; L = Letters[,jdx]; /*******************************************************/ /* Choose random sample of 20 quotes. Visualize counts */ call randseed(21); rows = sample(1:N, 20, "NoReplace"); Subset = F[rows,]; /* sort by total number of bigrams */ Count = Subset[,+]; call sortndx(ndx, Count, 1, 1); /* sort descending */ Subset = Subset[ndx,]; rows = rows[,ndx]; ramp = palette("Blues", 1+max(F)); ramp[1] = 'CXFFFFFF'; /* replace light blue with white */ ods graphics / width = 400px height=300px; call heatmapdisc(Subset) colorramp=ramp xvalues=L yvalues=rows title="Frequency of Double-Letter Bigrams in 20 Random Quotes"; *print (rows`) (Count[ndx]); /*******************************************************/ /* Compute descriptive statistics for each double-letter bigram */ /*******************************************************/ call qntl(q, F); mean = mean(F); std = std(F); max = F[<>,]; m = q[2:3,] // max // mean // std; print m[r={"Median" "Q3" "Max" "Mean" "StdDev"} c=L label="Descriptive Statistics per Quote" format=Best5.]; /*******************************************************/ /* Compute distribution of frequency of double-letter bigrams in quotes */ /*******************************************************/ Counts = F[,+]; call tabulate(Levels, QuoteFreq, Counts); title "Frequency Distribution of Double-Letter Bigrams"; title2 "N = 120"; call bar(Levels) freq=QuoteFreq grid={Y} label={"Number of Double-Letter Bigrams in Quote" "Number of Quotes"}; print (Levels // QuoteFreq)[r={"#Doubles" "#Quotes"}]; /***************************************************************/ /* Compute mean and 95% CI for the (full) collection of quotes */ /* Formulas are for CIs of binomial proportions */ TotalFreq = Freq[+,]; phat = TotalFreq / BigramCount; stderr = sqrt( phat#(1-phat) / BigramCount ); z = quantile("Normal", 1 - 0.05/2); /* two-sided 95% */ LowerCI = phat - z*stderr; UpperCI = phat + z*stderr; /* truncate CI within interval [0,1] */ LowerCI = choose(LowerCI<0, 0, LowerCI); UpperCI = choose(UpperCI>1, 1, UpperCI); varNames = {"Prop" "Estimate" "Lower95" "Upper95"}; m = Prop // phat // lowerCI // upperCI; m = m`; /* write estimates, CIs, and parameter values from corpus */ create BigramVar from m[rowname=Letters colname=varNames]; append from m[rowname=Letters]; close; quit; proc sort data=BigramVar; by decending Prop; run; title "Double-Letter Bigram Frequency"; title2 "Estimates from 120 Cryptograms vs Google Corpus"; ods graphics / width=600px height=400px; proc sgplot data=BigramVar; label Prop="Corpus Proportion"; scatter x=Letters y=Estimate / yerrorlower=lower95 yerrorupper=Upper95; series x=Letters y=Prop / markers markerattrs=(symbol=Diamond); yaxis grid; run;