/* SAS program to accompany the article "Visualize repetition in song lyrics" by Rick Wicklin, published 14MAR2018 on The DO Loop blog: https://blogs.sas.com/content/iml/2018/03/14/visualize-repetition-lyrics.html This program shows how to construct a "repetition matrix" for song lyrics or other text sources (speaches, poems, etc.) The program consists of two parts 1. Read the words into a SAS data set that contains one character variable (WORD). Strip out all punctuation and normalize the words to lowercase. 2. Use SAS/IML software to construct and visualize the repetition matrix. This idea is from Colin Morris and appeared in Significance magazine. See https://colinmorris.github.io/blog/weird-pop-songs and https://www.significancemagazine.com/570 */ data RowRowRow; length word $20; input word @@; word = lowcase( compress(word, ,'ps') ); /* remove punctuation and spaces */ datalines; Row, row, row your boat Gently down the stream Merrily, merrily, merrily, merrily Life is but a dream ; data Mulberry; length word $20; input word @@; word = lowcase( compress(word, ,'ps') ); datalines; Here we go round the mulberry bush, The mulberry bush, The mulberry bush. Here we go round the mulberry bush So early in the morning. ; data HeyJude; length word $20; input word @@; word = lowcase( compress(word, ,'ps') ); datalines; Hey Jude, don't make it bad Take a sad song and make it better Remember to let her into your heart Then you can start to make it better Hey Jude, don't be afraid You were made to go out and get her The minute you let her under your skin Then you begin to make it better And anytime you feel the pain, hey Jude, refrain Don't carry the world upon your shoulders For well you know that it's a fool who plays it cool By making his world a little colder Nah nah nah nah nah nah nah nah nah Hey Jude, don't let me down You have found her, now go and get her Remember to let her into your heart Then you can start to make it better So let it out and let it in, hey Jude, begin You're waiting for someone to perform with And don't you know that it's just you, hey Jude, you'll do The movement you need is on your shoulder Nah nah nah nah nah nah nah nah nah yeah Hey Jude, don't make it bad Take a sad song and make it better Remember to let her under your skin Then you'll begin to make it Better better better better better better, oh Nah nah nah nah nah nah, nah nah nah, hey Jude Nah nah nah nah nah nah, nah nah nah, hey Jude Nah nah nah nah nah nah, nah nah nah, hey Jude Nah nah nah nah nah nah, nah nah nah, hey Jude Nah nah nah nah nah nah, nah nah nah, hey Jude Nah nah nah nah nah nah, nah nah nah, hey Jude Nah nah nah nah nah nah, nah nah nah, hey Jude Nah nah nah nah nah nah, nah nah nah, hey Jude Nah nah nah nah nah nah, nah nah nah, hey Jude Nah nah nah nah nah nah, nah nah nah, hey Jude Nah nah nah nah nah nah, nah nah nah, hey Jude Nah nah nah nah nah nah, nah nah nah, hey Jude Nah nah nah nah nah nah, nah nah nah, hey Jude Nah nah nah nah nah nah, nah nah nah, hey Jude Nah nah nah nah nah nah, nah nah nah, hey Jude Nah nah nah nah nah nah, nah nah nah, hey Jude ; data LoveShack; length word $20; input word @@; word = lowcase( compress(word, ,'ps') ); datalines; If you see a faded sign by the side of the road that says Fifteen miles to the, Love Shack, Love Shack yeah I'm headin' down the Atlanta highway Lookin' for the love getaway Headed for the love getaway, love getaway I got me a car, it's as big as a whale And we're headin' on down to the Love Shack I got me a Chrysler, it seats about twenty So hurry up and bring your jukebox money The Love Shack is a little old place where We can get together Love Shack baby A Love Shack baby Love Shack, baby Love Shack Love Shack, baby Love Shack Love Shack, baby Love Shack Love Shack, baby Love Shack Sign says, woo, stay away fools 'Cause love rules at the Love Shack Well it's set way back in the middle of a field Just a funky old shack and I gotta get back Glitter on the mattress Glitter on the highway Glitter on the front porch Glitter on the hallway The Love Shack is a little old place where We can get together Love Shack, baby Love Shack, baby Love Shack, that's where it's at Love Shack, that's where it's at Huggin' and a-kissin', dancin' and a-lovin' Wearin' next to nothing 'cause it's hot as an oven The whole shack shimmies Yeah the whole shack shimmies The whole shack shimmies when everybody's movin' around And around and around and around Everybody's movin', everybody's groovin' baby Folks linin' up outside just to get down Everybody's movin', everybody's groovin' baby Funky little shack Funky little shack Hop in my Chrysler, it's as big as a whale And it's about to set sail I got me a car, it seats about twenty, so come on And bring your jukebox money The Love Shack is a little old place where We can get together Love Shack baby A Love Shack baby Love Shack, baby Love Shack Love Shack, baby Love Shack Love Shack, baby Love Shack (oh baby that's where it's at) Love Shack, baby Love Shack (baby that's where it's at) Bang bang bang on the door baby! Knock a little louder baby! Bang bang bang on the door baby! I can't hear you Bang bang bang on the door baby! Knock a little louder sugar! Bang bang bang on the door baby! I can't hear you Bang bang bang on the door baby, knock a little louder Bang bang on the door baby, bang bang! On the door baby, bang bang! On the door, bang bang! On the door baby, bang bang! You're what? Tin roof rusted! Love Shack, baby Love Shack! Love Shack, baby Love Shack! Love Shack, baby Love Shack! Love Shack, baby Love Shack! ; data CallMe; length word $20; input word @@; word = lowcase( compress(word, ,'ps') ); datalines; Color me your color, baby Color me your car Color me your color, darling I know who you are Come up off your color chart I know where you're comin' from Call me (call me) on the line Call me, call me any, anytime Call me (call me) my love You can call me any day or night Call me Cover me with kisses, baby Cover me with love Roll me in designer sheets I'll never get enough Emotions come, I don't know why Cover up love's alibi Call me (call me) on the line Call me, call me any, anytime Call me (call me) oh love When you're ready we can share the wine Call me Oh, oh, oh, oh, oh, he speaks the languages of love Oh, oh, oh, oh, oh, amore, chiamami, chiamami Oh, oh, oh, oh, oh, appelle-moi mon cherie, appelle-moi Anytime, anyplace, anywhere, any way Anytime, anyplace, anywhere, any day Call me (call me) my love Call me, call me any, anytime Call me (call me) for a ride Call me, call me for some overtime Call me (call me) my love Call me, call me in a sweet design Call me (call me), call me for your lover's lover's alibi Call me (call me) on the line Call me, call me any, anytime Call me (call me) Oh, call me, oh, oh, ah Call me (call me) my love Call me, call me any, anytime ; data StartTheFire; length word $20; input word @@; word = lowcase( compress(word, ,'ps') ); datalines; Harry Truman, Doris Day, Red China, Johnnie Ray South Pacific, Walter Winchell, Joe DiMaggio Joe McCarthy, Richard Nixon, Studebaker, television North Korea, South Korea, Marilyn Monroe Rosenbergs, H-bomb, Sugar Ray, Panmunjom Brando, "The King and I" and "The Catcher in the Rye" Eisenhower, vaccine, England's got a new queen Marciano, Liberace, Santayana goodbye We didn't start the fire It was always burning Since the world's been turning We didn't start the fire No we didn't light it But we tried to fight it Joseph Stalin, Malenkov, Nasser and Prokofiev Rockefeller, Campanella, Communist Bloc Roy Cohn, Juan Peron, Toscanini, Dacron Dien Bien Phu falls, "Rock Around the Clock" Einstein, James Dean, Brooklyn's got a winning team Davy Crockett, Peter Pan, Elvis Presley, Disneyland Bardot, Budapest, Alabama, Krushchev Princess Grace, "Peyton Place", trouble in the Suez We didn't start the fire It was always burning Since the world's been turning We didn't start the fire No we didn't light it But we tried to fight it Little Rock, Pasternak, Mickey Mantle, Kerouac Sputnik, Chou En-Lai, "Bridge on the River Kwai" Lebanon, Charlse de Gaulle, California baseball Starkweather, homicide, children of thalidomide Buddy Holly, "Ben Hur", space monkey, Mafia Hula hoops, Castro, Edsel is a no-go U2, Syngman Rhee, payola and Kennedy Chubby Checker, "Psycho", Belgians in the Congo We didn't start the fire It was always burning Since the world's been turning We didn't start the fire No we didn't light it But we tried to fight it Hemingway, Eichmann, "Stranger in a Strange Land" Dylan, Berlin, Bay of Pigs invasion "Lawrence of Arabia", British Beatlemania Ole Miss, John Glenn, Liston beats Patterson Pope Paul, Malcolm X, British politician sex JFK, blown away, what else do I have to say We didn't start the fire It was always burning Since the world's been turning We didn't start the fire No we didn't light it But we tried to fight it Birth control, Ho Chi Minh, Richard Nixon back again Moonshot, Woodstock, Watergate, punk rock Begin, Reagan, Palestine, terror on the airline Ayatollah's in Iran, Russians in Afghanistan "Wheel of Fortune", Sally Ride, heavy metal, suicide Foreign debts, homeless vets, AIDS, crack, Bernie Goetz Hypodermics on the shores, China's under martial law Rock and roller cola wars, I can't take it anymore We didn't start the fire It was always burning Since the world's been turning We didn't start the fire But when we are gone Will it still burn on, and on, and on, and on We didn't start the fire It was always burning Since the world's been turning We didn't start the fire No we didn't light it But we tried to fight it We didn't start the fire It was always burning Since the world's been turning We didn't start the fire No we didn't light it But we tried to fight it ; data IHaveADream; length word $20; input word @@; word = lowcase( compress(word, ,'ps') ); datalines; I have a dream that one day this nation will rise up and live out the true meaning of its creed: "We hold these truths to be self-evident, that all men are created equal." I have a dream that one day on the red hills of Georgia, the sons of former slaves and the sons of former slave owners will be able to sit down together at the table of brotherhood. I have a dream that one day even the state of Mississippi, a state sweltering with the heat of injustice, sweltering with the heat of oppression, will be transformed into an oasis of freedom and justice. I have a dream that my four little children will one day live in a nation where they will not be judged by the color of their skin but by the content of their character. I have a dream today! I have a dream that one day, down in Alabama, with its vicious racists, with its governor having his lips dripping with the words of "interposition" and "nullification", one day right there in Alabama little black boys and black girls will be able to join hands with little white boys and white girls as sisters and brothers. I have a dream today! ; /* preliminary investigation and exploration */ ods graphics / width=500 height=500 NXYBINSMAX=1000000; proc iml; use RowRowRow; read all var _CHAR_ into Word; close; N = nrow(Word); M = j(N,N,0); do i = 1 to N; M[,i] = element(Word, Word[i]); end; run heatmapdisc(M) title="Row, Row, Row Your Boat" colorramp={white black} displayoutlines=1 showlegend=0; upperIdx = loc(col(M)>row(M)); size = ncol(upperIdx); repTot = M[upperIdx][+]; repScore = M[upperIdx][:]; print repTot size repScore; quit; /* Use the following SAS/IML function to visualize lyrics in songs */ proc iml; /* define a function that creates and visualizes the repetition matrix */ start VizLyrics(DSName, Title); use (DSName); read all var _CHAR_ into Word; close; N = nrow(Word); M = j(N,N,0); do i = 1 to N; M[,i] = element(Word, Word[i]); end; run heatmapdisc(M) title=Title colorramp={white black} displayoutlines=0 showlegend=0; upperIdx = loc(col(M)>row(M)); return ( M[upperIdx][:] ); /* proportion of words that are repeated */ finish; score = VizLyrics("Mulberry", "Here We Go Round the Mulberry Bush"); print score; score = VizLyrics("HeyJude", "Hey Jude (The Beatles)"); print score; score = VizLyrics("LoveShack", "Love Shack (The B-52s)"); print score; score = VizLyrics("CallMe", "Call Me (Blondie)"); print score; score = VizLyrics("StartTheFire", "We Didn't Start the Fire (Billy Joel)"); print score; score = VizLyrics("IHaveADream", "I Have a Dream (Exerpt, MLK Jr)"); print score;