In a wide-ranging and arrogant blog post to promote his new book, Christian Rudder, the co-founder of the OKCupid dating website, described several experiments the site performed on its customers:
"... chose to celebrate the app’s release by removing all the pictures from OkCupid on launch day. “Love Is Blind Day” on OkCupid—January 15, 2013... But by comparing Love Is Blind Day to a normal Tuesday, we learned some very interesting things. In those 7 hours without photos: people responded to first messages 44% more often; conversations went deeper; contact details were exchanged more quickly; in short, OKCupid worked better..."
In another experiment, the OKCupid site changed its display parameters telling some users with poor matched that the matches were excellent and the reverse:
"... the “match percentage” we calculate for users is very good at predicting relationships. It correlates with message success, conversation length, whether people actually exchange contact information, and so on... To test this, we took pairs of bad matches (actual 30% match) and told them they were exceptionally good for each other (displaying a 90% match.)† Not surprisingly, the users sent more first messages when we said they were compatible..."
"Because of a diagnostic test, your match percentage with XXX was misstated as 31%. It is really 91%. We wanted to let you know."
Diagnostic test? That explanation doesn't sound entirely accurate. It sounds like some type of error-checking routine, and not a true admission or notification of an intentional marketing test. Were customers offered refunds for "misstated" compatibility matches? If I were an OKCupid customer, I'd demand a refund as the service didn't seem to deliver what was promised.
Rudder's blog post provides plenty of statistics about what the company learned from its live tests with customers. Rudder's blog post gave the impression that the ends justify the means -- that the wealth of data the company collected justified the test approach. Rudder also defended Facebook, after that social networking site had been criticized for performing experiments on its members without notice nor explicit consent:
"We noticed recently that people didn’t like it when Facebook “experimented” with their news feed. Even the FTC is getting involved. But guess what, everybody: if you use the Internet, you’re the subject of hundreds of experiments at any given time, on every site. That’s how websites work."
All websites? For sure, at least Facebook and OKCupid.
I am no prude. I fully expect websites to explore and implement new services, content, and functionality. How one does it matters. The ends do not justify the means.
During the last 20 years, as a usability professional I have built dozens of websites in a variety of industries: telecommunications, petroleum, travel, banking, insurance, higher education, food, consumer packaged goods, and more. In all instances, we used a variety of standard, proven test methods to collect users' opinions and reactions to proposed website features and functionality. Usually, we started by asking users -- customers and prospective customers -- what they wanted in the site that they couldn't get today. Many users will tell you. Man users are happy to tell you.
Frankly, it makes sense -- time wise and financially -- to build features that users want. No matter how curious OKCupid executives may be, I highly doubt that the site's users wanted the service to lie to them about compatibility matches.
After compiling a list of requested features (e.g., content and/or functionality), we tested implementation approaches... not on the live site, but in usability sessions with mockups or with prototypes. That approach builds users' trust. Many users appreciated the opportunity to view and comment on new features before those features are added to the live site.
In other cases, we used focus group sessions to uncover users' needs and to explore their reactions and attitudes. We often used rigorous questionnaires (sometimes in combination with other test methods), so that we could analyze the results later. In some instances, we included survey forms with the live site.
My point: we never adjusted the live site's core functions and contents without notice. We didn't add new features to live sites until after all testing was finished, the new features were built, and all "bugs" or code glitches were fixed. Anyone experienced with website development knows that it takes time to get the bugs out. When you add new features, they often affect, or break, something else -- unintended consequences.
Users' trust and reliability are critical. Frankly, we trusted uses to ask them what they wanted. We trusted users enough to inform them of tests. We respected users enough to compensate test participants for their time. We respected users enough to acknowledge that some have a right to not participate in tests.
After reading Mr. Rudder's blog post, I began to wonder how trustworthy the OKCupid site really is. The good: OKCupid executives are curious, want to continually improve their site, and act quickly. The not-so-good: curiosity and acting quickly aren't enough. Users rely on the live site to to operate as advertised and promised. Deviations from that with unannounced tests that users can't opt out of, erode users' confidence and trust.
All of the tests Rudder described could have been performed with standard testing methods, some of which I have described above; without directly changing the live site. Maybe the OKCupid executives aren't aware of or wanted to skip the costs and time of traditional testing methods. Maybe speed is their primary goal. In their rush to improve things, Mr. Rudder and his executive team seem comfortable to unnecessarily risk consumers' trust and respect.
If this is the current state of social networking sites, then the industry has fallen. It has moved beyond simply collecting, archiving, and analyzing massive amounts of consumers' personal information for advertising revenues. It also operates arrogantly: making any changes they please to live sites, while ignoring users' trust nor respect. That's not something I look for in a site. Nor will I buy Mr. Rudder's book.
What are your opinions of OKCupid's tests?