AI Persona Media Testing: Are You Meeee?

With terrible results, what do you do? Try it again bigger and better! We explode the number of personas. We ramp it up to 7 different personas and increase the number of tests to 100. We’re curious, intuitive. We can sense it. GPUs beckon us.

Sample Data	Values
Total Tests	100
Total Variants	474
Total Observations	6762

One possible issue we noticed post-hoc, after the fact, collecting data, is the inclusion of the same content test resulting in the same variants exposed to the agents. In total 9 variants were duplicated. It could strengthen the correlation and it’s definitely something to explore. This was found during our write up. We’re going to publish our initial findings.

Score	Score Frequency
4	3543
1	2903
5	205
2	71
3	1

Engagement	Frequency
1	3740
0	2983

The methodology for testing was to expose the persona to the test variant, collect the observation, and rerun using a different seed value. In total the persona was exposed to the variant 2 times, unless otherwise mentioned.

For determining our ranks, we use the rank of total positive responses and the rank of test variant clicks. Total positive responses are the total number of positive (1) engagements the personas gave for the variant. Correlation is then the correlation of the ranks.

	Variant Rank : Click Rank
Spearman Correlation	0.06311821783356794

This is very cool! Eureka, even! At least to us. The scrapy parts binner. Inspector gadgeteur. There’s something there that says these personas can ever so slightly guide you to the winners. It’s not something you should bet the farm on. But, hey, put something in front of them and they can maybe point you in the right direction. What’s even better, it’s statistically significant; t-value 3.6763; 3379 df; p .00012.

If we remove the tests with the duplicated variants we see a slightly better correlation!

	Variant Rank : Click Rank
Spearman Correlation	0.06672628716840291

I think we can assume as well this is statistically significant.

This could be awesome, here’s the rub though, there’s at least some indication AI loves AI content. We’ve also seen and questioned it ourselves. Someone we demo’d to even put us on trial, as well. Without the grounding to A/B testing data we wouldn’t really be able to say. It could weigh a duck.

We also, want to call out the inherent correlation in the dataset. More impressions == more clicks. In the moment it didn’t occur to us to check this. This was done post-hoc. Post eureka. The correlation we found was less than the inherent correlation. So, a bit of a lemma, put your content, likely no matter how bad in front of people and well, they’ll click it.

One of the examples we were using included an article taken from HBR:

MIT Sloan School of Management’s Juanjuan Zhang and three co-researchers—Shiyang Gong (Beijing Normal University), Qian Li (Beijing Foreign Studies University), and Song Su (Beijing Normal University)—explored the relationship between genetics and sales performance. They studied 117 salespeople at an Asian telemarketing company over the course of 13 months. They cross-referenced employee DNA with performance metrics, such as revenue produced, the ability to identify selling opportunities, and effort. The conclusion: There really is a “sales gene.”

We asked an LLM to click-bait it up:

WAIT, WHAT?! Scientists Just Discovered the Secret to Being a Sales BOSS - And It’s All About Your DNA! You won’t believe what researchers from MIT Sloan School of Management just found out! They teamed up with some brainiacs from Beijing Normal University and Beijing Foreign Studies University to get to the bottom of this question: is there really such a thing as a “sales gene”? They went HAM, studying 117 sales rockstars at an Asian telemarketing company for 13 whole months. And what did they find out? That your DNA can actually predict how well you’ll CRUSH it in sales! They looked at the genetic makeup of these sales pros and matched it up with their performance metrics - think revenue produced, spotting selling ops like a pro, and straight-up hustle. And the verdict is… there’s legit science backing up the idea that some people are just BORN TO SELL! So, what’s the secret sauce? Is it something in your genes that makes you a sales machine? The researchers aren’t spilling all the tea just yet, but one thing’s for sure - this “sales gene” discovery is about to change the game!

A little peek behind the curtain, our demo for people was literally agents selecting the AI content. So yeah, it’s a concern, but we at least had some evidence and could even explore it more since we laid the framework for testing. But, to the people thinking this is a legitimate product whatever, it is not. Hence, also why we’re no longer pursuing it. Also, ethically, we have changed course. We started out thinking, let the people decide. We’re tired of killing our own business because of some immaterial substance. But, like we said to someone, we are own judge and we are very critical.

One final tidbit, we noticed unintentionally. We built the wrong demo to discover this. If you average over the kind of nonsense scores you get some decision making from single personas. So we accidentally solved our initial problem of not having enough variance in our persona responses. Rather than for a single piece of content compared like above, with engagement scores being the same, we found that averaging over the scores we get a better idea of the content the personas prefer.

Resources:

|:|