AI Persona Media Testing: Are You Meeee?
With terrible results, what do you do? Try it again bigger and better! We explode the number of personas. We ramp it up to 7 different personas and increase the number of tests to 100. Weāre curious, intuitive. We can sense it. GPUs beckon us.
| Sample Data | Values |
|---|---|
| Total Tests | 100 |
| Total Variants | 474 |
| Total Observations | 6762 |
One possible issue we noticed post-hoc, after the fact, collecting data, is the inclusion of the same content test resulting in the same variants exposed to the agents. In total 9 variants were duplicated. It could strengthen the correlation and itās definitely something to explore. This was found during our write up. Weāre going to publish our initial findings.
| Score | Score Frequency |
|---|---|
| 4 | 3543 |
| 1 | 2903 |
| 5 | 205 |
| 2 | 71 |
| 3 | 1 |
| Engagement | Frequency |
|---|---|
| 1 | 3740 |
| 0 | 2983 |
The methodology for testing was to expose the persona to the test variant, collect the observation, and rerun using a different seed value. In total the persona was exposed to the variant 2 times, unless otherwise mentioned.
For determining our ranks, we use the rank of total positive responses and the rank of test variant clicks. Total positive responses are the total number of positive (1) engagements the personas gave for the variant. Correlation is then the correlation of the ranks.
| Ā | Variant Rank : Click Rank |
|---|---|
| Spearman Correlation | 0.06311821783356794 |
This is very cool! Eureka, even! At least to us. The scrapy parts binner. Inspector gadgeteur. Thereās something there that says these personas can ever so slightly guide you to the winners. Itās not something you should bet the farm on. But, hey, put something in front of them and they can maybe point you in the right direction. Whatās even better, itās statistically significant; t-value 3.6763; 3379 df; p .00012.
If we remove the tests with the duplicated variants we see a slightly better correlation!
| Ā | Variant Rank : Click Rank |
|---|---|
| Spearman Correlation | 0.06672628716840291 |
I think we can assume as well this is statistically significant.
This could be awesome, hereās the rub though, thereās at least some indication AI loves AI content. Weāve also seen and questioned it ourselves. Someone we demoād to even put us on trial, as well. Without the grounding to A/B testing data we wouldnāt really be able to say. It could weigh a duck.
We also, want to call out the inherent correlation in the dataset. More impressions == more clicks. In the moment it didnāt occur to us to check this. This was done post-hoc. Post eureka. The correlation we found was less than the inherent correlation. So, a bit of a lemma, put your content, likely no matter how bad in front of people and well, theyāll click it.
One of the examples we were using included an article taken from HBR:
MIT Sloan School of Managementās Juanjuan Zhang and three co-researchersāShiyang Gong (Beijing Normal University), Qian Li (Beijing Foreign Studies University), and Song Su (Beijing Normal University)āexplored the relationship between genetics and sales performance. They studied 117 salespeople at an Asian telemarketing company over the course of 13 months. They cross-referenced employee DNA with performance metrics, such as revenue produced, the ability to identify selling opportunities, and effort. The conclusion: There really is a āsales gene.ā
We asked an LLM to click-bait it up:
WAIT, WHAT?! Scientists Just Discovered the Secret to Being a Sales BOSS - And Itās All About Your DNA! You wonāt believe what researchers from MIT Sloan School of Management just found out! They teamed up with some brainiacs from Beijing Normal University and Beijing Foreign Studies University to get to the bottom of this question: is there really such a thing as a āsales geneā? They went HAM, studying 117 sales rockstars at an Asian telemarketing company for 13 whole months. And what did they find out? That your DNA can actually predict how well youāll CRUSH it in sales! They looked at the genetic makeup of these sales pros and matched it up with their performance metrics - think revenue produced, spotting selling ops like a pro, and straight-up hustle. And the verdict isā¦ thereās legit science backing up the idea that some people are just BORN TO SELL! So, whatās the secret sauce? Is it something in your genes that makes you a sales machine? The researchers arenāt spilling all the tea just yet, but one thingās for sure - this āsales geneā discovery is about to change the game!
A little peek behind the curtain, our demo for people was literally agents selecting the AI content. So yeah, itās a concern, but we at least had some evidence and could even explore it more since we laid the framework for testing. But, to the people thinking this is a legitimate product whatever, it is not. Hence, also why weāre no longer pursuing it. Also, ethically, we have changed course. We started out thinking, let the people decide. Weāre tired of killing our own business because of some immaterial substance. But, like we said to someone, we are own judge and we are very critical.
One final tidbit, we noticed unintentionally. We built the wrong demo to discover this. If you average over the kind of nonsense scores you get some decision making from single personas. So we accidentally solved our initial problem of not having enough variance in our persona responses. Rather than for a single piece of content compared like above, with engagement scores being the same, we found that averaging over the scores we get a better idea of the content the personas prefer.