How VWO Uses GPT-3
To generate copy variants that beat their human counterparts
The Company
VWO is like Marpipe for websites. With VWO, you can run a/b/n and multivariate tests on any part of your website, including in-app (if you have one). They also happen to be the industry leader with over 2500 customers across 90+ countries.
The Experiment
OpenAI’s GPT-3 API is an AI that can perform “any language task — semantic search, summarization, sentiment analysis, content generation, translation, and more.” That open-ended “and more” is the exciting part. My first thoughts went to copywriting.
And while others are making the news for their creative uses — Gwern’s poetry, a college kid’s #1 HN post, a business idea generator, and a controversial Guardian article written completely with GPT-3 — VWO immediately moved to test the AI against us lowly humans in a game of making money with words in verticals as far ranging as SaaS to Travel, Education to Logistics, Finance to Automotive.
You can see the official rules below, but the key (besides VWO’s brilliant marketing play) is to test the % improvement of website copy with a clear measurable goal.
The Results
With 13 confirmed participants, 4 results are currently in and the early numbers fare well for our robotic counterpart, Mr. GPT-3. In two contests, GPT-3 emerged as a clear winner. What’s interesting, is that one test was a traditional A/B test while the other was a full-blown Multi-Armed Bandit showing early signs of the AI’s sheer adaptability. You can see the comparisons below.
The other two contests, VWO ruled “inconclusive” because both companies stopped the test midway. Perhaps I misinterpreted the situation, but I’d argue that the humans behind those company’s copy were embarrassed to be losing to an AI and paused the campaigns to avoid any negative PR backlash. But for now, we’ll have to wait another few months to see how the competition pans out.
The Application
If by any chance you can get your hands on GPT-3, DO IT. Just do it. If you’re a copywriter, the earlier you learn how to wield AI like a wand, the sooner you’ll overtake your competitors.
Originally published at https://www.marpipe.com.
