TOPIC
Not All Data is Valuable
The central message in this issue is that collecting more data doesn’t automatically lead to better results. The real value lies in asking the right questions and using the answers to drive meaningful action.
Data becomes valuable when it helps marketers make smarter, more informed decisions, ultimately improving the quality and relevance of email content and strategy.
THEME
Data, like rocks, can weigh you down if you don’t stack it with intention. The Rock’s strength is legendary, but even he wouldn’t carry around a bunch of useless stones.
This theme reinforces the idea that marketers need to be selective, stacking only the data that supports growth and drives action.
What Was Tested?
This experiment compared two formats for collecting subscriber data:
- A short, text-only email with a single CTA: “Update my profile”
- A long-form, content-rich newsletter with visuals and the same CTA
The goal was to determine which approach would drive more subscribers to update their profile with their biggest email challenge.
Hypothesis
A concise, text-only email with a clear ask will generate higher engagement and action compared to a longer, visually rich newsletter, as subscribers may be more likely to respond to a straightforward request without additional content distractions.
Key Insights
Apple Mail Behavior: iOS 18’s AI-generated preview displayed “Know your role.” The first name personalization was removed.
Segment Behavior: Of the subscribers who were sent the initial send and identified their email challenge as “Using data to make smarter decisions,” 36.11% opened the email (excluding Apple MPP), likely due to the subject line “Data Rocks!“.
OUTCOME
Out of 26 subscribers expected to participate in the test:
- 8 updated their profile from the text-only email
- 1 updated via the newsletter
While the text-only email outperformed the newsletter, with 34.6% of subscribers updating their profile, the overall result is still considered a success.
Campaign Details
Subject line: Data Rocks!
Preview: Know your role, *|FNAME|*.
GOALS
- Educate subscribers on the value of collecting the right data.
- Encourage engagement by prompting subscribers to update their preferences.
- Collect data to better personalize future emails based on subscriber challenges.
- Track engagement specific to subscribers who identified “using data to make smarter decisions” as their challenge, by monitoring clicks on the “See the data” CTA to provide insights into how this group interacts with data-driven content.
Audience Size & Segmentation
- Segment 1 – Subscribers who identified a challenge: 45 subscribers
- Segment 2 – Subscribers did not identify a challenge and also received the text-only version: 26 subscribers
- Segment 3 – New Subscribers who completed the welcome journey after the initial send: 27 subscribers
Test VARIABLES
- The text-only and newsletter-formatted emails used different subject lines and preview text.
- Text-only – Subject line: Experiment update, Preview: *|FNAME|*, Your profile is missing information.
- Newsletter – Subject line: Data Rocks!, Preview: Know your role, *|FNAME|*.
- To ensure a clear comparison, subscribers who received the text-only email were sent this newsletter separately from those who had already identified their challenge via the latest opt-in form.
- This newsletter displayed each subscriber’s current selection. Those with a blank field could update, while those who previously selected an option could choose to revise their response.
- Variations & Adjustments: The resend version included an intro message informing new subscribers they joined after the initial test and reinforcing they were receiving the newsletter after the test was completed providing them with the outcome. The goal was to leverage high engagement levels from new subscribers who had just completed the welcome journey, making this their first newsletter.
Engagement KPIs
| Initial & Resend Combined excludes Apple MPP | Combined Total | Combined Rate |
|---|---|---|
| Sent | 98 | — |
| Delivered | 98 | 100% |
| Unique opens | 58 | 59.18% |
| Unique clicks | 16 | 16.3% |
| Clicks per unique opens | 16 | 22.54% |
| Bounces | 0 | 0.00% |
| Unsubscribes | 0 | 0.00% |
| Replies | 4 | 4.08% |
Mailchimp. powers this experiment. Information about Mailchimp’s Metrics can be viewed here.
Qualitative Feedback
💬 Direct Subscriber Comments:
- “I am interested in the data 😎“
- “love the tryandreply. Saw the dashboard, that is pretty cool.” – This subscriber thoughtfully pointed out a misalignment of the flags in the dashboard and even included a screenshot for reference. I truly appreciate this kind of feedback!
- “I love pulling the curtain back to show your subscriber data by segment—very cool, Danielle!“
💡 Long-Form Feedback:
“Shocker… I loved the long form! It felt so much more personal…I understood why you were asking for more information and why it would benefit me. And considering the reason why people signed up for your newsie in the first place, I feel like teaching them about the “why” and providing all that sweet, sweet data makes so much sense. It was a great read, and I love the data you shared! In comparison, the short one felt like homework that benefits you more than me. ¯\_(ツ)_/¯
FWIW – I was ready to take action as soon as I checked out the text version, so I started scrolling (skipping over “The Balancing Act: Asking for Data vs. Keeping Sign-Ups Easy” because you already had me hooked on following your CTA.
Also an FYI – it went to my inbox! (Gmail Updates tab)“
💭 Observations on Subscriber Interaction:
- The topic rating at the bottom of the email received five clicks for “Love it! 😍.” Two of these five subscribers who loved the topic also identified “using data to make smarter decisions” as their biggest email challenge.
- Additionally, three of the four subscribers who clicked on the “See the data” CTA had also identified “using data to make smarter decisions” as their main challenge.
- The most engaged link was the CTA to “View the text-only email,” indicating there was a strong interest in comparing it to the newsletter version they were reading.
📩 Recipient Inquiry:
“I’m currently contemplating a weird shift for my own newsletter and would love your take on it (after I flesh it out a bit more) if you don’t mind. 😅“
Questions, suggestions or takeaways to share? Email sender@tryandreply.email
Danielle Cambio
Marketing Technologist
Execute. Measure. Assess. Improve. Learn.
If you enjoyed this content, consider subscribing to the newsletter!
I experiment with what I see in the inbox and share the results. As a subscriber, you can participate in real-time, and your feedback helps shape the email experiment.