Sign in Subscribe

AI Experiment 009: Can AI Audit My Newsletter Audience?

AI Experiment 009: Can AI Audit My Newsletter Audience?

The short answer: I asked my AI assistant a simple question: "which of my Substack subscribers are missing from my Ghost subscriber list?" It compared my Substack export against my Ghost members, surfaced a 532-person import I had completely forgotten I did, reconciled all 832 Ghost members numerically, and imported the 20 subscribers who were actually missing. My total contribution: uploading one CSV and answering three questions.

The question

I started in old-mind mode. My first ask was, "Can you look and see what date I uploaded my last subscriber list?" My plan was to just cross reference that date with every new subscriber I got on Substack since and add them to my Ghost subscriber list manually.

Shamus (my muse)'s response: "just get me the list and I'll run the diff," which prompted the thought, "oh yeah, duh. Why am I making this harder on myself than it needs to be? AI can see both sub lists and just do this for me!"

Even as a relatively advanced user of AI, I still experience these aha moments regularly. Old habits die hard, I guess...

I was still thinking like the operator of my tools. Decades of software trained me to decompose everything into human-sized steps:

  • Find the date,
  • Pull the list
  • Run the comparison
  • Eyeball the overlap

I was doing the orchestrating and asking the AI to fetch. The new move is the reverse. Hand over the goal and the raw material, and let the AI do the orchestrating. My job becomes judgment. Which list? What's in scope? Go or no go?

It's going to require a lot of unwiring from how we're accustomed to using technology. In the end I think it makes us more productive and frees us to focus on our strengths instead of the administrative work we do in support of our strengths today.

The first answer (and why I didn't trust it)

As liberating as it is to be able to offload the rote, manual work I'm used to doing manually is, it's still so important to think critically anytime you delegate something to AI, because as intelligent as it is, it can also hallucinate and make things up - happens all the time - including in this example.

The Substack export had 843 subscribers. My Ghost site showed 832 members, with 277 of them created on February 12, 2025, the obvious import spike. Diff the export against those 277 and you get 588 subscribers "not in the batch."

So, the first answer was "588 people to import."

Didn't sound right. I had a memory of uploading ~700 when I decided to migrate my newsletter to Ghost. If the upload was 700 and only 277 carried February dates, where were the other 400-plus?

My instinct was right, which is the whole point.

AI can help us do all sorts of things better, but we must still use our mind and to some extent our gut to make sure the output we receive really is accurate.

The forgotten import

The breakthrough came from ignoring the dates and reading the labels instead. Every Ghost member can carry labels, and the label filter told the real story:

  • "Migrated": 809 members
  • "Import 2026-06-12 20:50": 809 members, the identical set
  • "Import 2026-08-16 16:50": 18 members, a separate set

And there it was.

On June 12, 2026, I imported 532 subscribers from Substack, and that import preserved everyone's original Substack signup dates instead of stamping the import date. That's why no single-day spike existed anywhere. Those 532 members are spread across February 2025 to June 2026, each wearing an original signup date like nothing happened. My "like 700" memory was the 809 total migrated members. Close enough.

The full reconciliation: 277 (February 2025) plus 532 (June 2026) plus 18 (August 2026) plus 5 organic signups equals 832. Every member accounted for in the totals.

Glad I pushed back.

The import

With the real baseline of 809 migrated members, the candidate diff shrank from 588 to 61: 37 people who subscribed after the June 2026 import, plus 24 early-2025 edge cases the imports missed. That 61 was a reconstructed list, built from an imperfect transcription plus date ranges and labels, not a verified email-by-email diff.

The import itself would be the real test.

I held out two rows: myself (the author row, pointless to import) and one yearly paid subscriber whose billing lives on Substack and deserves its own decision. The remaining 59 went in tagged "substack." Ghost reported "59 people successfully added or updated." The member count moved from 832 to 852, a net gain of 20. The other 39 matched existing members and were updated instead of duplicated, which is exactly how a good import should behave. It also confirmed the 61 was an overcount because 39 of those addresses were already in Ghost.

While we were in there, my AI picked up on something else worth paying attention to. The newsletter sender name was "LIAM DARMODY" in all caps like it is on LinkedIn. I do that because it makes it obvious when people are using automation to reach out to me. No one actually types "Hey LIAM" - that's a sign of a machine.

But this isn't LinkedIn, this is my personal newsletter and my muse pointed out that "all caps in the inbox reads like shouting." So I changed it to "Liam Darmody," which reads like a person writing to you.

Just another subtle way AI picks up on things you might be overlooking.

The Caveats

First, the first answer was wrong. 588 was an honest diff against an incomplete baseline, and I nearly ran with it. A diff is only as good as what you compare against. The fix wasn't a better diff, rather, questioning the baseline.

Second, the June 2026 import was nearly invisible by design. Preserved signup dates are a feature for the member record and a trap for the auditor. Created dates don't always mean what you think they mean. I only caught it because the member count refused to make sense.

Third, this worked because both platforms export clean email lists and Ghost dedupes imports on email address. If either side had messy data, the "import everyone and let it sort itself out" move wouldn't be safe. Know your platforms and make sure you clean the data before you trust the merge.

Fourth, the 61 was never exact either. It was reconstructed, not verified, and the import proved 39 of those addresses were already members. The final verified number is 20 genuinely new. Precision matters more in the retelling than it did in the doing, because the platform deduped safely either way.

Key Takeaways

  1. State the outcome, not the steps. "Which subscribers am I missing?" beats "what date did I upload the list?" Start by telling AI what you want, not what you think you need to get what you want. Let the AI decompose the task. That is the new skill to build.
  2. When the numbers don't add up, check the labels. Metadata outlives your memory of what you clicked.
  3. Let the platform dedupe. Ghost matches imports on email, so over-including is safe and under-including is the risk. I did not know this until AI told me.
  4. Keep a human on the judgment calls. The paid subscriber, the sender name, the go-ahead to import. Let the AI do the arithmetic, but you do the deciding.

I hope you found this experiment interesting. The moral if the story is: if you find yourself asking AI to help find information to enable you to do something, try just asking it to do the thing first and see what happens.

Chances are you'll be pleasantly surprised by its capabilities.

Hi5 & thanks for stopping by,
LD
🌶️