Horrifying AI Hallucination from a big AI company
I walked away from a $7K deal
Got BDE?⚡️
Hi data baddies 💅🏼
Most people grind for months trying to break into data and never land the role. Usually, because they are focused on the wrong things. 🙅🏻♀️ My FREE 5-day Data Career Kickstart course fixes that. 🔥
You can access the FREE course:
While everyone is arguing about AI models, Postgres is building the actual data layer that makes AI reliable in production, supporting vector search, agent memory, and production RAG pipelines. And it runs on the same open source database all in SQL.
I just got to be part of POSETTE: An Event for Postgres organized by the Postgres team at Microsoft, and it was incredible.
These were my 3 favorite sessions:
↳ From Queries to Agents: how Postgres is becoming the backbone for AI agent data retrieval and why it matters for production
systems
↳ Why We Built Azure HorizonDB: Microsoft’s new cloud-native Postgres service built specifically for AI workloads
↳ pg_duckdb in Action: running analytics directly inside Postgres without ever moving your data
If you missed the event, all the sessions are available on replay here:
⚠️ Content warning ⚠️
This post contains references to violence, slurs, and self-harm language in the context of AI-generated content.
I've worked with more AI tools than most people have heard of
And I love talking about the efficiencies AI brings and the cool new things it can do. But I try to focus on the foundations and concepts rather than just chasing trends and hyping everything that drops.
Yes, some things are cool proof of concepts. But I wouldn’t post tutorials on tools with no guardrails just because they’re trendy, because the average person reading my content isn’t going to know the difference. And that’s exactly the problem.
What I will always do is speak out when something is alarming because people need to understand this to have a real and honest picture of AI.
That’s what I’m here to do today.
I’m not here to blast a company publicly. What I’m here to do is talk about something that shook me, because I think it says a lot about where AI actually is right now versus where people think it is.
Dangerous hallucinations are a much bigger deal than your chatbot just confidently citing a fake study. And having no guardrails is a product choice, NOT an accident, and how a company responds when something goes wrong matters just as much as the thing that went wrong in the first place (sometimes more).
Here’s what happened to me.
I was working on a brand deal with a well-known AI avatar company for a paid campaign.
Not a small one, but a well-known company. I was hired to make a video for $7,000, and I was excited about it.
I went through the full approval process: concept, script, filming, and multiple rounds of revisions. They asked for one final edit: record myself saying a single line so their tool could capture my voice, transcribe it, and produce an AI avatar of me saying it.
Simple enough.
Except my dumba** forgot to turn my microphone on (yes, really).
The file I uploaded was 17 seconds of complete silence. But there were no words, no audio, NOTHING intelligible at all. Here’s a video of the faulty audio I recorded and submitted:
What came back
Even though the audio was completely blank and null, the voice-to-text model returned a transcription anyway.
Except I hadn’t said anything. So what it produced was in absolutely no way a reflection of the audio I gave it.
It was full of cuss words, references to violence, and self-harm language. I’m not going to show you it in full, but I need you to understand what I actually saw on my screen.
There were phrases like “k** yourself.” It was calling me a “c**t” and a “b***h.” There were multiple slurs and explicit descriptions of violence directed at a person. When this appeared on my screen at first, I felt like I was being hacked or personally threatened.
I feared for my own safety. Then I had to step back and think about what was actually happening underneath the product, and I realized: this was a hallucination. Which was still terrifying but comforting knowing there wasn’t a real human threatening me on the other end of it.
Why it happens
ASR models (Automatic Speech Recognition, the tech that converts spoken audio into text) are trained on human speech. Their whole job is to find patterns in audio and produce a transcription.
When you feed them silence or a near-null file, they don’t stop and say, “I didn’t hear anything.” They do what the model is designed to do, regardless: predict what’s most likely.
They generate whatever sequence of words is statistically most likely, with nothing real to anchor them to. That’s what hallucination actually is at the model level. It’s a model throwing darts at a wall because there’s nothing to ground it. And when there’s no input at all, it can produce anythinggggggg… and it depends on the training data and guardrails. Here’s what was produced:
The output made absolutely no sense. It was WAY longer than anything that could ever fit in a 17-second voice note, and it was full of content that had no business being produced EVER especially since it’s a product anyone can use without any AI literacy.
What I did not understand was what happened next.
The response that made me walk
When I brought this concern to the company’s marketing team that I’ve been working with on the campaign, they accused me of having a TV on in the background that produced the content. I was horrified they were blaming me and thought I was watching something so vulgar on TV. Why are we blaming the victim? I sent them the full video recording of my session, screenshots, and EVERYTHING, but they still blamed and questioned me.
Then I filed a separate ticket through customer support, explained I was a creator working on a paid campaign with them, and asked to escalate to their safety team. They responded with tips on how to use the product better. When I told them I had no interest in continuing and asked if they had even looked at the images I had sent, they still didn’t show any concern.
There was no apology, no accountability, and not even a “thanks for letting us know, we’re looking into this.”
So I walked away from the partnership and money. All the work was done, but I knew I wasn't comfortable promoting this to my audience.
Honestly, it was not a hard decision. My integrity isn't worth $7,000.
The part that kept haunting me
Even after I’d already made my decision, I kept coming back to this.
Even if I HAD intentionally typed those harmful messages (which again, this was a hallucination from a user mistake), a product that turns them into a realistic avatar of a real person should NEVER allow that output.
This type of content that is threatening, dangerous, deceptive, and manipulative, rendered in a realistic human face and voice that can be used to cause real harm to real people.
What’s why content moderation and safety guardrails are so important. They sit in between what a model can generate and what a user actually sees. They catch harmful content before it’s even rendered, regardless of whether it came from a hallucinating model or someone with bad intentions. That layer is NOT optional; it’s essential in AI products.
The fact that it can happen from a USER ERROR, from an accidentally blank audio file, makes the absence of that guardrail even less excusable.
Cool demos are not the same as responsible products.
Every technology fails eventually. The question is what a company has built to catch it and how they respond when something slips through.
This one failed both tests.
Before you partner with, promote, or build on any AI product: ask what happens when it breaks. Not if, when. Does the company have a real escalation path? Do they take accountability? Or do they question the user and blame them?
I walked away from $7,000, and I’d do it again.
Bye BDEs 💅🏼
Jess Ramos 💕
⚡️ Social Highlights:
⚡️If you’re new here:
💁🏽♀️ Who Am I?
I’m Jess Ramos, the founder of Big Data Energy and the creator of the BEST SQL course and community: Big SQL Energy⚡️. Check me out on socials: 🔗YouTube, 🔗LinkedIn, 🔗Instagram, and 🔗TikTok. And of course subscribe to my 🔗newsletter here for all my upcoming lessons and updates— all for free!