I Cloned My Own Voice. Now My Family Needs a Safe Word.
I recently cloned my own voice with Voicebox, and the first few seconds were mostly fun.
There I was, listening to a computer say words I had never recorded in a voice that sounded a lot like me. It had my tone, my pacing, and enough of my little speech habits to make my brain do a double take.
My first thought was, “This would make a great Colin soundboard.”
My second thought was, “Well, this could be a problem.”
The same technology that can narrate one of my videos, read an article in my voice, or make a ridiculous button say one of my favorite phrases can also create a panicked phone message that sounds familiar enough to short-circuit a family member’s common sense.
That is the part I think regular people need to understand. You do not have to become an audio engineer or learn how to spot every artificial pause. You need a family rule that still works when the voice sounds real.
TLDR
- I used the free, open-source Voicebox app to clone my own voice locally, and the result was convincing enough to change how I think about family scam calls.
- Voicebox can run on your own computer and supports several speech engines. That is useful for privacy and creative work, but local software can also remove the central safeguards a hosted service might enforce.
- ElevenLabs offers instant and professional voice cloning. Its professional process uses longer, cleaner recordings to improve consistency and requires voice verification.
- A February 2026 study of 22 people found an average accuracy of only 37.5% when participants judged 16 human and AI scam-style voice clips. The study was small, but the warning is useful: confidence in your ears is not a security plan.
- If a call creates urgency, hang up and call the person back using a number already saved in your phone. Never send money, share a verification code, or reveal sensitive information based only on an incoming voice.
- Pick a private family phrase, but treat it as a backup—not magic. The callback rule is still the strongest habit.
The Moment My Own Voice Stopped Being Proof
I have worked with enough AI tools to know the basic trick. A model is not “recording me saying everything.” It learns characteristics of my voice and uses those patterns to synthesize new speech.
Knowing that did not make hearing it less strange.
The clone did not have to be perfect. It only had to sound like me for a few seconds while saying something emotional and urgent. A scammer does not need to produce the audiobook version of Colin. They need one believable moment:
Hey, it’s me. Something happened. Please do not tell anyone yet.
That kind of line does most of the work before the technology has to do very much. The familiar voice opens the door. Fear, secrecy, and urgency push the person through it.
That is also why this is not just a technology story. It is a family-habit story.
Interactive Demo: The Colin AI Voice Soundboard
https://hear-the-hook.captaincolin.chatgpt.site/
I am building a separate interactive Colin soundboard for this spot in the post. It will let readers compare a short human recording with clearly disclosed AI-generated examples made from my own voice.
The useful version is not just a row of funny buttons—although there should absolutely be a row of funny buttons. It should include a small “real or cloned?” challenge, reveal the source after each guess, and then demonstrate the family rule that defeats both recordings: stop and verify through a known channel.
Planned examples:
- Human baseline: a short, newly recorded introduction.
- Voicebox clone: a casual line generated locally from my authorized voice profile.
- Urgency test: a safe demonstration that sounds important but tells the listener to use the family callback rule before acting.
- Optional ElevenLabs comparison: the same neutral script generated from my own verified voice, if I decide to include a hosted-service comparison.
- The fun buttons: a few harmless Colin phrases so the module still feels like something I would actually build.
The public demo should use only my voice, disclose that the synthetic clips are AI-generated, avoid payment instructions or real personal details, and keep the original clean training sample out of the downloadable package. A short web clip can still be captured, so publishing any voice demo is a risk decision, not just a design decision.
Voicebox Made This Surprisingly Easy—and Local
Voicebox describes itself as a free, open-source, local-first AI voice studio. Its current documentation says it supports seven text-to-speech engines, voice cloning, speech generation, dictation, multi-voice projects, and local integrations for AI agents.
The part that got my attention was the combination of local control and a simple desktop interface. Voicebox’s quick-start guide recommends a clean 10-to-30-second sample for a profile. You can record or upload the sample, choose an available engine, type a line, and generate new speech.
For someone like me, that opens up useful possibilities:
- Narrate a draft without recording every revision.
- Give a local app a familiar voice.
- Make temporary voiceovers for a video edit.
- Prototype accessibility features.
- Build the deeply necessary Colin soundboard the world has somehow survived without.
Because the app can run on my machine, my models, samples, and generations do not need to be sent to a cloud service for the normal local workflow. The project is open source under the MIT license, so people can inspect it, modify it, and connect it to their own tools.
That privacy advantage is real. So is the other side of it.
A local, open-source program is software on the user’s computer. There is no universal company account standing between the person and the generate button. Voicebox’s current terms describe its Voice ID system as a consent and publishing standard, while also acknowledging that the open-source desktop software cannot be technically gated by that cloud identity system.
That does not make open source bad. It means responsibility moves closer to the person running the software.
I cloned my own voice with permission from the extremely cooperative voice owner: me. The ethical rule should be just as simple for everyone else—do not clone or publish another person’s voice without clear permission.
What ElevenLabs Does Differently
ElevenLabs is the hosted name many people recognize in realistic AI speech. Its documentation explains voice cloning as capturing characteristics such as timbre, cadence, accent, and pronunciation, then applying those learned patterns to new speech.
It currently offers two different cloning paths:
- Instant Voice Cloning uses a short sample as a conditioning signal. It can create a usable clone quickly without fine-tuning a separate model for that voice.
- Professional Voice Cloning fine-tunes a dedicated model using a much larger set of high-quality recordings. ElevenLabs recommends at least 30 minutes when consistency and production quality matter.
That second path helps explain why a strong professional clone can feel less like a novelty filter and more like a real performance. More clean material gives the system a better view of how a person handles different sounds, pacing, emphasis, and delivery. Modern speech models also reproduce the pauses, emotional variation, and little imperfections people used to treat as evidence that a recording was human.
ElevenLabs has also added safeguards around that power. Its current documentation says both instant and professional clones include voice verification. Its safety page says the company monitors for policy violations, uses automated and human review, blocks cloning of celebrity and other high-risk voices, and offers tools to check whether audio was generated by its systems.
As of June 2026, ElevenLabs also began rolling out SynthID watermarking to its own generated audio. The company’s documentation says the watermark currently covers all text-to-speech generations by free users and selected paid features, with wider coverage rolling out in phases.
Those details matter, but they do not create a universal AI detector. ElevenLabs says its detector is for ElevenLabs output, does not detect audio from other platforms, and may not cover third-party models available through its service. A family member receiving a frightening call is not going to stop the conversation, export the audio, identify the possible provider, and run a forensic tool anyway.
The practical defense still has to work in the moment.
The Research Says Our Ears Are Easy to Overrate
A February 2026 paper called “Can You Tell It’s AI? Human Perception of Synthetic Voices in Vishing Scenarios” tested how people judged synthetic and human voices in scam-style situations.
The study included 22 participants and 16 clips: eight AI-generated and eight human-recorded. Average accuracy was 37.5%, which was below chance for a two-choice task. Six of the eight AI clips were labeled human by a majority of participants, while five of the eight real human clips were labeled AI by a majority.
The especially uncomfortable part was not just that people were wrong. They were often wrong with moderate or high confidence.
Participants listened for pauses, filler words, cadence, emotional expression, and vocal variation. Those are exactly the kinds of signals modern speech systems can now reproduce.
This is a small controlled study, not proof that every person will fail every time. It also tested short vishing-style clips rather than every kind of live call. I would not turn 22 participants into a giant universal statistic.
I would take the lesson seriously, though: “It sounded like them” is no longer enough.
The Family Rule I Want to Use
The Federal Trade Commission recommends calling the person back at a number you know belongs to them. If you cannot reach them, contact another family member or friend. The FBI also recommends independently verifying the caller and creating a private word or phrase with family members.
Here is the version I want my own family to remember:
1. Urgency triggers a pause
If the caller says there has been an accident, arrest, medical problem, lost phone, frozen account, or other emergency, slow down. Real emergencies can survive a 60-second verification step.
2. Hang up and call a saved number
Do not call a number the caller gives you. Do not trust caller ID by itself. Use the number already saved in your contacts or reach the person through another previously confirmed channel.
3. Use a family phrase
Pick a phrase that is memorable, not publicly posted, and not based on an obvious birthday, pet name, school, or vacation. Do not text it casually or use it as a password anywhere else.
A safe phrase is a useful second check, but it can be exposed, guessed, overheard, or socially engineered. It should support the callback rule, not replace it.
4. No money or codes on an unverified call
No gift cards. No cryptocurrency. No wire transfer. No payment-app rush. No account password. No one-time authentication code. No “do not tell Mom” secrecy rule.
If the request involves money, account access, personal documents, or a verification code, the answer is automatically: “I will call you back.”
5. Confirm through a second person when needed
If the family member does not answer, call someone who should know where they are. A second trusted person can break the emotional tunnel vision a scammer is trying to create.
A Five-Minute Family Voice-Scam Plan
You can set this up today without installing anything:
- Family phrase: ____________________________________
- Primary callback rule: Always hang up and use a saved contact.
- Backup contact: ____________________________________
- Money rule: No payment based only on an incoming call or voice message.
- Code rule: Never read a login or two-factor code to a caller.
- If we are unsure: Stop, call another family member, and verify.
Put the rule somewhere the people in your family will actually see it. That may be a shared note for some families and a card next to the kitchen phone for others. The best security system is the one everybody remembers while somebody is trying to scare them.
What I Learned From Cloning Myself
I still think AI voice generation is incredible.
Voicebox gives developers and creators a private, local playground. ElevenLabs shows how polished, expressive, and consistent synthetic speech can become when a service combines strong models, better training material, production controls, and a serious safety program. These tools can help with accessibility, narration, translation, creative projects, and communication.
The danger is not that every generated voice is flawless. The danger is that it can be good enough, for long enough, at exactly the moment somebody is scared.
My Colin soundboard will be funny. It will also make one point very clearly: if a button on a website can sound like me, a familiar voice on an unexpected call cannot be the thing that proves it is me.
The new proof is behavior.
Slow down. Hang up. Call back. Use the family rule.
That may feel slightly awkward the first time you do it to a real family member. I would rather explain an unnecessary callback than explain why we sent money to a computer wearing my voice.
Sources and Further Reading
- Can You Tell It’s AI? Human Perception of Synthetic Voices in Vishing Scenarios
- Voicebox: Open-Source AI Voice Studio
- Voicebox Quick Start
- ElevenLabs: How Voice Cloning Works
- ElevenLabs Safety
- ElevenLabs Audio Detector and SynthID
- FTC: Scammers Use AI to Enhance Family Emergency Schemes
- FBI: Impersonation and AI-Generated Voice Safety Guidance