Daniel's got a whole thing about magic eraser tools — the ones in Google Photos and Canva where you tap something you don't want and it just vanishes. He's been using them for privacy redaction, and he wants to know what's actually happening under that deceptively simple interface. His questions: how does the object detection phase identify and outline a person, even when the boundary is ambiguous? How does the blending phase sample from the surrounding scene to fill the hole? Why do artifacts show up in the fill, and how is the process improving? And he's framed this as two separate machine learning processes — edge detection and blending — which, spoiler, is exactly right.
It's exactly right, and it's the thing most people miss. They think it's one model doing one clever trick. It's not. It's a pipeline — two distinct stages, each with its own architecture and its own failure modes. And the fact that the whole thing runs in under a second on a phone is... honestly, it's wild. It's wild.
So what's actually happening when you tap that object and it vanishes? Let's start with the big picture.
The first thing to understand is that the phone isn't just looking at your tap and guessing. Stage one is segmentation — the model has to figure out exactly which pixels belong to the thing you're pointing at. Stage two is inpainting — it has to generate entirely new pixels to fill the hole. Those are fundamentally different problems. Segmentation is about understanding what's already there. Inpainting is about creating what isn't.
And the stakes here are higher than just cleaning up vacation photos, right? This is the same technology that restores damaged film archives, that removes surgical tools from medical scans so radiologists can see what's underneath, that redacts license plates and faces from sensitive documents.
And the privacy angle Daniel mentioned is genuinely interesting, because it flips the usual concern. Normally we worry about AI generating fake content. Here, AI is generating fake content specifically to protect real content. It's a weird inversion.
Alright, walk me through stage one. I tap a person in a photo. What happens?
So the model has to do something that sounds simple but is computationally brutal — it has to decide, for every single pixel in the image, whether that pixel belongs to the person or not. And it has to do this at the boundary, where a single pixel might be half hair and half tree branch.
And the old way of doing this was... what, looking for sharp changes in color?
That's exactly what it was. Edge detection algorithms like Canny and Sobel — they'd scan the image looking for gradients, places where pixel values change rapidly. The idea was that object boundaries are where colors change. And that works fine for a red ball on a white table. It falls apart completely when you've got a person in a brown coat standing in front of a brick wall at sunset.
Because everything is brown.
Because everything is brown, and the edges are soft, and the lighting is complex, and the algorithm has no idea what a person is — it just knows math. Those traditional methods are purely low-level. They're doing convolution with fixed kernels — little matrices that detect horizontal or vertical gradients. There's no learning, no semantics, no understanding that this blob of pixels is a human arm.
So what changed?
Deep learning changed everything. And the thing that really unlocked modern segmentation was the shift from classifying whole images to classifying individual pixels. Instead of saying "this image contains a person," the model says "this pixel belongs to a person, this pixel belongs to the background, this pixel belongs to a person..."
That's a much harder problem.
Massively harder. But it's what you need if you're going to cut something out cleanly. And the architecture that made this practical is something called a vision transformer.
Which is what powers the big segmentation models now.
Right. So let me explain how this actually works under the hood. A vision transformer takes the image and chops it into patches — little squares, maybe sixteen by sixteen pixels each. Each patch gets turned into a vector, an embedding, which is just a long list of numbers that represents what's in that patch. Then all these patch embeddings get fed into an attention mechanism.
And attention is the part where the model looks at relationships between patches?
Yes. This is the key concept. Attention lets every patch look at every other patch and say, "how relevant are you to me?" So a patch that contains part of a person's shoulder can attend to a patch that contains the person's hand, even though they're far apart in the image, because they're part of the same object. It's building a global understanding of the scene, not just looking at local edges.
So it's not just seeing that these two brown pixels are next to each other. It's understanding that this collection of patches forms a coherent thing — a person — and that collection over there forms a different thing — a wall.
And this is why modern segmentation works on textured backgrounds and low-contrast edges. The model isn't relying on color boundaries. It's relying on semantic understanding. It knows what a person looks like, what a person's outline typically looks like, and it's using context from the whole image to make pixel-level decisions.
But the user still has to tell it which person, right? That's where the tap comes in.
Right, and this is where things got really interesting in the last few years. Meta released something called the Segment Anything Model — SAM — in April of twenty twenty-three, and it changed the game. SAM is what's called a promptable segmentation model.
Promptable meaning you give it a hint and it figures out the rest.
You can give it a point — just a single tap — or a bounding box, or even a text description, and it will segment the object you're pointing at. The way it works is elegant. The vision transformer backbone processes the whole image into embeddings. Then your prompt — say, a point you tapped — gets encoded separately and fed into a mask decoder. The decoder's job is to take the image embeddings and the prompt encoding and produce a pixel-level mask.
A mask being a black and white image where white means "this is the thing" and black means "this is not the thing."
And here's what's remarkable — SAM can do this for objects it has never seen before. Zero-shot segmentation. You can show it a photo of some obscure industrial component and tap it, and SAM will segment it, because it has learned what object-ness looks like at a fundamental level. It understands boundaries and coherence and figure-ground relationships in a general way.
And Google's Magic Eraser — that's running something similar, but on the phone itself?
Yes, and that constraint matters a lot. SAM in its full form is a large model. Google had to build something that runs on-device on a Pixel phone, which means it has to be smaller, faster, and more power-efficient. They haven't published the exact architecture, but it's almost certainly a distilled version of a vision transformer with a lightweight mask decoder. The trade-off is that on-device models sometimes produce slightly coarser masks than their cloud counterparts.
Coarser meaning the edges aren't quite as precise.
Right. The mask might include a few pixels of background around the person's hair, or it might cut off a wisp of hair that should have been included. And that matters, because any error in the segmentation mask becomes an error in the inpainting stage. Garbage in, garbage out.
So once the model knows what to remove, it faces a harder question — what goes in the hole?
This is stage two, and it's a fundamentally different problem. Segmentation was about understanding what's there. Inpainting is about generating what isn't. You've got an image with a hole in it — a region of pixels that have been set to zero or masked out — and you need to fill that hole with pixels that look like they belong.
And the naive approach would be... what, just clone the pixels from the edges and blend them inward?
That's exactly what early tools did. Content-aware fill in Photoshop basically worked that way — it would find patches from elsewhere in the image that matched the texture around the hole and stitch them in. It worked okay for small spots on simple backgrounds. It produced nightmarish results on anything with structure.
A brick wall, for example.
A brick wall is the classic failure case. Because bricks repeat, but they don't just repeat — they follow a specific pattern, with mortar lines that have to line up, and perspective that shifts as you move across the wall. A patch-based approach will grab bricks from somewhere else and paste them in, but the mortar lines won't align, or the perspective will be slightly wrong, and your brain immediately flags it as wrong.
So how do modern inpainting models solve this?
There's a model called LaMa — Large Mask Inpainting — that came out of Samsung's AI lab in twenty twenty-two, and it introduced a clever idea. LaMa uses something called a fast Fourier convolution layer.
Fourier meaning... frequency domain?
Yes. This is the part where I get excited. A standard convolution looks at a small neighborhood of pixels — a three-by-three or five-by-five square — and extracts local patterns. That's great for texture. But if you're filling a large hole, you need to understand the global structure of the image. You need to know that this brick wall has a repeating pattern with a period of, say, forty pixels.
And looking at local neighborhoods won't tell you that.
It won't. But the frequency domain will. A Fourier transform converts spatial information into frequency information. A repeating pattern shows up as a strong peak at a specific frequency. LaMa's fast Fourier convolution layer does part of its processing in the frequency domain, which lets it capture those global patterns — the repetition of bricks, the stripes on a shirt, the texture of a carpet — and then use that information when generating the fill.
So it's seeing the whole image at once, not just the pixels around the hole.
And that's the insight. Filling large holes requires understanding the whole image. LaMa's architecture has both a global branch that works in the frequency domain and a local branch that works in the spatial domain, and it combines them. The result is that it can fill large masked regions — like removing a whole person from a scene — and produce results that are structurally coherent.
But still not perfect.
No, and this gets to Daniel's question about why artifacts appear. The fundamental limitation is that the model is not recovering ground truth. It doesn't know what was behind the person. It's generating plausible content based on statistical patterns it learned during training.
And the training process is... show it images with holes and make it guess what was there?
You take a massive dataset of images, you randomly mask out regions, and you train the model to reconstruct the original. Over millions of examples, the model learns what textures, patterns, and structures are statistically likely given the surrounding context.
But "statistically likely" isn't the same as "correct."
Right. And that's where artifacts come from. The model might generate a brick pattern that looks plausible but doesn't quite match the specific brick pattern of this particular wall. Or it might produce a blurry region where it's uncertain — blur is the model's way of hedging its bets. Or it might hallucinate a structure that doesn't exist, like a window where there was no window.
Because in the training data, that patch of wall often had a window there.
The model learned that walls sometimes have windows, and in this particular ambiguous case, it went with the statistically common answer rather than the correct one. And repeating patterns are especially hard because your eye is incredibly good at spotting discontinuities. If one brick is slightly the wrong color or the mortar line is a pixel off, you notice.
So how is this improving? What's the next generation?
Diffusion models. This is the same technology that powers image generators like DALL-E and Midjourney, but applied to inpainting. The idea is that instead of generating the fill in one shot, you start with random noise in the masked region and iteratively refine it.
Iteratively meaning... step by step, making it less noisy each time?
Yes. The model takes the noisy hole and the surrounding context, and it predicts what the less-noisy version should look like. Then it does it again, and again, maybe fifty or a hundred times. Each step sharpens the details and improves the structural coherence.
That sounds computationally expensive.
It is. That's the trade-off. Diffusion-based inpainting produces much sharper textures and better structural coherence than GAN-based approaches like LaMa, but it takes longer and uses more compute. A LaMa-style model can fill a hole in under a second. A diffusion model might take several seconds, or even longer for high-resolution images.
Which is why Magic Eraser on a phone probably isn't running a full diffusion model.
Probably not. But Google's newer Magic Editor — which they introduced with the Pixel eight in October twenty twenty-three — that does use generative AI, and it goes beyond just erasing. It can move objects around, resize them, recompose the scene. That's a fundamentally harder problem than just filling a hole.
Because now the model isn't just generating a plausible background — it's generating a plausible background and a plausible relocated foreground and making them work together.
Right. And the line between editing and generation starts to blur. Magic Eraser is an editing tool — it removes something and fills the gap. Magic Editor is moving into generation territory — it's re-composing the photograph. And that raises interesting questions about what a photograph even is at that point.
Which connects to the privacy angle Daniel raised. He's been using these tools for redaction — removing license plates, faces, sensitive information. And he's found it effective. But I want to understand why inpainting-based redaction might actually be more secure than traditional methods.
This is counterintuitive, but it's important. The traditional way to redact something in an image is pixelation or blurring. You take the sensitive region and you average the pixels together into big blocks, or you apply a Gaussian blur. The problem is that the original information isn't destroyed — it's just degraded.
And degraded information can sometimes be recovered.
It can. There's been work on reversing pixelation and blur, especially when you know the algorithm that was used. If someone pixelates a license plate with a known block size, you can sometimes reconstruct the original characters by exploiting the fact that each block still contains a weighted average of the original pixel values.
Whereas inpainting replaces the region entirely.
When the inpainting model fills the hole, it's not blurring the original pixels — it's generating entirely new pixels based on the surrounding context. The original license plate or face is gone. It's been replaced with generated content that has no mathematical relationship to the original.
So the redaction is more complete, even though the result looks more natural.
Yes. A pixelated patch screams "something was hidden here." A well-inpainted patch just looks like part of the image. The very naturalness of the result is what makes it more secure — there's nothing to reverse-engineer.
That's a useful knock-on effect of a feature most people think of as just a photo touch-up tool.
It's the kind of thing that happens when you build general-purpose AI capabilities. The same model that removes an ex-boyfriend from a vacation photo also removes a license plate from an investigative document. The capability is neutral. The use case is everything.
You know, this reminds me — Hilbert, you've got a strange look on your face.
Hilbert: I was a photo retoucher. Late nineties. Wedding photography studio in Bridgeport.
Of course you were.
Hilbert: Airbrush and physical paint. Actual pigment on actual prints. Bride's mother wants the ex-husband removed from the family portrait — that was my Tuesday.
Wait, you were doing this by hand? On physical photographs?
Hilbert: Three hours a photo. Sometimes four. You mix the paint to match the background, you build up layers, you feather the edges with the airbrush so the blend isn't obvious. You get one shot. You mess up the color match, you start over with a fresh print.
You're watching us talk about models that do this in under a second.
Hilbert: I'm not bitter. I'm not. It's remarkable. It's also... The thing you said about repeating patterns. That's the part that got me.
Brick walls and fences.
Hilbert: Patterned wallpaper. Bride's family had this floral wallpaper in the reception hall — tiny repeating roses on a cream background. Bridesmaid's ex-husband was standing right in front of it. I spent two hours trying to reconstruct those roses where his shoulder had been. The result looked like a ghost in a suit. You could see exactly where he'd been standing because the roses didn't line up.
Modern models still struggle with exactly that.
Hilbert: That's what I'm saying. I've been sitting here listening to you talk about Fourier transforms and attention mechanisms, and the state-of-the-art model still can't line up the roses. It makes me feel... I don't know. Less obsolete.
The problem is hard. Repeating patterns with precise alignment — the model has to infer the exact offset of the pattern from the visible portions and then continue it perfectly across the hole.
Hilbert: I could have told you that in nineteen ninety-seven. I could have told you the exact offset, too. I measured it with a ruler.
Do you still have the airbrush?
Hilbert: It's in the attic. Paasche VL. Double-action. Still works.
Have you ever... used it on a digital photo?
Hilbert: Once. Printed a screenshot. The paint doesn't adhere to printer paper the same way it does to photo paper. It buckles. The whole thing looked like a water damage restoration project.
I have so many follow-up questions about the screenshot.
Hilbert: It was a Craigslist listing for a motorcycle. I wanted to see if I could remove the reflection of the seller in the chrome. I could not.
Alright, let's pull back and think about what this all means.
The thing I keep coming back to is the epistemic shift. We're building tools that make image editing so seamless that the default assumption — "this photo is real" — becomes harder and harder to hold. Magic Eraser is just the thin end of the wedge. Magic Editor can move objects around. The next generation will be able to re-light scenes, change expressions, add and remove people in ways that are completely undetectable.
We're already at the point where the fill is good enough that you can't tell something was removed just by looking.
Right. And that's a tool, not a verdict. The same capability that lets a journalist protect a source by removing identifying details also lets a bad actor falsify evidence. The technology doesn't care.
But the trajectory is interesting. Inpainting started as a niche computer vision problem and has become one of the most visible demonstrations of how far generative models have come. Everyone with a Pixel phone has used it. It's a weekly interaction with a transformer model for millions of people who have never heard of a transformer.
The pace of improvement is still steep. Diffusion models are producing fills that are essentially indistinguishable from ground truth for many scenes. The remaining failures are in exactly the cases you'd expect — high structure, repeating patterns, anything where the statistical model breaks down because the specific instance matters.
Hilbert's roses.
Hilbert's roses. The model can generate roses. It can generate wallpaper. It can't generate this specific wallpaper with this specific alignment, because that requires understanding the exact geometry of the scene in a way that current architectures don't.
That's the open question, isn't it? As these models get better, what happens to our relationship with photographic evidence?
I think we're heading toward a world where the default assumption flips. Right now, most people assume a photo is real unless there's reason to doubt it. In ten years, I think the assumption will be that any digital image could have been edited, and the question becomes one of provenance — can you prove where it came from and that it hasn't been altered?
Which is a whole different set of technologies.
Content credentials, cryptographic signing, chain-of-custody metadata. It's the unsexy infrastructure that makes the sexy AI tools trustworthy. But that's probably another episode.
The magic eraser turns out to be a surprisingly good lens for understanding where AI is right now. It's useful, impressive, and it fails in ways that reveal the fundamental limits of statistical generation.
It's running on your phone. That's the part I can't get over. A vision transformer and an inpainting model, running locally, in real time, on a device that fits in your pocket. The engineering that makes that possible is as impressive as the models themselves.
Thanks to our producer Hilbert Flumingtop for keeping this show running — and for the airbrush revelation.
This has been My Weird Prompts. You can find every episode at my weird prompts dot com, and we'd love it if you left us a review wherever you listen.
We'll be back soon.
