When PencilKit Meets AI: The Fastest Way to Turn Drawings Into Stories

How to build an app that transforms any doodle into art and a magical tale, step by step.

Sometimes a side project is just an excuse to tinker with frameworks.

Sometimes, it leads to something genuinely surprising.

That’s what I found when I started to play with PencilKit, Apple's open-ended drawing framework for iOS, and connected it to the bleeding edge of AI models.

Let me walk you through the whole process. By the end, you will understand how you can build an app where anyone (especially kids) can doodle at will, hit a button, and in seconds see their crude doodle transformed into an impressive image, with an original story produced by a large language model, in real-time.

This isn't a "concept" piece. It's a hands-on, production-ready pattern you can put in your own apps, teaching, or just to play around.

Why do I want to connect PencilKit and AI?

PencilKit, if you haven't played with it, gives you a blank screen with Apple Pencil support, and it feels natural - excellent for freehand sketches, diagrams, or allowing kids to draw anything they can imagine.

The only problem is that most people aren't artists, and images that resemble doodles - well, they're doodles.

That's where AI comes in.

The models like GPT-4o and DALL-E 3 can take almost anything and turn it into something visually beautiful, or take nothing and tell a story from it.

The magic is when you combine those things too:

• You lower the barrier to creative output. Kids (and adults) don't have to draw on the white board perfectly; they just need an idea.

• You get surprising, delightful results every time.

Execution with the app is less of a "tool", and more of the magic of creation.

If you're simply looking for a fun learning project or a sort of feature for the next app you're working on, this pattern is surprisingly easy to implement.

Step 1: Building the whiteboard with SwiftUI and PencilKit

You're going to start with a blank whiteboard. PencilKit is based on UIKit, but depending on what you're trying to accomplish you can wrap the drawing tool for SwiftUI with a few lines of code.

Here's the simplest yet most complete example:

You now have a legit drawing surface in SwiftUI. The tool picker will show up, Apple Pencil works for drawing, and users will be able to export whatever they draw as a .png.

If you want to add undo/redo/clear features to your whiteboard, here is a way to add extensions for PKCanvasView:

And in your SwiftUI view, wire up the controls into your view:

Need to get the current drawing? Just do this:

Exporting the users doodle is as simple as:

It would only take a couple lines to convert the image to Data or base64 string and prepare for the next step!