A (quasi-) real-time video processing on iOS
In previous posts, I showed you how to create a custom camera using AVFoundation and how to process an image with the accelerate framework. Let’s now combine both results to create a (quasi-) real-time (I’ll explain later what I mean with quasi) video processing.
Custom camera preview
To appreciate what we are going to do, we need to build a custom camera preview. If we want to process a video buffer and show the result in real-time, we cannot use the AVCaptureVideoPreviewLayer as shown in this post, because that camera preview renders the signal directly and does not offer any way to process it, before the rendering. To make this possible, you need to take the video buffer, process it and then render it on a custom CALayer. Let’s see how to do that.
As I already demonstated here, setting the AVFoundation stack is quite straightfoward (thank you, Apple): you need to create a capture session (AVCaptureSession), then a capture device (AVCaptureDevice) and add it to the session as a device input (AVCaptureDeviceInput). Translating this in source code, this becomes:
// Create the capture session
AVCaptureSession *captureSession = [AVCaptureSession new];
[captureSession setSessionPreset:AVCaptureSessionPresetLow];
// Capture device
AVCaptureDevice *captureDevice = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];
// Device input
AVCaptureDeviceInput *deviceInput = [AVCaptureDeviceInput deviceInputWithDevice:captureDevice error:nil];
if ( [captureSession canAddInput:deviceInput] )
[captureSession addInput:deviceInput];
The output buffer
Until here, nothing is new with respct to the previous post. Here, instead, where the new stuffs come in place. First of all, we need to define a video data output (AVCaptureVideoDataOutput) and add it to the session:
AVCaptureVideoDataOutput *dataOutput = [AVCaptureVideoDataOutput new];
dataOutput.videoSettings = [NSDictionary dictionaryWithObject:[NSNumber numberWithUnsignedInt:kCVPixelFormatType_420YpCbCr8BiPlanarFullRange] forKey:(NSString *)kCVPixelBufferPixelFormatTypeKey];
[dataOutput setAlwaysDiscardsLateVideoFrames:YES];
if ( [captureSession canAddOutput:dataOutput]
[captureSession addOutput:dataOutput];
Here, I defined the output format as YUV (YpCbCr 4:2:0). If you don’t know what I am talking about, I suggest you to give a look at this article. YUV or, more correctly, YCbCr is a very common video format and I use it here, because, except when the color brings some usefull information, you usually use graylevel images for image processing. So, the YUV format provides a signal with the intensity component (the Y) and 2 cromatic components (the U and the V).
The destination layer
Additionally, we need to create a new layer and use it as our rendering destination:
CALayer *customPreviewLayer = [CALayer layer];
customPreviewLayer.bounds = CGRectMake(0, 0, self.view.frame.size.height, self.view.frame.size.width);
customPreviewLayer.position = CGPointMake(self.view.frame.size.width/2., self.view.frame.size.height/2.);
customPreviewLayer.affineTransform = CGAffineTransformMakeRotation(M_PI/2);
We can add this layer to any other layer. I’m going to add it to my view controller view layer:
[self.view.layer addSublayer:customPreviewLayer];
Let’s go
The last step of the initial configuration is to create a GCD queue that is going to manage the video buffer and set our class as delegate of the video data output sample buffer (AVCaptureVideoDataOutputSampleBufferDelegate):
dispatch_queue_t queue = dispatch_queue_create("VideoQueue", DISPATCH_QUEUE_SERIAL);
[dataOutput setSampleBufferDelegate:self queue:queue];
Final setup
Now, remeber to add the following frameworks to your project:
- AVFoundation
- CoreMedia
- CoreVideo
- CoreGraphics
Video rendering
Since the view controller is now the delegate of the capture video data output, you can implement the following callback:
- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection
AVFoundation fires this delegate method as soon as it has a data buffer available. So, you can use it to collet the video buffer frames, process them and render them on the layer that we previously created. For the moment, let’s collect the video buffer frames and render them on the layer. Later, we’ll give a look at the image processing.
The previous delegate method returns the sampleBuffer of type CMSampleBufferRef. This is a Core Media object we can bring into Core Video:
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
Let’s lock the buffer base address:
CVPixelBufferLockBaseAddress(imageBuffer, 0);
Then, let’s extract some useful image information:
size_t width = CVPixelBufferGetWidthOfPlane(imageBuffer, 0);
size_t height = CVPixelBufferGetHeightOfPlane(imageBuffer, 0);
size_t bytesPerRow = CVPixelBufferGetBytesPerRowOfPlane(imageBuffer, 0);
Remember the video buffer is in YUV format, so I extract the luma component from the buffer in this way:
Pixel_8 *lumaBuffer = CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 0);
Now, let’s render this buffer on the layer. To do so, we need to use Core Graphics: create a color space, create a graphic context and render the buffer onto the graphic context using the created color space:
CGColorSpaceRef grayColorSpace = CGColorSpaceCreateDeviceGray();
CGContextRef context = CGBitmapContextCreate(lumaBuffer, width, height, 8, bytesPerRow, grayColorSpace, kCGImageAlphaNone);
CGImageRef dstImage = CGBitmapContextCreateImage(context);
So, the dstImage is a Core Graphics image (CGImage), created from the captured buffer. Finally, we render this image on the layer, changing its contents. We do that on the main queue:
dispatch_sync(dispatch_get_main_queue(), ^{
customPreviewLayer.contents = (__bridge id)dstImage;
});
Now, let’s do some clean-up (we are good citizens, right?).
CGImageRelease(dstImage);
CGContextRelease(context);
CGColorSpaceRelease(grayColorSpace);
If you build and run, you’ll see the camera in action with your camera preview.
Image processing
Now, let’s start with some funny stuffs. Let’s process the buffer before rendering it. For this, I am going to use the Accelerate framework. The Pixel_8 *lumaBuffer would be the input of my algorithm. I need to convert the this buffer into a vImage_Buffer and prepare a vImage_Buffer for the output of the image processing algorithm.
Add this code after the line generating the lumaBuffer:
...
Pixel_8 *lumaBuffer = CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 0);
const vImage_Buffer inImage = { lumaBuffer, height, width, bytesPerRow };
Pixel_8 *outBuffer = (Pixel_8 *)calloc(width*height, sizeof(Pixel_8));
const vImage_Buffer outImage = { outBuffer, height, width, bytesPerRow };
[self maxFromImage:inImage toImage:outImage];
...
The -maxFromImage:toImage: method does all the work. Just for fun, I process the input image with a morphological operator that minimizes a region of interest within the image. Here it is:
- (void)maxFromImage:(const vImage_Buffer)src toImage:(const vImage_Buffer)dst
{
int kernelSize = 7;
vImageMin_Planar8(&src, &dst, NULL, 0, 0, kernelSize, kernelSize, kvImageDoNotTile);
}
If you now run it, rendering the outImage on the custom preview, you should obtain something like this:
![]()
You can download from here the example.
Final considerations
As I mentioned at the beginning of this post, this processing is done in quasi- real-time. The limitiation derives from the accelerate framework. This framework is optimized for the CPU that is anyway a limited resource. Depending on the final application, this limitation could not be important. However, if you start to add more processing before the rendering, you will see what I mean. Again, the result is really dependent on the application, but if you really want to process and display the processed results in real-time, maybe you should think of using the GPU… but this is something for a future post.
Geppy
A crash course on Core Graphics
Core Graphics (CG), a.k.a. Quartz, is an old API included in both Cocoa and Cocoa Touch. It allows you to draw paths, shapes, shadows and gradients on the so-called graphic destinations or Graphic Contexts (we will come back later on this concept).
The idea behind CG is very simple. If you want to modify any view you have to subclass it and override its drawRect: method. Every view has a drawRect: method. So, if you want to customize a view you need to overwrite that method adding your code there, combining paths, shadows and gradients. You can do this with any UIView subclass. In this way, you can customize buttons, labels, maps and, in general, any graphic object descendant of UIView. In a similar way, you can use CG “to paint” on an image or on a PDF file.
You could tell me: “Hey, just a moment! Why should I write a bounce of code, if I have my graphic designer or if I can create my graphics using powerful tools like Photoshop?” Well, that’s right, you don’t need to use CG code to draw everywhere. Moreover, trying to reproduce simple Photoshop effects could be really hard to achieve and would require many lines of code.
However, there are some advantages in using CG instead of importing your very nice prepared graphics. For example, if you need to draw something in realtime that changes dynamically its visual appearance (a rounded square changing its color according to the sound input of the microphone), then you should think of using CG (yeah, you can also use Core Animation for this particular case).
Additionally, CG reduces the application memory footprints (think of the new iPad with the Retina Diaplay). Including every graphic object in your application increases the size of the application bundle. For the retina devices, this size can be quite significative. So, instead of loading images from the memory, you could use CG to draw at runtime your objects.
Another significant advantage of using CG instead of a pre-loaded graphic object is the resolution independence. This means the same source code can run on the iPhone with and without the retina display, on the iPad with and without the retina display and on the Mac with or without HiDPI. All this, with no need to create the same object in three or four different resolutions. Additionally, when you scale an image at runtime, the aliasing effect can make your object looking ugly. CG re-renders the object without this problem.
Now, before we get into the details on how to draw on a view using CG, I need to review with you some basic concepts. To understand CG, you need to understand three basics things: the Graphic Context, the Path and the Graphic State. But, even before that, we need go through how CG “paints” on a view.
There are two approaches: Painter Model and Crayon Model. The Painter Model is the way an artist paints on a canvas. Artists use to draw the background of a scene first and then proceed adding the foreground objects on top of each others until the top-front foreground object is drawn. When we will start using CG, we will follow the same approach. For example, to draw a simple square with a dark shadow in the background, we need to tell CG that you need a shadow with a given shape, color and blur and then tell CG about the foreground square. This is the sequence to draw a complex set of graphic objects. But what about each single object? In this case, we will use the Crayon Model. Kids usually use this method when painting something on a piece of paper. They usually stroke a shape and than they fill it with some colors. Using CG, you will follow the same approach: you define the shape of an object, first and then you have to color it.
Ok, let’s go now through the above mentioned basic concepts: Graphic Context, Path and Graphic State.
Graphic Context
The Graphic Context is your graphic destination. If you want to draw on a view, the view is your Graphic Context. Instead, if you want to draw on an image, the image is your Graphic Context. So, the first thing you should do, before trying to draw anything is to define the Graphic Context. Each Core Graphics function has a pointer to the current graphic context.
You get the current Graphic Context by using the UIGraphicsGetCurrentContext function (Yes, I said function and not method. This is because CG is a C-based API. Sorry, no ObjC). So, after subclassing your UIView, you override the drawRect: method (don’t forget to execute the method on super) and get the current graphic context. In this way:
- (void)drawRect:(CGRect)rect {
[super drawRect:rect];
CGContextRef ctx = UIGraphicsGetCurrentContext();
...
}
Now that we have a reference to the current Graphic Context, we could start to draw.
Path
Each drawing is composed of some basic graphic elements: points, lines, arcs. All of them are referred to as Paths. A path is a set of lines, arcs and curves you can draw on the current graphic context to build complex objects. Only when the path description is completed, you can draw the path on the screen and choose to fill it or stroke it or both.
Core Graphics provides you with specific functions to draw paths. Some of these functions are specific and can be used to draw a line, a rectangle, an ellipse, an arc, etc.
Graphic State
Before we move to a practical example, the last concept you should know is related on how Quartz draws on the graphic destination. The Graphic State keeps track of the colors, the size of the path, the transformations and many other features you want to apply to a path or group of paths.
States are maintained in stack. You save the state using CGContextSaveCGState. In this way, you push the graphic state onto the stack. When you want to reload the previous graphic state, you use CGContextRestoreCGState. In this way, the state is popped from the stack.
Hands-on
Let’s see now how to prepare our development environment. Let’s create a basic iOS project. I prefer to use the iPhone template for this exercise, but feel free to use the iPad template, if you prefer.
Launch Xcode and create a Single-View Application project. Call it DigitalPicasso and use ARC. Once the project window opens, open the ViewController.xib file. This view controller has an associated view that we are going to use as our Graphic Context. To do so, we’d need to override its drawRect: method as it has been previously discussed. Select the view in the ViewController.xib file and change the Class field to MyView in the Indentity Inspector.
Now, let’s add the class files. Select File -> New -> Files.... In the window panel, select Cocoa Touch -> Objective-C class and press the Next button. Give “MyClass” as name and UIView as “Subclass of”. Click Next and save the files in the DigitalPicasso folder.
Drawing paths
Now, you have two new files in the Xcode project (MyView.h and MyView.m). Open the MyView.m and add the following code:
- (void)drawRect:(CGRect)rect
{
[super drawRect:rect];
CGContextRef ctx = UIGraphicsGetCurrentContext();
CGContextBeginPath(ctx);
CGContextMoveToPoint(ctx, 100.0, 100.0);
CGContextAddLineToPoint(ctx, 200.0, 200.0);
CGContextAddLineToPoint(ctx, 200.0f, 300.0f);
CGContextAddLineToPoint(ctx, 100.0f, 300.0f);
CGContextClosePath(ctx);
CGContextStrokePath(ctx);
}
Now, just Run. If everything was done correctly, you get the same result shown in the following figure.

Let’s give a look at what we have done. After executing the drawRect: method on super, we get the current Graphic Context using the UIGraphicsGetCurrentContext function. The pointer ctx is very important, because we need to pass it as first argument to the Core Graphics functions. After this, we tell CG we want to begin a new path. Take into account that nothing is drawn on the screen until the last function CGContextStrokePath is executed. The rest of the code is just a preparation. CGContextMoveToPoint is used to place your pencil on a selected point of the screen. After that, you start to add lines to the path, using the CGContextAddLineToPoint until you finally close the path. Using the CGContextClosePath you connect the last point of your path with the first one. For Core Graphics a path can be a line, a rectangle, an arc or an ellipse. Once you have create a path, you can either stroke it or fill in or both. Here, I write for you some functions that could be useful to create a path. Give a look at them and check the documentation. In the same doc page, you will find other useful functions organized by type.
CGContextBeginPath - creates a new path
CGContextAddArc - creates an arc
CGContextAddEllipseInRect - creates an ellipse that fits inside a rectagle
CGContextAddLineToPoint - creates a line from the current point to the specified point
CGContextAddRect - creates a rect
CGContextMoveToPoint - moves to a point without drawing
CGContextClosePath - draws from the last point to the first one to close the path
CGContextFillPath - closes the path and fills it
CGContextStrokePath - strokes the path
CGContextClearRect - erases a rect
CGContextFillRect - fills a rect
CGContextStrokeRect - strokes a rect
CGContextStrokeRectWithWidth - strokes a rect with a specific line width
Let’s play a little bit with these functions. After the CGContextStrokePath function, add the following lines of code:
CGContextBeginPath(ctx);
CGContextAddArc(ctx, 100.0, 100.0, 100.0, 0, 2*M_PI, 0);
CGContextStrokePath(ctx);
Build and Run. Cool, right? Now, try to use the previous functions. Next time, I will show you how to create reusable paths and change the colors of your pencil.

Again on the Graphic States
The only problem we have now is that everything you try to draw on the screen has the same color, the same pencil size and type. A little bit borying, right? This means that everything is in the same graphic state. Obviuosly, we can modify this. When you change anything related with the status of your environment (colors, type of line, line width and so on), it is a good idea to save the current state before applying any change. In this way, you can always go back without redefining all the possible features. To store the current graphic state you use the CGContextSaveCGState function. Instead, use CGContextRestoreCGState is used to to revert to the previous state. Graphic states are stored in a LIFO (Last In - First Out) stack. So, every time you save a state, this is added onto the stack. If you save 2 times the state, if you want to restore the first state, you need to restore the state 2 times. If you need to go back to the previous state, then you simply restore the previous state using the CGContextRestoreCGState function.
Colors
Let’s try to save the state, change the pen colors and revert to the previous state. Before that, let’s talk a little bit about how to manage colors in Core Graphics. There is a set of Core Graphics functions you can use to modify the color of your pen. However, I suggest you to use UIKit as much as you can, since UIKit has very easy methods to create colors. You can then use the CGColor method to convert a UIColor to a CGColor. For example, you can simply use this line of code to create a red CG color:
CGColorRef aColor = [[UIColor redColor] CGColor];
This is the most convenient way to create a CG color. The other approach is to use one of the following Core Graphics functions:
CGContextSetRGBFillColor sets an RGBA color to fill a path
CGContextSetRGBStrokeColor sets an RGBA color to stroke a path
CGContextSetCMYKFillColor sets a CMYK color to fill a path
CGContextSetCMYKStrokeColor sets a CMYK color to stroke a path
CGContextSetFillColorWithColor sets a fill to a CGColor
CGContextSetStrokeColorWithColor sets a stroke color to a CGColor
So, let’s modify our DigitalPicasso project in this way.
- (void)drawRect:(CGRect)rect {
[super drawRect:rect];
CGContextRef ctx = UIGraphicsGetCurrentContext();
CGContextBeginPath(ctx);
// Add path
CGContextMoveToPoint(ctx, 100.0, 100.0);
CGContextAddLineToPoint(ctx, 200.0, 200.0);
CGContextAddLineToPoint(ctx, 200.0f, 300.0f);
CGContextAddLineToPoint(ctx, 100.0f, 300.0f);
CGContextClosePath(ctx);
CGContextStrokePath(ctx);
//Create circle and fill it in
CGContextAddArc(ctx, 200, 100.0, 50.0, 0, 2*M_PI, 0);
CGContextSetFillColorWithColor(ctx, [[UIColor whiteColor] CGColor]);
CGContextFillPath(ctx);
CGContextAddArc(ctx, 200, 100.0, 50.0, 0, 2*M_PI, 0);
CGContextSetStrokeColorWithColor(ctx, [[UIColor redColor] CGColor]);
CGContextStrokePath(ctx);
}
You should get something similar to this:

Now, the problem is that if you want to go back to the black pen, you need to set again the black color. In this case, it is very simple, you need just to add a line of code. But what happens if you want to change again to many other pen features? Here where the states help you. So, before changing the black pen to a white one, as we did in the previous code, we should save the current state. Give a look at the following code:
- (void)drawRect:(CGRect)rect {
[super drawRect:rect];
CGContextRef ctx = UIGraphicsGetCurrentContext();
CGContextBeginPath(ctx);
// Add path
CGContextMoveToPoint(ctx, 100.0, 100.0);
CGContextAddLineToPoint(ctx, 200.0, 200.0);
CGContextAddLineToPoint(ctx, 200.0f, 300.0f);
CGContextAddLineToPoint(ctx, 100.0f, 300.0f);
CGContextClosePath(ctx);
CGContextStrokePath(ctx);
// save the state
CGContextSaveGState(ctx);
//Create circle and fill it in
CGContextAddArc(ctx, 200, 100.0, 50.0, 0, 2*M_PI, 0);
CGContextSetFillColorWithColor(ctx, [[UIColor whiteColor] CGColor]);
CGContextFillPath(ctx);
CGContextAddArc(ctx, 200, 100.0, 50.0, 0, 2*M_PI, 0);
CGContextSetStrokeColorWithColor(ctx, [[UIColor redColor] CGColor]);
CGContextStrokePath(ctx);
//restore the state
CGContextRestoreGState(ctx);
CGContextAddArc(ctx, 200, 300.0, 50.0, 0, 2*M_PI, 0);
CGContextStrokePath(ctx);
}
So, we saved the state and then, we restored it. And this is the result:

Now, you see the new circle is again a black stroke, because we simply restored the previous CG state.
Extra settings
Beside the color you can also change other pen features. For example, you can change the pen size using CGContextSetLineWidth. Add this line, before line 20 of the previous code.
CGContextSetLineWidth(ctx, 10);
You get a thicker circle stroke as in the following figure. You can also change the transparency of your pen, using CGContextSetAlpha. Try this code and see what happens:
CGContextSetAlpha(ctx, .3);
Reusable paths
Reusable paths are very interesting Core Graphics tools, since they can help you saving a lot of time when you try to draw complex objects on the screen. A reusable path is a path you create just once and then, you can quickly and easily apply it later. Using reusable paths is very simple. You need to use CGPath functions in place of the CGContext ones. First, create a CGPathRef using the CGPathCreateMutable function. Then, add paths to the CGPathRef and finally close the path using CGPathCloseSubpath. To draw the path in a context you need to use CGContextAddPath. The following table shows the equivalence between reusable and not-reusable path functions:
CGPathCreateMutable <- CGContextBeginPath
CGPathAddArc <- CGContextAddArc
CGPathAddEllipseInRect <- CGContextAddEllipseInRect
CGPathAddLineToPoint <- CGContextAddLineToPoint
CGPathAddRect <- CGContextAddRect
CGPathMoveToPoint <- CGContextMoveToPoint
CGPathCloseSubpath <- CGContextClosePath
So, you can now try to convert the code we wrote untill now to use resuable paths.
- (void)drawRect:(CGRect)rect {
CGMutablePathRef myPath = CGPathCreateMutable();
CGPathAddArc(myPath, NULL, 110, 50, 30, 0. 2*M_PI, 1, YES);
...
CGContextRef ctx = UIGraphicsGetCurrentContext();
CGContextBeginPath(myPath);
CGContextAddPath(myPath);
CGContextStrokePath(myPath);
CFRelease(myPath);
}
As you can see in the above example, I created a new reusable path. Later, whenever I need to draw that path again, I simply use the reference myPath. Reusable paths are very useful when you want to draw different layers with the same shape. For example, if you want to create a square with a shadow, you need first to create a square shadow (do you remember the Crayon model?) and then, draw your square path on top of the shadow. If you use reusable paths, this is very easy: you just need to create the reusable square and draw it once as a shadow (I will show you later how to do that) and once again as a squared shape.
Color Gradients
A color gradient is a smooth transition from any color to a different one. This transition makes your drawings more realistic and look gorgeous. There are essentially 2 ways to draw a gradient in Core Graphics. You can either use the CGGradientRef object or the CGShadingRef object. The first approach is very simple and allows you to draw linear gradients. The second approach is a bit more complex, since the user has to specify a CGFunctionRef object to draw a gradient with any shape. So, to draw a color gradient with CG, you need to:
- Define a color space
- Define the list of colors you want to draw
- Define the locations where you want the colors appear
- Draw (finally) your gradient; and, finally
- Free up your memory
Color space
So, first let’s give a look at the color spaces. A color space specifies how color values are interpreted by the graphic engine. A color space is usually a multi-dimensional space, each dimension of which represents a specific color component. Defining a color space in Core Graphics is quite straightforward. Give a look at the class CGColorSpace. I usually use the CGColorSpaceCreateDeviceRGB function, but you can use any of the other functions provided in the class.
Memory management
Just a small interruption here, since I missed an important point until now. The memory management in Core Graphics follows the simple rules of the retain count mechanism. Sorry, no ARC. You would ask me: “Ehy, wait a minute didn’t you say CG is C-based? So, no objects?”. You are right. And indeed, we are not using objects. However, it is really common to call objects the CG references. These are the variables of a type ending with word Ref. For example, when you create a color space, you create a reference (a pointer) of type CGColorSpaceRef. This is simply a pointer, but we usually call it an object. Now, whenever you create a CG object (or in general, a Core Foundation object) with a function whose name contains the word Create or Copy, you are responsible to release that object. In general, you can use the Core Foundation function, CFRelease. However, in the case of the color space, there is a specialized function for that: CGColorSpaceRelease. So, if you create a color space with the above mentioned functions, you need to release it as show in this chunk of code:
CGColorSpaceRef colorSpace = CGColorSpaceCreateWithName(kCGColorSpaceGenericRGB);
// other code here
CGColorSpaceRelease(colorSpace);
Gradient example
So, let’s see how we can create a linear gradient, first. Let’s color the whole screen. So, open Xcode, create a new View-based project, create a new class subclassing UIView and add a view to the main view of your custom view controller. In the MyView class, edit the drawRect: method as follows:
- (void)drawRect:(CGRect)rect
{
[super drawRect:rect];
CGContextRef ctx = UIGraphicsGetCurrentContext();
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
CGFloat components [12] = {
1.0, 0.0, 0.0, 1.0,
0.0, 0.0, 0.0, 1.0,
0.0, 1.0, 0.0, 1.0
};
CGFloat locations[3] = { 0.0, 0.5, 1. };
CGGradientRef gradient = CGGradientCreateWithColorComponents(
colorSpace,
components,
locations,
(size_t)3);
CGContextDrawLinearGradient(
ctx,
gradient,
CGPointMake(320, 0),
CGPointMake(0, 460),
(CGGradientDrawingOptions)NULL);
CGColorSpaceRelease(colorSpace);
}
So, what is this chunk of code? First, we get the reference of the current graphic context. Then, we create a reference to the color space. I decided to use the RGB color space. This multi-dimentional space has 4 components: Red, Green, Blue and Alpha (the transparency). Each component has a value between 0 and 1. After creating the color space, we create the color components. This is a vector containing the 2 or more colors you want to use in your gradient. In the above example, I used just 3 colors. The first component is (R, G, B, A) = (1, 0, 0, 1). The second component is (R, G, B, A) = (0, 0, 0, 1). And the third component is(R, G, B, A) = (0, 1, 0, 1).
Now, we need to tell CG where to draw these colors. To draw the linear gradient, we use the CGContextDrawLinearGradient function. This function asks to specify the 2 points in the view where to the gradient starts and ends. You can see I created 2 CGPoints. Now, imagine to connect these 2 points with a segment. The aim of the locations vector is to define the percentage of that segment where the components are placed along it. In the above example, the first color is at the 0% of the segment (i.e., the beginning), the second component is at the 50% (just in the middle of the segment) and the last color component is just at the end of the segment. If you now run your project, you should get this:

Very cool, eh? Now, imagine to combine this with shadow and transparencies. You can make really cool stuffs. Try to play a little bit with this and apply it to the paths. Our approach is to combine the graphics prepared by our graphic designer and the real time drawing of Core Graphics. You can make really beautiful pieces of arts.
Flags: very useful when debugging with Instruments
There is a very nice feature in Instruments that helps you to point your loupe to a very precise spot in your code.
I am writing an app that process images captured by the camera in real-time. I am using GCD to launch different queues both in asynchronous and synchrous mode and when it is time to debug, it’s a mess. Most of the time I need to know extactly what is doing what and where something is executing.
Using the breakpoints interrupting the execution and analyze the status of your variable, it is not always useful in a multi-threaded environment, because you alter the real flow of your application.
My mind went immediately to Instruments and I found a very cool tool that you can use to know when something is executing.
Performance Session
To use this feature, you need to add to your Xcode project the DTPerfomanceSession.framework and #import <DTPerformanceSession/DTSignalFlag.h> into the class you want to analyze.
Now, wherever you need to send a signal to Instruments from your app, you just add this line of code:
DTSendSignalFlag("com.invasivecode.mytracepoints.app.point", DT_POINT_SIGNAL, TRUE);
You can also mark the start and the end of something, inserting these lines at the beginning and at the end of your code chunk:
// Put this line at the beggining
DTSendSignalFlag("com.invasivecode.mytracepoints.app.start", DT_START_SIGNAL, TRUE);
// ... more code here
// put this at the end
DTSendSignalFlag("com.invasivecode.mytracepoints.app.end", DT_END_SIGNAL, TRUE);
Done that, you can now lunch your Instruments, select the tool or the set of tools (Allocation, Profiler, etc.) you need and start to analyze your code. When the execution hits the above macros, Instruments will show a flag in the timeline as illustrated here:

You can also click on each flag and it will provide you with some detail information:

You can imagine how this is useful during the debugging and especially in a multi-threaded environment, where lots of things are happening in parallel. If you combine this with small tricks like putting a short sleep in your code, you can really get very close to each bit of execution.
Geppy
A very cool custom video camera with AVFoundation
AVFoundation is a very cool framework that allows you to collect multimedia data generated by different input sources (camera, microphone, etc.) and redirect them to any output destinations (screen, speakers, etc.).
You can create custom playback and capture solutions for audio, video and still images. The advantage of using this framework with respect to the out-of-the-shelf solutions such as the MPMoviePlayerController or the UIImagePickerController is that you get access to the raw data of the camera. In this way, you can apply effects in real-time to the input signals for different purposes.
I have prepared for you a small app to show you how to use this framework and create a very cool video camera.
Session
AVFoundation is based on the concept of the session. A session is used to control the flow of the data from an input to an output device. The creation of a session is really straightforward:
AVCaptureSession *session = [[AVCaptureSession alloc] init];
The session allows you to define the audio and video recording quality, using the sessionPreset property of the AVCaptureSession class. For this example, it’s fine to go for low quality data (so we save some battery cycle):
[session setSessionPreset:AVCaptureSessionPresetLow];
Capture Device
After the capture session has been created, you need to define the capture device you want to use. It can be the camera or the microphone. In this case, I am going to use the AVMediaTypeVideo type that supports videos and images:
AVCaptureDevice *audioCaptureDevice = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];
Capture Device Input
Next step you need to define the input of the capture device and add it to the session. Here you go:
AVCaptureDeviceInput *deviceInput = [AVCaptureDeviceInput deviceInputWithDevice:inputDevice error:&error];
if ( [session canAddInput:deviceInput] )
[session addInput:deviceInput];
You check if you can add the device input to the session and if you can, you add it.
Preview
Before defining the device output, I want to show you how to preview the camera buffer. This will be the viewfinder of your camera, i.e. the preview of what the input device is seeing.
We can quickly render the raw data collected by the camera on the screen using the AVCaptureVideoPreviewLayer. We can create this preview layer using the session we defined above and then add it to our main view layer:
AVCaptureVideoPreviewLayer *previewLayer = [[AVCaptureVideoPreviewLayer alloc] initWithSession:session];
[previewLayer setVideoGravity:AVLayerVideoGravityResizeAspectFill];
CALayer *rootLayer = [[self view] layer];
[rootLayer setMasksToBounds:YES];
[previewLayer setFrame:CGRectMake(-70, 0, rootLayer.bounds.size.height, rootLayer.bounds.size.height)];
[rootLayer insertSublayer:previewLayer atIndex:0];
You don’t need to do any additional work. You can now display the camera signal on your screen.
If you instead want to do some more cool stuffs, for example, if you want to process the camera signal to create nice video effects with Core Image or the Accelerate framework (give a look at this post), you need to collect the raw data generated by the camera, process them, and, if you like it, display them on the screen.
Go baby, go!!!
We are ready to go. The last thing you need to do is to start the session:
[session startRunning];
Cool stuffs
Since the AVCaptureVideoPreviewLayer is a layer, you can obviously add animations to it. I am attaching here a very simple Xcode project showing the previous concepts. It creates a custom video camera with the preview rotating in the 3D space.
Real-time processing
If you want to do some image processing with the raw data captured by the the camera and display the result on the screen, you need to collect those data, process them and render them on the screen without using the AVCaptureVideoPreviewLayer. Depending on what you want to achieve, you have two main strategies:
- Either you capture a still picture as soon as you need one; or
- You capture continuously the video buffer
Now, the first approach is the simplest one: whenever you need to know what the camera is looking at, you just shoot a picture. Instead, if you want to process the video buffer, that’s more tricky, especially if your image processing algorithm is slower than the camera framerate output. Here, you need to evaluate which solution is more suitable for you case. Take into account that depending on the device you can get different image resolution. For example, the iPhone 4s can provide images up to 8 mega pixels. Now, that’s a lot of data to process in real-time. So, if you are doing real-time image processing, you need to accept some lower quality images. But all these considerations are a topic for a next post.
Keep coding,
Geppy
iOS image processing with the accelerate framework
Sometime ago, my friend John Fox asked me how to reproduce a blurring image effect in the iOS SDK. Core Image on the iOS does not provide that effect. You can find in the Internet a couple of solutions for the iOS performing the convolution as matrix multiplication. That’s an ok approach, but it does not take advantage of the hardware acceleration.
I show here briefly how to apply a blurring filter to any image using the Accelerate framework and vImage.
Before iOS 5, image processing on the iOS was hard. You were required to build your own set of basic tools to perform convolution, fft, dct, scaling, rotation, histogram equalization and so on. That was amazing difficult and time consuming, especially because you had to keep in mind the hardware limitations of your device. Many developers opted to send an image to a remote server, process it and send the result back to the iPhone. Obviously, that solution was only suitable for special and limited cases. If you wanted real-time processing, then you had to really fight hard against the clock cycles.
Nowadays, iOS 5 allows you to do image processing operation on board of your device. You can achieve this either using the Core Image or the Accelerate framework. Both frameworks were already available on the Mac, but only recently they shipped with iOS.
Core Image offers a set of predefined image processing filters. Unfortunately, differently than the Mac version, the iOS version does not offer the possibility to build custom filters. Additionally, it does not provide a blurring filter (and that’s what my friend John was looking for). So, the alternative is to use the Accelerate framework. Let’s see how to do it.
vImage
The basic data structure used by the Accelerate framework to process images is vImage. It’s essentially a C structure containing four elements: the image buffer (it can be either the image intensity or the values of the red, green, blue and alpha channels), the image height and width and the number of bytes for each image row.
Any method belonging to the Accelerate framework uses vImage as input and/or output. So, before applying any processing to an image, you have to create a vImage. Core Graphics can help with that. The quickest way to create a vImage structure is the following:
CGImageRef imageCGSource = [[UIImage imageNamed:@"input.png"] CGImage];
Now, you need to extract from this CGImage the pixels that will constitute the buffer of our vImage. You can do that in different ways. In this case, I am showing you the simplest approach, i.e. extracting the pixel intensities from the image (you can also apply a similar approach to colored images).
Here how to do it:
// Compute the image size
size_t width = CGImageGetWidth(image);
size_t height = CGImageGetHeight(image);
// create a reference to a color space
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceGray();
// allocate some memory for the bitmap buffer
Pixel_8 *bitmap = (Pixel_8 *)malloc(width * height * sizeof(Pixel_8));
long bytesPerPixel = 1;
long bytesPerRow = bytesPerPixel * width;
long bitsPerComponent = 8;
// create a context
CGContextRef context = CGBitmapContextCreate(bitmap,
width,
height,
bitsPerComponent,
bytesPerRow,
colorSpace,
kCGImageAlphaNone);
// draw the image on the context
CGContextDrawImage(context, CGRectMake(0, 0, width, height), image);
// create the vImage buffer
const vImage_Buffer srcBuffer = { bitmap, height, width, bytesPerRow };
// release the memory
CGColorSpaceRelease(colorSpace);
CGContextRelease(context);
Now, the srcBuffer is our vImage and it’s ready to be processed.
The convolution can be performed using one of the convolution functions offered by the Accelerate framework. Since we are using a gray level image, I will use here the vImageConvolve_Planar8 function. Here, Planar8 means that the image is treated as a simple matrix with each element representing a pixel intensity.
Before the convolution, you need to prepare (allocate) some memory space for the final result:
size_t width = srcBuffer.width;
size_t height = srcBuffer.height;
size_t bytesPerRow = srcBuffer.rowBytes;
Pixel_8 *outData = (Pixel_8 *)malloc( bytesPerRow * height );
const vImage_Buffer dstBuffer = { outData, height, width, bytesPerRow };
Then, you need to create a blurring filter:
int16_t *kernel = (int16_t *)malloc(size * size * sizeof(int16_t));
int16_t *tempKernel = kernel;
for (int i = 0; i < (size*size); i++) {
*tempKernel++ = 1;
}
And finally, convolve the image and the filter:
vImageConvolve_Planar8(&srcBuffer,
&dstBuffer,
NULL,
0,
0,
kernel,
size,
size,
size*size,
0,
kvImageBackgroundColorFill);
I suppose that you need to display the result somewhere. So, you need to convert the resulting dstBuffer to a UIImage. I am attaching here a simple project. I load an image and apply a blurring filter with different kernel sizes.
I hope my friend John (and you) enjoyed this post.

![A (quasi-) real-time video processing on iOS
In previous posts, I showed you how to create a custom camera using AVFoundation and how to process an image with the accelerate framework. Let’s now combine both results to create a (quasi-) real-time (I’ll explain later what I mean with quasi) video processing.
Custom camera preview
To appreciate what we are going to do, we need to build a custom camera preview. If we want to process a video buffer and show the result in real-time, we cannot use the AVCaptureVideoPreviewLayer as shown in this post, because that camera preview renders the signal directly and does not offer any way to process it, before the rendering. To make this possible, you need to take the video buffer, process it and then render it on a custom CALayer. Let’s see how to do that.
As I already demonstated here, setting the AVFoundation stack is quite straightfoward (thank you, Apple): you need to create a capture session (AVCaptureSession), then a capture device (AVCaptureDevice) and add it to the session as a device input (AVCaptureDeviceInput). Translating this in source code, this becomes:
// Create the capture session
AVCaptureSession *captureSession = [AVCaptureSession new];
[captureSession setSessionPreset:AVCaptureSessionPresetLow];
// Capture device
AVCaptureDevice *captureDevice = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];
// Device input
AVCaptureDeviceInput *deviceInput = [AVCaptureDeviceInput deviceInputWithDevice:captureDevice error:nil];
if ( [captureSession canAddInput:deviceInput] )
[captureSession addInput:deviceInput];
The output buffer
Until here, nothing is new with respct to the previous post. Here, instead, where the new stuffs come in place. First of all, we need to define a video data output (AVCaptureVideoDataOutput) and add it to the session:
AVCaptureVideoDataOutput *dataOutput = [AVCaptureVideoDataOutput new];
dataOutput.videoSettings = [NSDictionary dictionaryWithObject:[NSNumber numberWithUnsignedInt:kCVPixelFormatType_420YpCbCr8BiPlanarFullRange] forKey:(NSString *)kCVPixelBufferPixelFormatTypeKey];
[dataOutput setAlwaysDiscardsLateVideoFrames:YES];
if ( [captureSession canAddOutput:dataOutput]
[captureSession addOutput:dataOutput];
Here, I defined the output format as YUV (YpCbCr 4:2:0). If you don’t know what I am talking about, I suggest you to give a look at this article. YUV or, more correctly, YCbCr is a very common video format and I use it here, because, except when the color brings some usefull information, you usually use graylevel images for image processing. So, the YUV format provides a signal with the intensity component (the Y) and 2 cromatic components (the U and the V).
The destination layer
Additionally, we need to create a new layer and use it as our rendering destination:
CALayer *customPreviewLayer = [CALayer layer];
customPreviewLayer.bounds = CGRectMake(0, 0, self.view.frame.size.height, self.view.frame.size.width);
customPreviewLayer.position = CGPointMake(self.view.frame.size.width/2., self.view.frame.size.height/2.);
customPreviewLayer.affineTransform = CGAffineTransformMakeRotation(M_PI/2);
We can add this layer to any other layer. I’m going to add it to my view controller view layer:
[self.view.layer addSublayer:customPreviewLayer];
Let’s go
The last step of the initial configuration is to create a GCD queue that is going to manage the video buffer and set our class as delegate of the video data output sample buffer (AVCaptureVideoDataOutputSampleBufferDelegate):
dispatch_queue_t queue = dispatch_queue_create("VideoQueue", DISPATCH_QUEUE_SERIAL);
[dataOutput setSampleBufferDelegate:self queue:queue];
Final setup
Now, remeber to add the following frameworks to your project:
AVFoundation
CoreMedia
CoreVideo
CoreGraphics
Video rendering
Since the view controller is now the delegate of the capture video data output, you can implement the following callback:
- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection
AVFoundation fires this delegate method as soon as it has a data buffer available. So, you can use it to collet the video buffer frames, process them and render them on the layer that we previously created. For the moment, let’s collect the video buffer frames and render them on the layer. Later, we’ll give a look at the image processing.
The previous delegate method returns the sampleBuffer of type CMSampleBufferRef. This is a Core Media object we can bring into Core Video:
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
Let’s lock the buffer base address:
CVPixelBufferLockBaseAddress(imageBuffer, 0);
Then, let’s extract some useful image information:
size_t width = CVPixelBufferGetWidthOfPlane(imageBuffer, 0);
size_t height = CVPixelBufferGetHeightOfPlane(imageBuffer, 0);
size_t bytesPerRow = CVPixelBufferGetBytesPerRowOfPlane(imageBuffer, 0);
Remember the video buffer is in YUV format, so I extract the luma component from the buffer in this way:
Pixel_8 *lumaBuffer = CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 0);
Now, let’s render this buffer on the layer. To do so, we need to use Core Graphics: create a color space, create a graphic context and render the buffer onto the graphic context using the created color space:
CGColorSpaceRef grayColorSpace = CGColorSpaceCreateDeviceGray();
CGContextRef context = CGBitmapContextCreate(lumaBuffer, width, height, 8, bytesPerRow, grayColorSpace, kCGImageAlphaNone);
CGImageRef dstImage = CGBitmapContextCreateImage(context);
So, the dstImage is a Core Graphics image (CGImage), created from the captured buffer. Finally, we render this image on the layer, changing its contents. We do that on the main queue:
dispatch_sync(dispatch_get_main_queue(), ^{
customPreviewLayer.contents = (__bridge id)dstImage;
});
Now, let’s do some clean-up (we are good citizens, right?).
CGImageRelease(dstImage);
CGContextRelease(context);
CGColorSpaceRelease(grayColorSpace);
If you build and run, you’ll see the camera in action with your camera preview.
Image processing
Now, let’s start with some funny stuffs. Let’s process the buffer before rendering it. For this, I am going to use the Accelerate framework. The Pixel_8 *lumaBuffer would be the input of my algorithm. I need to convert the this buffer into a vImage_Buffer and prepare a vImage_Buffer for the output of the image processing algorithm.
Add this code after the line generating the lumaBuffer:
...
Pixel_8 *lumaBuffer = CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 0);
const vImage_Buffer inImage = { lumaBuffer, height, width, bytesPerRow };
Pixel_8 *outBuffer = (Pixel_8 *)calloc(width*height, sizeof(Pixel_8));
const vImage_Buffer outImage = { outBuffer, height, width, bytesPerRow };
[self maxFromImage:inImage toImage:outImage];
...
The -maxFromImage:toImage: method does all the work. Just for fun, I process the input image with a morphological operator that minimizes a region of interest within the image. Here it is:
- (void)maxFromImage:(const vImage_Buffer)src toImage:(const vImage_Buffer)dst
{
int kernelSize = 7;
vImageMin_Planar8(&src, &dst, NULL, 0, 0, kernelSize, kernelSize, kvImageDoNotTile);
}
If you now run it, rendering the outImage on the custom preview, you should obtain something like this:
You can download from here the example.
Final considerations
As I mentioned at the beginning of this post, this processing is done in quasi- real-time. The limitiation derives from the accelerate framework. This framework is optimized for the CPU that is anyway a limited resource. Depending on the final application, this limitation could not be important. However, if you start to add more processing before the rendering, you will see what I mean. Again, the result is really dependent on the application, but if you really want to process and display the processed results in real-time, maybe you should think of using the GPU… but this is something for a future post.
Geppy](http://24.media.tumblr.com/tumblr_m4391x5J8H1qae4fpo1_1280.png)
![A crash course on Core Graphics
Core Graphics (CG), a.k.a. Quartz, is an old API included in both Cocoa and Cocoa Touch. It allows you to draw paths, shapes, shadows and gradients on the so-called graphic destinations or Graphic Contexts (we will come back later on this concept).
The idea behind CG is very simple. If you want to modify any view you have to subclass it and override its drawRect: method. Every view has a drawRect: method. So, if you want to customize a view you need to overwrite that method adding your code there, combining paths, shadows and gradients. You can do this with any UIView subclass. In this way, you can customize buttons, labels, maps and, in general, any graphic object descendant of UIView. In a similar way, you can use CG “to paint” on an image or on a PDF file.
You could tell me: “Hey, just a moment! Why should I write a bounce of code, if I have my graphic designer or if I can create my graphics using powerful tools like Photoshop?” Well, that’s right, you don’t need to use CG code to draw everywhere. Moreover, trying to reproduce simple Photoshop effects could be really hard to achieve and would require many lines of code.
However, there are some advantages in using CG instead of importing your very nice prepared graphics. For example, if you need to draw something in realtime that changes dynamically its visual appearance (a rounded square changing its color according to the sound input of the microphone), then you should think of using CG (yeah, you can also use Core Animation for this particular case).
Additionally, CG reduces the application memory footprints (think of the new iPad with the Retina Diaplay). Including every graphic object in your application increases the size of the application bundle. For the retina devices, this size can be quite significative. So, instead of loading images from the memory, you could use CG to draw at runtime your objects.
Another significant advantage of using CG instead of a pre-loaded graphic object is the resolution independence. This means the same source code can run on the iPhone with and without the retina display, on the iPad with and without the retina display and on the Mac with or without HiDPI. All this, with no need to create the same object in three or four different resolutions. Additionally, when you scale an image at runtime, the aliasing effect can make your object looking ugly. CG re-renders the object without this problem.
Now, before we get into the details on how to draw on a view using CG, I need to review with you some basic concepts. To understand CG, you need to understand three basics things: the Graphic Context, the Path and the Graphic State. But, even before that, we need go through how CG “paints” on a view.
There are two approaches: Painter Model and Crayon Model. The Painter Model is the way an artist paints on a canvas. Artists use to draw the background of a scene first and then proceed adding the foreground objects on top of each others until the top-front foreground object is drawn. When we will start using CG, we will follow the same approach. For example, to draw a simple square with a dark shadow in the background, we need to tell CG that you need a shadow with a given shape, color and blur and then tell CG about the foreground square.
This is the sequence to draw a complex set of graphic objects. But what about each single object? In this case, we will use the Crayon Model. Kids usually use this method when painting something on a piece of paper. They usually stroke a shape and than they fill it with some colors. Using CG, you will follow the same approach: you define the shape of an object, first and then you have to color it.
Ok, let’s go now through the above mentioned basic concepts: Graphic Context, Path and Graphic State.
Graphic Context
The Graphic Context is your graphic destination. If you want to draw on a view, the view is your Graphic Context. Instead, if you want to draw on an image, the image is your Graphic Context. So, the first thing you should do, before trying to draw anything is to define the Graphic Context. Each Core Graphics function has a pointer to the current graphic context.
You get the current Graphic Context by using the UIGraphicsGetCurrentContext function (Yes, I said function and not method. This is because CG is a C-based API. Sorry, no ObjC). So, after subclassing your UIView, you override the drawRect: method (don’t forget to execute the method on super) and get the current graphic context. In this way:
- (void)drawRect:(CGRect)rect {
[super drawRect:rect];
CGContextRef ctx = UIGraphicsGetCurrentContext();
...
}
Now that we have a reference to the current Graphic Context, we could start to draw.
Path
Each drawing is composed of some basic graphic elements: points, lines, arcs. All of them are referred to as Paths. A path is a set of lines, arcs and curves you can draw on the current graphic context to build complex objects. Only when the path description is completed, you can draw the path on the screen and choose to fill it or stroke it or both.
Core Graphics provides you with specific functions to draw paths. Some of these functions are specific and can be used to draw a line, a rectangle, an ellipse, an arc, etc.
Graphic State
Before we move to a practical example, the last concept you should know is related on how Quartz draws on the graphic destination. The Graphic State keeps track of the colors, the size of the path, the transformations and many other features you want to apply to a path or group of paths.
States are maintained in stack. You save the state using CGContextSaveCGState. In this way, you push the graphic state onto the stack. When you want to reload the previous graphic state, you use CGContextRestoreCGState. In this way, the state is popped from the stack.
Hands-on
Let’s see now how to prepare our development environment. Let’s create a basic iOS project. I prefer to use the iPhone template for this exercise, but feel free to use the iPad template, if you prefer.
Launch Xcode and create a Single-View Application project. Call it DigitalPicasso and use ARC. Once the project window opens, open the ViewController.xib file. This view controller has an associated view that we are going to use as our Graphic Context. To do so, we’d need to override its drawRect: method as it has been previously discussed. Select the view in the ViewController.xib file and change the Class field to MyView in the Indentity Inspector.
Now, let’s add the class files. Select File -> New -> Files.... In the window panel, select Cocoa Touch -> Objective-C class and press the Next button. Give “MyClass” as name and UIView as “Subclass of”. Click Next and save the files in the DigitalPicasso folder.
Drawing paths
Now, you have two new files in the Xcode project (MyView.h and MyView.m). Open the MyView.m and add the following code:
- (void)drawRect:(CGRect)rect
{
[super drawRect:rect];
CGContextRef ctx = UIGraphicsGetCurrentContext();
CGContextBeginPath(ctx);
CGContextMoveToPoint(ctx, 100.0, 100.0);
CGContextAddLineToPoint(ctx, 200.0, 200.0);
CGContextAddLineToPoint(ctx, 200.0f, 300.0f);
CGContextAddLineToPoint(ctx, 100.0f, 300.0f);
CGContextClosePath(ctx);
CGContextStrokePath(ctx);
}
Now, just Run. If everything was done correctly, you get the same result shown in the following figure.
Let’s give a look at what we have done. After executing the drawRect: method on super, we get the current Graphic Context using the UIGraphicsGetCurrentContext function. The pointer ctx is very important, because we need to pass it as first argument to the Core Graphics functions. After this, we tell CG we want to begin a new path. Take into account that nothing is drawn on the screen until the last function CGContextStrokePath is executed. The rest of the code is just a preparation. CGContextMoveToPoint is used to place your pencil on a selected point of the screen. After that, you start to add lines to the path, using the CGContextAddLineToPoint until you finally close the path. Using the CGContextClosePath you connect the last point of your path with the first one. For Core Graphics a path can be a line, a rectangle, an arc or an ellipse. Once you have create a path, you can either stroke it or fill in or both. Here, I write for you some functions that could be useful to create a path. Give a look at them and check the documentation. In the same doc page, you will find other useful functions organized by type.
CGContextBeginPath - creates a new path
CGContextAddArc - creates an arc
CGContextAddEllipseInRect - creates an ellipse that fits inside a rectagle
CGContextAddLineToPoint - creates a line from the current point to the specified point
CGContextAddRect - creates a rect
CGContextMoveToPoint - moves to a point without drawing
CGContextClosePath - draws from the last point to the first one to close the path
CGContextFillPath - closes the path and fills it
CGContextStrokePath - strokes the path
CGContextClearRect - erases a rect
CGContextFillRect - fills a rect
CGContextStrokeRect - strokes a rect
CGContextStrokeRectWithWidth - strokes a rect with a specific line width
Let’s play a little bit with these functions. After the CGContextStrokePath function, add the following lines of code:
CGContextBeginPath(ctx);
CGContextAddArc(ctx, 100.0, 100.0, 100.0, 0, 2*M_PI, 0);
CGContextStrokePath(ctx);
Build and Run. Cool, right? Now, try to use the previous functions. Next time, I will show you how to create reusable paths and change the colors of your pencil.
Again on the Graphic States
The only problem we have now is that everything you try to draw on the screen has the same color, the same pencil size and type. A little bit borying, right? This means that everything is in the same graphic state. Obviuosly, we can modify this. When you change anything related with the status of your environment (colors, type of line, line width and so on), it is a good idea to save the current state before applying any change. In this way, you can always go back without redefining all the possible features. To store the current graphic state you use the CGContextSaveCGState function. Instead, use CGContextRestoreCGState is used to to revert to the previous state. Graphic states are stored in a LIFO (Last In - First Out) stack. So, every time you save a state, this is added onto the stack. If you save 2 times the state, if you want to restore the first state, you need to restore the state 2 times. If you need to go back to the previous state, then you simply restore the previous state using the CGContextRestoreCGState function.
Colors
Let’s try to save the state, change the pen colors and revert to the previous state. Before that, let’s talk a little bit about how to manage colors in Core Graphics. There is a set of Core Graphics functions you can use to modify the color of your pen. However, I suggest you to use UIKit as much as you can, since UIKit has very easy methods to create colors. You can then use the CGColor method to convert a UIColor to a CGColor. For example, you can simply use this line of code to create a red CG color:
CGColorRef aColor = [[UIColor redColor] CGColor];
This is the most convenient way to create a CG color. The other approach is to use one of the following Core Graphics functions:
CGContextSetRGBFillColor sets an RGBA color to fill a path
CGContextSetRGBStrokeColor sets an RGBA color to stroke a path
CGContextSetCMYKFillColor sets a CMYK color to fill a path
CGContextSetCMYKStrokeColor sets a CMYK color to stroke a path
CGContextSetFillColorWithColor sets a fill to a CGColor
CGContextSetStrokeColorWithColor sets a stroke color to a CGColor
So, let’s modify our DigitalPicasso project in this way.
- (void)drawRect:(CGRect)rect {
[super drawRect:rect];
CGContextRef ctx = UIGraphicsGetCurrentContext();
CGContextBeginPath(ctx);
// Add path
CGContextMoveToPoint(ctx, 100.0, 100.0);
CGContextAddLineToPoint(ctx, 200.0, 200.0);
CGContextAddLineToPoint(ctx, 200.0f, 300.0f);
CGContextAddLineToPoint(ctx, 100.0f, 300.0f);
CGContextClosePath(ctx);
CGContextStrokePath(ctx);
//Create circle and fill it in
CGContextAddArc(ctx, 200, 100.0, 50.0, 0, 2*M_PI, 0);
CGContextSetFillColorWithColor(ctx, [[UIColor whiteColor] CGColor]);
CGContextFillPath(ctx);
CGContextAddArc(ctx, 200, 100.0, 50.0, 0, 2*M_PI, 0);
CGContextSetStrokeColorWithColor(ctx, [[UIColor redColor] CGColor]);
CGContextStrokePath(ctx);
}
You should get something similar to this:
Now, the problem is that if you want to go back to the black pen, you need to set again the black color. In this case, it is very simple, you need just to add a line of code. But what happens if you want to change again to many other pen features? Here where the states help you. So, before changing the black pen to a white one, as we did in the previous code, we should save the current state. Give a look at the following code:
- (void)drawRect:(CGRect)rect {
[super drawRect:rect];
CGContextRef ctx = UIGraphicsGetCurrentContext();
CGContextBeginPath(ctx);
// Add path
CGContextMoveToPoint(ctx, 100.0, 100.0);
CGContextAddLineToPoint(ctx, 200.0, 200.0);
CGContextAddLineToPoint(ctx, 200.0f, 300.0f);
CGContextAddLineToPoint(ctx, 100.0f, 300.0f);
CGContextClosePath(ctx);
CGContextStrokePath(ctx);
// save the state
CGContextSaveGState(ctx);
//Create circle and fill it in
CGContextAddArc(ctx, 200, 100.0, 50.0, 0, 2*M_PI, 0);
CGContextSetFillColorWithColor(ctx, [[UIColor whiteColor] CGColor]);
CGContextFillPath(ctx);
CGContextAddArc(ctx, 200, 100.0, 50.0, 0, 2*M_PI, 0);
CGContextSetStrokeColorWithColor(ctx, [[UIColor redColor] CGColor]);
CGContextStrokePath(ctx);
//restore the state
CGContextRestoreGState(ctx);
CGContextAddArc(ctx, 200, 300.0, 50.0, 0, 2*M_PI, 0);
CGContextStrokePath(ctx);
}
So, we saved the state and then, we restored it. And this is the result:
Now, you see the new circle is again a black stroke, because we simply restored the previous CG state.
Extra settings
Beside the color you can also change other pen features. For example, you can change the pen size using CGContextSetLineWidth. Add this line, before line 20 of the previous code.
CGContextSetLineWidth(ctx, 10);
You get a thicker circle stroke as in the following figure. You can also change the transparency of your pen, using CGContextSetAlpha. Try this code and see what happens:
CGContextSetAlpha(ctx, .3);
Reusable paths
Reusable paths are very interesting Core Graphics tools, since they can help you saving a lot of time when you try to draw complex objects on the screen. A reusable path is a path you create just once and then, you can quickly and easily apply it later. Using reusable paths is very simple. You need to use CGPath functions in place of the CGContext ones. First, create a CGPathRef using the CGPathCreateMutable function. Then, add paths to the CGPathRef and finally close the path using CGPathCloseSubpath. To draw the path in a context you need to use CGContextAddPath. The following table shows the equivalence between reusable and not-reusable path functions:
CGPathCreateMutable <- CGContextBeginPath
CGPathAddArc <- CGContextAddArc
CGPathAddEllipseInRect <- CGContextAddEllipseInRect
CGPathAddLineToPoint <- CGContextAddLineToPoint
CGPathAddRect <- CGContextAddRect
CGPathMoveToPoint <- CGContextMoveToPoint
CGPathCloseSubpath <- CGContextClosePath
So, you can now try to convert the code we wrote untill now to use resuable paths.
- (void)drawRect:(CGRect)rect {
CGMutablePathRef myPath = CGPathCreateMutable();
CGPathAddArc(myPath, NULL, 110, 50, 30, 0. 2*M_PI, 1, YES);
...
CGContextRef ctx = UIGraphicsGetCurrentContext();
CGContextBeginPath(myPath);
CGContextAddPath(myPath);
CGContextStrokePath(myPath);
CFRelease(myPath);
}
As you can see in the above example, I created a new reusable path. Later, whenever I need to draw that path again, I simply use the reference myPath. Reusable paths are very useful when you want to draw different layers with the same shape. For example, if you want to create a square with a shadow, you need first to create a square shadow (do you remember the Crayon model?) and then, draw your square path on top of the shadow. If you use reusable paths, this is very easy: you just need to create the reusable square and draw it once as a shadow (I will show you later how to do that) and once again as a squared shape.
Color Gradients
A color gradient is a smooth transition from any color to a different one. This transition makes your drawings more realistic and look gorgeous. There are essentially 2 ways to draw a gradient in Core Graphics. You can either use the CGGradientRef object or the CGShadingRef object. The first approach is very simple and allows you to draw linear gradients. The second approach is a bit more complex, since the user has to specify a CGFunctionRef object to draw a gradient with any shape. So, to draw a color gradient with CG, you need to:
Define a color space
Define the list of colors you want to draw
Define the locations where you want the colors appear
Draw (finally) your gradient; and, finally
Free up your memory
Color space
So, first let’s give a look at the color spaces. A color space specifies how color values are interpreted by the graphic engine. A color space is usually a multi-dimensional space, each dimension of which represents a specific color component. Defining a color space in Core Graphics is quite straightforward. Give a look at the class CGColorSpace. I usually use the CGColorSpaceCreateDeviceRGB function, but you can use any of the other functions provided in the class.
Memory management
Just a small interruption here, since I missed an important point until now. The memory management in Core Graphics follows the simple rules of the retain count mechanism. Sorry, no ARC. You would ask me: “Ehy, wait a minute didn’t you say CG is C-based? So, no objects?”. You are right. And indeed, we are not using objects. However, it is really common to call objects the CG references. These are the variables of a type ending with word Ref. For example, when you create a color space, you create a reference (a pointer) of type CGColorSpaceRef. This is simply a pointer, but we usually call it an object. Now, whenever you create a CG object (or in general, a Core Foundation object) with a function whose name contains the word Create or Copy, you are responsible to release that object. In general, you can use the Core Foundation function, CFRelease. However, in the case of the color space, there is a specialized function for that: CGColorSpaceRelease. So, if you create a color space with the above mentioned functions, you need to release it as show in this chunk of code:
CGColorSpaceRef colorSpace = CGColorSpaceCreateWithName(kCGColorSpaceGenericRGB);
// other code here
CGColorSpaceRelease(colorSpace);
Gradient example
So, let’s see how we can create a linear gradient, first. Let’s color the whole screen. So, open Xcode, create a new View-based project, create a new class subclassing UIView and add a view to the main view of your custom view controller. In the MyView class, edit the drawRect: method as follows:
- (void)drawRect:(CGRect)rect
{
[super drawRect:rect];
CGContextRef ctx = UIGraphicsGetCurrentContext();
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
CGFloat components [12] = {
1.0, 0.0, 0.0, 1.0,
0.0, 0.0, 0.0, 1.0,
0.0, 1.0, 0.0, 1.0
};
CGFloat locations[3] = { 0.0, 0.5, 1. };
CGGradientRef gradient = CGGradientCreateWithColorComponents(
colorSpace,
components,
locations,
(size_t)3);
CGContextDrawLinearGradient(
ctx,
gradient,
CGPointMake(320, 0),
CGPointMake(0, 460),
(CGGradientDrawingOptions)NULL);
CGColorSpaceRelease(colorSpace);
}
So, what is this chunk of code? First, we get the reference of the current graphic context. Then, we create a reference to the color space. I decided to use the RGB color space. This multi-dimentional space has 4 components: Red, Green, Blue and Alpha (the transparency). Each component has a value between 0 and 1. After creating the color space, we create the color components. This is a vector containing the 2 or more colors you want to use in your gradient. In the above example, I used just 3 colors. The first component is (R, G, B, A) = (1, 0, 0, 1). The second component is (R, G, B, A) = (0, 0, 0, 1). And the third component is(R, G, B, A) = (0, 1, 0, 1).
Now, we need to tell CG where to draw these colors. To draw the linear gradient, we use the CGContextDrawLinearGradient function. This function asks to specify the 2 points in the view where to the gradient starts and ends. You can see I created 2 CGPoints. Now, imagine to connect these 2 points with a segment. The aim of the locations vector is to define the percentage of that segment where the components are placed along it. In the above example, the first color is at the 0% of the segment (i.e., the beginning), the second component is at the 50% (just in the middle of the segment) and the last color component is just at the end of the segment. If you now run your project, you should get this:
Very cool, eh? Now, imagine to combine this with shadow and transparencies. You can make really cool stuffs. Try to play a little bit with this and apply it to the paths. Our approach is to combine the graphics prepared by our graphic designer and the real time drawing of Core Graphics. You can make really beautiful pieces of arts.](http://25.media.tumblr.com/tumblr_m2eon1CWU51qae4fpo1_1280.png)

![A very cool custom video camera with AVFoundation
AVFoundation is a very cool framework that allows you to collect multimedia data generated by different input sources (camera, microphone, etc.) and redirect them to any output destinations (screen, speakers, etc.).
You can create custom playback and capture solutions for audio, video and still images. The advantage of using this framework with respect to the out-of-the-shelf solutions such as the MPMoviePlayerController or the UIImagePickerController is that you get access to the raw data of the camera. In this way, you can apply effects in real-time to the input signals for different purposes.
I have prepared for you a small app to show you how to use this framework and create a very cool video camera.
Session
AVFoundation is based on the concept of the session. A session is used to control the flow of the data from an input to an output device. The creation of a session is really straightforward:
AVCaptureSession *session = [[AVCaptureSession alloc] init];
The session allows you to define the audio and video recording quality, using the sessionPreset property of the AVCaptureSession class. For this example, it’s fine to go for low quality data (so we save some battery cycle):
[session setSessionPreset:AVCaptureSessionPresetLow];
Capture Device
After the capture session has been created, you need to define the capture device you want to use. It can be the camera or the microphone. In this case, I am going to use the AVMediaTypeVideo type that supports videos and images:
AVCaptureDevice *audioCaptureDevice = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];
Capture Device Input
Next step you need to define the input of the capture device and add it to the session. Here you go:
AVCaptureDeviceInput *deviceInput = [AVCaptureDeviceInput deviceInputWithDevice:inputDevice error:&error];
if ( [session canAddInput:deviceInput] )
[session addInput:deviceInput];
You check if you can add the device input to the session and if you can, you add it.
Preview
Before defining the device output, I want to show you how to preview the camera buffer. This will be the viewfinder of your camera, i.e. the preview of what the input device is seeing.
We can quickly render the raw data collected by the camera on the screen using the AVCaptureVideoPreviewLayer. We can create this preview layer using the session we defined above and then add it to our main view layer:
AVCaptureVideoPreviewLayer *previewLayer = [[AVCaptureVideoPreviewLayer alloc] initWithSession:session];
[previewLayer setVideoGravity:AVLayerVideoGravityResizeAspectFill];
CALayer *rootLayer = [[self view] layer];
[rootLayer setMasksToBounds:YES];
[previewLayer setFrame:CGRectMake(-70, 0, rootLayer.bounds.size.height, rootLayer.bounds.size.height)];
[rootLayer insertSublayer:previewLayer atIndex:0];
You don’t need to do any additional work. You can now display the camera signal on your screen.
If you instead want to do some more cool stuffs, for example, if you want to process the camera signal to create nice video effects with Core Image or the Accelerate framework (give a look at this post), you need to collect the raw data generated by the camera, process them, and, if you like it, display them on the screen.
Go baby, go!!!
We are ready to go. The last thing you need to do is to start the session:
[session startRunning];
Cool stuffs
Since the AVCaptureVideoPreviewLayer is a layer, you can obviously add animations to it. I am attaching here a very simple Xcode project showing the previous concepts. It creates a custom video camera with the preview rotating in the 3D space.
Real-time processing
If you want to do some image processing with the raw data captured by the the camera and display the result on the screen, you need to collect those data, process them and render them on the screen without using the AVCaptureVideoPreviewLayer. Depending on what you want to achieve, you have two main strategies:
Either you capture a still picture as soon as you need one; or
You capture continuously the video buffer
Now, the first approach is the simplest one: whenever you need to know what the camera is looking at, you just shoot a picture. Instead, if you want to process the video buffer, that’s more tricky, especially if your image processing algorithm is slower than the camera framerate output. Here, you need to evaluate which solution is more suitable for you case. Take into account that depending on the device you can get different image resolution. For example, the iPhone 4s can provide images up to 8 mega pixels. Now, that’s a lot of data to process in real-time. So, if you are doing real-time image processing, you need to accept some lower quality images.
But all these considerations are a topic for a next post.
Keep coding,
Geppy](http://24.media.tumblr.com/tumblr_m02z6blEv61qae4fpo1_1280.png)
![iOS image processing with the accelerate framework
Sometime ago, my friend John Fox asked me how to reproduce a blurring image effect in the iOS SDK. Core Image on the iOS does not provide that effect. You can find in the Internet a couple of solutions for the iOS performing the convolution as matrix multiplication. That’s an ok approach, but it does not take advantage of the hardware acceleration.
I show here briefly how to apply a blurring filter to any image using the Accelerate framework and vImage.
Before iOS 5, image processing on the iOS was hard. You were required to build your own set of basic tools to perform convolution, fft, dct, scaling, rotation, histogram equalization and so on. That was amazing difficult and time consuming, especially because you had to keep in mind the hardware limitations of your device. Many developers opted to send an image to a remote server, process it and send the result back to the iPhone. Obviously, that solution was only suitable for special and limited cases. If you wanted real-time processing, then you had to really fight hard against the clock cycles.
Nowadays, iOS 5 allows you to do image processing operation on board of your device. You can achieve this either using the Core Image or the Accelerate framework. Both frameworks were already available on the Mac, but only recently they shipped with iOS.
Core Image offers a set of predefined image processing filters. Unfortunately, differently than the Mac version, the iOS version does not offer the possibility to build custom filters. Additionally, it does not provide a blurring filter (and that’s what my friend John was looking for). So, the alternative is to use the Accelerate framework. Let’s see how to do it.
vImage
The basic data structure used by the Accelerate framework to process images is vImage. It’s essentially a C structure containing four elements: the image buffer (it can be either the image intensity or the values of the red, green, blue and alpha channels), the image height and width and the number of bytes for each image row.
Any method belonging to the Accelerate framework uses vImage as input and/or output. So, before applying any processing to an image, you have to create a vImage. Core Graphics can help with that. The quickest way to create a vImage structure is the following:
CGImageRef imageCGSource = [[UIImage imageNamed:@"input.png"] CGImage];
Now, you need to extract from this CGImage the pixels that will constitute the buffer of our vImage. You can do that in different ways. In this case, I am showing you the simplest approach, i.e. extracting the pixel intensities from the image (you can also apply a similar approach to colored images).
Here how to do it:
// Compute the image size
size_t width = CGImageGetWidth(image);
size_t height = CGImageGetHeight(image);
// create a reference to a color space
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceGray();
// allocate some memory for the bitmap buffer
Pixel_8 *bitmap = (Pixel_8 *)malloc(width * height * sizeof(Pixel_8));
long bytesPerPixel = 1;
long bytesPerRow = bytesPerPixel * width;
long bitsPerComponent = 8;
// create a context
CGContextRef context = CGBitmapContextCreate(bitmap,
width,
height,
bitsPerComponent,
bytesPerRow,
colorSpace,
kCGImageAlphaNone);
// draw the image on the context
CGContextDrawImage(context, CGRectMake(0, 0, width, height), image);
// create the vImage buffer
const vImage_Buffer srcBuffer = { bitmap, height, width, bytesPerRow };
// release the memory
CGColorSpaceRelease(colorSpace);
CGContextRelease(context);
Now, the srcBuffer is our vImage and it’s ready to be processed.
The convolution can be performed using one of the convolution functions offered by the Accelerate framework. Since we are using a gray level image, I will use here the vImageConvolve_Planar8 function. Here, Planar8 means that the image is treated as a simple matrix with each element representing a pixel intensity.
Before the convolution, you need to prepare (allocate) some memory space for the final result:
size_t width = srcBuffer.width;
size_t height = srcBuffer.height;
size_t bytesPerRow = srcBuffer.rowBytes;
Pixel_8 *outData = (Pixel_8 *)malloc( bytesPerRow * height );
const vImage_Buffer dstBuffer = { outData, height, width, bytesPerRow };
Then, you need to create a blurring filter:
int16_t *kernel = (int16_t *)malloc(size * size * sizeof(int16_t));
int16_t *tempKernel = kernel;
for (int i = 0; i < (size*size); i++) {
*tempKernel++ = 1;
}
And finally, convolve the image and the filter:
vImageConvolve_Planar8(&srcBuffer,
&dstBuffer,
NULL,
0,
0,
kernel,
size,
size,
size*size,
0,
kvImageBackgroundColorFill);
I suppose that you need to display the result somewhere. So, you need to convert the resulting dstBuffer to a UIImage.
I am attaching here a simple project. I load an image and apply a blurring filter with different kernel sizes.
I hope my friend John (and you) enjoyed this post.](http://24.media.tumblr.com/tumblr_lz0gys47mV1qae4fpo1_1280.png)