Nokia 808 PureView in focus: An interview with Damian Dinning

Nokia 808 PureView in focus: An interview with Damian Dinning Introduction The Nokia 808 PureView certainly made a splash at this year's Mobile World Congress, pinning the company back on the cameraphone map in an extremely advantageous position. If you've been living in a cave these past few days, you might have missed some of the juicy details about the Nokia 808 and what makes it so special. So we'll make a quick recap. The Nokia 808 PureView has a 4-inch AMOLED display with the rather lowly resolution of 360 x 640 pixels. Spiritually succeeding the Nokia N8, it runs Nokia Belle (previously known as Symbian) and it's centered around a 1.3 GHz processor. While this smartphone might technically revolve around that processor, what's got the publics imagination is actually the never-before-seen 41MP multi-aspect camera sensor.

Capable of shooting full-resolution shots of up to 38 megapixels and utilizing the industry's first pixel oversampling technology for delivering vastly superior lower-resolution photos (5MP and 8MP), the Nokia 808 camera is a “quantum leap forward in cameraphone performance,” as Nokia themselves put it. Another industry's first is the lossless digital zooming in less-than-maximum resolution shots and video. The garnishes on this excellent dish are the Carl Zeiss lens, the xenon flash, a relatively large F2.4 aperture, the mechanical shutter coupled with a neutral density filter, and there's also, of course, FullHD video recording. We had Damian Dinning, Head of Imaging Experience - Nokia Smart Devices, to tell us more about his team's baby and how it came to be. Damian has played a key role in the development of iconic devices such as the Nokia N95, Nokia N82, Nokia N86 8MP, and N8, among others, and he's also head of the team that delivered the Nokia 808. We did delve into the inner workings of the Nokia 808 PureView 41MP camera sensor yesterday, however we are more than eager to hear the story behind the product from the man himself. We rarely do interviews but the Nokia 808 is a real whopper. The sheer hype it generated yesterday almost took our site down for a moment. The camera enthusiast inside of us can't get enough of it either and Damian Dinning is the man to see to learn everything about this exciting new smartphone (and about cameraphones in general). The story behind the Nokia 808 PureView

GSMArena: The Nokia 808 PureView is currently the hottest topic on our website. We are interested in every detail. Please tell us how the project came about. D. Dinning: It all started about five years ago. We wanted to do optical zoom—at this time we thought it necessary. When we were thinking about the future development of the N93 and N93i, we could see that the expectation of the performance of such a device would be high. We didn't feel that we could achieve the right balance of a product that was small enough with the performance that we thought would be required in the future, after the N93. You could have good image quality, but the device would be really big, you could make it smaller but the image quality was lousy. Whichever way you looked at it...we actually had a number of concepts which were taken quite far in the development stage. We looked at folded optics for example, we looked at straight optics; various different configurations, and we couldn't make any of those work well enough. It was clear to us that we were going to have to do something quite different. One of our guys was reading an article on satellite imaging, and he found it kind of inspiring how satellite imaging was done, where you have a very high resolution image, and you only use pieces of it. He wondered: could we use an idea along those lines, and actually zoom? And they actually created one idea, which wasn't taken forward at that particular time. Later on, when we were thinking about this problem again, we connected to the original idea, in a way that the original idea then became improved upon. We did some very crude sketching, and we realized that there were the possibilities to actually do zooming with a very high resolution image. Originally, we wanted to do a 38 megapixel sensor with a 4:3 aspect ratio. Then, when we were developing it in the early days, we had this idea to do this extended 16:9, like we've done on the N9, the Lumia 800 and now the 900. And that meant we had to increase it from 38 to 41 megapixels, to give it that extended capability. That's how we came to the idea, and it was fundamentally driven by what things we were able to do. Now along the way, we realized that there would be other benefits of using this very high resolution sensor, other than just zoom. One of those is the oversampling capability. This is where, when you are using the default 5 MP setting, we take up to 7 pixels, take the information that you want from those pixels, and process that using some Nokia algorithms. We are then able to throw away most of the visual noise that you don't want, and what that leaves you with are pixels that are extremely precise, so you see far more detail with a 5 MP image than you could capture with any other device, when you're using no zoom. That's when you get the best quality—when you're not using zoom, because you get all the benefits from the oversampling. So in good lighting conditions, typically you won't see any noise at all. They look like SLR images. As you zoom, the oversampling reduces, so when you reach the maximum zoom, it will no longer apply any oversampling. We have a very good sensor, and we want to preserve as much of the information as possible. The noise will probably be no worse than N8's was in daylight. In low-light, it's significantly better than N8. Even in full zoom, we have better per pixel quality than on the N8. GSMArena: But you have more pixels than on the N8 and the pixels in the Nokia 808 are smaller. How are you able to achieve better per pixel quality? D. Dinning: Partially because of the larger F2.4 aperture. That gives us 50% more light. It's also a later generation sensor, and we also have some additional processing that goes on some of the data that comes straight from that sensor, and that gives us a cleaner output. GSMArena: But why has no one else done this? Why has no one else produced a 41MP sensor? Is it that expensive? D. Dinning: It's not cheap, but when you compare it to optical zoom, you would have to put more cost into the optics. You'd have moving components. You'd also have a device that becomes potentially more fragile, and bigger. There are a number of other advantages as well, for example with typical optical zoom, as you zoom, you end up with 5.6 aperture for example, so that means we have 5.4 times more light reaching our sensor with our maximum zoom versus an optical zoom. GSMArena: Is there something which ties this product with this particular operating system? D. Dinning: In theory, no. When we started this project some time ago, it was initially a technology development, then it became a product development as well, and switching operating systems would have delayed deployment. Our thinking was that there's a lot of people out there with N8's, for example, who want a follow-up device, and we couldn't see any rational argument for not doing that. I think it's a really great smartphone. GSMArena: At one stage or another, most companies stopped actively pushing the boundaries of what was possible with mobile imaging. Currently Nokia seems to be the only company pushing in this direction. Are you concerned about not having any competition? D.Dinning: I obviously can't guess or speculate what our competitors are going to be able to do. Our intent is to lead in the area of imaging, and this is one way that we can do that. I think we demonstrated today quite firmly that we can differentiate in this area. How long it would take for anybody to redeploy that technology, I honestly don't know. There are not many people that can manufacture optics at that level of precision, and you need optics, otherwise you just can't make it work. This is by far and away the highest precision optics we've ever introduced. D. Dinning: Yes, it is more complicated than binning. We looked at a number of different methods how we could do this, and some methods preserve more detail, but there are generally a couple of problems here. One is that there's just something not right about the images when you use binning. I won't use the term natural, the term that we use is they are not pure. When you look at the picture, you just see the pixels that relate to the exact detail. What you end up with is you never have strong contrast and there's always a certain softness between pixels—you see three or four pixels leading from black to white. You never see black then white pixels next to each other. Whereas with oversampling, typically you can see pixels literally switch from one color to another. It's really clean. GSMArena: We noticed that the 808's PureView mode only refers to the lower resolution images, which are oversampled and then downsized? D. Dinning: What we wanted to be able to do is make it easy for people to understand the difference between these two camera modes. You have PureView where you get the oversampling and zooming capability, as opposed to the full resolution capability. We are interested to see how people will use the different resolution options. One way we think people will make use of that, is that they'll take pictures quickly, and then do the composition and framing later. Another way would be, you can take a picture, and then reframe it or crop it in multiple ways. It gives you that flexibility when you need it. Whereas with optical zoom, once you take it, you're done, and you can't really change it other than with a little bit of extra cropping. We're intrigued to see if people will put that into use. Maybe they'll do it just for fun shots, maybe some people will do it all the time, which is why we made that setting quite accessible. GSMArena: If we only needed a 2MP image, for instance, we would traditionally take a full resolution shot, and then downsample it with Photoshop. Does PureView provide downsampling algorithms superior to those of Photoshop or other image editing software, effectively eliminating this step? D. Dinning: We haven't really made that comparison. What I can say is, depending on the interpolation and downsampling you use in Photoshop, it may be possible that you'll get similar performance. But you're then handling the JPG file that was saved, so you're probably better off doing it at point of capture. I think you'll find that we probably have a different balance to what you achieve with Photoshop. How we optimize the algorithms is to retain as much of the detail as possible that we think is represented in the object, but also filter as much of the noise as possible. In Photoshop, you typically might get a sharpness that looks higher, but you might get more noise, for example. We struck a slightly different balance when we use our algorithms. GSMArena: So PureView technology essentially allows you to skip shooting in full resolution and relying on post-cropping on a computer, saving yourself from filling your memory card with large files? D.Dinning: Yes, you won't get any benefit by shooting in full resolution and downsizing on a computer, other than the flexibility of being able to crop whatever portions of the image you like later.

D. Dinning: As far as I know, the way of doing it this way is unique to us. The companion chip takes the information directly from the sensor and scales it, before that data goes to the GPU. The GPU has less load because of this scaling chip. What that means in practice is that in video we are able to handle over one billion pixels per second. GSMArena: Does PureView technology help in any way with image stabilization, in video for example? D. Dinning: We do have a digital stabilizer for video, which we are still tuning at the moment. We had to modify the algorithm a little bit, but otherwise there weren't any additional significant challenges for us, other than, obviously, handling many more pixels. GSMArena: Is it actually working on the units demoed at the booth? D.Dinning: Not very well at this stage. It's one of the last things we are going to tune. GSMArena: Will you be able to shoot stills while shooting video? D. Dinning: No, we're not supporting that at this stage. GSMArena: To use zooming in video you either use the volume key or drag your finger across the screen. Will there be an option to set the rate of the slide zoom? D. Dinning: Unfortunately not. That's something that we wanted to be able to do, but we have some technical limitations. One thing we are looking are looking at is to fine tune the speed of the zoom. I think we're going to make it slower than it currently is. It's a bit too fast. We're going to try a number of different speeds, and then pick what we think is the best balance of smoothness and speed. There is no technical limitation for us to be able to adjust the speed; the limitation is for us to be able to vary the speed. GSMArena: When you zoom digitally, you typically need high resolution just at the center, at the expense of the corners. Why are you using a grid with uniform resolution? Wasn't it better to have a lower resolution in the corners and a higher resolution at the center? D. Dinning: That was something that we actually tried. In the end, we wanted to have really high sharpness across the entire image. The original theory was that we could have a higher resolution center, but when we started to think more about the opportunities that people may have with a full resolution sensor, we decided to make the performance at such a level that you could get edge-to-edge sharpness, so that if you wanted to get view full resolution images, you'll still get good image quality when you zoom to the edges. GSMArena: There was an issue in the N8 we discovered where not every step of the zoom is of equal quality. Is that a problem with the 808? D. Dinning: We don't have that problem here. You'll have degradation when you don't oversample, which you'll only really see in low light, otherwise you probably won't see it. When you zoom from minimum to maximum, you may still see that, because we have to change the sensor mode. The sensor and the scaling run in different combinations, and we're trying to mask that as much as possible, so that it's not as noticeable as in the N8. An engineering feat

GSMArena: Let's talk about the optics. Carl Zeiss, right? D. Dinning: Carl Zeiss, yes. The [manufacturing] precision is ten times greater than with SLR [lenses]. There are five lenses, quite small at the top, and then get progressively larger as they approach the sensor. There's also a mechanical ND filter which can be controlled in Creative mode, similar to the N8. In addition to the precision of the optics, one of the things we've been working on for some time is precision alignment. We have a live image sensor setup when we assemble the optics, so that each and every single module we make is built and then assembled with live feedback from the sensor to ensure proper alignment. Obviously, there are always differences. There are five different elements inside each optics module and each set of five elements will be different from other sets. That's why we've implemented very small manufacturing tolerances to make them as identical as possible. By aligning them actively we can get the very best from them. The optics would already have to be within the minimum tolerances that we define, but even if there are differences, we would align the lenses to make the best of each specific set. Fundamentally, what that means is that we can get the very best from each and every one. If we don't do that, we can't achieve the performance in the optics that we need to on a mass production scale. And that was one of the greatest challenges for mass production. GSMArena: Are you guys the only one using this live sensor production method? D. Dinning: I don't know anyone else who uses that manufacturing technique. The sensors come from one company, the optics from another, and it's in the assembly of the module that that alignment process happens. The design is so complex—it relies on very high precision. If you don't achieve that level of high precision, then it becomes very bad, and that's why we had to do this. Otherwise we were having too much variation; we had some really good ones, and some really bad ones. GSMArena: Regarding the 41MP sensor, we were told that it was entirely Nokia technology. D.Dinning: We co-developed the sensor. GSMArena: Any word on who else is involved? D. Dinning: Unfortunately, not at this time. GSMArena: Are you going to share the technology? D.Dinning: No. GSMArena: How excited are Microsoft about this technology? D. Dinning: That's a really leading question [laughs].

Best in class audio GSMArena: You advertise 140 db audio recording. What's special about the technology behind that? D. Dinning: First of all, we have new microphones, which we position at the ends of the device to give as much stereo-spatial awareness as possible, as well as getting them as far away from the camera as possible. Secondly, some algorithms developed by Nokia allow us to extend the upper scale of the recording level. In the N8, we had an algorithm which had a fixed dynamic range, but we would shift that according to the audio level. The problem you may have experienced using that algorithm, is that it couldn't adapt quickly enough to sudden bursts of sound. What it would do is, it would hear that sudden burst and then adjust itself, but it would be too late. Whereas with this capability, rather than being limited to a certain level, it's always able to resolve this much higher dynamic range, in terms of volume. My understanding of how the algorithm works, is that it handles the audio in two different ways—a regular level and a higher level—and then combines the information. GSMArena: So, it something like HDR for audio? D. Dinning: Yes, it is basically very similar to how HDR works for images. Typically you would need an external microphone to get similar performance capability. In addition to that, the frequency response range that we have is almost the same as CD, which is why we say it has CD-like audio recording quality. The road ahead GSMArena: Should we expect a similar, but scaled down system? For example, a 20MP sensor used to produce 3MP PureView photos? D. Dinning: What I can say today is that PureView is a combination of high performance optics, high performance sensor, and Nokia image processing algorithms. What we will do in the future, is you will see multiple combinations of those in different products. The 808 represents one direction that we want to explore in terms of our imaging offerings. We believe that there are other ways that we can exploit these offerings, but that's all I can say at this time. GSMArena: Any word on when the Nokia 808 will be available? D. Dinning: We're still optimizing the camera; we don't have the final hardware yet. We're still developing the algorithms—particularly low-light. We still have plenty of work still to do in low-light. While the low-light images are significantly better than on other devices, we believe we can make them even better. http://www.gsmarena.com/d_dinning_interview_nokia_808-review-728.php ____________________________________________________________________________________________________________________________