Cell shading

Source : cellviewer.tar.gz (browse)
Windows binary : cellviewer_win.zip
Data : betty.zip

Overview

I wrote this application for Mekensleep as an attempt to explore cell shading with modern hardware. It needs both vertex and pixel programmability, and is sadly based on ATI proprietary GL extensions (anyone buy me a Radeon 9700 ?), namely :

You need at least a Radeon 8500, and both GNU/Linux (with the ATI proprietary driver) and Windows will do. The viewer will load any 3DS or PRJ file. It will ignore texturing data.

It is based on NGL and NUI (http://ngl.sf.net). This sample code should compile nicely when laid out in the nui/examples/cellviewer folder.

The cell viewer with the ramp editors
The cell viewer with the ramp editors

Concepts

The lighting is basically a phong model, ie. normals and light vectors are interpolated at the fragment level. The resulting lighting intensity at every pixel is then run through a lookup table. The light editor interface is a facility meant to tune this lookup table with intuitive parameters.

The shade count discretizes the light ramp in the desired number of tones. flatness provides a smooth transition between a continuous and a simple edge-leveled ramp. The sampling points are taken from a gamma ramp, both along the X and Y axis, providing an easy way to change the tones span and contrast.

The following serie of images illustrate the flatness parameter :

Flat shading (edge level)
Flat shading (edge level)
Mid shading
Mid shading
Regular shading
Regular shading

While the following shows the shade count parameter off :

Monotone (no lighting)
Monotone (no lighting)
3 tones shading
3 tones shading
4 tones shading
4 tones shading

The edge detection is also done at the fragment level : it computes the scalar product of the current (interpolated) normal and the eye vector, and then run it through a lookup table. The edge editor control this lookup table.

The 0 value (leftmost on the ramp) means that viewing direction and normal are orthogonal, and hence we are on a silhouette point. The 1 value (rightmost) means that the fragment is facing right at us. We just have to built a ramp that filters the interesting case, around the 0 value.

Soft edges
Soft edges
Medium edges
Medium edges
Strong edges
Strong edges

Other experiments

Source : nuiMesh_crude_cellshader.patch

This is a crude but small patch against NUI, which adds a rendering pass to the regular mode of nuiMeshEngine. It redraws the back facing polygons as pure black with a slight Z offset towards the viewer. The result is that the polygons which are on the limit of the culling equation will slightly overdraw the front facing polygons. It needs no software or hardware special feature at all. But it suffers from nasty artefacts when objects intersects, due to obvious Z fighting. Hence the hack gives a nice sketching effect on regular surfaces, but breaks badly on composite objects. Worth noting is that this method works well on low poly models while the hardware oriented cell shader prefers highly tesselated surfaces.

Fails badly
Fails badly
Sort of work (interesting artefacts)
Sort of work (interesting artefacts)
Low poly is okay
Low poly is okay