Why Go Stylized?

Photorealism is an arms race. The more realistic your game looks, the more every imperfection stands out. A slightly wrong skin shader, a stiff animation, a flat-lit surface — players notice because their brain compares to reality.

Stylized rendering sidesteps this entirely. A cel-shaded character doesn't compete with reality — it's its own aesthetic. And stylized games age better. Wind Waker still looks great two decades later.

For indie developers especially, stylized rendering offers a practical advantage: distinctive visuals with smaller art teams and lower asset requirements.

Approach 1: Cel Shading (Toon Shading)

The Principle

Replace smooth light-to-dark gradients with discrete steps:

Photorealistic:  |████████████████████████████|  (smooth gradient)
Cel-shaded:      |████████████|        |████|  (2-3 discrete bands)

Material Implementation

Create a cel-shading material function:

In the Material Editor, build a lighting ramp:

1. Calculate the dot product of the surface normal and light direction (N·L)
2. Instead of using this directly, quantize it into bands:

// Material graph (conceptual)
NdotL = Dot(WorldNormal, LightDirection)
Step1 = NdotL > 0.5 ? 1.0 : 0.0    // Lit vs shadow
Step2 = NdotL > 0.8 ? 1.0 : Step1   // Highlight band
FinalShading = lerp(ShadowColor, LitColor, Step2)

For more control, use a 1D Lookup Texture (Ramp Texture):

1. Create a small gradient texture (256x1 pixels) in Photoshop/GIMP
2. Paint the bands you want (dark → mid → bright with hard edges)
3. In the material, use the NdotL value as the UV coordinate into this texture
4. Different ramp textures = different shading styles

Unlit vs Lit Approach

Unlit material + custom lighting: Full control over shading, but you must handle all lighting yourself. Good for extreme stylization.

Lit material + post-process: Use UE5's standard lighting, then quantize in post-processing. Easier to set up, works with existing lights.

Shadow Color

In cel shading, shadows aren't just "darker" — they're a different color:

• Anime style: Shadows are a purple/blue-shifted version of the base color
• Western cartoon: Shadows are warmer (brown-shifted)
• Flat style: Shadow is a darker but equally saturated version

Approach 2: Outline Rendering

Post-Process Outlines

The most common approach for toon outlines:

Create a post-process material that detects edges using:

Depth-based edges: Compare neighboring pixel depths. Large depth differences = edge.

Normal-based edges: Compare neighboring pixel normals. Large normal differences = edge.

Combined: Both depth and normal detection for the most complete outlines.

// Post-process material (conceptual)
// Sample depth at current pixel and 4 neighbors
DepthCenter = SceneDepth
DepthUp = SceneDepth(UV + (0, PixelSize.y))
DepthDown = SceneDepth(UV - (0, PixelSize.y))
DepthLeft = SceneDepth(UV - (PixelSize.x, 0))
DepthRight = SceneDepth(UV + (PixelSize.x, 0))

// Sobel-like edge detection
DepthEdge = abs(DepthUp - DepthDown) + abs(DepthLeft - DepthRight)

// If edge detected, output outline color
OutputColor = DepthEdge > Threshold ? OutlineColor : SceneColor

Inverted Hull Outlines

A mesh-based approach that creates outlines by rendering a slightly enlarged copy of the mesh behind it:

1. Duplicate the mesh
2. Scale it slightly larger (or push vertices along normals)
3. Flip the normals (back-face becomes front-face)
4. Apply a solid black material
5. Render behind the original mesh

This produces per-object outlines that vary with mesh shape. The outline width is consistent in screen space when using a material that accounts for camera distance.

Outline Width Control

• Distance scaling: Outlines get thinner at distance (prevents them from overwhelming distant objects)
• Importance-based: Thicker outlines on characters and interactive objects, thinner on environment
• Edge type: Silhouette edges thicker than internal edges

Approach 3: Painterly and Watercolor Effects

Painterly Post-Processing

Transform the rendered image into a painting:

1. Kuwahara filter: Smooths the image while preserving edges, creating a painted look
2. Oil paint effect: Directional smearing along the structure tensor (follows surface curves)
3. Paper texture overlay: Multiply a canvas or watercolor paper texture over the final image
4. Color palette reduction: Posterize colors to a limited palette

Brush Stroke Materials

Apply visible brush strokes to surfaces:

1. Use a brush stroke normal map on all surfaces
2. World-space projection ensures strokes don't swim with camera movement
3. Vary stroke direction and scale by surface normal
4. Animate subtly for a "living painting" effect

Approach 4: Flat/Minimalist Shading

Completely Flat

No lighting response. Every surface is a solid color:

Material: Unlit
Base Color = Flat color per material
No normal maps, no roughness variation

This requires strong silhouettes and color design since form is communicated entirely through shape and color, not light.

Low-Poly Aesthetic

Deliberately low-polygon meshes with flat shading:

1. Model with visible hard edges (no smoothing groups)
2. Apply solid colors per face or vertex colors
3. No textures needed (vertex color is the texture)
4. Lighting can be either flat or standard (with hard normals, standard lighting creates faceted shading)

Color Theory for Stylized Games

Limited Palette

Restrict your color palette for visual coherence:

• Triadic: Three evenly spaced hues (red/blue/yellow)
• Analogous: Adjacent hues (blue/teal/green)
• Complementary: Opposite hues (orange/blue, red/green)
• Monochromatic: Variations of one hue

Saturation as Depth Cue

In stylized rendering, use saturation instead of atmospheric fog for depth:

• Foreground: Full saturation
• Midground: Slightly desaturated
• Background: Heavily desaturated (approaching grey)

Color for Gameplay

Use color to communicate gameplay information:

• Interactive objects: Higher saturation than environment
• Hazards: Warm/red tones
• Safe zones: Cool/green tones
• Collectibles: Bright, contrasting accent color

Technical Considerations

Performance Advantages

Stylized rendering is often cheaper than photorealistic:

• Simpler materials (fewer texture samples)
• Can disable Lumen in favor of simple lighting
• Lower texture resolution requirements
• No need for photoscanned mega-textures
• Simpler meshes (fewer vertices for low-poly styles)

Art Pipeline Simplification

Stylized art pipelines are leaner:

• No need for PBR texture baking (roughness/metallic)
• Vertex colors replace texture maps
• Fewer unique textures per asset
• Faster iteration (less detail to author per asset)

Consistency Challenges

The biggest risk in stylized rendering: inconsistency. When one asset doesn't match the art style, it's immediately obvious.

Maintain consistency by:

• Creating a style guide document with reference images
• Building a master material with style-defining parameters
• Using material instances (not unique materials) for all assets
• Regular art reviews comparing new assets to the style guide
• Limiting the number of artists per art style (fewer hands = more consistency)

Inspiration and References

Study these games for stylized rendering techniques:

• Zelda: Breath of the Wild / Tears of the Kingdom: Cel shading with painterly environment textures
• Gris: Watercolor aesthetic with dynamic paint effects
• Cuphead: 1930s cartoon hand-drawn style
• Okami: Japanese ink painting (sumi-e) style
• Journey: Minimalist with rich atmospheric lighting
• Firewatch: Flat illustration style with strong color palette
• Return of the Obra Dinn: 1-bit dithered rendering

Stylized rendering isn't a compromise — it's a creative decision. It lets indie developers create visually distinctive games that stand out in a market flooded with "realistic" UE5 showcases. Choose a style that serves your game's mood, and commit to it fully.