Ambient Occlusion in XNormal
Before starting an Ambient Occlusion render, you will first have to create a floor for light to bounce off of. A large plane works most of the time, but sometimes it doesn't work at the right angles. If this happens, I create a Sphere and delete half of it to make a bowl. If your mesh is placed too close to the bottom, your ambient occlusion map will be very dark, so I line up the top of my model with the rim of the bowl. Sometimes I render out both of these maps and combine them together. The occlusion floor should always be loaded with your High Poly Mesh.
Threshold: Occlusion Map accuracy. The lower the number, the more precise the occlusion render.
Clicking the ... button after loading Ambient Occlusion in the Maps to Render field opens up a new option window.
Rays and Adaptive interval: The higher the number, the more accurate. Both should be the same number though.
Distribution: I like to leave this on Uniform
Min and Max conrols the value range you want your map to have.
I leave the Bias alone at 0.00001.
A higher Spread Angle will give a softer map with less contrast around the forms.
Uncheck Limit Ray Distance. I tend to get almost white maps if I have it checked.
Unchecking Weighting will give more intense shadows.
Attentuation The higher any of the values, the less shadows you and more white you get. Keeping these values as close to 0, or at the lowest allowed value usually results in better occlusion maps. The only value I change is the middle column.
Changing the First Attentuation Value
Changing the Middle Attentuation value
Changing the Last Attentuation value
Jitter will introduce noise into your shadows.
Settings I usually use
Using Your Normal Map to Create Cavity Maps
Ambient occlusion only captures general lighting information, so it does not capture small details well. This is why a cavity map is needed. In XNormal, I like to render out 2 cavity maps: one to balance out the darks from my Ambient Occlusion, and the other to capture the fine details the Ambient Occlusion map missed.
Cavity maps are easy and fast to render, and they can add a lot to your base texture.
Under the Tools tab in XNormal, load the Tangent Space Normals to Cavity Map tool.
Right click in the left window and load your normal map file. Be sure to change the Y coordinations to Y-.
Right click in the right Cavity Map window to generate a cavity map or save it.
The Bright field allows you to control the overall value of your cavity map. The minimum and maximum values are -1 to 1.
Contrast controls the sharpness of the details.
XNormal offers several different cavity map generation methods, but the ones I find most useful are EDT and EMB.
EDT - Looks very similar to an Ambient Occlusion map. The benefit is that it creates shadows based on how your surface changes, not from a light source, so your shadows will never be too dark.
I personally like a "Bright" value of -.1, as the default of 0 has bright whites. I also lower the "Contast" down to 0 because I will get my details in my next cavity map.
EMB - Creates a cavity map that only gives you darks and lights in areas in areas of small detail.
Bright in EMB works similar to the way it does in EDT, and will establish the base value for the cavity map. The Contrast controls how strong and sharp your details appear, but it will also influence the brightness. If the Contrast is set at 0, you'll get a black image. I find a contrast value of 1 to be a little too sharp, so I stick to something like .8
You can produce the same kind of Cavity Map as the EMB using your normal map in Photoshop, but XNormal is nice because the result is cleaner and automated.
At this point, I open up all my maps in Photoshop. I use the Ambient Occlusion as a base, and on top of that, I'll place the EDT Cavity Map and set the layer's blending mode to "Screen" or "Soft Light" (use whatever you like). This will help neutralize the darks from the Ambient Occlusion map. Then I place my EMB Cavity map on top, and set the blending mode to "Overlay."
Base Occlusion -> EDT Cavity Map -> EMB Cavity Map
Using a different base occlusion render + cavity maps
You could also try and skip rendering an Ambient Occlusion map completely, and use the Cavity maps together to start your Diffuse texture