Hi! I'm Teeh. After a long wait, a new version of Poiyomi Shaders is finally ready to release! There's been a lot of bug fixes and new features added, so let me show you the highlights for Poiyomi Toon 9.0.60, and Poiyomi Pro 9.1.12.
See the video patch notes below for more information.
Some previously Pro-only AMD fixes are being rolled back into Toon. There's still work to be done, but we're not ignoring you guys! Keep an eye out for more details later on.
Compatibility with Unity 2019 has been fixed! This should now allow you to use the Shader for either ChilloutVR, Beat Saber, and VSeeFace (or other V-tubing software of your preference).
If a Locked Material is missing for whatever reason, the UI will now indicate this and prompt you to unlock it. This should help alleviate issues with Materials unable to Unlock in cases where the generated OptimizedShaders files went missing or were accidentally deleted.
Last, but not least, is the Poiyomi Material Translator. The goal of this Tool is to easily translate Materials from any other Shader, including from older versions of Poiyomi, as accurate as possible. Shaders such as lilToon can be translated to Poiyomi Shaders easily, with all properties completely converted without any major differences. No more fiddling with MatCap masks and shading settings!
To use the new feature, a new context menu option was added in Unity. Right-Click on an Avatar or Mesh in the Hierarchy OR on a Material in your Files, and in the context menu, click Poiyomi -> Materials -> Translate to Poiyomi Toon to run the program. This will seamlessly convert the configuration of those Materials to appear properly in Poiyomi Shaders.
You can also have it make a Copy of the Material(s) with the converted properties. To use that, select Translate Copy to Poiyomi Toon from the same context menu instead. If there are multiple Materials, they will be organized automatically.
Added new Documentation Links in the UI to the following sections (thanks @BluWizard10 for all the hard work on the Docs!)
Shading/Anisotropics
Shading/Backlight
Shading/LTCGI
Special FX/Depth FX
AudioLink
AudioLink/AL Spectrum
AudioLink/AL Volume Color
Global Modifiers & Data/Global Data & Masks/Global Mask
Global Modifiers & Data/UVs/Parallax Heightmapping
You can now translate from LilToon to Poiyomi. Right-Click your Avatar or any Material basically anywhere, then click Poiyomi -> Materials -> Translate to Poiyomi Toon.
The Translate to Poiyomi Toon function will translate the Materials in-place.
If you select Translate Copy to Poiyomi Toon, the program will create copies of the Materials and then translate them. Use this if you want to preserve the original Materials.
Pro Users: You can choose to Translate to Poiyomi Pro if you so wish.
Shader: Alpha Mask Blend Strength was internally named _AlphaMaskScale and had compatibility problems when converting from other Shaders to Poiyomi. It has been renamed to _AlphaMaskBlendStrength. If you were using this, the value will be reset to 1.0.
Shader: Added Mirrored UV Mode to Decals.
This helps control the visibility of Decals on models with overlapping UVs (or mirrored UVs). Set it to Flip on one side, or be on one side only.
Shader: Added Symmetry Mode to Decals.
If enabled, will put the Decal on both sides by dividing the UV down the center.
Works only on Models with perfect Symmetrized UVs.
Shader: Added Color Space dropdown to various Color Adjust settings.
OKLab provides better and more accurate perceptive color changes. This can eliminate issues with noise and colors being washed out when adjusting the Hue.
HSV is the "legacy" method, reflecting how the Hue Shift behaved in 8.1 and older.
Fix: Fixed Alpha Offset being backwards when Alpha was inverted. It now functions the same way, inverted or not.
Fix: Grayscale for Cloth Shading now works as intended.
Fix: The first round of fixes for AMD computers have been implemented.
Fix: Improved compatibility with Unity 2019 all the way thru Unity 2022.
Fix: Fixed lightmapping code in Poiyomi World shaders, causing Emissive Materials to not work properly with Bakery.
UI: Added an Unlock Button to Locked Materials that have missing files, fixing an issue with Material properties being lost when the OptimizedShaders folder goes missing.
Greetings! My name is BluWizard. I've been one of the recent active contributors to the Poiyomi Shaders Documentation, bringing everything up-to-date with 9.0 and finishing up some areas that needed attention.
I would like to talk about some of the major updates that I'm bringing to the Documentation. These overhauls will help bring it more in line with our goals with the Poiyomi Shaders Documentation, as well as bring it more in-line with how similar Documentations organize their pages and helpful resources. We hope these updates will encourage more frequent usage of the Documentation, rather than just trying to search Discord for an answer.
Let's start of with the big one... A brand new Home Page!
Instead of providing just completion status, I made the Home Page look more presentable and professional. Not only it introduces what known features you can do with the Shader, but has two new Buttons that will redirect you where to Download & Install the shader, as well as a link to Join the Discord Server.
I have appended some of the relevant information back to the Introduction page, which will now act as the Docs Hub. It talks about how to navigate the Documentation, as well as our Completion Status. As for links to Download the Shader, it has been moved into an entirely dedicated page.
Speaking of, let's talk about the new Download & Install page!
This is going to be your new one-stop shop to learn how to Download, Install, and Update Poiyomi Shaders. Since we both maintain a classic Manual Unity Package version and a VCC Version, we now detail instructions on how to use either one. We list both Method 1. and Method 2. as the instructions.
We prefer that the users choose which method they wish to use when downloading and installing the Shader. While Method 1. Manual Unity Package is preferred, a growing number of users may have a better experience using Method 2. Creator Companion to install the Shader due to it's ease-of-use.
Regardless, whichever method you use is up to you, but keep in mind that the latest version of the Shader will always be available first via Discord before it's distributed on both GitHub, BOOTH, and in the VCC Repo. If you are a Poiyomi Pro user, they will exclusively be available via Discord as always. This is also mentioned on the page.
There are even more changes, down to the small details. Let's break down all the notable visual changes:
Overhauled the Home Page.
A more professional, presentable Home Page that gets right to the point and contains relevant information.
Added a new Logo.
Added Buttons that redirect to Download & Install and as the Discord Server.
Added Download & Install Page.
Two Methods with Instructions on how to install Poiyomi Shaders, either as a Unity Package or through the Creator Companion.
Download Links.
Info on the Pro Shader.
Improved SEO on various pages.
I have given many pages more descriptions and keywords for SEO (Search Engine Optimization), so that the Documentation and the website as a whole can appear clearly on Search Engines.
Fixed a lot of Image and Video Sizing to be more consistent.
For a while, this Documentation was not very mobile-friendly. I've implemented some changes to React JS that makes the embedded videos have more responsive width, regardless of display size. This eliminates an issue where the embedded videos would extend far beyond the width constrains on a Mobile Web Browser. By default, all Images and Videos will now be automatically responsive based on your browser's width, with some exceptions on certain Documentation entries.
Upgraded Docusaurus to v3.3.2.
Docusaurus v3.3.2 introduces major improvements, bug fixes, and new features for us to use in the near future.
React, MDX, and Node.js have been updated as a result of the Docusaurus v3 Upgrade.
The Light and Dark Theme will now automatically match the user's client by default.
Removed irrelevant information and old pages in favor of the new Download & Install page.
The Poiyomi VCC Repo Page is no longer necessary, as I've implemented a button in the Download & Install page for the VCC Version that directly opens the Creator Companion App, adding the Repository.
Redirects added to the website configuration to handle old links.
We are continuously working on improving the Documentation with as much information as possible in an easy-to-understand fashion. If you have any feedback, feel free to drop us a line in the Discord Server. Other than that, we hope you enjoy these new updates!
I am very excited to announce that version 9.0 of Poiyomi Shaders is now fully available!
This major update contains a vast amount of new features and improvements to the shader. While some areas may look similar to how they were in 8.0, it's worth noting that some sections have significantly changed. I highly recommend you take the time to look them over.
Much of the existing Documentation pages are still relevant, with some notes if applicable. The pages will be updated accordingly to be more in-line with 9.0 in the coming weeks. Some pages have already been updated as well (thanks, BluWizard and Teeh!).
All older versions, including 7.3, 8.0, 8.1 and 8.2, have all been moved under .poiyomi/Old Versions.
This means the current main version of Poiyomi Shaders will be .poiyomi/Poiyomi Toon instead.
UNLOCK YOUR MATERIALS BEFORE UPGRADING to ensure you have an easy transition to the new naming scheme!
RGBA Color Masking
RGBA Color Masking has been completely redone from scratch. With this, a much better user interface and workflow is available for you to use.
Each Channel has more properties available now compared to before. To see what new areas do exactly, check the Documentation.
Matcaps
Introducing Gradient UV Mode.
This feature works very similarly to Iridescence for those upgrading to 9.0.
Use a Gradient Texture in the Matcap texture slot when using Gradient, or use the Thry Gradient Editor to make your own.
Increased Matcap Slots from 2 to 4.
Added AudioLink to Matcaps.
Adjust your Matcap Alpha, Emission, and Intensity with the music Band.
This feature was added to accomodate users switching over from using Iridescence.
Outlines
All Outlines features have been reintroduced to the main shader variant.
This means you no longer have to select a separate shader variant in order to use Outlines!
Added a new Outline Fixed Size Mode that basically makes Outlines look better at a closer range. This will be on by default on new Materials.
Added Color Adjust properties to Outlines.
Added more AudioLink settings to Outlines.
Enable Color to change the Color of your Outlines with the music Band.
Glitter / Sparkle
Glitter has received a major UI rework.
Added Layers to Glitter.
Added Random Rotation Speed Offset to Glitter.
Altered the Linear Emission version of Glitter.
Added a slider allowing Glitter to scale with Lighting.
This would make it sparkly in the light, and dark in the dark.
Added AudioLink to Glitter.
Play around with Max Brightness, Alpha Mod, Size Mod, Chrono Sparkle Speed, and Chrono Rotation Speed with music Band to create some incredible effects.
Iridescence
Iridescence has been deprecated in favor of Matcaps. Use Matcaps with Gradient UV Mode instead.
This was done because Matcaps can technically do the same thing as Iridescence with a few adjustments.
To accomodate users switching over, all AudioLink features have been ported over to Matcaps as well.
Rim Lighting
Reorganized everything. Same properies, better user experience.
Additionally, the Blend Mode property name was changed due to incompatibilities with lilToon naming. Check the Upgrade Guide in the Documentation to see what the default settings were set to.
LTCGI
Added experimental LTCGI support. Check the Documentation for more information.
This marks the first time it is available in the Free Shader version.
Varous Elements in the Material UI have been redesigned to make things look better and easier to visually distinguish between other sections.
Color Adjust
Hue Shift is now using the human perception of color.
This means better stability of the Brightness and Saturation with your base color.
Added Color Grading.
Module Toggles
Module Toggles let you remove modules from your shader. Handy if you don't want to deal with all the things you don't use or want to be able to check boxes and see a little less cyan.
Rendering
Reorganized Rendering Options instead of being randomly ordered. The new order of appearance is now: Blending, Outline Blending, Stencil, and Outline Stencil.
Global Masks, Themes, UV Settings, and Post Processing are now organized under the Global Data and Masks Category.
Changed Min Brightness for Additive Lights to now function with the luminance value of the additive light rather than the direct RGB values.
For reference, "additive lights" are Point and Spot Lights.
Decals
Added Back Face and Front Face Masking.
Cubemap
Section rework, including better organization.
Added Blurring and Color Adjustment.
Reflections & Specular
Added Global Masking support.
Lit Fallback is now On by default to improve lighting in maps with no Reflection Probe at all.
Light Data
Added Mirror Vertex Lights (Non-Important) Checkbox to Light Data. This enables separate additive lighting to appear on your Avatar's Mirror Clone.
This is enabled by default.
Adjusted Vertex Lights so they're more in line with how Additive Lights function.
Early Z
Added Early Z option to almost every shader rather than having it be it's own shader.
To use Early Z, enable it under Rendering on your Material.
Grab Pass
Massive changes to how Grab Pass works in Poiyomi Shaders. It now allows for all features to be used on it with correct lighting for Opaque surface and Transparent shading.
Defaults have been changed so they function correctly.
AudioLink
Added animatable Overrides to the individual AudioLink Bands for debugging and custom AudioLink inputs.
Emissions
Added Saturation to Color Adjust.
Changed the name of the Hue Shift section to Color Adjust.
Vertex Options
Added a stupid sphere mode to vertex options because someone really wanted it.
Fixed a bug where Materials de-select themselves after Locking and Unlocking.
Fixed a bunch of duplicate named things that was definitely not copy pasted and was forgotten to be changed.
Fixed duplicate sampler names.
Fixed Vertex Light issues in Shading and Stylized Specular.
Fixed Grab Pass blending at various transparencies.
Fixed a problem with additive lighting breaking when Point Light Passthrough was set to 0.
Corrected a problem with Packed Map behavior.
Fixed Object Position being set incorrectly.
It was set to 0,0,0 in World Space and is now correctly set tot he Model's position.
Fixed Glitter Color Alpha. It actually now does something.
Fixed Grab Pass on the Toon Grab Pass shader.
Fixed Export Checker throwing a Null Reference Exception.
Fixed Point Lights being too dark at longer ranges. They were recalculated to be more correct.
Fixed Light Data Maps not having RGBA Packers.
Fixed a script error in the Toon Shader.
Fixed some buggy UI in Rim Lighting.
Fixed an issue where Reset, Copy, and Pasting of sections were not handling the reference properties correctly.
Fixed various Shader Warnings.
Turns out that sometimes the Shader Warnings can actually crash the compiler!
Fixed a bunch of crashes.
Fixed an issue where Unlocked Shaders were included in the Upload when they weren't supposed to due to a failed Auto-Lock or other issue. It will now throw a callback to stop the Upload entirely if this situation is detected.
A bunch of other fixes that would take ages to document.
Added isPlaying check to Avatar Preprocessor. This should prevent Locking in cases where preprocessing is called for in Editor Test Builds that are not being Uploaded.
Added Cross Shader Editor.
Use this Tool to edit multiple Poiyomi Materials, all at once.
Summon the panel by clicking Thry -> Cross Shader Editor from the Unity Menu.
Added organizers which are currently used in Geometric Dissolve and Multilayer Shading.
Added Decal Placement Tools, found in the Decal section.
Fixed Presets from not functioning or updating correctly.
VectorToSlidersDrawer negative support using np. If used all floats in the drawer declaration need to have either n or p in front of it to convert the parameter to a string. n will be converted to a negative float.
Fixed a null pointer exception involving the Shader Translator.
When utilizing the Poiyomi Toon Shader through VCC, you have the option to set it up for your project. However, there are some essential considerations to keep in mind, and we'll provide a detailed guide to help you through the process.
To initiate the use of the Poiyomi Toon Shader via VCC, follow these steps to ensure a smooth setup:
Check Existing Installations: Prior to proceeding, it's crucial to verify whether Poiyomi is already installed in your project. This step is essential to prevent conflicts or unexpected behavior.
Adding the Shader to Repo Listing: If you have the latest VCC installed, follow these steps:
Reduce Resolutionin the texture import settings! Not every texture needs to be 2k or 4k.
Don't disable mipmaps - Your materials will look and run better with them on!
Crunch doesn't change VRAM usage but can reduce download size. Not a magic bullet!
For Normal Maps, Use RG Compressed BC5 (From Platform overrides menu1). Normal Quality (DXT5nm) and High Quality (BC7) use the same VRAM but look worse.
For textures with Alpha, Use High Quality (BC7). Normal Quality (DXT5) uses the same VRAM but looks worse.
Don't use JPG (or other lossy formats) to store your source textures! Lower quality, with no benefit to VRAM/Download Size.
In VRChat Creation, texture assets comprise a large portion (often the majority!) of map/avatar video memory (VRAM) usage, and a significant chunk of map/avatar download size (assetbundle size). In the latest update, VRChat has added an estimate of texture memory to the avatar stats menu, as it's an important component of GPU performance.
When the GPU can't store more assets in its memory, it has to swap them back and forth to the system memory (RAM). This works fine, but at best slows things down and causes hitching, and at worst, it can cause terribly low framerates and crashing.
Here's some notes on optimizing texture size, on both worlds and avatars, to reduce VRAM usage and download size. These are geared toward PC optimization, but outside of specific formats, they apply to mobile platforms (like Quest) as well.
In the VRC creation pipeline, textures are compressed multiple times: when they're imported to Unity, to a gpu-readable format (determining the VRAM size), and when an assetbundle is built, with LZMA compression (kind of like a zip archive!), which reduces download size. Technically, there's also the source asset's compression (most formats have some, for efficiency), but as long as it's lossless, we don't need to care about it.
Knowing this, we can safely store source assets at their original resolution, losslessly (using formats like PSD, TGA, PNG, etc - don't use lossy ones like JPG, especially with data textures like normal maps!). The source asset file size on disk doesn't matter as the image data will be re-encoded in its entirety.
GPU texture formats almost always have a fixed amount of data per pixel when loaded in memory. This means that (for example) a square BC7 texture at 512px with mipmapping on will always be 341.4 KB in VRAM, no matter what the texture data contains. This isn't as true for download sizes - LZMA compression can be more or less effective on different data, but we don't have as much control over that.
For non-crunched textures, the size in the texture preview will give the VRAM usage of an imported texture asset. For crunched textures, the preview size doesn't match VRAM usage! the usage will be the same as "Normal Quality" (DXT1/DXT5) for that texture. VRAM usage can be calculated using the Bits per pixel for a given format, which can be found in the Unity Documentation.
You can use the preview and game build views as a guide for (non-crunched) VRAM, and a very rough indicator of disk size, but verify with lox9973's Assetbundle Stat tool to confirm both VRAM usage contribution and final compressed bundle size. Thry's VRAM estimator and the VRAM estimation built into ThryEditor (used in Poiyomi) are also useful for this.
Crunch compression is frequently talked about around the topic of avatar and world optimization. When optimizing filesize, you can use crunch compression for things that you can afford to lose quality on, and verify visually that it still looks OK. When using Crunch, test different crunch quality levels and measure the difference it makes in visual quality and filesize, and choose what works out best for your application.
Crunch compression does not affect VRAM usage.
Crunch pre-compresses textures in a way that's easy for the CPU to decompress at runtime, so it brings disk size down a bit (though not as much as it would appear from the preview, since there's assetbundle LZMA compression too!), but it always has to be uncrunched into DXT1 or DXT5 anyway.
Left-to-Right: DXT1 Crunch (25 Quality), DXT1 Crunch (75 Quality), DXT1, BC7(Open for full quality)
DXT1Crunch/DXT5Crunch ("Use Crunch Compression") for simple colors that can take it, if you need the disk space. Always look at the visual quality, and again, Crunch does not affect VRAM usage, so this will use the same memory as "Normal Quality".
DXT1 ("Normal Quality") for simple color textures. This setting should not be used if the source has an alpha channel, or if it contains smooth gradients - skin textures often look especially bad, as well as anything that's being hue shifted.
BC7 ("High Quality") for anything else, especially if it has an alpha channel. It's higher quality and uses 8 bpp instead of 4, so textures using BC7 have twice the VRAM usage of ones using DXT1, so as noted above, consider using High Quality at half the resolution instead of Normal Quality at the "full" resolution, as it may be an acceptable tradeoff.
Note that setting a texture to "High Quality" will override the crunch compression setting.
If you are using an alpha channel, either use BC7 ("High Quality") or crunch it (DXT5), depending on the application. using "Normal Quality" on a texture with an alpha channel is just a waste, as the DXT5 format it uses has the same VRAM usage as BC7. If your source texture has an alpha channel, but you don't need alpha, you can set "Alpha Source" to "None" in the texture import settings, which will let you either use DXT1 ("Normal quality") or BC7 with higher quality.
DXTnm Crunch ("Use Crunch Compression") for normal maps that do not need to be high quality. Crunch is not ideal for normal maps, so it often results in low quality results.
When not crunching, always use BC51 or alternatively BC7 ("High Quality") for Normal maps. BC7 can cause some artfiacts and banding that BC51, which stores the R and G channels as two separate BC4 channels, can help reduce. The default setting of DXT5/DXTnm ("Normal Quality") uses the same memory but looks worse!
DXT1 ("Normal Quality") for simple masks that don't have a lot of edges or high frequency detail.
BC7 for high-detail masks
BC4/BC51 can be used for specific use cases, like single and dual channel masks. These formats can give high quality when you only need specific channels.
"None" for very small (smaller than 256px) textures that benefit from storing exact colors, like color ramps used for lighting and small gradients. These formats tend to be large, so be careful!
To optimize both VRAM usage and download size, reduce resolution as much as possible for your application.
Textures in Unity are always re-encoded on import according to the texture import settings. That means you can non-destructively change the resolution size - you can always set the resolution in the texture import settings lower to optimize, without modifying the source texture. The vast majority of materials don't need high resolution textures (4k+) to look good.
Every time you halve the resolution of an image, you reduce the amount of pixels it has by a quarter - that means that while a square 4k BC7-encoded texture may take over 21MB in memory, the 2k version would be just a bit over 5MB!
It's important to note that while reducing resolution makes textures lower-fidelity up close, they will often look exactly the same at from even a little distance away, due to screen/headset resolution being limited. This is why AAA games don't use super high resolution textures everywhere - even in the most advanced modern games. Zoom out a bit when adjusting resolution - it might have less of an impact than you think!
If choosing between lower quality compression at higher resolution, and higher quality compression at lower resolution (ex. Normal Quality/Crunched at 2k vs High Quality at 1k), the latter option will often look better at all but the closest distance, due to how mipmapping works. This reduces VRAM usage, keeps download size roughly the same, and can improve the relative quality! Tools like the texture compression analysis tool included with d4rk's Avatar Optimizer can make this easier to analyze.
To maintain close-range details, one technique is to use low resolution base textures and tiled, low-resolution detail textures (especially normals). This lets you use multiple much smaller textures instead of a single larger one.
If you're having trouble maintaining details (lines or designs get blurry when lowering resolution), the mesh amy simply not have good texel density. This can be improved by improving the UV mapping of the mesh. Read more about this topic here and here.
Textures used for PBR (Physically Based Rendering) materials often fit into 3 categories: Color, Normal, and Data. While it's beneficial to reduce res in general, PBR data textures can be an especially good place to reduce resolution.
Color: textures that directly use the color for the material appearance, like Base Color (diffuse/albedo) and emission map textures.
Normal: Normal maps, used to modify how the surface responds to light.
Data: textures that contain data, like Metallic, Smoothness/Glossiness/Glossiness, Ambient Occlusion (AO) maps, and Height maps.
Color and normal textures often benefit directly from higher resolution, but you can often get away with a lot less resolution for data textures - try half (or a quarter) of the res of the main (color/normal) textures.
In terms of what overall usage to shoot for, we can think in terms of average GPU VRAM. According to the September 2022 Steam Hardware Survey, the most common VRAM amount for steam users is 8 GB, with about 35-40% of users having 8 GB or more - we'll use 8 GB as our reference number.
Doing some quick-and-dirty math, in a an instance with 80 avatars, if someone were to show every avatar and have it in view, each model would need to be under 100 MB of VRAM usage to stay below 8 GB total. That's before accounting for overhead from the operating system, the game itself, and the world hosting the instance. With that in mind, optimized avatars should shoot for under this - I usually aim for 80 MB or less on optimized avatars, but higher can be ok on models that will be used in less crowded spaces.
For worlds, it depends entirely on the content and intended usecase of the world. If you're building an event or party venue that's intended to host 80 people and heavy effects, you'll want to drop VRAM usage as low as it can go. The same is true for mobile-optimized worlds. However, for a more atmospheric world, intended to be experienced in small groups or alone, or where the focus is on beautiful, rich environments, higher resource usage can be justified. Exercise judgement and keep low-spec users in mind.
You should only disable mipmaps if it's something like a lookup table (LUT) or data texture (ex. font textures) that always needs the full texture resolution. While disabling them may reduce VRAM usage, it's not worth the hit to performance, and will make things look worse.
Please don't turn off mipmaps. Image courtesy Ben Golus
Unity's editor log will contain a breakdown of the build process, which some tools break out into a nice interface. While this can be useful, it won't be entirely accurate in describing how much space assets take up in the final bundle.
This comes from a subtle technical detail - due to how DXT1 represents colors, it ends up having a limited hue palette in the red and blue channels. As a result, hue shifts applied to DXT1 encoded color textures can end up with blocky and bad results. (thanks to d4rkplayer for pointing this out!)
Choosing to optimize your creations benefits everyone that uses and experiences them, and improves the platform in general - when everything runs better, everything feels better.
Recognize that how materials, and their textures, are viewed is different between the creator building it (and often heavily self-scrutinizing) and other people that experience it. Make sure to step back and recognize that not everyone is going to be looking at the tiny details - focus on what matters in reasonable use cases.
When using the Outlines module, outlines are generated by offsetting the original mesh along its normals, with front face culling (known as the Inverse Hull or Inverted Hull technique) This generally works well, but outlines will be visible anywhere there's mesh borders - this is especially visible for things like pupils, eyelashes, and other interior geometry. While you can use an outline thickness mask for this, for many applications, we only want outlines on the outside of the mesh.
To fix this, we can use stencils. Stencils are a DirectX feature that allow values to be set by a material in a special buffer, which later passes and materials can use to determine whether a pixel should be drawn or not. Stencils are very flexible and can be used for a lot of different effects - we're going to use them to set up our outline so that it only draws outside the mesh.
When the Outlines module is enabled, an additional section for Outlines is added to the Stencil section at the bottom of the shader. We can set up these options a few different ways to get the desired result; here's one way that works well.
Stencilled Outline Settings
The specific value of the Stencil Reference Value doesn't matter; it just has to be something other than 0, and that it matches between both stencil settings.
If you just wanted to get the effect, you're done! If you want to understand more about why this works, read on.
By default, the stencil buffer is set to 0 for every pixel. For the base pass of the material, we set the stencil value to something other than 0, with a stencil compare function of Always. This means that the stencil buffer will be set to our value for every pixel the material draws.
For stencils to work, one pass/material has to be written before another pass/material reads from it. In this case, the outline pass is drawn after the base pass, so it can read the modified stencil buffer. If the value is NotEqual to the reference value, the outline is drawn - meaning wherever the base material drew a pixel, the outline will not render.
This setup has the same result, but works in a slightly different way. It writes a non-zero value to the stencil buffer wherever the base material draws. The outline pass reads from the stencil buffer and only renders where the stencil buffer has its default value of 0.
This setup would result in outlines potentially not rendering where other materials use the stencil buffer, which may or may not be desirable.
