Control Guide

Introduction

SDNext's Control tab is our long awaited effort to bring ControlNet, IP-Adapters, T2I Adapter, ControlNet XS, and ControlNet LLLite to our users.

After doing that, we decided that we would add everything else under the sun that we could squeeze in there, and place it directly into your hands with greater options and flexibility than ever before, to allow you to Control your image and video generation with as little effort, as as much power, as possible.

This guide will attempt to explain how to use it so that anyone can understand it and put it to work for themselves.
We'll start with the... Control controls!

Controls

Input

The Input control is exactly what it sounds like, it controls what input images (or videos) are contributing to your image generation, by default that is just the image in Control input pane, however if you select Separate init image, another image pane will appear below, allowing you to use that as well.

Note: When using a Control input image as well as a Init input image, the Init input dominates. Adjusting denoise to >=0.9 is recommended, as that will allow the Control input to balance with the Init input. Higher values will increase the strength of Control input further, giving it dominance.

Show Preview is simple, it controls the visibility of the preview window in the far right of the middle row. You'll want this on if you're doing any kind of masking or manipulations that you would want to preview before generating.

There are 3 different Input types:

• Control only: This uses only the Control input below as a source for any ControlNet or IP Adapter type tasks based on any of our various options.

• Init image same as control: This option will additionally treat any image placed into the Control input window as a source for img2img type tasks, an image to modify for example.

• Separate init image: This option creates an additional window next to Control input labeled Init input, so you can have a separate image for both Control operations and an init source.

Denoising strength is the same as if you were doing any img2img operation. The higher the value, the more denoising that will take place, and the greater any source image will be modified.

Size

This can be a little confusing at first because of the Before and After subtabs, however it's really quite simple and extremely powerful.
The Control size menu allows you to manipulate the size of your input images before and after inference takes place.

The Before subtab does 2 things:

• If you do not select any Resize method, it is only controlling the output image size width and height in pixels as it would in any text2img or img2img operation.

• However, if you do select a Resize method, Nearest for example, you can upscale or downscale the Control input image before any other operations take place. This will be the size of any image used in further operations. Second Pass is not entirely functional yet, but will be part of this.

For example, you might have a much larger image, such as 2048x3072, that you want to use with canny or depth map, but you do not want an image that large to manipulate or guide your generation, that would be prohibitive, slower, and possibly cause an OOM.

This is where Resize method comes in, you would simply select a resize method, typically Nearest or Lanczos, and then either set the pixel width or height you want to resize to under Fixed, or switch over to Scale and select a number below 1. A setting of 0.5 would make your input image effectively 1024x1536 pixels, which would be used as input for later operations.

The After subtab controls any upscaling or downscaling that would take place at the end of your image generation process, most commonly this would either be latent upscaling, and ESRGAN model such as 4x Ultrasharp, or one of the various chaiNNer models we provide. This is the same as it would be in a standard upscaling via text2img or img2img.

Mask

The Mask controls are where we start getting into the real meat of Control, not only does it allow a plethora of different options to mask, segment, and control the view of your masking with various preview types, but it comes with 22 different colormaps for your viewing pleasure! (And I think vlad made some of those words up 🤫)

• Live update: With this checked, your masking will update as you make changes to it, if this is off, you will need to hit the Refresh button to the right to have your preview pane update, making more changes to it while it is processing may lead to it being desynchronized, just hit the refresh button if it does not look correct.

• Inpaint masked only: Inpainting will apply only to areas you have masked if this is checked. You must actually inpaint something, otherwise it's just img2img.

• Invert mask: Inverts the masking, things you mark with the brush will be excluded from a full mask of the image.

• Auto-mask: There are three options here, Threshold, Edge, and Greyscale. Each provides a different method of auto-masking your images.

• Auto-segment: Just like Auto-mask, we have provided an extensive list of Auto-segmentation models, they don't require ControlNet to handle the process, but may take a few seconds to process, depending on your GPU.

• Preview: You can select the preview type here, we have provided 5 modes, Masked, Binary, Greyscale, Color, and Composite, which is the default. See the chart below for more information.

• Colormap: You can select the style/color scheme of the preview here. There are 22 fantastic color schemes!

• Blur: This blurs the edges of what you have masked, to allow some flexibility. Play with it.

• Erode: This slider controls the reduction of your auto-masking or auto-segmentation border.

• Dilate: This slider controls the expansion of your auto-masking or auto-segmentation border.

Segmentation Models

There are 8 Auto-segmentation models available:

• Facebook SAM ViT Base (357MB)
• Facebook SAM ViT Large (1.16GB)
• Facebook SAM ViT Huge (2.56GB)
• SlimSAM Uniform (106MB)
• SlimSAM Uniform Tiny (37MB)
• Rembg Silueta
• Rembg U2Net
• Rembg ISNet