Control Guide

Introduction

SDNext's Control tab is our long awaited effort to bring ControlNet, IP-Adapters, T2I Adapter, ControlNet XS, and ControlNet LLLite to our users.

After doing that, we decided that we would add everything else under the sun that we could squeeze in there, and place it directly into your hands with greater options and flexibility than ever before, to allow you to Control your image and video generation with as little effort, as as much power, as possible.

This guide will attempt to explain how to use it so that anyone can understand it and put it to work for themselves.
We'll start with the... Control controls!

Controls

Input

The Input control is exactly what it sounds like, it controls what input images (or videos) are contributing to your image generation, by default that is just the image in Control input pane, however if you select Separate init image, another image pane will appear below, allowing you to use that as well.

Note: When using a Control input image as well as a Init input image, the Init input dominates. Adjusting denoise to >=0.9 is recommended, as that will allow the Control input to balance with the Init input. Higher values will increase the strength of Control input further, giving it dominance.

Show Preview is simple, it controls the visibility of the preview window in the far right of the middle row. You'll want this on if you're doing any kind of masking or manipulations that you would want to preview before generating.

There are 3 different Input types:

• Control only: This uses only the Control input below as a source for any ControlNet or IP Adapter type tasks based on any of our various options.

• Init image same as control: This option will additionally treat any image placed into the Control input window as a source for img2img type tasks, an image to modify for example.

• Separate init image: This option creates an additional window next to Control input labeled Init input, so you can have a separate image for both Control operations and an init source.

Denoising strength is the same as if you were doing any img2img operation. The higher the value, the more denoising that will take place, and the greater any source image will be modified.

Size

This can be a little confusing at first because of the Before and After subtabs, however it's really quite simple and extremely powerful.
The Control size menu allows you to manipulate the size of your input images before and after inference takes place.

The Before subtab does 2 things:

• If you do not select any Resize method, it is only controlling the output image size width and height in pixels as it would in any text2img or img2img operation.

• However, if you do select a Resize method, Nearest for example, you can upscale or downscale the Control input image before any other operations take place. This will be the size of any image used in further operations. Second Pass is not entirely functional yet, but will be part of this.

For example, you might have a much larger image, such as 2048x3072, that you want to use with canny or depth map, but you do not want an image that large to manipulate or guide your generation, that would be prohibitive, slower, and possibly cause an OOM.

This is where Resize method comes in, you would simply select a resize method, typically Nearest or Lanczos, and then either set the pixel width or height you want to resize to under Fixed, or switch over to Scale and select a number below 1. A setting of 0.5 would make your input image effectively 1024x1536 pixels, which would be used as input for later operations.

The After subtab controls any upscaling or downscaling that would take place at the end of your image generation process, most commonly this would either be latent upscaling, and ESRGAN model such as 4x Ultrasharp, or one of the various chaiNNer models we provide. This is the same as it would be in a standard upscaling via text2img or img2img.

Mask

The Mask controls are where we start getting into the real meat of Control, not only does it allow a plethora of different options to mask, segment, and control the view of your masking with various preview types, but it comes with 22 different colormaps for your viewing pleasure! (And I think vlad made some of those words up 🤫)

• Live update: With this checked, your masking will update as you make changes to it, if this is off, you will need to hit the Refresh button to the right to have your preview pane update, making more changes to it while it is processing may lead to it being desynchronized, just hit the refresh button if it does not look correct.

• Inpaint masked only: Inpainting will apply only to areas you have masked if this is checked. You must actually inpaint something, otherwise it's just img2img.

• Invert mask: Inverts the masking, things you mark with the brush will be excluded from a full mask of the image.

• Auto-mask: There are three options here, Threshold, Edge, and Greyscale. Each provides a different method of auto-masking your images.

• Auto-segment: Just like Auto-mask, we have provided an extensive list of Auto-segmentation models, they don't require ControlNet to handle the process, but may take a few seconds to process, depending on your GPU.

• Preview: You can select the preview type here, we have provided 5 modes, Masked, Binary, Greyscale, Color, and Composite, which is the default.

• Colormap: You can select the style/color scheme of the preview here. There are 22 fantastic color schemes!

• Blur: This blurs the edges of what you have masked, to allow some flexibility. Play with it.

• Erode: This slider controls the reduction of your auto-masking or auto-segmentation border.

• Dilate: This slider controls the expansion of your auto-masking or auto-segmentation border.

Video

The Video controls are quite exciting and fun to play with, with our tools now you can, if you wished, turn any video into an anime version for example, frame by frame. There are three output options, GIF (not JIF!), PNG, and MP4. You must select one of these to have video output.
With these simple controls, you can tweak your video output with surprising flexibility.
Some video output methods provide more controls, try them all.

• Skip input frames: This setting controls how many frames are processed from input instead of every frame. Setting it to 2 would process every other frame, 3 would process every third frame.

• Video file: You select the type of output you want here, animated GIF, animated PNG, or MP4 video, all provided via FFMPEG of course.

• Duration: The length in seconds you want your output video to be.

• Pad frames: This setting

• Interpolate frames: The number of frames you want generated (via RIFE interpolation) between existing generated SD frames in a video sequence, to fill in between the generated output frames and make the video output more smooth.

• Loop: This is purely for animated GIF and PNG output, it enables the classic looping that you would expect.

Extensions

These are some nice goodies that we have cooked up so that no actual installed extensions are necessary, you may even find that our version works better!

AnimateDiff

This is the new home of our Vlad-created implementation of the AnimateDiff extension. Now with FREE FreeInit!

I honestly don't know how to use this, so I'll update this when I do. My apologies! But if you already do, enjoy!

IP-Adapter

This is our IP Adapter implementation, with 10 available models for your image or face cloning needs!

Self-portrait example image provided by our loveliest Henchman by far, Nyxxia.

Image Panes

You may notice small icons above the image panes that look like pencils, these are Interrogate buttons. The left one is BLIP, and the right one is DeepBooru. Click one of the buttons to interrogate the image in the pane below it. The results will appear in your prompt area.

Control Input

This is the heart of Control, you may put any image or even video here to be processed by our system, that means any and all scripts, extensions, even the various Controlnet variants below, though you can individually add guidance images to each of those. If an image is placed here, the system will assume you are performing an img2img process of some sort. If you upload a video to SDNext via the Control input pane, you will see that you can play the video, both input and resultant output. Batching and folders should work as expected.

Note below there are 2 other buttons, Inpainting and Outpainting, below.

ControlNet+

At the very bottom of the Control page, we have what you've all been waiting for, full ControlNet! This includes SD1x and SD-XL.