Beginner's Guide to Stable Diffusion (Text To Image Generation)

Have you ever thought of converting your thoughts and dreams into an image? Well, you are in the right place. This is a beginner’s guide for everyone who wants to convert text to images, whether you are a non-tech person or a programming geek. This guide will walk you through three different ways to set up and run Stable Diffusion, explain its key parameters, and more.

1. WebUI (AUTOMATIC1111): For beginners and non-techs, WebUI is the most suitable. It loads a page on the browser that gives you an interface to write a prompt (text), select parameters, and have results. Easiest to set up, learn, and experiment on to generate images.

2. AUTOMATIC1111 Rest API: AUTOMATIC1111 also provides an API to generate images. This can be easily integrated with custom UI’s.

3. Python (diffusers): Using the Python library diffusers to generate images; recommended for research and automated pipelines.

1. WebUI (AUTOMATIC1111)

WebUI is recommended for users who just want to play with text-to-image generation on their local machine. I will recommend having an Nvidia GPU with at least 16GB VRAM (like 3090ti, 4060ti, 5060ti, etc). You can run this on a CPU, but it will be painfully slow. I tested it on the CPU, and it took me 1 hour to generate a portrait image of resolution 832x1152. Plus, setting it up requires some additional effort as these are generally optimized to be used with a GPU.

I will guide you step by step to set up Automatic1111 and generate images.

1. Clone the repo: For non-technical folks, this may sound scary, but this is just downloading the necessary files. If you haven’t installed Git before, download git for your OS from their website and install it like any other software. Then open Terminal/Command Prompt (depending upon OS) and run the following command

git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui

2. When the clone is completed, depending on the OS, run webui-user.bat(for Windows) or webui-user.sh(for Linux and Mac). When you run it for the first time, this script will take some time because it sets up the environment and makes the WebUI ready for use.

3. In the meantime, go to this link and download the model sd_xl_base_1.0.safetensor file. Place this file in the models/Stable-diffusion directory inside the stable-diffusion-webui folder.

4. After successful setup, if you see a page loaded in the browser like this, you are ready to generate images.

Automatic1111 WebUI

5. In Stable Diffusion Checkpoint dropdown, select the model you downloaded. Add your thoughts and imagination to the Prompt field. In Negative Prompt field, add what you don’t want to see in the image. For example, if you are creating a realistic scenery, you can add realistic scenery, mountains, running water, beautiful skya prompt and anime, cartoon, drawing in negative prompt. Set height and width to 1024. Set Sampling Steps to 30 . Set CFG Scale to 7.5 and click Generate .

6. It will start generating the image. Once completed, you will see output like this

7. Generated images are automatically saved in the output/txt2img-images/{date}/ directory.

Now you can make your imagination become a reality. Just think and imagine, and you can visualize it as an image. If your images don’t come out as expected, you can understand different parameters, their role, and how to fine-tune them in the next section.

Understanding Stable Diffusion Parameters

Sometimes, defining a prompt and a negative prompt is not enough. We need to optimize different parameters to get the perfect image

• Prompt: This is the main text input that tells Stable Diffusion what you want to generate. Whatever you can think of, write it in this field. The best way is to add some important details and leave the rest for creativity.
• Negative Prompt: This tells the model what you don’t want in the image. Some models have a bias of adding/removing objects. For example, if you are generating an image of a human/animal, you don’t want weird anatomy like a human with 6 legs and 10 arms or a lion with 1 leg. You can add the following to the negative prompt bad anatomy, low quality, blurred, extra hands, extra limbs etc.
• CFG Scale (Classifier Free Guidance Scale): Tells how accurately to follow the prompt. Lower values represent creativity and freedom, whereas higher values represent strict adherence to text. Skewing away from the optimized values will result in a bad image or unwanted object.
• Sampling Method (aka Sampler): This is an algorithm that tells how to convert noise into a proper image. Popular choices are Euler a , DPM++ 2M , DPM++ 2M Karras etc.
• Scheduler: Controls how much noise should be reduced in each step.
• Sampling Steps: Number of iterations used to refine/improve the image. If you use a small number of sampling steps, your image will look exceptionally blurred. If the sampling steps are too large, either it will not have much of an effect, or it will destroy colors on the output. The ideal number is around 30–40 for most models (But keep in mind that some models can generate an image around 10 steps, whereas some can take 60 or above as well).
• Seed: Seed is a starting number for random noise. This noise is decreased in each step to reveal the complete image. This parameter is extremely useful for reproducibility. If you have the same parameters, you will get the same image every time. If you keep all parameters the same except the seed, it will generate a different image for the same concept. In WebUI, -1 means random noise.

Note: Each model has its own set of best parameters. If you decide to use another model, look for the parameters that work best for it. Best parameters are often listed in the repository or as a comment from where you download the model.

2. AUTOMATIC1111 Rest API

Note: This section requires some basic understanding of API calls.

Automatic1111 also provides an API to generate images. Let's look step by step at how to generate an image using the Automatic1111 API.

**Prerequisite: **Basic understanding of sending API calls and handling responses, and Automatic1111 setup. See the first method if you haven’t set up Automatic1111.

1. We need to enable the API in Automatic1111 to use it. Edit the file webui-user.bat and add a command-line argument — apilike COMMANDLINE_ARGS=--api (or in webui-user.sh If using Mac/Linux, remove # at the beginning of the line and set export COMMANDLINE_ARGS= — api. Save the file and re-run Automatic1111

2. Now send a POST call to this url http://127.0.0.1:7860/sdapi/v1/txt2img with the following data. It doesn’t automatically save the image to the default path. It returns the image as base64 encoded. To save the image, we need to set it save_images to True

{
  "prompt": "realistic scenery, mountains, running water, beautiful sky",
  "negative_prompt": "anime, cartoon, drawing",
  "steps": 30,
  "cfg_scale": 7.5,
  "sampler_name": "DPM++ 2M",
  "scheduler": "Automatic",
  "width": 1024,
  "height": 1024,
  "seed": -1,
  "save_images": True
}

3. Here’s Python code for the above API call

import requests, base64
from io import BytesIO
from PIL import Image

url = "http://127.0.0.1:7860/sdapi/v1/txt2img"
payload = {
    "prompt": "realistic scenery, mountains, running water, beautiful sky",
    "negative_prompt": "anime, cartoon, drawing",
    "steps": 30,
    "cfg_scale": 7.5,
    "sampler_name": "DPM++ 2M",
    "scheduler": "Automatic",
    "width": 1024,
    "height": 1024,
    "seed": -1
}

response = requests.post(url, json=payload)
json_data = response.json()

image_base64 = json_data['images'][0]
image = Image.open(BytesIO(base64.b64decode(image_base64)))
image.save("output.png")

4. Now you can create any UI with this to generate images, maybe deploy it on production.

I prefer this method the most when generating multiple images. Just create a UI as per your needs and call the API to get results

3. Python (diffusers by HuggingFace)

If you are a Python developer and you want more control, not just generating images and fine-tuning some parameters, then this is the best option. Unlike WebUI or RestAPI, where we can set some parameters, diffusers allowing us full control of pipeline, model, etc., e.g., you can create your custom scheduler and add it to this pipeline. Recommended for those who are familiar with torch and other related libraries.

1. Let's install the required libraries

pip install diffusers transformers accelerate safetensors

You will also need to install torch. You can use their website to check which command to run for your specific device

2. I will be using the same parameters from before

import torch
from diffusers import StableDiffusionXLPipeline, DPMSolverMultistepScheduler

pipe = StableDiffusionXLPipeline.from_single_file(
    "sd_xl_base_1.0.safetensors",
    torch_dtype=torch.float16
).to("cuda")

# Replaces the default scheduler with DPM++ 2M
pipe.scheduler = DPMSolverMultistepScheduler.from_config(
    pipe.scheduler.config,
    algorithm_type="dpmsolver++",
    solver_type="midpoint"
)

image = pipe(
    prompt="realistic scenery, mountains, running water, beautiful sky",
    negative_prompt="anime, cartoon, drawing",
    height=1024,
    width=1024,
    num_inference_steps=30,   # Sampling Steps
    guidance_scale=7.5,       # CFG
    generator=torch.Generator(device="cuda").manual_seed(torch.randint(0, 2**32, (1,)).item())
).images[0]
image.save("output.png")

3. You will have your generated image in your current working directory with the filename output.png

Final Thoughts

The barrier between imagination and creation is getting lower day by day. It doesn’t matter whether you are a beginner or an expert; you can easily generate highly detailed, realistic images with some fine-tuning. Start simple, i.e., simple prompts, basic negative prompts, and default parameters. Keep experimenting with those until you find a perfect balance that fits your style. Dozens of models are released daily that can generate certain types of images with perfection. Their generation style is not limited to anime or realism. The more you play with this tool, the more you will come to know how limitless its imagination can be.