home/ modules/ authoring-comfyui-generative-pipelines

Authoring Generative Pipelines in ComfyUI

L2First instrument 5h K · AI & real-time generative AV 18 atoms

Learning objectives

learner can build a diffusion pipeline as a typed ComfyUI node graph and tune CFG, denoise, LoRA, and prompt weighting
learner can use wildcards, node bypass, workflow-in-PNG, and partial re-execution efficiently
learner can drive ComfyUI headlessly via its JSON HTTP API and write a minimal custom node

Capstone — one whole task that evidences the objectives

Author a reusable ComfyUI workflow and expose it as a service: build a text2img+img2img graph with LoRA and prompt-emphasis/wildcard variation, tune CFG and denoise strength, rely on partial re-execution and node bypass while iterating, save the workflow embedded in a PNG, then submit it programmatically through the JSON HTTP API — and add one custom node with proper INPUT_TYPES/RETURN_TYPES.

Prerequisite modules

The Text-Conditioned Image Generation Stack

In an audiovisual live-coding rig, ComfyUI is the visual counterpart to your pattern engine: a diffusion pipeline you author once, then vary and trigger from code mid-set. The whole task here is exactly that — a workflow sturdy enough to be a reusable instrument, exposed as a service so a script (or your live-coding environment) can queue images without touching the GUI. That is what turns a “generate a picture” toy into a stage asset that produces themed visuals on demand between scenes.

The arc starts fully supported: wire a basic text2img graph and internalize that ComfyUI is a DAG of typed nodes, not a linear script — “ComfyUI models a diffusion pipeline as a directed graph of typed nodes” is your map. Then make it expressive: dial CFG against denoise strength to move between from-scratch generation and img2img refinement, blend in a LoRA, and shape prompts with (term:weight) emphasis and {a|b} wildcards for controlled variation. Iteration speed is a skill of its own here — lean on partial re-execution (only changed nodes re-run) and Ctrl+B bypass to audition variants in seconds, the same reflexes you need under performance pressure. The final, unsupported stretch goes headless: export the graph in API format, POST it to /prompt per “ComfyUI’s HTTP API accepts workflows as JSON,” and write one custom node following the INPUT_TYPES/RETURN_TYPES contract.

Every required atom is a gate: the capstone literally cannot be completed — or completed well — without the graph model, the four tuning surfaces, the iteration shortcuts, PNG-embedded workflows, the API format, and the node-definition contract. The supporting atoms deepen judgment rather than gate: knowing SD is three cooperating models and that diffusion runs in latent space explains why your knobs behave as they do, while TAESD previews, VRAM offloading, quantization, and the localhost threat model matter once this pipeline joins a real performance machine.