Carnegie Mellon University IntroDL Project

Expanding StyleCLIP: Enhancing Text-Driven Image Manipulation Beyond Facial Datasets

Team Members: Andy Tang, Pranav Setlur, Youjeong Roh, Minwoo Oh

Goal: Enhancing StyleCLIP beyond facial images

Requirements

• Python 3.8

How to use test2.py - latent vector extractor

• git clone e4e repo
  Here is the link
• give image path and pretrained e4e model path
• can give result's path and name in the code

References / Resources we refer to

• StyleGAN2 implemented on tensorflow 1.x
• StyleGAN2 pytorch
• encoder4editing a.k.a. e4e

How to run optimization

If you want to use existing latent vectors

1. Prepare all pretrained model weights
   You can download it from this link

2. python run_optimization.py with proper options
   These are some options you can use

   • --description: A text prompt describing the features of the image to be generated or edited.
     • Type: str
     • Default: "a person with purple hair"
   • --ckpt: Path to the pretrained StyleGAN2 model weights file.
     • Type: str
     • Default: "../pretrained_models/stylegan2-ffhq-config-f.pt"
   • --stylegan_size: Resolution of the output image generated by StyleGAN.
     • Type: int
     • Default: 1024
   • --lr_rampup: Fraction of the optimization process spent ramping up the learning rate.
     • Type: float
     • Default: 0.05
   • --lr: Learning rate used in the optimization process.
     • Type: float
     • Default: 0.1
   • --step: Number of optimization steps.
     • Type: int
     • Default: 300
   • --mode: Determines the operation mode: edit an existing image or generate a new one.
     • Type: str
     • Default: "edit"
     • Choices: ["edit", "free_generation"]
   • --l2_lambda: Weight for the latent space distance constraint (used in edit mode).
     • Type: float
     • Default: 0.008
   • --id_lambda: Weight for the identity (ID) loss (used in edit mode to preserve identity).
     • Type: float
     • Default: 0.000
   • --latent_path: Path to the initial latent code file, if provided. If not provided, uses a random latent code (edit mode) or the mean latent vector (free generation mode).
     • Type: str
     • Default: None

   Example: python run_optimization.py --mode edit --description "A person with green hair"

If you want to run on new images

1. Extract latent vector from new image
2. Just do exactly as above, but give --latent_path new image's latent vector
   Example: python run_optimization.py --mode edit --description "A person with green hair" --latent_path ./latent.pt