Pix2Pix-PracticalML

Co-Author: Marcelo Padovani

Goal of the project:

To explore and try out practical usages of Pix2Pix Machine Learning approach on dancing that could be used by dancers or choreographers to create and visualize new movements and choreographies.

Here's a demo of the final results:

PML.-.Group.3.mp4

How It's Made

Process of Implementation and tools used:

To create the training database we used a videoclip from Mc Dadinho: we extracted his position using posenet and Processing where we matched with the frame of the video, 10 frames per second of video, and exported the images of the pose + video frame.

Then, we trained Tensor Flow Implementation of Pix2Pix using Google Colab and generated the images using skeletons extracted using posenet from another video of a person dancing.

Processing's Grafica library was used to create graphs so that we could analyze the training of the model. Below are the graphs indicating the Discrimination, Gan and L1 losses during the last training in which we used 164 images for 463 epochs.

Lessons Learned:

Based on the observation of the graph, since the result are somewhat linear, it means that both “Discriminator” and “GAN” are still advancing. According to other users running models on Pix2Pix, we haven’t reached the point where “Discriminator” starts to fluctuate along with “GAN”. This means that the Generator hasn’t really won over the DIscriminator in a lot of the epochs so far, and more training time can be done to further enhance the output.

---

📚 [Slides] Pix2Pix & How it works

Below are some slides explaining how Pix2Pix works in our project.

Click here to view

How it works:

Differences between different methods of training:

Noise Vector = cGAN - Conditional GAN, uses Observed image along with Noise vector to generate the output image.

Generator interaction with Discriminator:

Creating the Training Data Sets

Results

Initial Tests of ML Model Training

Trained with 30 images, 100 Epochs (15 min)

With 300 images & 6hrs Training Time (Epochs: 166)

With 164 images & 8hrs Training Time (Epochs: 463)