diff --git a/BE2_GAN_and_Diffusion.ipynb b/BE2_GAN_and_Diffusion.ipynb
new file mode 100644
index 0000000000000000000000000000000000000000..d0181703393a3d78ef4ef5e81c66b6175e2ba32d
--- /dev/null
+++ b/BE2_GAN_and_Diffusion.ipynb
@@ -0,0 +1,1581 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "UGwKsKS4GMTN"
+ },
+ "source": [
+ "<h1 ><big><center>MSO 3.4 - Deep Structured Learning</center></big></h1>\n",
+ "\n",
+ "<h2><big><center> BE 2 - GANs and Diffusion </center></big></h2>\n",
+ "\n",
+ "<h5><center>GANs section adapted from the <i>Projet d'Option</i> of : Mhamed Jabri, Martin Chauvin, Ahmed Sahraoui, Zakariae Moustaïne and Taoufik Bouchikhi \n",
+ "<h5><center>Diffusion section implemented by Bruno Machado<br>\n",
+ "\n",
+ "\n",
+ "<p align=\"center\">\n",
+ "<img height=300px src=\"https://cdn-images-1.medium.com/max/1080/0*tJRy5Chmk4XymxwN.png\"/></p>\n",
+ "<p align=\"center\"></p>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "16aVF81lJuiP"
+ },
+ "source": [
+ "The aim of this assignment is to discover generative models for images, understand how they are implemented and then explore some specific architectures and training strategies which allows us to perform image generation with and without conditioning (the picture above exemplifies image generation conditioned by another image)\n",
+ "\n",
+ "The deliverable for this assignment is this notebook, with the required modifications : \n",
+ "* In this notebook you'll find <font color='red'>**questions**</font> that aim to test your understanding of the concepts studied. You should answer them by editing the Markdown cells (or adding new ones). \n",
+ "* In some of the code cells, you'll have to complete the code and you'll find a \"TODO\" explaining what you should implement.\n",
+ "* There are also some <font color='green'>**bonus**</font> exercises. Those exercises are **optional** and should be performed only **AFTER** all the rest of the assignment is complete."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "M-WNKvhOP1ED"
+ },
+ "source": [
+ "# Part 1: DC-GAN"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "y_r8nMTGQI9a"
+ },
+ "source": [
+ "In this part, we aim to learn and understand the basic concepts of **Generative Adversarial Networks** through a DCGAN. For this purpose, please study the following tutorial: https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "jiHCy4_UUBFb"
+ },
+ "source": [
+ "We want to generate handwritten digits using the MNIST dataset. It is available within the torvision package (https://pytorch.org/vision/stable/generated/torchvision.datasets.MNIST.html#torchvision.datasets.MNIST)\n",
+ "\n",
+ "Adapt the code given in the tutorial in order to use it with the MNIST dataset. Present the loss curves in function of the gradient steps for the generator and discriminator and also compare the generated images with the images from the dataset."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {},
+ "colab_type": "code",
+ "id": "sIL7UvYAZx6L"
+ },
+ "outputs": [],
+ "source": [
+ "# TODO: adapt the code from the tutorial to experiment on the MNIST dataset"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The results for some images might be convincing, but probably you can spot some bad results as well. We will see how we can use different architectures and training objectives to get better results. More importantly, we generate images directly from noise, not knowing what number (if any) will come out on the output."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "<font color='red'>**Question**</font> \n",
+ "How could we change the architecture above to control which number the generator should output ?"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "<font color='green'>**Bonus**</font> \n",
+ "**Copy** the code you implemented above into a new Python cell and adapt it to implement your idea to control the number the generator outputs. Plot the training loss curves and a few examples of your model's outputs (e.g. generate(number=7) # Generates a handwritten 7). Note that we still want to be able to generate different images of the same class ! (i.e. generate(number=4) shoud not generate always the same handwritten 4, but a different one at each time) "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "5fbSgsrE1GqC"
+ },
+ "source": [
+ "# Part 2: Conditional GAN (cGAN)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "7SjXNoT7BUey"
+ },
+ "source": [
+ "Let's take the example of the set described in the following picture: \n",
+ "\n",
+ "<p align=\"center\">\n",
+ "<img height=300px src=\"https://raw.githubusercontent.com/Neyri/Projet-cGAN/master/BE/img/map_streetview.png\"/></p>\n",
+ "<p align=\"center\"></p>\n",
+ "\n",
+ "We have a picture of a map (from Google Maps) and we want to create an image of what the satellite view may look like.\n",
+ "\n",
+ "Therefore, we do not only want to generate a picture from random noise, but rather generate a picture from another picture. For this purpose we will use a cGAN instead of a vanilla GAN, which was introduced in this [paper](https://arxiv.org/pdf/1611.07004.pdf).\n",
+ "\n",
+ "A cGAN is a supervised GAN aiming at mapping a label picture to a real one or a real picture to a label one. As you can see in the diagram below, the discriminator will take as input a pair of images and try to predict if the pair was generated or not. The generator will not generate an image from noise but will intead use an image (label or real) to generate another one (real or label). \n",
+ "\n",
+ "<p align=\"center\">\n",
+ "<img height=300px src=\"https://raw.githubusercontent.com/Neyri/Projet-cGAN/master/BE/img/cgan_map.png\"/></p>\n",
+ "<p align=\"center\"></p>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "0JRaeHfzl6cO"
+ },
+ "source": [
+ "### Generator\n",
+ "\n",
+ "In the cGAN architecture, the generator chosen is a U-Net.\n",
+ "\n",
+ "<p align=\"center\">\n",
+ "<img height=300px src=\"https://raw.githubusercontent.com/Neyri/Projet-cGAN/master/BE/img/unet.png\"/></p>\n",
+ "<p align=\"center\"></p>\n",
+ "\n",
+ "A U-Net takes as input an image, and outputs another image. \n",
+ "\n",
+ "It can be divided into 2 subparts : an encoder and a decoder. \n",
+ "* The encoder takes the input image and reduces its dimension to encode the main features into a vector. \n",
+ "* The decoder takes this vector and map the features stored into an image.\n",
+ "\n",
+ "A U-Net architecture is different from a vanilla convolutional encoder-decoder in that every layer of the decoder takes as input the previous decoded output as well as the output feature map from the encoder layers of the same level. This allows the decoder to map low frequencies information encoded during the descent as well as high frequencies from the original picture. \n",
+ "\n",
+ "<p align=\"center\">\n",
+ "<img height=300px src=\"https://www.researchgate.net/profile/Baris_Turkbey/publication/315514772/figure/fig2/AS:485824962797569@1492841105670/U-net-architecture-Each-box-corresponds-to-a-multi-channel-features-maps-The-number-of.png\"/></p>\n",
+ "<p align=\"center\"></p>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "xFqMOsoYwzFe"
+ },
+ "source": [
+ "The architecture we will implement is the following (the number in the square is the number of filters used).\n",
+ "\n",
+ "<p align=\"center\">\n",
+ "<img height=300px src=\"https://raw.githubusercontent.com/Neyri/Projet-cGAN/master/BE/img/unet_architecture.png\"/></p>\n",
+ "<p align=\"center\"></p>\n",
+ "\n",
+ "The encoder will take as input a colored picture (3 channels: RGB), it will pass through a series of convolution layers to encode the features of the picture. It will then be decoded by the decoder using transposed convolutional layers. These layers will take as input the previous decoded vector AND the encoded features of the same level. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "yzy7y4hmbbX3"
+ },
+ "source": [
+ "For this part the objective is to use a cGAN to generate facades from a template image. For this purpose, we will use the \"Facade\" dataset.\n",
+ "\n",
+ "<p align=\"center\">\n",
+ "<img height=250px src=\"https://storage.googleapis.com/kaggle-datasets-images/926166/1567332/1fd1b7ec805d92b48a1227c376044d0b/dataset-cover.png\"/></p>\n",
+ "<p align=\"center\"></p>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "Q_jf9H_NDESm"
+ },
+ "source": [
+ "Let's start by creating a few classes describing the layers we will use in the U-Net."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {},
+ "colab_type": "code",
+ "id": "uOKvYDyu0w8N"
+ },
+ "outputs": [],
+ "source": [
+ "# Importing all the libraries needed\n",
+ "import os\n",
+ "import glob\n",
+ "import torch\n",
+ "import kagglehub\n",
+ "\n",
+ "import numpy as np\n",
+ "import matplotlib.pyplot as plt\n",
+ "import torchvision.transforms as transforms\n",
+ "\n",
+ "from torch import nn\n",
+ "from PIL import Image\n",
+ "from torch.utils.data import Dataset, DataLoader"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {},
+ "colab_type": "code",
+ "id": "Zk5a6B5hILN2"
+ },
+ "outputs": [],
+ "source": [
+ "# code adapted from https://github.com/milesial/Pytorch-UNet/blob/master/unet/unet_parts.py\n",
+ "\n",
+ "# Input layer\n",
+ "class inconv(nn.Module):\n",
+ " def __init__(self, in_ch, out_ch):\n",
+ " super(inconv, self).__init__()\n",
+ " self.conv = nn.Sequential(\n",
+ " nn.Conv2d(in_ch, out_ch, kernel_size=4, padding=1, stride=2),\n",
+ " nn.LeakyReLU(negative_slope=0.2, inplace=True)\n",
+ " )\n",
+ "\n",
+ " def forward(self, x):\n",
+ " x = self.conv(x)\n",
+ " return x\n",
+ "\n",
+ "# Encoder layer\n",
+ "class down(nn.Module):\n",
+ " def __init__(self, in_ch, out_ch):\n",
+ " super(down, self).__init__()\n",
+ " self.conv = nn.Sequential(\n",
+ " nn.Conv2d(in_ch, out_ch, kernel_size=4, padding=1, stride=2),\n",
+ " nn.BatchNorm2d(out_ch),\n",
+ " nn.LeakyReLU(negative_slope=0.2, inplace=True)\n",
+ " )\n",
+ "\n",
+ " def forward(self, x):\n",
+ " x = self.conv(x)\n",
+ " return x\n",
+ "\n",
+ "# Decoder layer\n",
+ "class up(nn.Module):\n",
+ " def __init__(self, in_ch, out_ch, dropout=False):\n",
+ " super(up, self).__init__()\n",
+ " if dropout :\n",
+ " self.conv = nn.Sequential(\n",
+ " nn.ConvTranspose2d(in_ch, out_ch, kernel_size=4, padding=1, stride=2),\n",
+ " nn.BatchNorm2d(out_ch),\n",
+ " nn.Dropout(0.5, inplace=True),\n",
+ " nn.ReLU(inplace=True)\n",
+ " )\n",
+ " else:\n",
+ " self.conv = nn.Sequential(\n",
+ " nn.ConvTranspose2d(in_ch, out_ch, kernel_size=4, padding=1, stride=2),\n",
+ " nn.BatchNorm2d(out_ch),\n",
+ " nn.ReLU(inplace=True)\n",
+ " )\n",
+ "\n",
+ " def forward(self, x1, x2):\n",
+ " x1 = self.conv(x1)\n",
+ " x = torch.cat([x1, x2], dim=1)\n",
+ " return x\n",
+ "\n",
+ "# Output layer\n",
+ "class outconv(nn.Module):\n",
+ " def __init__(self, in_ch, out_ch):\n",
+ " super(outconv, self).__init__()\n",
+ " self.conv = nn.Sequential(\n",
+ " nn.ConvTranspose2d(in_ch, out_ch, kernel_size=4, padding=1, stride=2),\n",
+ " nn.Tanh()\n",
+ " )\n",
+ "\n",
+ " def forward(self, x):\n",
+ " x = self.conv(x)\n",
+ " return x"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "1rZ5Qz1mBUe8"
+ },
+ "source": [
+ "Now let's create the U-Net using the helper classes defined previously."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {},
+ "colab_type": "code",
+ "id": "4Tbp_535EVPW"
+ },
+ "outputs": [],
+ "source": [
+ "class U_Net(nn.Module):\n",
+ " ''' \n",
+ " Ck denotes a Convolution-BatchNorm-ReLU layer with k filters.\n",
+ " CDk denotes a Convolution-BatchNorm-Dropout-ReLU layer with a dropout rate of 50%\n",
+ " Encoder:\n",
+ " C64 - C128 - C256 - C512 - C512 - C512 - C512 - C512\n",
+ " Decoder:\n",
+ " CD512 - CD1024 - CD1024 - C1024 - C1024 - C512 - C256 - C128\n",
+ " '''\n",
+ " def __init__(self, n_channels, n_classes):\n",
+ " super(U_Net, self).__init__()\n",
+ " # Encoder\n",
+ " self.inc = inconv(n_channels, 64) # 64 filters\n",
+ " # TODO :\n",
+ " # Create the 7 encoder layers called \"down1\" to \"down7\" following this sequence\n",
+ " # C64 - C128 - C256 - C512 - C512 - C512 - C512 - C512\n",
+ " # The first one has already been implemented\n",
+ " \n",
+ " \n",
+ " # Decoder\n",
+ " # TODO :\n",
+ " # Create the 7 decoder layers called up1 to up7 following this sequence :\n",
+ " # CD512 - CD1024 - CD1024 - C1024 - C1024 - C512 - C256 - C128\n",
+ " # The last layer has already been defined\n",
+ " \n",
+ " \n",
+ " self.outc = outconv(128, n_classes) # 128 filters\n",
+ "\n",
+ " def forward(self, x):\n",
+ " x1 = self.inc(x)\n",
+ " x2 = self.down1(x1)\n",
+ " x3 = self.down2(x2)\n",
+ " x4 = self.down3(x3)\n",
+ " x5 = self.down4(x4)\n",
+ " x6 = self.down5(x5)\n",
+ " x7 = self.down6(x6)\n",
+ " x8 = self.down7(x7)\n",
+ " # At this stage x8 is our encoded vector, we will now decode it\n",
+ " x = self.up7(x8, x7)\n",
+ " x = self.up6(x, x6)\n",
+ " x = self.up5(x, x5)\n",
+ " x = self.up4(x, x4)\n",
+ " x = self.up3(x, x3)\n",
+ " x = self.up2(x, x2)\n",
+ " x = self.up1(x, x1)\n",
+ " x = self.outc(x)\n",
+ " return x"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {},
+ "colab_type": "code",
+ "id": "1hmcejTWJSYY"
+ },
+ "outputs": [],
+ "source": [
+ "# We take images that have 3 channels (RGB) as input and output an image that also have 3 channels (RGB)\n",
+ "generator = U_Net(3, 3)\n",
+ "\n",
+ "# Check that the architecture is as expected\n",
+ "print(generator)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "xIXFtHzcBUfO"
+ },
+ "source": [
+ "You should now have a working U-Net."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "RqD1katYBUfP"
+ },
+ "source": [
+ "<font color='red'>**Question**</font> \n",
+ "Knowing the input and output images will have the shape 256x256 with 3 channels, what will be the dimension of the feature map \"x8\" ?\n",
+ "\n",
+ "<font color='red'>**Question**</font> \n",
+ "As you can see, U-net has an encoder-decoder architecture with skip connections. Explain what is the point of the skip connections."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "cchTp3thBUfR"
+ },
+ "source": [
+ "### Discriminator\n",
+ "\n",
+ "In the cGAN architecture, the chosen discriminator is a Patch GAN. Instead of classifying if the whole image is fake or not (binary classification), this discriminator tries to classify if each N × N patch in an image is real or fake.\n",
+ "\n",
+ "\n",
+ "\n",
+ "The size N is given by the depth of the net. According to this table :\n",
+ "\n",
+ "| Number of layers | N |\n",
+ "| ---- | ---- |\n",
+ "| 1 | 16 |\n",
+ "| 2 | 34 |\n",
+ "| 3 | 70 |\n",
+ "| 4 | 142 |\n",
+ "| 5 | 286 |\n",
+ "| 6 | 574 |\n",
+ "\n",
+ "The number of layers actually means the number of layers with `kernel=(4,4)`, `padding=(1,1)` and `stride=(2,2)`. These layers are followed by 2 layers with `kernel=(4,4)`, `padding=(1,1)` and `stride=(1,1)`.\n",
+ "In our case we are going to create a 70x70 PatchGAN."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "ge6I7M0aBUfT"
+ },
+ "source": [
+ "Let's first create a few helping classes."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {},
+ "colab_type": "code",
+ "id": "RYqomFO8BUfV"
+ },
+ "outputs": [],
+ "source": [
+ "class conv_block(nn.Module):\n",
+ " def __init__(self, in_ch, out_ch, use_batchnorm=True, stride=2):\n",
+ " super(conv_block, self).__init__()\n",
+ " if use_batchnorm:\n",
+ " self.conv = nn.Sequential(\n",
+ " nn.Conv2d(in_ch, out_ch, kernel_size=4, padding=1, stride=stride),\n",
+ " nn.BatchNorm2d(out_ch),\n",
+ " nn.LeakyReLU(negative_slope=0.2, inplace=True)\n",
+ " )\n",
+ " else:\n",
+ " self.conv = nn.Sequential(\n",
+ " nn.Conv2d(in_ch, out_ch, kernel_size=4, padding=1, stride=stride),\n",
+ " nn.LeakyReLU(negative_slope=0.2, inplace=True)\n",
+ " )\n",
+ "\n",
+ " def forward(self, x):\n",
+ " x = self.conv(x)\n",
+ " return x\n",
+ " \n",
+ "\n",
+ "class out_block(nn.Module):\n",
+ " def __init__(self, in_ch, out_ch):\n",
+ " super(out_block, self).__init__()\n",
+ " self.conv = nn.Sequential(\n",
+ " nn.Conv2d(in_ch, 1, kernel_size=4, padding=1, stride=1),\n",
+ " nn.Sigmoid()\n",
+ " )\n",
+ "\n",
+ " def forward(self, x):\n",
+ " x = self.conv(x)\n",
+ " return x"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "5m4Dnup4BUfc"
+ },
+ "source": [
+ "Now let's create the Patch GAN discriminator.\n",
+ "As we want a 70x70 Patch GAN, the architecture will be as follows :\n",
+ "```\n",
+ "1. C64 - K4, P1, S2\n",
+ "2. C128 - K4, P1, S2\n",
+ "3. C256 - K4, P1, S2\n",
+ "4. C512 - K4, P1, S1\n",
+ "5. C1 - K4, P1, S1 (output)\n",
+ "```\n",
+ "Where Ck denotes a convolution block with k filters, Kk a kernel of size k, Pk is the padding size and Sk the stride applied.\n",
+ "*Note :* For the first layer, we do not use batchnorm."
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "AH6u5a-PBUfg"
+ },
+ "source": [
+ "<font color='red'>**Question**</font> \n",
+ "Knowing input images will be 256x256 with 3 channels each, how many learnable parameters this neural network has ?"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {},
+ "colab_type": "code",
+ "id": "g_9LxNhGBUfi"
+ },
+ "outputs": [],
+ "source": [
+ "class PatchGAN(nn.Module):\n",
+ " def __init__(self, n_channels, n_classes):\n",
+ " super(PatchGAN, self).__init__()\n",
+ " # TODO : create the 4 first layers named conv1 to conv4\n",
+ " self.conv1 = \n",
+ " self.conv2 = \n",
+ " self.conv3 = \n",
+ " self.conv4 = \n",
+ " # output layer\n",
+ " self.out = out_block(512, n_classes)\n",
+ " \n",
+ " def forward(self, x1, x2):\n",
+ " x = torch.cat([x2, x1], dim=1)\n",
+ " x = self.conv1(x)\n",
+ " x = self.conv2(x)\n",
+ " x = self.conv3(x)\n",
+ " x = self.conv4(x)\n",
+ " x = self.out(x)\n",
+ " return x"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {},
+ "colab_type": "code",
+ "id": "W_sevZRnBUfn"
+ },
+ "outputs": [],
+ "source": [
+ "# We have 6 input channels as we concatenate 2 images (with 3 channels each)\n",
+ "discriminator = PatchGAN(6, 1)\n",
+ "\n",
+ "print(discriminator)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "v_QubOycBUfv"
+ },
+ "source": [
+ "You should now have a working discriminator."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "DiI2CByRBUfz"
+ },
+ "source": [
+ "### Loss functions\n",
+ "\n",
+ "The global loss function will be made of 2 parts :\n",
+ "* The vanilla GAN loss, in which the discriminator tries to maximize the probability it correctly classifies reals and fakes and the generator tries to minimize the probability that the discriminator will predict its outputs are fake.\n",
+ "* An auxiliary image reconstruction objective, in which the generator not only has to fool the discriminator, but also generate an image that is near to the ground truth image.\n",
+ "\n",
+ "Therefore, the loss can be defined as: $$ G^* = arg\\ \\underset{G}{min}\\ \\underset{D}{max}\\ \\mathcal{L}_{cGAN}(G, D) + \\lambda \\mathcal{L}_{L1}(G)$$ \n",
+ "In which \n",
+ "$$ \\mathcal{L}_{cGAN}(G, D) = \\mathbb{E}_{x, y}\\big[logD(x, y)\\big] + \\mathbb{E}_{x, z}\\big[log(1 - D(x, G(x, z)))\\big] $$\n",
+ "$$ \\mathcal{L}_{L1}(G) = \\mathbb{E}_{x, y, z}\\big[||y - G(x, z)||_1\\big] $$"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {},
+ "colab_type": "code",
+ "id": "k4G_xewPBUf4"
+ },
+ "outputs": [],
+ "source": [
+ "# Loss functions\n",
+ "criterion_GAN = torch.nn.BCELoss()\n",
+ "criterion_pixelwise = torch.nn.L1Loss()\n",
+ "\n",
+ "# Loss weight of L1 pixel-wise loss between translated image and real image\n",
+ "lambda_pixel = 100"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "c12q2NwkBUf7"
+ },
+ "source": [
+ "### Training and evaluating models "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {},
+ "colab_type": "code",
+ "id": "vGKjO0UMBUf9"
+ },
+ "outputs": [],
+ "source": [
+ "# parameters\n",
+ "num_epochs = 200 # number of epochs of training\n",
+ "batch_size = 16 # size of the batches\n",
+ "lr = 2e-4 # learning rate\n",
+ "b1 = 0.5 # decay of first order momentum of gradient\n",
+ "b2 = 0.999 # decay of second order momentum of gradient\n",
+ "img_height = 256 # size of image height\n",
+ "img_width = 256 # size of image width\n",
+ "channels = 3 # number of image channels\n",
+ "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "PhPkU7BDYooV"
+ },
+ "source": [
+ "Download the Facades dataset\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {},
+ "colab_type": "code",
+ "id": "8wyPjAxPYsNF"
+ },
+ "outputs": [],
+ "source": [
+ "dataset_path = kagglehub.dataset_download(\"kokeyehya/cmp-facade-db-base\")\n",
+ "\n",
+ "print(\"Path to dataset files:\", dataset_path)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "6DHT9c0_BUgA"
+ },
+ "source": [
+ "Configure the dataloader"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {},
+ "colab_type": "code",
+ "id": "rxi_QIpgBUgB"
+ },
+ "outputs": [],
+ "source": [
+ "class FacadeDataset(Dataset):\n",
+ " def __init__(self, root, transforms_=None, mode='train'):\n",
+ " self.transform = transforms.Compose(transforms_)\n",
+ " self.images_path = sorted(glob.glob(root + '/base/*.jpg'))\n",
+ "\n",
+ " if mode == 'train':\n",
+ " self.images_path = self.images_path[:int(len(self.images_path) * 0.95)]\n",
+ " elif mode == 'val':\n",
+ " self.images_path = self.images_path[int(len(self.images_path) * 0.95):]\n",
+ " else:\n",
+ " raise Exception('Invalid mode! It must be either train or val')\n",
+ "\n",
+ " self.masks_path = [image.split(\".jpg\")[0] + \".png\" for image in self.images_path]\n",
+ " assert len(self.images_path) == len(self.masks_path), \"Number of images and masks must be the same\"\n",
+ "\n",
+ "\n",
+ " def __getitem__(self, index):\n",
+ " img = Image.open(self.images_path[index])\n",
+ " mask = Image.open(self.masks_path[index])\n",
+ " mask = mask.convert('RGB')\n",
+ "\n",
+ " img = self.transform(img)\n",
+ " mask = self.transform(mask)\n",
+ "\n",
+ " return img, mask\n",
+ "\n",
+ " def __len__(self):\n",
+ " return len(self.images_path)\n",
+ " \n",
+ "# Configure dataloaders\n",
+ "transforms_ = [transforms.Resize((img_height, img_width), Image.BICUBIC),\n",
+ " transforms.ToTensor()]\n",
+ "\n",
+ "dataloader = DataLoader(FacadeDataset(dataset_path, transforms_=transforms_),\n",
+ " batch_size=batch_size, shuffle=True, drop_last=True, num_workers=4)\n",
+ "\n",
+ "val_dataloader = DataLoader(FacadeDataset(dataset_path, transforms_=transforms_, mode='val'),\n",
+ " batch_size=batch_size, shuffle=False)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "Okb3LU76BUgG"
+ },
+ "source": [
+ "Check if the loading works and add few helper functions"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {},
+ "colab_type": "code",
+ "id": "xuxq4TZRBUgJ"
+ },
+ "outputs": [],
+ "source": [
+ "def plot2x2Array(image, mask):\n",
+ " f, axarr = plt.subplots(1, 2)\n",
+ " axarr[0].imshow(image)\n",
+ " axarr[1].imshow(mask)\n",
+ "\n",
+ " axarr[0].set_title('Image')\n",
+ " axarr[1].set_title('Mask')\n",
+ "\n",
+ "\n",
+ "def reverse_transform(image):\n",
+ " image = image.numpy().transpose((1, 2, 0))\n",
+ " image = np.clip(image, 0, 1)\n",
+ " image = (image * 255).astype(np.uint8)\n",
+ "\n",
+ " return image\n",
+ "\n",
+ "def plot2x3Array(image, mask,predict):\n",
+ " f, axarr = plt.subplots(1,3,figsize=(15,15))\n",
+ " axarr[0].imshow(image)\n",
+ " axarr[1].imshow(mask)\n",
+ " axarr[2].imshow(predict)\n",
+ " axarr[0].set_title('input')\n",
+ " axarr[1].set_title('real')\n",
+ " axarr[2].set_title('fake')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {},
+ "colab_type": "code",
+ "id": "m2NxLrQEBUgM"
+ },
+ "outputs": [],
+ "source": [
+ "images, masks = next(iter(dataloader))\n",
+ "\n",
+ "for i in range(5):\n",
+ " image = reverse_transform(images[i])\n",
+ " mask = reverse_transform(masks[i])\n",
+ " plot2x2Array(image, mask)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "zAvaxAbxBUgQ"
+ },
+ "source": [
+ "Initialize our GAN"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {},
+ "colab_type": "code",
+ "id": "dVgF3qfDBUgR"
+ },
+ "outputs": [],
+ "source": [
+ "# Calculate output of image discriminator (PatchGAN)\n",
+ "patch_size = (1, img_height//2**3-2, img_width//2**3-2)\n",
+ "\n",
+ "generator = generator.to(device)\n",
+ "discriminator = discriminator.to(device)\n",
+ " \n",
+ "# Optimizers\n",
+ "optimizer_G = torch.optim.Adam(generator.parameters(), lr=lr, betas=(b1, b2))\n",
+ "optimizer_D = torch.optim.Adam(discriminator.parameters(), lr=lr, betas=(b1, b2))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "rN3cbiWaBUgf"
+ },
+ "source": [
+ "Additional auxiliary functions"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {},
+ "colab_type": "code",
+ "id": "msmQQUX-BUgh"
+ },
+ "outputs": [],
+ "source": [
+ "def save_model(epoch, loss_D, loss_G):\n",
+ " # save your model weights\n",
+ " torch.save({\n",
+ " 'epoch': epoch,\n",
+ " 'model_state_dict': generator.state_dict(),\n",
+ " 'optimizer_state_dict': optimizer_G.state_dict(),\n",
+ " 'loss': loss_G,\n",
+ " }, 'generator_'+str(epoch)+'.pth')\n",
+ " torch.save({\n",
+ " 'epoch': epoch,\n",
+ " 'model_state_dict': discriminator.state_dict(),\n",
+ " 'optimizer_state_dict': optimizer_D.state_dict(),\n",
+ " 'loss': loss_D,\n",
+ " }, 'discriminator_'+str(epoch)+'.pth')\n",
+ " \n",
+ "def weights_init_normal(m):\n",
+ " classname = m.__class__.__name__\n",
+ " if classname.find('Conv') != -1:\n",
+ " torch.nn.init.normal_(m.weight.data, 0.0, 0.02)\n",
+ " elif classname.find('BatchNorm2d') != -1:\n",
+ " torch.nn.init.normal_(m.weight.data, 1.0, 0.02)\n",
+ " torch.nn.init.constant_(m.bias.data, 0.0)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "6UXrZLLNBUgq"
+ },
+ "source": [
+ "Train the model ! \n",
+ "But first, complete the loss function in the following training loop: "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {},
+ "colab_type": "code",
+ "id": "7NUuGcQ0SiJw"
+ },
+ "outputs": [],
+ "source": [
+ "# ----------\n",
+ "# Training\n",
+ "# ----------\n",
+ "\n",
+ "losses = []\n",
+ "\n",
+ "# Initialize weights\n",
+ "generator.apply(weights_init_normal)\n",
+ "discriminator.apply(weights_init_normal)\n",
+ "\n",
+ "# train the network\n",
+ "discriminator.train()\n",
+ "generator.train()\n",
+ "print_every = 400\n",
+ "\n",
+ "for epoch in range(num_epochs):\n",
+ " for i, batch in enumerate(dataloader):\n",
+ " # Model inputs\n",
+ " images, masks = batch\n",
+ " images = images.to(device)\n",
+ " masks = masks.to(device)\n",
+ "\n",
+ " # Discriminator labels\n",
+ " valid = torch.ones((images.size(0), *patch_size), requires_grad=False).to(device)\n",
+ " fake = torch.zeros((images.size(0), *patch_size), requires_grad=False).to(device)\n",
+ "\n",
+ " # ------------------\n",
+ " # Train Generators\n",
+ " # ------------------\n",
+ "\n",
+ " optimizer_G.zero_grad()\n",
+ "\n",
+ " # GAN loss\n",
+ " # TODO: Put here your GAN loss\n",
+ "\n",
+ " # Pixel-wise loss\n",
+ " # TODO: Put here your pixel loss\n",
+ "\n",
+ " # Total loss\n",
+ " # TODO: Put here your total loss for the generator\n",
+ "\n",
+ " loss_G.backward()\n",
+ " optimizer_G.step()\n",
+ "\n",
+ " # ---------------------\n",
+ " # Train Discriminator\n",
+ " # ---------------------\n",
+ "\n",
+ " optimizer_D.zero_grad()\n",
+ "\n",
+ " # Real loss\n",
+ " pred_real = discriminator(images, masks)\n",
+ " loss_real = criterion_GAN(pred_real, valid)\n",
+ "\n",
+ " # Fake loss\n",
+ " pred_fake = discriminator(generated_images.detach(), masks)\n",
+ " loss_fake = criterion_GAN(pred_fake, fake)\n",
+ "\n",
+ " # Total loss\n",
+ " loss_D = 0.5 * (loss_real + loss_fake)\n",
+ "\n",
+ " loss_D.backward()\n",
+ " optimizer_D.step()\n",
+ " \n",
+ " # Print some loss stats\n",
+ " if i % print_every == 0:\n",
+ " print(f'Epoch [{epoch+1}/{num_epochs}][{i}/{len(dataloader)}] | d_loss: {loss_D.item():6.4f} | g_loss: {loss_G.item():6.4f}')\n",
+ "\n",
+ " # Keep track of discriminator loss and generator loss\n",
+ " losses.append((loss_D.item(), loss_G.item()))\n",
+ " \n",
+ " if (epoch + 1) % 100 == 0:\n",
+ " print('Saving model...')\n",
+ " save_model(epoch, loss_D, loss_G)\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "Ed-ZbuVWBUgu"
+ },
+ "source": [
+ "Observation of the loss along the training"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {},
+ "colab_type": "code",
+ "id": "nOLW054DTLpg"
+ },
+ "outputs": [],
+ "source": [
+ "fig, ax = plt.subplots()\n",
+ "losses = np.array(losses)\n",
+ "plt.plot(losses.T[0], label='Discriminator')\n",
+ "plt.plot(losses.T[1], label='Generator')\n",
+ "plt.title(\"Training Losses\")\n",
+ "plt.legend()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "S58kJj9HBUgV"
+ },
+ "source": [
+ "If the training takes too much time, you can use a pretrained model in the meantime, to evaluate its performance.\n",
+ "\n",
+ "It is available at : https://partage.liris.cnrs.fr/index.php/s/xwEFmxn9ANeq4zY"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "i0TC5qK3BUg4"
+ },
+ "source": [
+ "### Evaluate your cGAN"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {},
+ "colab_type": "code",
+ "id": "fYBRR6NYBUg6"
+ },
+ "outputs": [],
+ "source": [
+ "def load_model(epoch=200):\n",
+ " if 'generator_'+str(epoch)+'.pth' in os.listdir() and 'discriminator_'+str(epoch)+'.pth' in os.listdir():\n",
+ " checkpoint_generator = torch.load('generator_'+str(epoch)+'.pth', map_location=device)\n",
+ " generator.load_state_dict(checkpoint_generator['model_state_dict'])\n",
+ " optimizer_G.load_state_dict(checkpoint_generator['optimizer_state_dict'])\n",
+ " epoch_G = checkpoint_generator['epoch']\n",
+ " loss_G = checkpoint_generator['loss']\n",
+ "\n",
+ " checkpoint_discriminator = torch.load('discriminator_'+str(epoch)+'.pth', map_location=device)\n",
+ " discriminator.load_state_dict(checkpoint_discriminator['model_state_dict'])\n",
+ " optimizer_D.load_state_dict(checkpoint_discriminator['optimizer_state_dict'])\n",
+ " epoch_D = checkpoint_discriminator['epoch']\n",
+ " loss_D = checkpoint_discriminator['loss']\n",
+ " \n",
+ " else:\n",
+ " print('There isn\\' a training available with this number of epochs')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {},
+ "colab_type": "code",
+ "id": "4V0DwQomBUg9"
+ },
+ "outputs": [],
+ "source": [
+ "load_model(epoch=200)\n",
+ "\n",
+ "# switching mode\n",
+ "generator.eval()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {},
+ "colab_type": "code",
+ "id": "gyvmvkIvBUhB"
+ },
+ "outputs": [],
+ "source": [
+ "# show a sample evaluation image on the training base\n",
+ "image, mask = next(iter(dataloader))\n",
+ "output = generator(mask.to(device))\n",
+ "output = output.cpu().detach()\n",
+ "\n",
+ "for i in range(8):\n",
+ " image_plot = reverse_transform(image[i])\n",
+ " output_plot = reverse_transform(output[i])\n",
+ " mask_plot = reverse_transform(mask[i])\n",
+ " plot2x3Array(mask_plot,image_plot,output_plot)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# show a sample evaluation image on the validation base\n",
+ "image, mask = next(iter(val_dataloader))\n",
+ "output = generator(mask.to(device))\n",
+ "output = output.cpu().detach()\n",
+ "\n",
+ "for i in range(8):\n",
+ " image_plot = reverse_transform(image[i])\n",
+ " output_plot = reverse_transform(output[i])\n",
+ " mask_plot = reverse_transform(mask[i])\n",
+ " plot2x3Array(mask_plot,image_plot,output_plot)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "qkFVjRsOBUhG"
+ },
+ "source": [
+ "<font color='red'>**Question**</font> \n",
+ "Compare results of your model after 100 and 200 epochs"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {},
+ "colab_type": "code",
+ "id": "k85Cl5_UDWyv"
+ },
+ "outputs": [],
+ "source": [
+ "# TODO : Your code here to load and evaluate with a few samples the 2 checkpoints (100 epochs and 200 epochs)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Part 3: Diffusion"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Diffusion models are probabilistic generative models which learn to generate data by iteratively refining random noise through a reverse diffusion process. Given a sample of data, noise is progressvely added in small steps until it becomes pure noise. Then, a neural network is trained to reverse this process and generate realistic data from noise. \n",
+ "\n",
+ "<p align=\"center\">\n",
+ "<img height=300px src=\"https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f5d7da7-52db-4104-9742-a0b4555d8dd6_1300x387.png\"/></p>\n",
+ "<p align=\"center\"></p>\n",
+ "\n",
+ "Diffusion models have gained popularity due to their ability to generate high-quality, diverse, and detailed content, surpassing GANs in the quality of the generated images. \n",
+ "\n",
+ "In this assignment we will focus on DDPMs, which were introduced in this [paper](https://arxiv.org/abs/2006.11239) and laid the foundation for generative diffusion models.\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "For this part, we will use the MNIST dataset, used in part 1"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# TODO: change the variable name to match the one you used in part 1 or reload the dataset\n",
+ "mnist_dataset = \n",
+ "mnist_dataloader ="
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Auxiliary function for plotting images"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def plot1xNArray(images, labels):\n",
+ " f, axarr = plt.subplots(1, len(images))\n",
+ " \n",
+ " for image, ax, label in zip(images, axarr, labels):\n",
+ " ax.imshow(image, cmap='gray')\n",
+ " ax.axis('off')\n",
+ " ax.set_title(label)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "In order to train the model with the diffusion process, we will use a noise scheduler, which will be in charge of the forward diffusion process. The scheduler takes an image, a sample of random noise and a timestep, and return a noisy image for the corresponding timestep. Noise is progressvely added to the image at each timestep, therefore a noisy image at timestep 0 will have barely any noise while a noisy image at the maximum timestep will be basically just noise. \n",
+ "\n",
+ "Let's create a noise scheduler with 1000 max timesteps and visualize some noise images. \n",
+ "\n",
+ "We will use the diffusers library, which provides several tools for training and using diffusion models. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from diffusers import DDPMScheduler\n",
+ "\n",
+ "# TODO: Create the scheduler\n",
+ "noise_scheduler = \n",
+ "\n",
+ "image, _ = mnist_dataset[0]\n",
+ "\n",
+ "# TODO: Create a noise tensor sampled from a normal distribution with the same shape as the image\n",
+ "noise = \n",
+ "\n",
+ "images, labels = [reverse_transform(image)], [\"Original\"]\n",
+ "\n",
+ "for i in [100, 250, 400, 900]:\n",
+ " timestep = torch.LongTensor([i])\n",
+ " noisy_image = noise_scheduler.add_noise(image, noise, timestep)\n",
+ " images.append(reverse_transform(noisy_image))\n",
+ " labels.append(f\"t={i}\")\n",
+ "\n",
+ "plot1xNArray(images, labels)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "For the reverse diffusion process we will use a neural network. Given a noisy image and the corresponding timestep, the goal of the neural network is to predict the noise, which allows for the denoising. \n",
+ "\n",
+ "For the model, we will have a similar architecture as we used for the cGAN generator, a 2D UNet with a few modifications. The main difference will be that we have to indicate to the model which timestep is currently being denoised. For that purpose a timestep embedding is added, therefore the model has 2 inputs, the noisy image and the corresponding timestep. \n",
+ " \n",
+ "In this exercise, we will use an UNet implementation from the diffusers library, which already has the timestep embedding included."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from diffusers import UNet2DModel\n",
+ "\n",
+ "# TODO: Complete the parameters\n",
+ "diffusion_backbone = UNet2DModel(\n",
+ " block_out_channels=(64, 128, 256, 512),\n",
+ " down_block_types=(\"DownBlock2D\", \"DownBlock2D\", \"DownBlock2D\", \"DownBlock2D\"),\n",
+ " up_block_types=(\"UpBlock2D\", \"UpBlock2D\", \"UpBlock2D\", \"UpBlock2D\"),\n",
+ " sample_size=,\n",
+ " in_channels=,\n",
+ " out_channels=,\n",
+ " ).to(device)\n",
+ " \n",
+ "# Optimizer\n",
+ "optimizer = torch.optim.AdamW(diffusion_backbone.parameters(), lr=1e-4)\n",
+ "\n",
+ "print(diffusion_backbone)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "<font color='red'>Question</font> \n",
+ "What are the differences between the UNet used for the cGAN generator and the one defined above ? \n",
+ "Indicate the differences in the architecture by analyzing both models \\_\\_str\\_\\_."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "<font color='green'>**Bonus**</font> \n",
+ "**Extend** the code of the UNet used in part 2 to add the timestep conditioning. Train it in the same way as the UNet2DModel from the diffusers library (using the DDPMScheduler) and compare the results. As in the first bonus exercise, add a label conditioning which allows to control the output of the model (i.e. generate a specific handwritten digit (e.g. \"4\") instead of a random one). Test the model, showcasing some examples."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now, let's train the model"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# ----------------\n",
+ "# Training Loop\n",
+ "# ----------------\n",
+ "torch.backends.cudnn.deterministic = True\n",
+ "\n",
+ "losses = []\n",
+ "num_epochs = 5\n",
+ "print_every = 100\n",
+ "\n",
+ "diffusion_backbone.train()\n",
+ "\n",
+ "for epoch in range(num_epochs):\n",
+ " for i, batch in enumerate(mnist_dataloader):\n",
+ " # Zero the gradients\n",
+ " optimizer.zero_grad()\n",
+ "\n",
+ " # Send input to device\n",
+ " images = batch[0].to(device)\n",
+ "\n",
+ " # Generate noisy images, different timestep for each image in the batch\n",
+ " timesteps = torch.randint(noise_scheduler.config.num_train_timesteps, (images.size(0),), device=device)\n",
+ "\n",
+ " # TODO: Complete the code\n",
+ " noise = \n",
+ " noisy_images = \n",
+ "\n",
+ " # Forward pass\n",
+ " residual = diffusion_backbone(noisy_images, timesteps).sample\n",
+ " \n",
+ " # TODO: Compute the loss\n",
+ " loss = \n",
+ " \n",
+ " loss.backward()\n",
+ " optimizer.step()\n",
+ "\n",
+ " # Print stats\n",
+ " if i % print_every == 0:\n",
+ " print(f'Epoch [{epoch+1}/{num_epochs}][{i}/{len(mnist_dataloader)}] | loss: {loss.item():6.4f}')\n",
+ "\n",
+ " losses.append(loss.item())\n",
+ " torch.save(diffusion_backbone.state_dict(), f\"diffusion_{epoch+1}.pth\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "If the training takes too long, you can download an already trained model from [this link](https://partage.liris.cnrs.fr/index.php/s/AP2t6b3w8SM4Bp5) and use it for inference. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# TODO: Add the path to the model checkpoint for loading the model\n",
+ "\n",
+ "diffusion_backbone.load_state_dict(torch.load())\n",
+ "diffusion_backbone.eval()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Time to generate some images. \n",
+ "\n",
+ "During training, for each data sample, we take a random timestep and correspondent noisy image to give it as input to our model. With suffitient training, the model should learn how to predict the noise in a noisy image for all possible timesteps. \n",
+ "\n",
+ "During inference, to generate an image, we will start from pure noise and step by step predict the noise to go from one noisy image to the next, progressively denoising the image until we reach the timestep 0, in which we should have an image without any noise."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from tqdm import tqdm\n",
+ "\n",
+ "# Start the image as random noise\n",
+ "image = torch.randn((10, 1, 64, 64)).to(device)\n",
+ "\n",
+ "# Create a list of images and labels for visualization\n",
+ "images, labels = [(image / 2 + 0.5).clamp(0, 1).cpu().permute(0, 2, 3, 1).numpy()], [\"Original\"]\n",
+ "\n",
+ "# Use the scheduler to iterate over timesteps\n",
+ "noise_scheduler.set_timesteps(1000)\n",
+ "\n",
+ "for timestep in tqdm(noise_scheduler.timesteps):\n",
+ " with torch.no_grad():\n",
+ " residual = diffusion_backbone(image, timestep).sample\n",
+ " image = noise_scheduler.step(residual, timestep, image).prev_sample\n",
+ "\n",
+ " if timestep.item() % 200 == 0:\n",
+ " images.append((image / 2 + 0.5).clamp(0, 1).cpu().permute(0, 2, 3, 1).numpy())\n",
+ " labels.append(f\"t={timestep.item()}\")\n",
+ "\n",
+ "for i in range(images[0].shape[0]):\n",
+ " plot1xNArray([img[i] for img in images], labels)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The diffusers library also provides *Pipeline* classes, which are wrappers around the model that abstracts the inference loop implemented above. \n",
+ " \n",
+ "We can create a pipeline, giving it the trained model and the noise scheduler, and use it to generate images. In this case, we will only have access to the final image, generated on the last timestep, but not the intermediary images from the denoising process. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from diffusers import DDPMPipeline\n",
+ "\n",
+ "pipeline = DDPMPipeline(diffusion_backbone, noise_scheduler)\n",
+ "generated_images = pipeline(10, output_type=\"np\")\n",
+ "\n",
+ "f, axarr = plt.subplots(1, len(generated_images[\"images\"]))\n",
+ "\n",
+ "for image, ax in zip(generated_images[\"images\"], axarr):\n",
+ " ax.imshow(image, cmap='gray')\n",
+ " ax.axis('off')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Part 4: What about those beautiful images ?"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "<p align=\"center\">\n",
+ "<img height=300px src=\"https://huggingface.co/stabilityai/stable-diffusion-3.5-large/media/main/sd3.5_large_demo.png\"/></p>\n",
+ "<p align=\"center\"></p>\n",
+ "\n",
+ "In this exercise we achieved decent results for very simple datasets. But we are quite far from those beautiful AI generated images we can find online. That is for 2 main reasons:\n",
+ "- Model size: due to the computation and time constrains, we can't really train very large models \n",
+ "- Dataset size: due to the same constrains, we can't use very complex and large datasets, which requires larger models and longer training times.\n",
+ "\n",
+ "Fortunatly, even though we can train those large models with the available hardware and time, we can at least use them for inference ! \n",
+ "\n",
+ "The goal of this part is to learn how to retrive and use a pre-trained diffusion models and also to get creative to come up with some nice prompts to generate cool images. \n",
+ "\n",
+ "We are going to use Stable Diffusion 3.5, which is a state of the art open-source text conditioned model. It takes a prompt in natural language and use it to guide the diffusion process. This type of models are trained with image-text pairs, but can generalize beyond the pairs seen during training, being able to mix several different concepts into a single image. \n",
+ "\n",
+ "In order to save memory, we will use quantization, which consists into converting the model weights types from float16 into float4. That simply means that each model weight will be stored using only 4 bits instead of 16 bits. That allows us to run the model in GPUs with less VRAM and have faster inference, with a small drop in the quality of the results."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "For this part of the assignment restart the notebook kernel to be sure your GPU memory is empty. The memory usage can be verified using the command `nvidia-smi` in a terminal. If your GPU has 2GB of VRAM or less, the model will probably not fit into memory even with quantization. In that case, use Google Colab for this part or use the smaller model indicated below. If you are not happy with the results and have plenty of VRAM available, feel free to increase the quantization to 8 bits or even load the model without quantization."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from diffusers import BitsAndBytesConfig, SD3Transformer2DModel\n",
+ "from diffusers import StableDiffusion3Pipeline\n",
+ "\n",
+ "model_id = \"stabilityai/stable-diffusion-3.5-medium\"\n",
+ "\n",
+ "nf4_config = BitsAndBytesConfig(\n",
+ " load_in_4bit=True,\n",
+ " bnb_4bit_quant_type=\"nf4\",\n",
+ " bnb_4bit_compute_dtype=torch.float16\n",
+ ")\n",
+ "model_nf4 = SD3Transformer2DModel.from_pretrained(\n",
+ " model_id,\n",
+ " subfolder=\"transformer\",\n",
+ " quantization_config=nf4_config,\n",
+ " torch_dtype=torch.float16\n",
+ ")\n",
+ "\n",
+ "pipeline = StableDiffusion3Pipeline.from_pretrained(\n",
+ " model_id, \n",
+ " transformer=model_nf4,\n",
+ " torch_dtype=torch.float16\n",
+ ")\n",
+ "pipeline.enable_model_cpu_offload()\n",
+ "\n",
+ "# TODO: test different prompts and visualize the generated images\n",
+ "# Once you are happy with the results, you can save 3 differet images as png file with the correspondent prompts in a text file\n",
+ "# Don't forget to add the images and prompts in your gitlab submission!\n",
+ "prompt = \n",
+ "\n",
+ "image = pipeline(\n",
+ " prompt=prompt,\n",
+ " num_inference_steps=40,\n",
+ " guidance_scale=4.5,\n",
+ " max_sequence_length=512\n",
+ ").images[0]\n",
+ "image.save(\"generated_image.png\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "If even with the quantization you still run out of GPU memory and you can't use Google Colab, you can use the following code instead, which uses a much smaller model (the results won't be as near as impressive, but it should be able to run even on a CPU, if you have a little patience)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from diffusers import DiffusionPipeline\n",
+ "\n",
+ "pipe = DiffusionPipeline.from_pretrained(\"OFA-Sys/small-stable-diffusion-v0\").to(device)\n",
+ "\n",
+ "prompt = \n",
+ "image = pipe(prompt).images[0]\n",
+ "\n",
+ "image.save(\"not_as_good_generated_image.png\")"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {
+ "colab_type": "text",
+ "id": "rVxSSPJgK60P"
+ },
+ "source": [
+ "# How to submit your Work ?\n",
+ "\n",
+ "This work must be done individually. The expected output is a private repository named gan-diffusion on https://gitlab.ec-lyon.fr. It must contain your notebook (or python files), a README.md file that explains briefly the successive steps performed, the model checkpoints, and the generated images. Don't forget to add your teacher as developer member of the project. The last commit is due before 11:59 pm on Wednesday, April 9th, 2025. Subsequent commits will not be considered."
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "collapsed_sections": [],
+ "name": "BE2 - GAN and cGAN.ipynb",
+ "provenance": []
+ },
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.12"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}
diff --git a/BE2_GAN_and_cGAN.ipynb b/BE2_GAN_and_cGAN.ipynb
deleted file mode 100644
index 01bf87603bd2aeee135f8ff4708167186d77ac36..0000000000000000000000000000000000000000
--- a/BE2_GAN_and_cGAN.ipynb
+++ /dev/null
@@ -1,1234 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "UGwKsKS4GMTN"
- },
- "source": [
- "<h1 ><big><center>MSO 3.4 - Deep Structured Learning</center></big></h1>\n",
- "\n",
- "<h2><big><center> BE 2 - GANs and cGAN </center></big></h2>\n",
- "\n",
- "<h5><big><center>Adapted from <i>Projet d'Option</i> of : Mhamed Jabri, Martin Chauvin, Ahmed Sahraoui, Zakariae Moustaïne and Taoufik Bouchikhi\n",
- "\n",
- "\n",
- "<p align=\"center\">\n",
- "<img height=300px src=\"https://cdn-images-1.medium.com/max/1080/0*tJRy5Chmk4XymxwN.png\"/></p>\n",
- "<p align=\"center\"></p>"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "16aVF81lJuiP"
- },
- "source": [
- "The aim of this assignment is to discover GANs, understand how they are implemented and then explore one specific architecture of GANs that allows us to perform image to image translation (which corresponds to the picture that you can see above this text ! )\n",
- "\n",
- "Before starting the exploration of the world of GANs, here's what students should do and send back for this assignement : \n",
- "* In the \"tutorial\" parts of this assignement that focus on explaining new concepts, you'll find <font color='red'>**questions**</font> that aim to test your understanding of those concepts. \n",
- "* In some of the code cells, you'll have to complete the code and you'll find a \"TO DO\" explaining what you should implement."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "M-WNKvhOP1ED"
- },
- "source": [
- "# Part1: DC-GAN"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "y_r8nMTGQI9a"
- },
- "source": [
- "In this part, we aim to learn and understand the basic concepts of **Generative Adversarial Networks** through a DCGAN and generate new celebrities from the learned network after showing it real celebrities. For this purpose, please study the tutorial here: https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "jiHCy4_UUBFb"
- },
- "source": [
- "##Work to do\n",
- "Now we want to generate handwritten digits using the MNIST dataset. It is available within torvision package (https://pytorch.org/vision/stable/generated/torchvision.datasets.MNIST.html#torchvision.datasets.MNIST)\n",
- "\n",
- "Please re-train the DCGAN and display some automatically generated handwritten digits.\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {},
- "colab_type": "code",
- "id": "sIL7UvYAZx6L"
- },
- "outputs": [],
- "source": [
- "#TO DO: your code here to adapt the code from the tutorial to experiment on MNIST dataset"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "5fbSgsrE1GqC"
- },
- "source": [
- "# Part2: Conditional GAN (cGAN)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "7SjXNoT7BUey"
- },
- "source": [
- "Let's take the example of the set described in the next picture.\n",
- "\n",
- "\n",
- "We have a picture of a map (from Google Maps) and we want to create an image of what the satellite view may look like.\n",
- "\n",
- "As we are not only trying to generate a random picture but a mapping between a picture to another one, we can't use the standard GAN architecture. We will then use a cGAN introduced in this [paper](https://arxiv.org/pdf/1611.07004.pdf).\n",
- "\n",
- "A cGAN is a supervised GAN aiming at mapping a label picture to a real one or a real picture to a label one. As you can see in the diagram below, the discriminator will take as input a pair of images and try to predict if the pair was generated or not. The generator will not only generate an image from noise but will also use an image (label or real) to generate another one (real or label).\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "0JRaeHfzl6cO"
- },
- "source": [
- "### Generator\n",
- "\n",
- "In the cGAN architecture, the generator chosen is a U-Net.\n",
- "\n",
- "\n",
- "A U-Net takes as input an image, and outputs another image. \n",
- "\n",
- "It can be divided into 2 subparts : an encoder and a decoder. \n",
- "* The encoder takes the input image and reduces its dimension to encode the main features into a vector. \n",
- "* The decoder takes this vector and map the features stored into an image.\n",
- "\n",
- "A U-Net architecture is different from a classic encoder-decoder in that every layer of the decoder takes as input the previous decoded output as well as the output vector from the encoder layers of the same level. It allows the decoder to map low frequencies information encoded during the descent as well as high frequencies from the original picture. \n",
- "\n",
- ""
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "xFqMOsoYwzFe"
- },
- "source": [
- "The architecture we will implement is the following (the number in the square is the number of filters used).\n",
- "\n",
- "\n",
- "The encoder will take as input a colored picture (3 channels: RGB), it will pass through a series of convolution layers to encode the features of the picture. It will then be decoded by the decoder using transposed convolutional layers. These layers will take as input the previous decoded vector AND the encoded features of the same level. "
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "yzy7y4hmbbX3"
- },
- "source": [
- "Now, let's create or cGAN to generate facades from a template image. For this purpose, we will use the \"Facade\" dataset available at http://cmp.felk.cvut.cz/~tylecr1/facade/.\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "Q_jf9H_NDESm"
- },
- "source": [
- "Let's first create a few classes describing the layers we will use in the U-Net."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {},
- "colab_type": "code",
- "id": "uOKvYDyu0w8N"
- },
- "outputs": [],
- "source": [
- "# Importing all the libraries needed\n",
- "import matplotlib.pyplot as plt\n",
- "import imageio\n",
- "import glob\n",
- "import random\n",
- "import os\n",
- "import numpy as np\n",
- "import math\n",
- "import itertools\n",
- "import time\n",
- "import datetime\n",
- "import cv2\n",
- "from pathlib import Path\n",
- "from PIL import Image\n",
- "\n",
- "from torch.utils.data import Dataset, DataLoader\n",
- "import torchvision.transforms as transforms\n",
- "from torchvision.utils import save_image, make_grid\n",
- "from torchvision import datasets\n",
- "from torch.autograd import Variable\n",
- "\n",
- "import torch.nn as nn\n",
- "import torch.nn.functional as F\n",
- "import torch"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {},
- "colab_type": "code",
- "id": "Zk5a6B5hILN2"
- },
- "outputs": [],
- "source": [
- "# code adapted from https://github.com/milesial/Pytorch-UNet/blob/master/unet/unet_parts.py\n",
- "\n",
- "# Input layer\n",
- "class inconv(nn.Module):\n",
- " def __init__(self, in_ch, out_ch):\n",
- " super(inconv, self).__init__()\n",
- " self.conv = nn.Sequential(\n",
- " nn.Conv2d(in_ch, out_ch, kernel_size=4, padding=1, stride=2),\n",
- " nn.LeakyReLU(negative_slope=0.2, inplace=True)\n",
- " )\n",
- "\n",
- " def forward(self, x):\n",
- " x = self.conv(x)\n",
- " return x\n",
- "\n",
- "# Encoder layer\n",
- "class down(nn.Module):\n",
- " def __init__(self, in_ch, out_ch):\n",
- " super(down, self).__init__()\n",
- " self.conv = nn.Sequential(\n",
- " nn.Conv2d(in_ch, out_ch, kernel_size=4, padding=1, stride=2),\n",
- " nn.BatchNorm2d(out_ch),\n",
- " nn.LeakyReLU(negative_slope=0.2, inplace=True)\n",
- " )\n",
- "\n",
- " def forward(self, x):\n",
- " x = self.conv(x)\n",
- " return x\n",
- "\n",
- "# Decoder layer\n",
- "class up(nn.Module):\n",
- " def __init__(self, in_ch, out_ch, dropout=False):\n",
- " super(up, self).__init__()\n",
- " if dropout :\n",
- " self.conv = nn.Sequential(\n",
- " nn.ConvTranspose2d(in_ch, out_ch, kernel_size=4, padding=1, stride=2),\n",
- " nn.BatchNorm2d(out_ch),\n",
- " nn.Dropout(0.5, inplace=True),\n",
- " nn.ReLU(inplace=True)\n",
- " )\n",
- " else:\n",
- " self.conv = nn.Sequential(\n",
- " nn.ConvTranspose2d(in_ch, out_ch, kernel_size=4, padding=1, stride=2),\n",
- " nn.BatchNorm2d(out_ch),\n",
- " nn.ReLU(inplace=True)\n",
- " )\n",
- "\n",
- " def forward(self, x1, x2):\n",
- " x1 = self.conv(x1)\n",
- " x = torch.cat([x1, x2], dim=1)\n",
- " return x\n",
- "\n",
- "# Output layer\n",
- "class outconv(nn.Module):\n",
- " def __init__(self, in_ch, out_ch):\n",
- " super(outconv, self).__init__()\n",
- " self.conv = nn.Sequential(\n",
- " nn.ConvTranspose2d(in_ch, out_ch, kernel_size=4, padding=1, stride=2),\n",
- " nn.Tanh()\n",
- " )\n",
- "\n",
- " def forward(self, x):\n",
- " x = self.conv(x)\n",
- " return x"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "1rZ5Qz1mBUe8"
- },
- "source": [
- "Now let's create the U-Net using the helper classes defined previously."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {},
- "colab_type": "code",
- "id": "4Tbp_535EVPW"
- },
- "outputs": [],
- "source": [
- " class U_Net(nn.Module):\n",
- " ''' \n",
- " Ck denotes a Convolution-BatchNorm-ReLU layer with k filters.\n",
- " CDk denotes a Convolution-BatchNorm-Dropout-ReLU layer with a dropout rate of 50%\n",
- " Encoder:\n",
- " C64 - C128 - C256 - C512 - C512 - C512 - C512 - C512\n",
- " Decoder:\n",
- " CD512 - CD1024 - CD1024 - C1024 - C1024 - C512 - C256 - C128\n",
- " '''\n",
- " def __init__(self, n_channels, n_classes):\n",
- " super(U_Net, self).__init__()\n",
- " # Encoder\n",
- " self.inc = inconv(n_channels, 64) # 64 filters\n",
- " # TO DO :\n",
- " # Create the 7 encoder layers called \"down1\" to \"down7\" following this sequence\n",
- " # C64 - C128 - C256 - C512 - C512 - C512 - C512 - C512\n",
- " # The first one has already been implemented\n",
- " \n",
- " \n",
- " # Decoder\n",
- " # TO DO :\n",
- " # Create the 7 decoder layers called up1 to up7 following this sequence :\n",
- " # CD512 - CD1024 - CD1024 - C1024 - C1024 - C512 - C256 - C128\n",
- " # The last layer has already been defined\n",
- " \n",
- " \n",
- " self.outc = outconv(128, n_classes) # 128 filters\n",
- "\n",
- " def forward(self, x):\n",
- " x1 = self.inc(x)\n",
- " x2 = self.down1(x1)\n",
- " x3 = self.down2(x2)\n",
- " x4 = self.down3(x3)\n",
- " x5 = self.down4(x4)\n",
- " x6 = self.down5(x5)\n",
- " x7 = self.down6(x6)\n",
- " x8 = self.down7(x7)\n",
- " # At this stage x8 is our encoded vector, we will now decode it\n",
- " x = self.up7(x8, x7)\n",
- " x = self.up6(x, x6)\n",
- " x = self.up5(x, x5)\n",
- " x = self.up4(x, x4)\n",
- " x = self.up3(x, x3)\n",
- " x = self.up2(x, x2)\n",
- " x = self.up1(x, x1)\n",
- " x = self.outc(x)\n",
- " return x"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {},
- "colab_type": "code",
- "id": "1hmcejTWJSYY"
- },
- "outputs": [],
- "source": [
- "# We take images that have 3 channels (RGB) as input and output an image that also have 3 channels (RGB)\n",
- "generator=U_Net(3,3)\n",
- "# Check that the architecture is as expected\n",
- "generator"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "xIXFtHzcBUfO"
- },
- "source": [
- "You should now have a working U-Net."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "RqD1katYBUfP"
- },
- "source": [
- "<font color='red'>**Question 1**</font> \n",
- "Knowing the input and output images will be 256x256, what will be the dimension of the encoded vector x8 ?\n",
- "\n",
- "<font color='red'>**Question 2**</font> \n",
- "As you can see, U-net has an encoder-decoder architecture with skip connections. Explain why it works better than a traditional encoder-decoder."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "cchTp3thBUfR"
- },
- "source": [
- "### Discriminator\n",
- "\n",
- "In the cGAN architecture, the chosen discriminator is a Patch GAN. It is a convolutional discriminator which enables to produce a map of the input pictures where each pixel represents a patch of size NxN of the input.\n",
- "\n",
- "\n",
- "\n",
- "The size N is given by the depth of the net. According to this table :\n",
- "\n",
- "| Number of layers | N |\n",
- "| ---- | ---- |\n",
- "| 1 | 16 |\n",
- "| 2 | 34 |\n",
- "| 3 | 70 |\n",
- "| 4 | 142 |\n",
- "| 5 | 286 |\n",
- "| 6 | 574 |\n",
- "\n",
- "The number of layers actually means the number of layers with `kernel=(4,4)`, `padding=(1,1)` and `stride=(2,2)`. These layers are followed by 2 layers with `kernel=(4,4)`, `padding=(1,1)` and `stride=(1,1)`.\n",
- "In our case we are going to create a 70x70 PatchGAN."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "ge6I7M0aBUfT"
- },
- "source": [
- "Let's first create a few helping classes."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {},
- "colab_type": "code",
- "id": "RYqomFO8BUfV"
- },
- "outputs": [],
- "source": [
- "class conv_block(nn.Module):\n",
- " def __init__(self, in_ch, out_ch, use_batchnorm=True, stride=2):\n",
- " super(conv_block, self).__init__()\n",
- " if use_batchnorm:\n",
- " self.conv = nn.Sequential(\n",
- " nn.Conv2d(in_ch, out_ch, kernel_size=4, padding=1, stride=stride),\n",
- " nn.BatchNorm2d(out_ch),\n",
- " nn.LeakyReLU(negative_slope=0.2, inplace=True)\n",
- " )\n",
- " else:\n",
- " self.conv = nn.Sequential(\n",
- " nn.Conv2d(in_ch, out_ch, kernel_size=4, padding=1, stride=stride),\n",
- " nn.LeakyReLU(negative_slope=0.2, inplace=True)\n",
- " )\n",
- "\n",
- " def forward(self, x):\n",
- " x = self.conv(x)\n",
- " return x\n",
- " \n",
- "\n",
- "class out_block(nn.Module):\n",
- " def __init__(self, in_ch, out_ch):\n",
- " super(out_block, self).__init__()\n",
- " self.conv = nn.Sequential(\n",
- " nn.Conv2d(in_ch, 1, kernel_size=4, padding=1, stride=1),\n",
- " nn.Sigmoid()\n",
- " )\n",
- "\n",
- " def forward(self, x):\n",
- " x = self.conv(x)\n",
- " return x"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "5m4Dnup4BUfc"
- },
- "source": [
- "Now let's create the Patch GAN discriminator.\n",
- "As we want a 70x70 Patch GAN, the architecture will be as follows :\n",
- "```\n",
- "1. C64 - K4, P1, S2\n",
- "2. C128 - K4, P1, S2\n",
- "3. C256 - K4, P1, S2\n",
- "4. C512 - K4, P1, S1\n",
- "5. C1 - K4, P1, S1 (output)\n",
- "```\n",
- "Where Ck denotes a convolution block with k filters, Kk a kernel of size k, Pk is the padding size and Sk the stride applied.\n",
- "*Note :* For the first layer, we do not use batchnorm."
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "AH6u5a-PBUfg"
- },
- "source": [
- "<font color='red'>**Question 3**</font> \n",
- "Knowing input images will be 256x256 with 3 channels each, how many parameters are there to learn ?"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {},
- "colab_type": "code",
- "id": "g_9LxNhGBUfi"
- },
- "outputs": [],
- "source": [
- "class PatchGAN(nn.Module):\n",
- " def __init__(self, n_channels, n_classes):\n",
- " super(PatchGAN, self).__init__()\n",
- " # TODO :\n",
- " # create the 4 first layers named conv1 to conv4\n",
- " self.conv1 =\n",
- " self.conv2 =\n",
- " self.conv3 =\n",
- " self.conv4 =\n",
- " # output layer\n",
- " self.out = out_block(512, n_classes)\n",
- " \n",
- " def forward(self, x1, x2):\n",
- " x = torch.cat([x2, x1], dim=1)\n",
- " x = self.conv1(x)\n",
- " x = self.conv2(x)\n",
- " x = self.conv3(x)\n",
- " x = self.conv4(x)\n",
- " x = self.out(x)\n",
- " return x"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {},
- "colab_type": "code",
- "id": "W_sevZRnBUfn"
- },
- "outputs": [],
- "source": [
- "# We have 6 input channels as we concatenate 2 images (with 3 channels each)\n",
- "discriminator = PatchGAN(6,1)\n",
- "discriminator"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "v_QubOycBUfv"
- },
- "source": [
- "You should now have a working discriminator."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "DiI2CByRBUfz"
- },
- "source": [
- "### Loss functions\n",
- "\n",
- "As we have seen in the choice of the various architectures for this GAN, the issue is to map both low and high frequencies.\n",
- "To tackle this problem, this GAN rely on the architecture to map the high frequencies (U-Net + PatchGAN) and the loss function to learn low frequencies features. The global loss function will indeed be made of 2 parts :\n",
- "* the first part to map hight frequencies, will try to optimize the mean squared error of the GAN.\n",
- "* the second part to map low frequencies, will minimize the $\\mathcal{L}_1$ norm of the generated picture.\n",
- "\n",
- "So the loss can be defined as $$ G^* = arg\\ \\underset{G}{min}\\ \\underset{D}{max}\\ \\mathcal{L}_{cGAN}(G,D) + \\lambda \\mathcal{L}_1(G)$$"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {},
- "colab_type": "code",
- "id": "k4G_xewPBUf4"
- },
- "outputs": [],
- "source": [
- "# Loss functions\n",
- "criterion_GAN = torch.nn.MSELoss()\n",
- "criterion_pixelwise = torch.nn.L1Loss()\n",
- "\n",
- "# Loss weight of L1 pixel-wise loss between translated image and real image\n",
- "lambda_pixel = 100"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "c12q2NwkBUf7"
- },
- "source": [
- "### Training and evaluating models "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {},
- "colab_type": "code",
- "id": "vGKjO0UMBUf9"
- },
- "outputs": [],
- "source": [
- "# parameters\n",
- "epoch = 0 # epoch to start training from\n",
- "n_epoch = 200 # number of epochs of training\n",
- "batch_size =10 # size of the batches\n",
- "lr = 0.0002 # adam: learning rate\n",
- "b1 =0.5 # adam: decay of first order momentum of gradient\n",
- "b2 = 0.999 # adam: decay of first order momentum of gradient\n",
- "decay_epoch = 100 # epoch from which to start lr decay\n",
- "img_height = 256 # size of image height\n",
- "img_width = 256 # size of image width\n",
- "channels = 3 # number of image channels\n",
- "sample_interval = 500 # interval between sampling of images from generators\n",
- "checkpoint_interval = -1 # interval between model checkpoints\n",
- "cuda = True if torch.cuda.is_available() else False # do you have cuda ?"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "PhPkU7BDYooV"
- },
- "source": [
- "Download the dataset. \n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {},
- "colab_type": "code",
- "id": "8wyPjAxPYsNF"
- },
- "outputs": [],
- "source": [
- "import urllib.request\n",
- "from tqdm import tqdm\n",
- "import os\n",
- "import zipfile\n",
- "\n",
- "def download_hook(t):\n",
- " \"\"\"Wraps tqdm instance.\n",
- " Don't forget to close() or __exit__()\n",
- " the tqdm instance once you're done with it (easiest using `with` syntax).\n",
- " Example\n",
- " -------\n",
- " >>> with tqdm(...) as t:\n",
- " ... reporthook = my_hook(t)\n",
- " ... urllib.request.urlretrieve(..., reporthook=reporthook)\n",
- " \"\"\"\n",
- " last_b = [0]\n",
- "\n",
- " def update_to(b=1, bsize=1, tsize=None):\n",
- " \"\"\"\n",
- " b : int, optional\n",
- " Number of blocks transferred so far [default: 1].\n",
- " bsize : int, optional\n",
- " Size of each block (in tqdm units) [default: 1].\n",
- " tsize : int, optional\n",
- " Total size (in tqdm units). If [default: None] remains unchanged.\n",
- " \"\"\"\n",
- " if tsize is not None:\n",
- " t.total = tsize\n",
- " t.update((b - last_b[0]) * bsize)\n",
- " last_b[0] = b\n",
- "\n",
- " return update_to\n",
- "\n",
- "def download(url, save_dir):\n",
- " filename = url.split('/')[-1]\n",
- " with tqdm(unit = 'B', unit_scale = True, unit_divisor = 1024, miniters = 1, desc = filename) as t:\n",
- " urllib.request.urlretrieve(url, filename = os.path.join(save_dir, filename), reporthook = download_hook(t), data = None)\n",
- "\n",
- "if __name__ == '__main__':\n",
- " # Download ground truth\n",
- " if not os.path.exists(\"CMP_facade_DB_base.zip\"):\n",
- " download(\"http://cmp.felk.cvut.cz/~tylecr1/facade/CMP_facade_DB_base.zip\", \"./\")\n",
- " # Extract in the correct folder\n",
- " with zipfile.ZipFile(\"CMP_facade_DB_base.zip\", 'r') as zip_ref:\n",
- " zip_ref.extractall(\"./facades\")\n",
- " os.rename(\"./facades/base\", \"./facades/train\")\n",
- "\n",
- " # Download ground truth\n",
- " if not os.path.exists(\"CMP_facade_DB_extended.zip\"):\n",
- " download(\"http://cmp.felk.cvut.cz/~tylecr1/facade/CMP_facade_DB_extended.zip\", \"./\")\n",
- " # Extract in the correct folder\n",
- " with zipfile.ZipFile(\"CMP_facade_DB_extended.zip\", 'r') as zip_ref:\n",
- " zip_ref.extractall(\"./facades\")\n",
- " os.rename(\"./facades/extended\", \"./facades/val\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "6DHT9c0_BUgA"
- },
- "source": [
- "Configure the dataloader"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {},
- "colab_type": "code",
- "id": "rxi_QIpgBUgB"
- },
- "outputs": [],
- "source": [
- "class ImageDataset(Dataset):\n",
- " def __init__(self, root, transforms_=None, mode='train'):\n",
- " self.transform = transforms.Compose(transforms_)\n",
- "\n",
- " self.files_img = sorted(glob.glob(os.path.join(root, mode) + '/*.jpg'))\n",
- " if mode == 'val':\n",
- " self.files_img.extend(\n",
- " sorted(glob.glob(os.path.join(root, 'val') + '/*.jpg')))\n",
- "\n",
- " self.files_mask = sorted(glob.glob(os.path.join(root, mode) + '/*.png'))\n",
- " if mode == 'val':\n",
- " self.files_mask.extend(\n",
- " sorted(glob.glob(os.path.join(root, 'val') + '/*.png')))\n",
- " \n",
- " assert len(self.files_img) == len(self.files_mask)\n",
- "\n",
- " def __getitem__(self, index):\n",
- "\n",
- " img = Image.open(self.files_img[index % len(self.files_img)])\n",
- " mask = Image.open(self.files_mask[index % len(self.files_img)])\n",
- " mask = mask.convert('RGB')\n",
- "\n",
- " img = self.transform(img)\n",
- " mask = self.transform(mask)\n",
- "\n",
- " return img, mask\n",
- "\n",
- " def __len__(self):\n",
- " return len(self.files_img)\n",
- " \n",
- "# Configure dataloaders\n",
- "transforms_ = [transforms.Resize((img_height, img_width), Image.BICUBIC),\n",
- " transforms.ToTensor()] # transforms.Normalize((0.5,0.5,0.5), (0.5,0.5,0.5))\n",
- "\n",
- "dataloader = DataLoader(ImageDataset(\"facades\", transforms_=transforms_),\n",
- " batch_size=16, shuffle=True)\n",
- "\n",
- "val_dataloader = DataLoader(ImageDataset(\"facades\", transforms_=transforms_, mode='val'),\n",
- " batch_size=8, shuffle=False)\n",
- "\n",
- "# Tensor type\n",
- "Tensor = torch.cuda.FloatTensor if cuda else torch.FloatTensor"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "Okb3LU76BUgG"
- },
- "source": [
- "Check the loading works and a few helper functions"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {},
- "colab_type": "code",
- "id": "xuxq4TZRBUgJ"
- },
- "outputs": [],
- "source": [
- "def plot2x2Array(image, mask):\n",
- " f, axarr = plt.subplots(1, 2)\n",
- " axarr[0].imshow(image)\n",
- " axarr[1].imshow(mask)\n",
- "\n",
- " axarr[0].set_title('Image')\n",
- " axarr[1].set_title('Mask')\n",
- "\n",
- "\n",
- "def reverse_transform(image):\n",
- " image = image.numpy().transpose((1, 2, 0))\n",
- " image = np.clip(image, 0, 1)\n",
- " image = (image * 255).astype(np.uint8)\n",
- "\n",
- " return image\n",
- "\n",
- "def plot2x3Array(image, mask,predict):\n",
- " f, axarr = plt.subplots(1,3,figsize=(15,15))\n",
- " axarr[0].imshow(image)\n",
- " axarr[1].imshow(mask)\n",
- " axarr[2].imshow(predict)\n",
- " axarr[0].set_title('input')\n",
- " axarr[1].set_title('real')\n",
- " axarr[2].set_title('fake')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {},
- "colab_type": "code",
- "id": "m2NxLrQEBUgM"
- },
- "outputs": [],
- "source": [
- "image, mask = next(iter(dataloader))\n",
- "image = reverse_transform(image[0])\n",
- "mask = reverse_transform(mask[0])\n",
- "plot2x2Array(image, mask)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "zAvaxAbxBUgQ"
- },
- "source": [
- "Initialize our GAN"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {},
- "colab_type": "code",
- "id": "dVgF3qfDBUgR"
- },
- "outputs": [],
- "source": [
- "# Calculate output of image discriminator (PatchGAN)\n",
- "patch = (1, img_height//2**3-2, img_width//2**3-2)\n",
- "\n",
- "if cuda:\n",
- " generator = generator.cuda()\n",
- " discriminator = discriminator.cuda()\n",
- " criterion_GAN.cuda()\n",
- " criterion_pixelwise.cuda()\n",
- " \n",
- "# Optimizers\n",
- "optimizer_G = torch.optim.Adam(generator.parameters(), lr=lr, betas=(b1, b2))\n",
- "optimizer_D = torch.optim.Adam(discriminator.parameters(), lr=lr, betas=(b1, b2))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "rN3cbiWaBUgf"
- },
- "source": [
- "Start training"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {},
- "colab_type": "code",
- "id": "msmQQUX-BUgh"
- },
- "outputs": [],
- "source": [
- "def save_model(epoch):\n",
- " # save your work\n",
- " torch.save({\n",
- " 'epoch': epoch,\n",
- " 'model_state_dict': generator.state_dict(),\n",
- " 'optimizer_state_dict': optimizer_G.state_dict(),\n",
- " 'loss': loss_G,\n",
- " }, 'generator_'+str(epoch)+'.pth')\n",
- " torch.save({\n",
- " 'epoch': epoch,\n",
- " 'model_state_dict': discriminator.state_dict(),\n",
- " 'optimizer_state_dict': optimizer_D.state_dict(),\n",
- " 'loss': loss_D,\n",
- " }, 'discriminator_'+str(epoch)+'.pth')\n",
- " \n",
- "def weights_init_normal(m):\n",
- " classname = m.__class__.__name__\n",
- " if classname.find('Conv') != -1:\n",
- " torch.nn.init.normal_(m.weight.data, 0.0, 0.02)\n",
- " elif classname.find('BatchNorm2d') != -1:\n",
- " torch.nn.init.normal_(m.weight.data, 1.0, 0.02)\n",
- " torch.nn.init.constant_(m.bias.data, 0.0)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "6UXrZLLNBUgq"
- },
- "source": [
- "<font color='red'>Complete the loss function </font> in the following training code and train your network: "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {},
- "colab_type": "code",
- "id": "7NUuGcQ0SiJw"
- },
- "outputs": [],
- "source": [
- "# ----------\n",
- "# Training\n",
- "# ----------\n",
- "\n",
- "losses = []\n",
- "num_epochs = 200\n",
- "\n",
- "# Initialize weights\n",
- "generator.apply(weights_init_normal)\n",
- "discriminator.apply(weights_init_normal)\n",
- "epoch_D = 0\n",
- "epoch_G = 0\n",
- "\n",
- "# train the network\n",
- "discriminator.train()\n",
- "generator.train()\n",
- "print_every = 400\n",
- "\n",
- "for epoch in range(epoch_G, num_epochs):\n",
- " for i, batch in enumerate(dataloader):\n",
- "\n",
- " # Model inputs\n",
- " real_A = Variable(batch[0].type(Tensor))\n",
- " real_B = Variable(batch[1].type(Tensor))\n",
- "\n",
- " # Adversarial ground truths\n",
- " valid = Variable(Tensor(np.ones((real_B.size(0), *patch))), requires_grad=False)\n",
- " fake = Variable(Tensor(np.zeros((real_B.size(0), *patch))), requires_grad=False)\n",
- "\n",
- " # ------------------\n",
- " # Train Generators\n",
- " # ------------------\n",
- "\n",
- " optimizer_G.zero_grad()\n",
- "\n",
- " # GAN loss\n",
- " # TO DO: Put here your GAN loss\n",
- "\n",
- " # Pixel-wise loss\n",
- " # TO DO: Put here your pixel loss\n",
- "\n",
- " # Total loss\n",
- " # TO DO: Put here your total loss\n",
- "\n",
- " loss_G.backward()\n",
- "\n",
- " optimizer_G.step()\n",
- "\n",
- " # ---------------------\n",
- " # Train Discriminator\n",
- " # ---------------------\n",
- "\n",
- " optimizer_D.zero_grad()\n",
- "\n",
- " # Real loss\n",
- " pred_real = discriminator(real_A, real_B)\n",
- " loss_real = criterion_GAN(pred_real, valid)\n",
- "\n",
- " # Fake loss\n",
- " pred_fake = discriminator(fake_A.detach(), real_B)\n",
- " loss_fake = criterion_GAN(pred_fake, fake)\n",
- "\n",
- " # Total loss\n",
- " loss_D = 0.5 * (loss_real + loss_fake)\n",
- "\n",
- " loss_D.backward()\n",
- " optimizer_D.step()\n",
- " \n",
- " # Print some loss stats\n",
- " if i % print_every == 0:\n",
- " # print discriminator and generator loss\n",
- " print('Epoch [{:5d}/{:5d}] | d_loss: {:6.4f} | g_loss: {:6.4f}'.format(\n",
- " epoch+1, num_epochs, loss_D.item(), loss_G.item()))\n",
- " ## AFTER EACH EPOCH##\n",
- " # append discriminator loss and generator loss\n",
- " losses.append((loss_D.item(), loss_G.item()))\n",
- " if epoch % 100 == 0:\n",
- " print('Saving model...')\n",
- " save_model(epoch)\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "Ed-ZbuVWBUgu"
- },
- "source": [
- "Observation of the loss along the training"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {},
- "colab_type": "code",
- "id": "nOLW054DTLpg"
- },
- "outputs": [],
- "source": [
- "fig, ax = plt.subplots()\n",
- "losses = np.array(losses)\n",
- "plt.plot(losses.T[0], label='Discriminator')\n",
- "plt.plot(losses.T[1], label='Generator')\n",
- "plt.title(\"Training Losses\")\n",
- "plt.legend()\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "S58kJj9HBUgV"
- },
- "source": [
- "If the training takes too much time, you can use a pretrained model in the meantime, to evaluate its performance.\n",
- "\n",
- "It is available at : https://partage.liris.cnrs.fr/index.php/s/xwEFmxn9ANeq4zY"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "i0TC5qK3BUg4"
- },
- "source": [
- "### Evaluate your cGAN"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {},
- "colab_type": "code",
- "id": "fYBRR6NYBUg6"
- },
- "outputs": [],
- "source": [
- "def load_model(epoch=200):\n",
- " if 'generator_'+str(epoch)+'.pth' in os.listdir() and 'discriminator_'+str(epoch)+'.pth' in os.listdir():\n",
- " if cuda:\n",
- " checkpoint_generator = torch.load('generator_'+str(epoch)+'.pth')\n",
- " else:\n",
- " checkpoint_generator = torch.load('generator_'+str(epoch)+'.pth', map_location='cpu')\n",
- " generator.load_state_dict(checkpoint_generator['model_state_dict'])\n",
- " optimizer_G.load_state_dict(checkpoint_generator['optimizer_state_dict'])\n",
- " epoch_G = checkpoint_generator['epoch']\n",
- " loss_G = checkpoint_generator['loss']\n",
- "\n",
- " if cuda:\n",
- " checkpoint_discriminator = torch.load('discriminator_'+str(epoch)+'.pth')\n",
- " else:\n",
- " checkpoint_discriminator = torch.load('discriminator_'+str(epoch)+'.pth', map_location='cpu')\n",
- " discriminator.load_state_dict(checkpoint_discriminator['model_state_dict'])\n",
- " optimizer_D.load_state_dict(checkpoint_discriminator['optimizer_state_dict'])\n",
- " epoch_D = checkpoint_discriminator['epoch']\n",
- " loss_D = checkpoint_discriminator['loss']\n",
- " else:\n",
- " print('There isn\\' a training available with this number of epochs')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {},
- "colab_type": "code",
- "id": "4V0DwQomBUg9"
- },
- "outputs": [],
- "source": [
- "load_model(epoch=200)\n",
- "\n",
- "# switching mode\n",
- "generator.eval()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {},
- "colab_type": "code",
- "id": "gyvmvkIvBUhB"
- },
- "outputs": [],
- "source": [
- "# show a sample evaluation image on the training base\n",
- "image, mask = next(iter(dataloader))\n",
- "output = generator(mask.type(Tensor))\n",
- "output = output.view(16, 3, 256, 256)\n",
- "output = output.cpu().detach()\n",
- "for i in range(8):\n",
- " image_plot = reverse_transform(image[i])\n",
- " output_plot = reverse_transform(output[i])\n",
- " mask_plot = reverse_transform(mask[i])\n",
- " plot2x3Array(mask_plot,image_plot,output_plot)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {},
- "colab_type": "code",
- "id": "nqvrxBoGBUhD"
- },
- "outputs": [],
- "source": [
- "# show a sample evaluation image on the validation dataset\n",
- "image, mask = next(iter(val_dataloader))\n",
- "output = generator(mask.type(Tensor))\n",
- "output = output.view(8, 3, 256, 256)\n",
- "output = output.cpu().detach()\n",
- "for i in range(8):\n",
- " image_plot = reverse_transform(image[i])\n",
- " output_plot = reverse_transform(output[i])\n",
- " mask_plot = reverse_transform(mask[i])\n",
- " plot2x3Array(mask_plot,image_plot,output_plot)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "qkFVjRsOBUhG"
- },
- "source": [
- "<font color='red'>**Question 4**</font> \n",
- "Compare results for 100 and 200 epochs"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {},
- "colab_type": "code",
- "id": "k85Cl5_UDWyv"
- },
- "outputs": [],
- "source": [
- "# TO DO : Your code here to load and evaluate with a few samples\n",
- "# a model after 100 epochs\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {},
- "colab_type": "code",
- "id": "_GbMIfRXBUhH"
- },
- "outputs": [],
- "source": [
- "# And finally :\n",
- "if cuda:\n",
- " torch.cuda.empty_cache()"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {
- "colab_type": "text",
- "id": "rVxSSPJgK60P"
- },
- "source": [
- "# How to submit your Work ?\n",
- "\n",
- "This work must be done individually. The expected output is a private repository named gan-cgan on https://gitlab.ec-lyon.fr. It must contain your notebook (or python files) and a README.md file that explains briefly the successive steps of the project. Don't forget to add your teacher as developer member of the project. The last commit is due before 11:59 pm on Monday, April 1st, 2024. Subsequent commits will not be considered."
- ]
- }
- ],
- "metadata": {
- "colab": {
- "collapsed_sections": [],
- "name": "BE2 - GAN and cGAN.ipynb",
- "provenance": []
- },
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.8.8"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 1
-}
diff --git a/BE2_GAN_and_cGAN.py b/BE2_GAN_and_cGAN.py
deleted file mode 100644
index 8acf1626ea812c16daff356090f249a65deb74ab..0000000000000000000000000000000000000000
--- a/BE2_GAN_and_cGAN.py
+++ /dev/null
@@ -1,689 +0,0 @@
-# -*- coding: utf-8 -*-
-"""BE2 - GAN and cGAN.ipynb
-
-<h1 ><big><center>MSO 3.4 - Deep Structured Learning</center></big></h1>
-
-<h2><big><center> BE 2 - GANs and cGAN </center></big></h2>
-
-<h5><big><center>Adapted from <i>Projet d'Option</i> of : Mhamed Jabri, Martin Chauvin, Ahmed Sahraoui, Zakariae Moustaïne and Taoufik Bouchikhi
-
-
-<p align="center">
-<img height=300px src="https://cdn-images-1.medium.com/max/1080/0*tJRy5Chmk4XymxwN.png"/></p>
-<p align="center"></p>
-
-The aim of this assignment is to discover GANs, understand how they are implemented and then explore one specific architecture of GANs that allows us to perform image to image translation (which corresponds to the picture that you can see above this text ! )
-
-Before starting the exploration of the world of GANs, here's what students should do and send back for this assignement :
-* In the "tutorial" parts of this assignement that focus on explaining new concepts, you'll find <font color='red'>**questions**</font> that aim to test your understanding of those concepts.
-* In some of the code cells, you'll have to complete the code and you'll find a "TO DO" explaining what you should implement.
-
-# Part1: DC-GAN
-
-In this part, we aim to learn and understand the basic concepts of **Generative Adversarial Networks** through a DCGAN and generate new celebrities from the learned network after showing it real celebrities. For this purpose, please study the tutorial here: https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html
-
-##Work to do
-Now we want to generate handwritten digits using the MNIST dataset. It is available within torvision package (https://pytorch.org/vision/stable/generated/torchvision.datasets.MNIST.html#torchvision.datasets.MNIST)
-
-Please re-train the DCGAN and display some automatically generated handwritten digits.
-"""
-
-#TO DO: your code here to adapt the code from the tutorial to experiment on MNIST dataset
-
-"""# Part2: Conditional GAN (cGAN)
-
-Let's take the example of the set described in the next picture.
-
-
-We have a picture of a map (from Google Maps) and we want to create an image of what the satellite view may look like.
-
-As we are not only trying to generate a random picture but a mapping between a picture to another one, we can't use the standard GAN architecture. We will then use a cGAN.
-
-A cGAN is a supervised GAN aiming at mapping a label picture to a real one or a real picture to a label one. As you can see in the diagram below, the discriminator will take as input a pair of images and try to predict if the pair was generated or not. The generator will not only generate an image from noise but will also use an image (label or real) to generate another one (real or label).
-
-
-### Generator
-
-In the cGAN architecture, the generator chosen is a U-Net.
-
-
-A U-Net takes as input an image, and outputs another image.
-
-It can be divided into 2 subparts : an encoder and a decoder.
-* The encoder takes the input image and reduces its dimension to encode the main features into a vector.
-* The decoder takes this vector and map the features stored into an image.
-
-A U-Net architecture is different from a classic encoder-decoder in that every layer of the decoder takes as input the previous decoded output as well as the output vector from the encoder layers of the same level. It allows the decoder to map low frequencies information encoded during the descent as well as high frequencies from the original picture.
-
-
-
-The architecture we will implement is the following (the number in the square is the number of filters used).
-
-
-The encoder will take as input a colored picture (3 channels: RGB), it will pass through a series of convolution layers to encode the features of the picture. It will then be decoded by the decoder using transposed convolutional layers. These layers will take as input the previous decoded vector AND the encoded features of the same level.
-
-Now, let's create or cGAN to generate facades from a template image. For this purpose, we will use the "Facade" dataset available at http://cmp.felk.cvut.cz/~tylecr1/facade/.
-
-Let's first create a few classes describing the layers we will use in the U-Net.
-"""
-
-# Importing all the libraries needed
-import matplotlib.pyplot as plt
-import imageio
-import glob
-import random
-import os
-import numpy as np
-import math
-import itertools
-import time
-import datetime
-import cv2
-from pathlib import Path
-from PIL import Image
-
-from torch.utils.data import Dataset, DataLoader
-import torchvision.transforms as transforms
-from torchvision.utils import save_image, make_grid
-from torchvision import datasets
-from torch.autograd import Variable
-
-import torch.nn as nn
-import torch.nn.functional as F
-import torch
-
-# code adapted from https://github.com/milesial/Pytorch-UNet/blob/master/unet/unet_parts.py
-
-# Input layer
-class inconv(nn.Module):
- def __init__(self, in_ch, out_ch):
- super(inconv, self).__init__()
- self.conv = nn.Sequential(
- nn.Conv2d(in_ch, out_ch, kernel_size=4, padding=1, stride=2),
- nn.LeakyReLU(negative_slope=0.2, inplace=True)
- )
-
- def forward(self, x):
- x = self.conv(x)
- return x
-
-# Encoder layer
-class down(nn.Module):
- def __init__(self, in_ch, out_ch):
- super(down, self).__init__()
- self.conv = nn.Sequential(
- nn.Conv2d(in_ch, out_ch, kernel_size=4, padding=1, stride=2),
- nn.BatchNorm2d(out_ch),
- nn.LeakyReLU(negative_slope=0.2, inplace=True)
- )
-
- def forward(self, x):
- x = self.conv(x)
- return x
-
-# Decoder layer
-class up(nn.Module):
- def __init__(self, in_ch, out_ch, dropout=False):
- super(up, self).__init__()
- if dropout :
- self.conv = nn.Sequential(
- nn.ConvTranspose2d(in_ch, out_ch, kernel_size=4, padding=1, stride=2),
- nn.BatchNorm2d(out_ch),
- nn.Dropout(0.5, inplace=True),
- nn.ReLU(inplace=True)
- )
- else:
- self.conv = nn.Sequential(
- nn.ConvTranspose2d(in_ch, out_ch, kernel_size=4, padding=1, stride=2),
- nn.BatchNorm2d(out_ch),
- nn.ReLU(inplace=True)
- )
-
- def forward(self, x1, x2):
- x1 = self.conv(x1)
- x = torch.cat([x1, x2], dim=1)
- return x
-
-# Output layer
-class outconv(nn.Module):
- def __init__(self, in_ch, out_ch):
- super(outconv, self).__init__()
- self.conv = nn.Sequential(
- nn.ConvTranspose2d(in_ch, out_ch, kernel_size=4, padding=1, stride=2),
- nn.Tanh()
- )
-
- def forward(self, x):
- x = self.conv(x)
- return x
-
-"""Now let's create the U-Net using the helper classes defined previously."""
-
-class U_Net(nn.Module):
- '''
- Ck denotes a Convolution-BatchNorm-ReLU layer with k filters.
- CDk denotes a Convolution-BatchNorm-Dropout-ReLU layer with a dropout rate of 50%
- Encoder:
- C64 - C128 - C256 - C512 - C512 - C512 - C512 - C512
- Decoder:
- CD512 - CD1024 - CD1024 - C1024 - C1024 - C512 - C256 - C128
- '''
- def __init__(self, n_channels, n_classes):
- super(U_Net, self).__init__()
- # Encoder
- self.inc = inconv(n_channels, 64) # 64 filters
- # TO DO :
- # Create the 7 encoder layers called "down1" to "down7" following this sequence
- # C64 - C128 - C256 - C512 - C512 - C512 - C512 - C512
- # The first one has already been implemented
-
-
- # Decoder
- # TO DO :
- # Create the 7 decoder layers called up1 to up7 following this sequence :
- # CD512 - CD1024 - CD1024 - C1024 - C1024 - C512 - C256 - C128
- # The last layer has already been defined
-
-
- self.outc = outconv(128, n_classes) # 128 filters
-
- def forward(self, x):
- x1 = self.inc(x)
- x2 = self.down1(x1)
- x3 = self.down2(x2)
- x4 = self.down3(x3)
- x5 = self.down4(x4)
- x6 = self.down5(x5)
- x7 = self.down6(x6)
- x8 = self.down7(x7)
- # At this stage x8 is our encoded vector, we will now decode it
- x = self.up7(x8, x7)
- x = self.up6(x, x6)
- x = self.up5(x, x5)
- x = self.up4(x, x4)
- x = self.up3(x, x3)
- x = self.up2(x, x2)
- x = self.up1(x, x1)
- x = self.outc(x)
- return x
-
-# We take images that have 3 channels (RGB) as input and output an image that also have 3 channels (RGB)
-generator=U_Net(3,3)
-# Check that the architecture is as expected
-generator
-
-"""You should now have a working U-Net.
-
-<font color='red'>**Question 1**</font>
-Knowing the input and output images will be 256x256, what will be the dimension of the encoded vector x8 ?
-
-<font color='red'>**Question 2**</font>
-As you can see, U-net has an encoder-decoder architecture with skip connections. Explain why it works better than a traditional encoder-decoder.
-
-### Discriminator
-
-In the cGAN architecture, the chosen discriminator is a Patch GAN. It is a convolutional discriminator which enables to produce a map of the input pictures where each pixel represents a patch of size NxN of the input.
-
-
-
-The size N is given by the depth of the net. According to this table :
-
-| Number of layers | N |
-| ---- | ---- |
-| 1 | 16 |
-| 2 | 34 |
-| 3 | 70 |
-| 4 | 142 |
-| 5 | 286 |
-| 6 | 574 |
-
-The number of layers actually means the number of layers with `kernel=(4,4)`, `padding=(1,1)` and `stride=(2,2)`. These layers are followed by 2 layers with `kernel=(4,4)`, `padding=(1,1)` and `stride=(1,1)`.
-In our case we are going to create a 70x70 PatchGAN.
-
-Let's first create a few helping classes.
-"""
-
-class conv_block(nn.Module):
- def __init__(self, in_ch, out_ch, use_batchnorm=True, stride=2):
- super(conv_block, self).__init__()
- if use_batchnorm:
- self.conv = nn.Sequential(
- nn.Conv2d(in_ch, out_ch, kernel_size=4, padding=1, stride=stride),
- nn.BatchNorm2d(out_ch),
- nn.LeakyReLU(negative_slope=0.2, inplace=True)
- )
- else:
- self.conv = nn.Sequential(
- nn.Conv2d(in_ch, out_ch, kernel_size=4, padding=1, stride=stride),
- nn.LeakyReLU(negative_slope=0.2, inplace=True)
- )
-
- def forward(self, x):
- x = self.conv(x)
- return x
-
-
-class out_block(nn.Module):
- def __init__(self, in_ch, out_ch):
- super(out_block, self).__init__()
- self.conv = nn.Sequential(
- nn.Conv2d(in_ch, 1, kernel_size=4, padding=1, stride=1),
- nn.Sigmoid()
- )
-
- def forward(self, x):
- x = self.conv(x)
- return x
-
-"""Now let's create the Patch GAN discriminator.
-As we want a 70x70 Patch GAN, the architecture will be as follows :
-```
-1. C64 - K4, P1, S2
-2. C128 - K4, P1, S2
-3. C256 - K4, P1, S2
-4. C512 - K4, P1, S1
-5. C1 - K4, P1, S1 (output)
-```
-Where Ck denotes a convolution block with k filters, Kk a kernel of size k, Pk is the padding size and Sk the stride applied.
-*Note :* For the first layer, we do not use batchnorm.
-
-<font color='red'>**Question 3**</font>
-Knowing the input and output images will be 256x256, what will be the dimension of the encoded vector x8 ?Knowing input images will be 256x256 with 3 channels each, how many parameters are there to learn ?
-"""
-
-class PatchGAN(nn.Module):
- def __init__(self, n_channels, n_classes):
- super(PatchGAN, self).__init__()
- # TODO :
- # create the 4 first layers named conv1 to conv4
- self.conv1 =
- self.conv2 =
- self.conv3 =
- self.conv4 =
- # output layer
- self.out = out_block(512, n_classes)
-
- def forward(self, x1, x2):
- x = torch.cat([x2, x1], dim=1)
- x = self.conv1(x)
- x = self.conv2(x)
- x = self.conv3(x)
- x = self.conv4(x)
- x = self.out(x)
- return x
-
-# We have 6 input channels as we concatenate 2 images (with 3 channels each)
-discriminator = PatchGAN(6,1)
-discriminator
-
-"""You should now have a working discriminator.
-
-### Loss functions
-
-As we have seen in the choice of the various architectures for this GAN, the issue is to map both low and high frequencies.
-To tackle this problem, this GAN rely on the architecture to map the high frequencies (U-Net + PatchGAN) and the loss function to learn low frequencies features. The global loss function will indeed be made of 2 parts :
-* the first part to map hight frequencies, will try to optimize the mean squared error of the GAN.
-* the second part to map low frequencies, will minimize the $\mathcal{L}_1$ norm of the generated picture.
-
-So the loss can be defined as $$ G^* = arg\ \underset{G}{min}\ \underset{D}{max}\ \mathcal{L}_{cGAN}(G,D) + \lambda \mathcal{L}_1(G)$$
-"""
-
-# Loss functions
-criterion_GAN = torch.nn.MSELoss()
-criterion_pixelwise = torch.nn.L1Loss()
-
-# Loss weight of L1 pixel-wise loss between translated image and real image
-lambda_pixel = 100
-
-"""### Training and evaluating models"""
-
-# parameters
-epoch = 0 # epoch to start training from
-n_epoch = 200 # number of epochs of training
-batch_size =10 # size of the batches
-lr = 0.0002 # adam: learning rate
-b1 =0.5 # adam: decay of first order momentum of gradient
-b2 = 0.999 # adam: decay of first order momentum of gradient
-decay_epoch = 100 # epoch from which to start lr decay
-img_height = 256 # size of image height
-img_width = 256 # size of image width
-channels = 3 # number of image channels
-sample_interval = 500 # interval between sampling of images from generators
-checkpoint_interval = -1 # interval between model checkpoints
-cuda = True if torch.cuda.is_available() else False # do you have cuda ?
-
-"""Download the dataset."""
-
-import urllib.request
-from tqdm import tqdm
-import os
-import zipfile
-
-def download_hook(t):
- """Wraps tqdm instance.
- Don't forget to close() or __exit__()
- the tqdm instance once you're done with it (easiest using `with` syntax).
- Example
- -------
- >>> with tqdm(...) as t:
- ... reporthook = my_hook(t)
- ... urllib.request.urlretrieve(..., reporthook=reporthook)
- """
- last_b = [0]
-
- def update_to(b=1, bsize=1, tsize=None):
- """
- b : int, optional
- Number of blocks transferred so far [default: 1].
- bsize : int, optional
- Size of each block (in tqdm units) [default: 1].
- tsize : int, optional
- Total size (in tqdm units). If [default: None] remains unchanged.
- """
- if tsize is not None:
- t.total = tsize
- t.update((b - last_b[0]) * bsize)
- last_b[0] = b
-
- return update_to
-
-def download(url, save_dir):
- filename = url.split('/')[-1]
- with tqdm(unit = 'B', unit_scale = True, unit_divisor = 1024, miniters = 1, desc = filename) as t:
- urllib.request.urlretrieve(url, filename = os.path.join(save_dir, filename), reporthook = download_hook(t), data = None)
-
-if __name__ == '__main__':
- # Download ground truth
- if not os.path.exists("CMP_facade_DB_base.zip"):
- download("http://cmp.felk.cvut.cz/~tylecr1/facade/CMP_facade_DB_base.zip", "./")
- # Extract in the correct folder
- with zipfile.ZipFile("CMP_facade_DB_base.zip", 'r') as zip_ref:
- zip_ref.extractall("./facades")
- os.rename("./facades/base", "./facades/train")
-
- # Download ground truth
- if not os.path.exists("CMP_facade_DB_extended.zip"):
- download("http://cmp.felk.cvut.cz/~tylecr1/facade/CMP_facade_DB_extended.zip", "./")
- # Extract in the correct folder
- with zipfile.ZipFile("CMP_facade_DB_extended.zip", 'r') as zip_ref:
- zip_ref.extractall("./facades")
- os.rename("./facades/extended", "./facades/val")
-
-"""Configure the dataloader"""
-
-class ImageDataset(Dataset):
- def __init__(self, root, transforms_=None, mode='train'):
- self.transform = transforms.Compose(transforms_)
-
- self.files_img = sorted(glob.glob(os.path.join(root, mode) + '/*.jpg'))
- if mode == 'val':
- self.files_img.extend(
- sorted(glob.glob(os.path.join(root, 'val') + '/*.jpg')))
-
- self.files_mask = sorted(glob.glob(os.path.join(root, mode) + '/*.png'))
- if mode == 'val':
- self.files_mask.extend(
- sorted(glob.glob(os.path.join(root, 'val') + '/*.png')))
-
- assert len(self.files_img) == len(self.files_mask)
-
- def __getitem__(self, index):
-
- img = Image.open(self.files_img[index % len(self.files_img)])
- mask = Image.open(self.files_mask[index % len(self.files_img)])
- mask = mask.convert('RGB')
-
- img = self.transform(img)
- mask = self.transform(mask)
-
- return img, mask
-
- def __len__(self):
- return len(self.files_img)
-
-# Configure dataloaders
-transforms_ = [transforms.Resize((img_height, img_width), Image.BICUBIC),
- transforms.ToTensor()] # transforms.Normalize((0.5,0.5,0.5), (0.5,0.5,0.5))
-
-dataloader = DataLoader(ImageDataset("facades", transforms_=transforms_),
- batch_size=16, shuffle=True)
-
-val_dataloader = DataLoader(ImageDataset("facades", transforms_=transforms_, mode='val'),
- batch_size=8, shuffle=False)
-
-# Tensor type
-Tensor = torch.cuda.FloatTensor if cuda else torch.FloatTensor
-
-"""Check the loading works and a few helper functions"""
-
-def plot2x2Array(image, mask):
- f, axarr = plt.subplots(1, 2)
- axarr[0].imshow(image)
- axarr[1].imshow(mask)
-
- axarr[0].set_title('Image')
- axarr[1].set_title('Mask')
-
-
-def reverse_transform(image):
- image = image.numpy().transpose((1, 2, 0))
- image = np.clip(image, 0, 1)
- image = (image * 255).astype(np.uint8)
-
- return image
-
-def plot2x3Array(image, mask,predict):
- f, axarr = plt.subplots(1,3,figsize=(15,15))
- axarr[0].imshow(image)
- axarr[1].imshow(mask)
- axarr[2].imshow(predict)
- axarr[0].set_title('input')
- axarr[1].set_title('real')
- axarr[2].set_title('fake')
-
-image, mask = next(iter(dataloader))
-image = reverse_transform(image[0])
-mask = reverse_transform(mask[0])
-plot2x2Array(image, mask)
-
-"""Initialize our GAN"""
-
-# Calculate output of image discriminator (PatchGAN)
-patch = (1, img_height//2**3-2, img_width//2**3-2)
-
-if cuda:
- generator = generator.cuda()
- discriminator = discriminator.cuda()
- criterion_GAN.cuda()
- criterion_pixelwise.cuda()
-
-# Optimizers
-optimizer_G = torch.optim.Adam(generator.parameters(), lr=lr, betas=(b1, b2))
-optimizer_D = torch.optim.Adam(discriminator.parameters(), lr=lr, betas=(b1, b2))
-
-"""Start training"""
-
-def save_model(epoch):
- # save your work
- torch.save({
- 'epoch': epoch,
- 'model_state_dict': generator.state_dict(),
- 'optimizer_state_dict': optimizer_G.state_dict(),
- 'loss': loss_G,
- }, 'generator_'+str(epoch)+'.pth')
- torch.save({
- 'epoch': epoch,
- 'model_state_dict': discriminator.state_dict(),
- 'optimizer_state_dict': optimizer_D.state_dict(),
- 'loss': loss_D,
- }, 'discriminator_'+str(epoch)+'.pth')
-
-def weights_init_normal(m):
- classname = m.__class__.__name__
- if classname.find('Conv') != -1:
- torch.nn.init.normal_(m.weight.data, 0.0, 0.02)
- elif classname.find('BatchNorm2d') != -1:
- torch.nn.init.normal_(m.weight.data, 1.0, 0.02)
- torch.nn.init.constant_(m.bias.data, 0.0)
-
-"""<font color='red'>Complete the loss function </font> in the following training code and train your network:"""
-
-# ----------
-# Training
-# ----------
-
-losses = []
-num_epochs = 200
-
-# Initialize weights
-generator.apply(weights_init_normal)
-discriminator.apply(weights_init_normal)
-epoch_D = 0
-epoch_G = 0
-
-# train the network
-discriminator.train()
-generator.train()
-print_every = 400
-
-for epoch in range(epoch_G, num_epochs):
- for i, batch in enumerate(dataloader):
-
- # Model inputs
- real_A = Variable(batch[0].type(Tensor))
- real_B = Variable(batch[1].type(Tensor))
-
- # Adversarial ground truths
- valid = Variable(Tensor(np.ones((real_B.size(0), *patch))), requires_grad=False)
- fake = Variable(Tensor(np.zeros((real_B.size(0), *patch))), requires_grad=False)
-
- # ------------------
- # Train Generators
- # ------------------
-
- optimizer_G.zero_grad()
-
- # GAN loss
- # TO DO: Put here your GAN loss
-
- # Pixel-wise loss
- # TO DO: Put here your pixel loss
-
- # Total loss
- # TO DO: Put here your total loss
-
- loss_G.backward()
-
- optimizer_G.step()
-
- # ---------------------
- # Train Discriminator
- # ---------------------
-
- optimizer_D.zero_grad()
-
- # Real loss
- pred_real = discriminator(real_A, real_B)
- loss_real = criterion_GAN(pred_real, valid)
-
- # Fake loss
- pred_fake = discriminator(fake_A.detach(), real_B)
- loss_fake = criterion_GAN(pred_fake, fake)
-
- # Total loss
- loss_D = 0.5 * (loss_real + loss_fake)
-
- loss_D.backward()
- optimizer_D.step()
-
- # Print some loss stats
- if i % print_every == 0:
- # print discriminator and generator loss
- print('Epoch [{:5d}/{:5d}] | d_loss: {:6.4f} | g_loss: {:6.4f}'.format(
- epoch+1, num_epochs, loss_D.item(), loss_G.item()))
- ## AFTER EACH EPOCH##
- # append discriminator loss and generator loss
- losses.append((loss_D.item(), loss_G.item()))
- if epoch % 100 == 0:
- print('Saving model...')
- save_model(epoch)
-
-"""Observation of the loss along the training"""
-
-fig, ax = plt.subplots()
-losses = np.array(losses)
-plt.plot(losses.T[0], label='Discriminator')
-plt.plot(losses.T[1], label='Generator')
-plt.title("Training Losses")
-plt.legend()
-
-"""If the training takes too much time, you can use a pretrained model in the meantime, to evaluate its performance.
-
-It is available at : https://partage.liris.cnrs.fr/index.php/s/xwEFmxn9ANeq4zY
-
-### Evaluate your cGAN
-"""
-
-def load_model(epoch=200):
- if 'generator_'+str(epoch)+'.pth' in os.listdir() and 'discriminator_'+str(epoch)+'.pth' in os.listdir():
- if cuda:
- checkpoint_generator = torch.load('generator_'+str(epoch)+'.pth')
- else:
- checkpoint_generator = torch.load('generator_'+str(epoch)+'.pth', map_location='cpu')
- generator.load_state_dict(checkpoint_generator['model_state_dict'])
- optimizer_G.load_state_dict(checkpoint_generator['optimizer_state_dict'])
- epoch_G = checkpoint_generator['epoch']
- loss_G = checkpoint_generator['loss']
-
- if cuda:
- checkpoint_discriminator = torch.load('discriminator_'+str(epoch)+'.pth')
- else:
- checkpoint_discriminator = torch.load('discriminator_'+str(epoch)+'.pth', map_location='cpu')
- discriminator.load_state_dict(checkpoint_discriminator['model_state_dict'])
- optimizer_D.load_state_dict(checkpoint_discriminator['optimizer_state_dict'])
- epoch_D = checkpoint_discriminator['epoch']
- loss_D = checkpoint_discriminator['loss']
- else:
- print('There isn\' a training available with this number of epochs')
-
-load_model(epoch=200)
-
-# switching mode
-generator.eval()
-
-# show a sample evaluation image on the training base
-image, mask = next(iter(dataloader))
-output = generator(mask.type(Tensor))
-output = output.view(16, 3, 256, 256)
-output = output.cpu().detach()
-for i in range(8):
- image_plot = reverse_transform(image[i])
- output_plot = reverse_transform(output[i])
- mask_plot = reverse_transform(mask[i])
- plot2x3Array(mask_plot,image_plot,output_plot)
-
-# show a sample evaluation image on the validation dataset
-image, mask = next(iter(val_dataloader))
-output = generator(mask.type(Tensor))
-output = output.view(8, 3, 256, 256)
-output = output.cpu().detach()
-for i in range(8):
- image_plot = reverse_transform(image[i])
- output_plot = reverse_transform(output[i])
- mask_plot = reverse_transform(mask[i])
- plot2x3Array(mask_plot,image_plot,output_plot)
-
-"""<font color='red'>**Question 4**</font>
-Compare results for 100 and 200 epochs
-"""
-
-# TO DO : Your code here to load and evaluate with a few samples
-# a model after 100 epochs
-
-# And finally :
-if cuda:
- torch.cuda.empty_cache()
-
-"""# How to submit your Work ?
-Your work should be uploaded within 3 weeks into the Moodle section "Devoir 2 - GAN et Conditional GAN". It can be either a notebook containing your code and a description of your work, experiments and results or a ".zip" file containing your report in a "pdf" format describing your work, experiments and results as well as your code (".py" Python files).
-"""
\ No newline at end of file
diff --git a/README.md b/README.md
index ecc207d01f5c34173ac1b563aa8ee1e65441ae45..ace09311e397fa83cd8fbe84f6c2e0c14804b87e 100644
--- a/README.md
+++ b/README.md
@@ -1,11 +1,9 @@
-# TD 2 : GAN & cGAN
+# TD 2 : GAN & Diffusion
MSO 3.4 Apprentissage Automatique
---
-We recommand to use the notebook (.ipynb) but the Python script (.py) is also provided if more convenient for you.
-
# How to submit your Work ?
-This work must be done individually. The expected output is a private repository named gan-cgan on https://gitlab.ec-lyon.fr. It must contain your notebook (or python files) and a README.md file that explains briefly the successive steps of the project. Don't forget to add your teacher as developer member of the project. The last commit is due before 11:59 pm on Wednesday, April 9th, 2025. Subsequent commits will not be considered.
+This work must be done individually. The expected output is a private repository named gan-diffusion on https://gitlab.ec-lyon.fr. It must contain your notebook (or python files) and a README.md file that explains briefly the successive steps of the project. Don't forget to add your teacher as developer member of the project. The last commit is due before 11:59 pm on Wednesday, April 9th, 2025. Subsequent commits will not be considered.