出处：Paddle文档平台 - 图像增广1. 数据增广在图像分类任务中，图像数据的增广是一种常用的正则化方法，主要用于增加训练数据集，让数据集尽可能的多样化，使得训练的模型具有更强的泛化能力，常用于数据量不足或者模型参数较多的场景。除了 ImageNet 分类任务标准数据增广方法外，还有8种数据增广方式非常常用，这里对其进行简单的介绍和对比，这8种数据增广方式在ImageNet上的精度指标如 图1 所示。图1 8种数据增广方法2. 常用数据增广方法注：如果没有特殊说明，所有示例均为 ImageNet 分类，并且假设最终输入网络的数据维度为：[batch-size, 3, 224, 224]在ImageNet 分类任务中，训练阶段的标准数据增广方法为以下几步：[1] - 图像解码：简写为 ImageDecode[2] - 随机裁剪到长宽均为 224 的图像：简写为 RandCrop[3] - 水平方向随机翻转：简写为 RandFlip[4] - 图像数据的归一化：简写为 Normalize[5] - 图像数据的重排，[224, 224, 3] 变为 [3, 224, 224]：简写为 Transpose[6] - 多幅图像数据组成 batch 数据，如 batch-size 个 [3, 224, 224] 的图像数据拼组成 [batch-size, 3, 224, 224]：简写为 Batch相比于上述标准的图像增广方法，研究者也提出了很多改进的图像增广策略，这些策略均是在标准增广方法的不同阶段插入一定的操作，基于这些策略操作所处的不同阶段，我们将其分为了三类：[1] - 对 RandCrop (上述的阶段2)后的 224 的图像进行一些变换: AutoAugment，RandAugment[2] - 对Transpose (上述的阶段5)后的 224 的图像进行一些裁剪: CutOut，RandErasing，HideAndSeek，GridMask[3] - 对 Batch(上述的阶段6) 后的数据进行混合: Mixup，Cutmix增广后的可视化效果如 图2 所示。图2 数据增广后可视化下文将介绍这些策略的原理与使用方法，其中，每种数据增广策略的参考论文与参考开源代码均在下面的介绍中列出。以 图3 为测试图像，第三节将基于测试图像进行变换，并将变换后的效果进行可视化。由于RandCrop是随机裁剪，变换前后的图像内容可能会有一定的差别，无法直观地对比变换前后的图像。因此，本节将 RandCrop 替换为 Resize。图3 测试图像3. 图像变换类图像变换类指的是对 RandCrop 后的224 的图像进行一些变换，主要包括：AutoAugment[1]RandAugment[2]3.1. AutoAugment论文：https://arxiv.org/abs/1805.09501v1Github：https://github.com/DeepVoltaire/AutoAugment不同于常规的人工设计图像增广方式，AutoAugment 是在一系列图像增广子策略的搜索空间中通过搜索算法找到的适合特定数据集的图像增广方案。针对 ImageNet 数据集，最终搜索出来的数据增广方案包含 25 个子策略组合，每个子策略中都包含两种变换，针对每幅图像都随机的挑选一个子策略组合，然后以一定的概率来决定是否执行子策略中的每种变换。结果如 图4 所示。图4 AutoAugment后图像可视化3.2. RandAugment论文：https://arxiv.org/pdf/1909.13719.pdfGithub：https://github.com/heartInsert/randaugmentAutoAugment 的搜索方法比较暴力，直接在数据集上搜索针对该数据集的最优策略，其计算量很大。在 RandAugment 文章中作者发现，一方面，针对越大的模型，越大的数据集，使用 AutoAugment 方式搜索到的增广方式产生的收益也就越小；另一方面，这种搜索出的最优策略是针对该数据集的，其迁移能力较差，并不太适合迁移到其他数据集上。在 RandAugment 中，作者提出了一种随机增广的方式，不再像 AutoAugment 中那样使用特定的概率确定是否使用某种子策略，而是所有的子策略都会以同样的概率被选择到，论文中的实验也表明这种数据增广方式即使在大模型的训练中也具有很好的效果。结果如 图5 所示。图5 RandAugment后图像可视化4. 图像裁剪类图像裁剪类主要是对Transpose 后的 224 的图像进行一些裁剪，并将裁剪区域的像素值置为特定的常数（默认为0），主要包括：CutOut[3]RandErasing[4]HideAndSeek[5]GridMask[6]图像裁剪的这些增广并非一定要放在归一化之后，也有不少实现是放在归一化之前的，也就是直接对 uint8 的图像进行操作，两种方式的差别是：如果直接对 uint8 的图像进行操作，那么再经过归一化之后被裁剪的区域将不再是纯黑或纯白（减均值除方差之后像素值不为0）。而对归一后之后的数据进行操作，裁剪的区域会是纯黑或纯白。上述的裁剪变换思路是相同的，都是为了解决训练出的模型在有遮挡数据上泛化能力较差的问题，不同的是他们的裁剪方式、区域不太一样。4.1. Cutout论文：https://arxiv.org/abs/1708.04552Github：https://github.com/uoguelph-mlrg/CutoutCutout 可以理解为 Dropout 的一种扩展操作，不同的是 Dropout 是对图像经过网络后生成的特征进行遮挡，而 Cutout 是直接对输入的图像进行遮挡，相对于Dropout，Cutout 对噪声的鲁棒性更好。作者在论文中也进行了说明，这样做法有以下两点优势：(1) 通过 Cutout 可以模拟真实场景中主体被部分遮挡时的分类场景；(2) 可以促进模型充分利用图像中更多的内容来进行分类，防止网络只关注显著性的图像区域，从而发生过拟合。结果如 图6 所示。图6 Cutout后图像可视化4.2. RandomErasing论文：https://arxiv.org/pdf/1708.04896.pdfGithub：https://github.com/zhunzhong07/Random-ErasingRandomErasing 与 Cutout 方法类似，同样是为了解决训练出的模型在有遮挡数据上泛化能力较差的问题，作者在论文中也指出，随机裁剪的方式与随机水平翻转具有一定的互补性。作者也在行人再识别（REID）上验证了该方法的有效性。与Cutout不同的是，在RandomErasing中，图片以一定的概率接受该种预处理方法，生成掩码的尺寸大小与长宽比也是根据预设的超参数随机生成。结果如 图7 所示。图7 RandomErasing后图像可视化4.3. HideAndSeek论文：https://arxiv.org/pdf/1811.02545.pdfGithub：https://github.com/kkanshul/Hide-and-SeekHideAndSeek论文将图像分为若干块区域(patch)，对于每块区域，都以一定的概率生成掩码，不同区域的掩码含义如 图8 所示。图8 HideAndSeek分块掩码图结果如 图9 所示。图9 HideAndSeek后图像可视化4.4. GridMask论文：https://arxiv.org/abs/2001.04086Github：https://github.com/akuxcw/GridMask作者在论文中指出，此前存在的基于对图像 crop 的方法存在两个问题，如 图10 所示：[1] - 过度删除区域可能造成目标主体大部分甚至全部被删除，或者导致上下文信息的丢失，导致增广后的数据成为噪声数据；[2] - 保留过多的区域，对目标主体及上下文基本产生不了什么影响，失去增广的意义。图10 增广后的噪声数据因此如果避免过度删除或过度保留成为需要解决的核心问题。GridMask是通过生成一个与原图分辨率相同的掩码，并将掩码进行随机翻转，与原图相乘，从而得到增广后的图像，通过超参数控制生成的掩码网格的大小。在训练过程中，有两种以下使用方法：[1] - 设置一个概率p，从训练开始就对图片以概率p使用GridMask进行增广。[2] - 一开始设置增广概率为0，随着迭代轮数增加，对训练图片进行GridMask增广的概率逐渐增大，最后变为p。论文中验证上述第二种方法的训练效果更好一些。结果如 图11 所示。图11 GridMask后图像可视化5. 图像混叠图像混叠主要对 Batch 后的数据进行混合，包括：Mixup[7]Cutmix[8]前文所述的图像变换与图像裁剪都是针对单幅图像进行的操作，而图像混叠是对两幅图像进行融合，生成一幅图像，两种方法的主要区别为混叠的方式不太一样。5.1. Mixup论文：https://arxiv.org/pdf/1710.09412.pdfGithub：https://github.com/facebookresearch/mixup-cifar10Mixup 是最先提出的图像混叠增广方案，其原理简单、方便实现，不仅在图像分类上，在目标检测上也取得了不错的效果。为了便于实现，通常只对一个 batch 内的数据进行混叠，在 Cutmix 中也是如此。如下是 imaug 中的实现，需要指出的是，下述实现会出现对同一幅进行相加的情况，也就是最终得到的图和原图一样，随着 batch-size 的增加这种情况出现的概率也会逐渐减小。结果如 图12 所示。图12 Mixup后图像可视化5.2. Cutmix论文：https://arxiv.org/pdf/1905.04899v2.pdfGithub：https://github.com/clovaai/CutMix-PyTorch与 Mixup 直接对两幅图进行相加不一样，Cutmix 是从一幅图中随机裁剪出一个 ROI，然后覆盖当前图像中对应的区域。结果如 图13 所示。图13 Cutmix后图像可视化6. 实验基于PaddleClas套件，使用上述几种数据增广方法在ImageNet1k数据集上进行了实验测试，每个方法的分类精度如下。模型初始学习率策略l2 decaybatch sizeepoch数据变化策略Top1 Acc论文中结论ResNet500.1/cosine_decay0.0001256300标准变换0.7731-ResNet500.1/cosine_decay0.0001256300AutoAugment0.77950.7763ResNet500.1/cosine_decay0.0001256300mixup0.78280.7790ResNet500.1/cosine_decay0.0001256300cutmix0.78390.7860ResNet500.1/cosine_decay0.0001256300cutout0.7801-ResNet500.1/cosine_decay0.0001256300gridmask0.77850.7790ResNet500.1/cosine_decay0.0001256300random-augment0.77700.7760ResNet500.1/cosine_decay0.0001256300random erasing0.7791-ResNet500.1/cosine_decay0.0001256300hide and seek0.77430.7720注意：在实验中，为了便于对比，将l2 decay固定设置为1e-4，在实际使用中，推荐尝试使用更小的l2 decay。结合数据增广，发现将l2 decay由1e-4减小为7e-5均能带来至少0.3~0.5%的精度提升。在使用数据增广后，由于训练数据更难，所以训练损失函数可能较大，训练集的准确率相对较低，但其拥有更好的泛化能力，所以验证集的准确率相对较高。在使用数据增广后，模型可能会趋于欠拟合状态，建议可以适当的调小l2_decay的值来获得更高的验证集准确率。参考文献[1] - Autoaugment: Learning augmentation strategies from data[2] - Randaugment: Practical automated data augmentation with a reduced search space[3] - Improved regularization of convolutional neural networks with cutout[4] - Random erasing data augmentation[5] - Hide-and-seek: Forcing a network to be meticulous for weakly-supervised object and action localization[6] - GridMask Data Augmentation[7] - mixup: Beyond empirical risk minimization[8] - Cutmix: Regularization strategy to train strong classifiers with localizable features)
论文：MoCov1 - Momentum Contrast for Unsupervised Visual Representation Learning - 2019论文：MoCov2 - Improved Baselines with Momentum Contrastive Learning - 2020Github - facebookresearch/mocoMoCo，Momentum Contrast，是一种无监督视觉表示学习算法. MoCo 基于对比学习(contrastive learning) 的角度，将问题转换为字典查询(dictionary look-up)，构建了基于队列的动态字典更新(dynamic dictionary with a queue)和滑动平均编码器(moving-averaged encoder)的算法.1. MoCoMoCo 基于的假设：可以通过覆盖大量负样本的大型字典来学习好的特征，且字典 keys 的编码器尽管在不断的更新，但仍应尽可能保持连续性.对于给定的一个样本x， 选择一个(或一批)正样本y(这里正样本对对于图像上的理解就是x的不同的data augmentation版本). 然后选择一批负样本(对于图像来说，就是除了x之外的图像), 然后设计loss function来将x与正样本之间的距离拉近，负样本之间的距离推开.MoCo 主要有如下特点：1.1. InfoNCE 损失函数[1] - 将对比学习问题转化为字典查询问题损失函数(InfoNCE)形式如：$$ L_q = -log \frac{exp(q \cdot k_+/ \tau)}{\sum_{i=0}^K exp(q \cdot k_i / \tau)} $$另一种形式如，$$ L_{q, k_+, \lbrace k_- \rbrace} = -log \frac{exp(q \cdot k_+/ \tau)}{exp(q \cdot k_+/ \tau) + \sum_{k_-} exp(q \cdot k_- / \tau)} $$其中，$q$ 是查询的特征表示；$k_+$ 是正样本(相似)的特征表示，$\lbrace k_- \rbrace$ 是负样本(不相似)的特征表示；$\tau$ 是超参数.损失函数的求和，是对一个正样本和K个负样本进行的，其类似于 (K+1) Softmax 分类的 log 损失函数，其目标是将$q$ 分类为 $k_+$. 参考 SoftmaxEntropyCross 损失：$$ L_{ce} = -log \frac{exp(x_i)}{\sum_j exp(x_j)} $$或，$$ L(x, class) = -log (\frac{exp(x[class])}{\sum_j exp(x[j])}) $$1.2. 动量更新记 $f_k$ 为 key 编码器；$f_q$ 为 q 查询编码器，$\theta_k$ 和 $\theta_q$ 分别为其参数，MoCo 提出的 $\theta_k$ 动量更新方式如：$$ \theta _k \leftarrow m \theta_k + (1 - m) \theta_q $$其中，只有 $\theta_q$ 是需要 BP 梯度计算的. $m$ 为动量系数，实验证明，相对大的动量(如，m=0.999) 比相对小的动量(如，m=0.9) 的效果好很多.1.3. 将字典作为队列MoCo 的核心是，将字典维护为数据样本的队列. 有利于重用训练过程中的历史 mini-batches 的编码 keys. 基于队列的实现，可以将字典大小和 mini-batch size 解耦，且更大. 队列中的样本是渐进式被替代的. 当前 mini-batch 入队列；最老的 mini-batch 出队列. 字典 keys 一般都所有数据的部分采样子集.1.4. Shuffling BNMoCo 论文所述，在标准 ResNet 的 BN 层，会影响模型的训练. 因为模型可能在试图欺骗任务，并容易地找到loss值比较低的方案.而，MoCo 采用的方式是 shuffling BN. 采用多个 GPU 进行训练，分别对每个 GPU 上的样本进行独立的进行BN. 即，对于 key 编码器 $f_k$，在将其分发到各 GPUs 上时，打乱当前 mini-batch 的采样次序；而查询编码器 $f_q$ 的mini-batch 的采样次序保持不变.1.5. MoCo 实现伪代码2. MoCoV2MoCoV2 的提出源于 SimCLR - A Simple Framework for Contrastive Learning of Visual Representations,SimCLR 是主要是有三方面的提升：[1] - 较大的batchsize(如4k或8k)，能够提供更多的负样本；[2] - 将输出 fc 层 head 网络替换为 MLP head；[3] - 更多的数据增强.而 MoCoV2 发现，MoCoV1 + MLP head + more data aug 也能取得更好的结果，而且还不需要大的训练 batchsize. 如图，从 batch 角度的理解, 基于 menmory queue 的机制，不需要很大的 batchsize，MoCo 也能提供更多的负样本进行对比学习.3. MoCo Pytorch 实现3.1. MoCo 构建moco/builder.pyimport torch import torch.nn as nn class MoCo(nn.Module): """ Build a MoCo model with: a query encoder, a key encoder, and a queue https://arxiv.org/abs/1911.05722 """ def __init__(self, base_encoder, dim=128, K=65536, m=0.999, T=0.07, mlp=False): """ dim: 特征维度feature dimension (default: 128) K: 队列大小queue size; 负样本keys数number of negative keys (default: 65536) m: moco momentum of updating key encoder (default: 0.999) T: softmax temperature (default: 0.07) """ super(MoCo, self).__init__() self.K = K self.m = m self.T = T # 创建编码器，这里是基于 resnet50 # num_classes：fc层输出的维度 self.encoder_q = base_encoder(num_classes=dim) self.encoder_k = base_encoder(num_classes=dim) if mlp: # hack: brute-force replacement; MoCov2 dim_mlp = self.encoder_q.fc.weight.shape[1] self.encoder_q.fc = nn.Sequential(nn.Linear(dim_mlp, dim_mlp), nn.ReLU(), self.encoder_q.fc) self.encoder_k.fc = nn.Sequential(nn.Linear(dim_mlp, dim_mlp), nn.ReLU(), self.encoder_k.fc) for param_q, param_k in zip(self.encoder_q.parameters(), self.encoder_k.parameters()): param_k.data.copy_(param_q.data) #初始化编码器k的参数 param_k.requires_grad = False #编码器k不需要梯度计算 #创建队列 self.register_buffer("queue", torch.randn(dim, K)) self.queue = nn.functional.normalize(self.queue, dim=0) self.register_buffer("queue_ptr", torch.zeros(1, dtype=torch.long)) @torch.no_grad() def _momentum_update_key_encoder(self): """ 动量更新编码器 key 的参数 """ for param_q, param_k in zip(self.encoder_q.parameters(), self.encoder_k.parameters()): param_k.data = param_k.data * self.m + param_q.data * (1. - self.m) @torch.no_grad() def _dequeue_and_enqueue(self, keys): #更新队列前，收集keys keys = concat_all_gather(keys) batch_size = keys.shape[0] ptr = int(self.queue_ptr) assert self.K % batch_size == 0 # for simplicity #在 ptr 处替换 keys，出队列和入队列 self.queue[:, ptr:ptr + batch_size] = keys.T ptr = (ptr + batch_size) % self.K # move pointer self.queue_ptr[0] = ptr @torch.no_grad() def _batch_shuffle_ddp(self, x): """ Batch shuffle, for making use of BatchNorm. *** Only support DistributedDataParallel (DDP) model. *** """ # gather from all gpus batch_size_this = x.shape[0] x_gather = concat_all_gather(x) batch_size_all = x_gather.shape[0] num_gpus = batch_size_all // batch_size_this # random shuffle index idx_shuffle = torch.randperm(batch_size_all).cuda() # broadcast to all gpus torch.distributed.broadcast(idx_shuffle, src=0) # index for restoring idx_unshuffle = torch.argsort(idx_shuffle) # shuffled index for this gpu gpu_idx = torch.distributed.get_rank() idx_this = idx_shuffle.view(num_gpus, -1)[gpu_idx] return x_gather[idx_this], idx_unshuffle @torch.no_grad() def _batch_unshuffle_ddp(self, x, idx_unshuffle): """ Undo batch shuffle. *** Only support DistributedDataParallel (DDP) model. *** """ # gather from all gpus batch_size_this = x.shape[0] x_gather = concat_all_gather(x) batch_size_all = x_gather.shape[0] num_gpus = batch_size_all // batch_size_this # restored index for this gpu gpu_idx = torch.distributed.get_rank() idx_this = idx_unshuffle.view(num_gpus, -1)[gpu_idx] return x_gather[idx_this] def forward(self, im_q, im_k): """ Input: im_q: a batch of query images im_k: a batch of key images Output: logits, targets """ # compute query features，计算查询特征 q = self.encoder_q(im_q) # queries: NxC q = nn.functional.normalize(q, dim=1) # compute key features with torch.no_grad(): # no gradient to keys self._momentum_update_key_encoder() # update the key encoder # shuffle for making use of BN im_k, idx_unshuffle = self._batch_shuffle_ddp(im_k) k = self.encoder_k(im_k) # keys: NxC k = nn.functional.normalize(k, dim=1) # undo shuffle k = self._batch_unshuffle_ddp(k, idx_unshuffle) # compute logits # Einstein sum is more intuitive # positive logits: Nx1，正样本 l_pos = torch.einsum('nc,nc->n', [q, k]).unsqueeze(-1) # negative logits: NxK，队列里都是负样本 l_neg = torch.einsum('nc,ck->nk', [q, self.queue.clone().detach()]) # logits: Nx(1+K) logits = torch.cat([l_pos, l_neg], dim=1) #第0列是正样本 # apply temperature logits /= self.T # labels: positive key indicators labels = torch.zeros(logits.shape[0], dtype=torch.long).cuda() # https://github.com/facebookresearch/moco/issues/24 # dequeue and enqueue self._dequeue_and_enqueue(k) return logits, labels # utils @torch.no_grad() def concat_all_gather(tensor): """ Performs all_gather operation on the provided tensors. *** Warning ***: torch.distributed.all_gather has no gradient. """ tensors_gather = [torch.ones_like(tensor) for _ in range(torch.distributed.get_world_size())] torch.distributed.all_gather(tensors_gather, tensor, async_op=False) output = torch.cat(tensors_gather, dim=0) return output3.2. MoCo 训练coco/loader.pyfrom PIL import ImageFilter import random class TwoCropsTransform: """ Take two random crops of one image as the query and key. """ def __init__(self, base_transform): self.base_transform = base_transform def __call__(self, x): q = self.base_transform(x) k = self.base_transform(x) return [q, k] class GaussianBlur(object): """ Gaussian blur augmentation in SimCLR https://arxiv.org/abs/2002.05709 """ def __init__(self, sigma=[.1, 2.]): self.sigma = sigma def __call__(self, x): sigma = random.uniform(self.sigma[0], self.sigma[1]) x = x.filter(ImageFilter.GaussianBlur(radius=sigma)) return x训练主函数:main_moco.py#!/usr/bin/python3 #!--*-- coding: utf-8 --*-- import argparse import builtins import math import os import random import shutil import time import warnings import torch import torch.nn as nn import torch.nn.parallel import torch.backends.cudnn as cudnn import torch.distributed as dist import torch.optim import torch.multiprocessing as mp import torch.utils.data import torch.utils.data.distributed import torchvision.transforms as transforms import torchvision.datasets as datasets import torchvision.models as models import moco.loader #数据加载与增强辅助函数 import moco.builder #MoCo模型函数 model_names = sorted(name for name in models.__dict__ if name.islower() and not name.startswith("__") and callable(models.__dict__[name])) parser = argparse.ArgumentParser(description='PyTorch ImageNet Training') parser.add_argument('data', metavar='DIR', help='path to dataset') parser.add_argument('-a', '--arch', metavar='ARCH', default='resnet50', choices=model_names, help='model architecture: ' + ' | '.join(model_names) + ' (default: resnet50)') parser.add_argument('-j', '--workers', default=32, type=int, metavar='N', help='number of data loading workers (default: 32)') parser.add_argument('--epochs', default=200, type=int, metavar='N', help='number of total epochs to run') parser.add_argument('--start-epoch', default=0, type=int, metavar='N', help='manual epoch number (useful on restarts)') parser.add_argument('-b', '--batch-size', default=256, type=int, metavar='N', help='mini-batch size (default: 256), this is the total ' 'batch size of all GPUs on the current node when ' 'using Data Parallel or Distributed Data Parallel') parser.add_argument('--lr', '--learning-rate', default=0.03, type=float, metavar='LR', help='initial learning rate', dest='lr') parser.add_argument('--schedule', default=[120, 160], nargs='*', type=int, help='learning rate schedule (when to drop lr by 10x)') parser.add_argument('--momentum', default=0.9, type=float, metavar='M', help='momentum of SGD solver') parser.add_argument('--wd', '--weight-decay', default=1e-4, type=float, metavar='W', help='weight decay (default: 1e-4)', dest='weight_decay') parser.add_argument('-p', '--print-freq', default=10, type=int, metavar='N', help='print frequency (default: 10)') parser.add_argument('--resume', default='', type=str, metavar='PATH', help='path to latest checkpoint (default: none)') parser.add_argument('--world-size', default=-1, type=int, help='number of nodes for distributed training') parser.add_argument('--rank', default=-1, type=int, help='node rank for distributed training') parser.add_argument('--dist-url', default='tcp://224.66.41.62:23456', type=str, help='url used to set up distributed training') parser.add_argument('--dist-backend', default='nccl', type=str, help='distributed backend') parser.add_argument('--seed', default=None, type=int, help='seed for initializing training. ') parser.add_argument('--gpu', default=None, type=int, help='GPU id to use.') parser.add_argument('--multiprocessing-distributed', action='store_true', help='Use multi-processing distributed training to launch ' 'N processes per node, which has N GPUs. This is the ' 'fastest way to use PyTorch for either single node or ' 'multi node data parallel training') # moco specific configs: parser.add_argument('--moco-dim', default=128, type=int, help='feature dimension (default: 128)') parser.add_argument('--moco-k', default=65536, type=int, help='queue size; number of negative keys (default: 65536)') parser.add_argument('--moco-m', default=0.999, type=float, help='moco momentum of updating key encoder (default: 0.999)') parser.add_argument('--moco-t', default=0.07, type=float, help='softmax temperature (default: 0.07)') # options for moco v2 parser.add_argument('--mlp', action='store_true', help='use mlp head') parser.add_argument('--aug-plus', action='store_true', help='use moco v2 data augmentation') parser.add_argument('--cos', action='store_true', help='use cosine lr schedule') def main(): args = parser.parse_args() if args.seed is not None: random.seed(args.seed) torch.manual_seed(args.seed) cudnn.deterministic = True warnings.warn('You have chosen to seed training. ' 'This will turn on the CUDNN deterministic setting, ' 'which can slow down your training considerably! ' 'You may see unexpected behavior when restarting ' 'from checkpoints.') if args.gpu is not None: warnings.warn('You have chosen a specific GPU. This will completely ' 'disable data parallelism.') if args.dist_url == "env://" and args.world_size == -1: args.world_size = int(os.environ["WORLD_SIZE"]) args.distributed = args.world_size > 1 or args.multiprocessing_distributed ngpus_per_node = torch.cuda.device_count() if args.multiprocessing_distributed: # Since we have ngpus_per_node processes per node, the total world_size # needs to be adjusted accordingly args.world_size = ngpus_per_node * args.world_size # Use torch.multiprocessing.spawn to launch distributed processes: the # main_worker process function mp.spawn(main_worker, nprocs=ngpus_per_node, args=(ngpus_per_node, args)) else: # Simply call main_worker function main_worker(args.gpu, ngpus_per_node, args) def main_worker(gpu, ngpus_per_node, args): args.gpu = gpu # suppress printing if not master if args.multiprocessing_distributed and args.gpu != 0: def print_pass(*args): pass builtins.print = print_pass if args.gpu is not None: print("Use GPU: {} for training".format(args.gpu)) if args.distributed: if args.dist_url == "env://" and args.rank == -1: args.rank = int(os.environ["RANK"]) if args.multiprocessing_distributed: # For multiprocessing distributed training, rank needs to be the # global rank among all the processes args.rank = args.rank * ngpus_per_node + gpu dist.init_process_group(backend=args.dist_backend, init_method=args.dist_url, world_size=args.world_size, rank=args.rank) #创建 MoCo 模型 print("=> creating model '{}'".format(args.arch)) model = moco.builder.MoCo( models.__dict__[args.arch], args.moco_dim, args.moco_k, args.moco_m, args.moco_t, args.mlp) print(model) if args.distributed: # For multiprocessing distributed, DistributedDataParallel constructor # should always set the single device scope, otherwise, # DistributedDataParallel will use all available devices. if args.gpu is not None: torch.cuda.set_device(args.gpu) model.cuda(args.gpu) # When using a single GPU per process and per # DistributedDataParallel, we need to divide the batch size # ourselves based on the total number of GPUs we have args.batch_size = int(args.batch_size / ngpus_per_node) args.workers = int((args.workers + ngpus_per_node - 1) / ngpus_per_node) model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu]) else: model.cuda() # DistributedDataParallel will divide and allocate batch_size to all # available GPUs if device_ids are not set model = torch.nn.parallel.DistributedDataParallel(model) elif args.gpu is not None: torch.cuda.set_device(args.gpu) model = model.cuda(args.gpu) # comment out the following line for debugging raise NotImplementedError("Only DistributedDataParallel is supported.") else: # AllGather implementation (batch shuffle, queue update, etc.) in # this code only supports DistributedDataParallel. raise NotImplementedError("Only DistributedDataParallel is supported.") #定义损失函数(criterion) criterion = nn.CrossEntropyLoss().cuda(args.gpu)#交叉熵损失函数 #定义优化器 optimizer = torch.optim.SGD(model.parameters(), args.lr, momentum=args.momentum, weight_decay=args.weight_decay) # optionally resume from a checkpoint if args.resume: if os.path.isfile(args.resume): print("=> loading checkpoint '{}'".format(args.resume)) if args.gpu is None: checkpoint = torch.load(args.resume) else: # Map model to be loaded to specified single gpu. loc = 'cuda:{}'.format(args.gpu) checkpoint = torch.load(args.resume, map_location=loc) args.start_epoch = checkpoint['epoch'] model.load_state_dict(checkpoint['state_dict']) optimizer.load_state_dict(checkpoint['optimizer']) print("=> loaded checkpoint '{}' (epoch {})" .format(args.resume, checkpoint['epoch'])) else: print("=> no checkpoint found at '{}'".format(args.resume)) cudnn.benchmark = True #数据加载 traindir = os.path.join(args.data, 'train') normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) if args.aug_plus: # MoCo v2's aug: similar to SimCLR https://arxiv.org/abs/2002.05709 augmentation = [ transforms.RandomResizedCrop(224, scale=(0.2, 1.)), transforms.RandomApply([ transforms.ColorJitter(0.4, 0.4, 0.4, 0.1) # not strengthened ], p=0.8), transforms.RandomGrayscale(p=0.2), transforms.RandomApply([moco.loader.GaussianBlur([.1, 2.])], p=0.5), transforms.RandomHorizontalFlip(), transforms.ToTensor(), normalize ] else: # MoCo v1's aug: the same as InstDisc https://arxiv.org/abs/1805.01978 augmentation = [ transforms.RandomResizedCrop(224, scale=(0.2, 1.)), transforms.RandomGrayscale(p=0.2), transforms.ColorJitter(0.4, 0.4, 0.4, 0.4), transforms.RandomHorizontalFlip(), transforms.ToTensor(), normalize ] #数据集加载，用的是 TwoCropsTransform train_dataset = datasets.ImageFolder( traindir, moco.loader.TwoCropsTransform(transforms.Compose(augmentation))) if args.distributed: train_sampler = torch.utils.data.distributed.DistributedSampler(train_dataset) else: train_sampler = None train_loader = torch.utils.data.DataLoader( train_dataset, batch_size=args.batch_size, shuffle=(train_sampler is None), num_workers=args.workers, pin_memory=True, sampler=train_sampler, drop_last=True) for epoch in range(args.start_epoch, args.epochs): if args.distributed: #用于 shuffling. 否则，每次处理都是相同的随机种子 train_sampler.set_epoch(epoch) #学习率调整 adjust_learning_rate(optimizer, epoch, args) # train for one epoch train(train_loader, model, criterion, optimizer, epoch, args) if not args.multiprocessing_distributed or (args.multiprocessing_distributed and args.rank % ngpus_per_node == 0): save_checkpoint({ 'epoch': epoch + 1, 'arch': args.arch, 'state_dict': model.state_dict(), 'optimizer' : optimizer.state_dict(), }, is_best=False, filename='checkpoint_{:04d}.pth.tar'.format(epoch)) def train(train_loader, model, criterion, optimizer, epoch, args): batch_time = AverageMeter('Time', ':6.3f') data_time = AverageMeter('Data', ':6.3f') losses = AverageMeter('Loss', ':.4e') top1 = AverageMeter('Acc@1', ':6.2f') top5 = AverageMeter('Acc@5', ':6.2f') progress = ProgressMeter( len(train_loader), [batch_time, data_time, losses, top1, top5], prefix="Epoch: [{}]".format(epoch)) # switch to train mode model.train() end = time.time() for i, (images, _) in enumerate(train_loader): # measure data loading time data_time.update(time.time() - end) if args.gpu is not None: images[0] = images[0].cuda(args.gpu, non_blocking=True) images[1] = images[1].cuda(args.gpu, non_blocking=True) # compute output,对比学习，输入是两张图片 output, target = model(im_q=images[0], im_k=images[1]) loss = criterion(output, target) # acc1/acc5 are (K+1)-way contrast classifier accuracy # measure accuracy and record loss acc1, acc5 = accuracy(output, target, topk=(1, 5)) losses.update(loss.item(), images[0].size(0)) top1.update(acc1[0], images[0].size(0)) top5.update(acc5[0], images[0].size(0)) # compute gradient and do SGD step optimizer.zero_grad() loss.backward() optimizer.step() # measure elapsed time batch_time.update(time.time() - end) end = time.time() if i % args.print_freq == 0: progress.display(i) def save_checkpoint(state, is_best, filename='checkpoint.pth.tar'): torch.save(state, filename) if is_best: shutil.copyfile(filename, 'model_best.pth.tar') class AverageMeter(object): """Computes and stores the average and current value""" def __init__(self, name, fmt=':f'): self.name = name self.fmt = fmt self.reset() def reset(self): self.val = 0 self.avg = 0 self.sum = 0 self.count = 0 def update(self, val, n=1): self.val = val self.sum += val * n self.count += n self.avg = self.sum / self.count def __str__(self): fmtstr = '{name} {val' + self.fmt + '} ({avg' + self.fmt + '})' return fmtstr.format(**self.__dict__) class ProgressMeter(object): def __init__(self, num_batches, meters, prefix=""): self.batch_fmtstr = self._get_batch_fmtstr(num_batches) self.meters = meters self.prefix = prefix def display(self, batch): entries = [self.prefix + self.batch_fmtstr.format(batch)] entries += [str(meter) for meter in self.meters] print('\t'.join(entries)) def _get_batch_fmtstr(self, num_batches): num_digits = len(str(num_batches // 1)) fmt = '{:' + str(num_digits) + 'd}' return '[' + fmt + '/' + fmt.format(num_batches) + ']' #学习率调整策略 def adjust_learning_rate(optimizer, epoch, args): """ Decay the learning rate based on schedule """ lr = args.lr if args.cos: # cosine lr schedule # cos lr lr *= 0.5 * (1. + math.cos(math.pi * epoch / args.epochs)) else: # stepwise lr schedule for milestone in args.schedule: lr *= 0.1 if epoch >= milestone else 1. for param_group in optimizer.param_groups: param_group['lr'] = lr def accuracy(output, target, topk=(1,)): """ Computes the accuracy over the k top predictions for the specified values of k """ with torch.no_grad(): maxk = max(topk) batch_size = target.size(0) _, pred = output.topk(maxk, 1, True, True) pred = pred.t() correct = pred.eq(target.view(1, -1).expand_as(pred)) res = [] for k in topk: correct_k = correct[:k].view(-1).float().sum(0, keepdim=True) res.append(correct_k.mul_(100.0 / batch_size)) return res if __name__ == '__main__': main()
Github - AutoAugment论文：AutoAugment - Learning Augmentation Policies from Data - 2018Google AI Blogpost - improving-deep-learning-performance图：ImageNet 数据集上成功的策略. 可以看出，其大部分策略只要是颜色变换.1. autoaugment.pyautoaugment.pyfrom PIL import Image, ImageEnhance, ImageOps import numpy as np import random class ImageNetPolicy(object): """ Randomly choose one of the best 24 Sub-policies on ImageNet. Example: >>> policy = ImageNetPolicy() >>> transformed = policy(image) Example as a PyTorch Transform: >>> transform=transforms.Compose([ >>> transforms.Resize(256), >>> ImageNetPolicy(), >>> transforms.ToTensor()]) """ def __init__(self, fillcolor=(128, 128, 128)): #数据增强策略列表 self.policies = [ SubPolicy(0.4, "posterize", 8, 0.6, "rotate", 9, fillcolor), SubPolicy(0.6, "solarize", 5, 0.6, "autocontrast", 5, fillcolor), SubPolicy(0.8, "equalize", 8, 0.6, "equalize", 3, fillcolor), SubPolicy(0.6, "posterize", 7, 0.6, "posterize", 6, fillcolor), SubPolicy(0.4, "equalize", 7, 0.2, "solarize", 4, fillcolor), SubPolicy(0.4, "equalize", 4, 0.8, "rotate", 8, fillcolor), SubPolicy(0.6, "solarize", 3, 0.6, "equalize", 7, fillcolor), SubPolicy(0.8, "posterize", 5, 1.0, "equalize", 2, fillcolor), SubPolicy(0.2, "rotate", 3, 0.6, "solarize", 8, fillcolor), SubPolicy(0.6, "equalize", 8, 0.4, "posterize", 6, fillcolor), SubPolicy(0.8, "rotate", 8, 0.4, "color", 0, fillcolor), SubPolicy(0.4, "rotate", 9, 0.6, "equalize", 2, fillcolor), SubPolicy(0.0, "equalize", 7, 0.8, "equalize", 8, fillcolor), SubPolicy(0.6, "invert", 4, 1.0, "equalize", 8, fillcolor), SubPolicy(0.6, "color", 4, 1.0, "contrast", 8, fillcolor), SubPolicy(0.8, "rotate", 8, 1.0, "color", 2, fillcolor), SubPolicy(0.8, "color", 8, 0.8, "solarize", 7, fillcolor), SubPolicy(0.4, "sharpness", 7, 0.6, "invert", 8, fillcolor), SubPolicy(0.6, "shearX", 5, 1.0, "equalize", 9, fillcolor), SubPolicy(0.4, "color", 0, 0.6, "equalize", 3, fillcolor), SubPolicy(0.4, "equalize", 7, 0.2, "solarize", 4, fillcolor), SubPolicy(0.6, "solarize", 5, 0.6, "autocontrast", 5, fillcolor), SubPolicy(0.6, "invert", 4, 1.0, "equalize", 8, fillcolor), SubPolicy(0.6, "color", 4, 1.0, "contrast", 8, fillcolor), SubPolicy(0.8, "equalize", 8, 0.6, "equalize", 3, fillcolor) ] def __call__(self, img): #随机选择策略 policy_idx = random.randint(0, len(self.policies) - 1) return self.policies[policy_idx](img) def __repr__(self): return "AutoAugment ImageNet Policy" class CIFAR10Policy(object): """ Randomly choose one of the best 25 Sub-policies on CIFAR10. Example: >>> policy = CIFAR10Policy() >>> transformed = policy(image) Example as a PyTorch Transform: >>> transform=transforms.Compose([ >>> transforms.Resize(256), >>> CIFAR10Policy(), >>> transforms.ToTensor()]) """ def __init__(self, fillcolor=(128, 128, 128)): self.policies = [ SubPolicy(0.1, "invert", 7, 0.2, "contrast", 6, fillcolor), SubPolicy(0.7, "rotate", 2, 0.3, "translateX", 9, fillcolor), SubPolicy(0.8, "sharpness", 1, 0.9, "sharpness", 3, fillcolor), SubPolicy(0.5, "shearY", 8, 0.7, "translateY", 9, fillcolor), SubPolicy(0.5, "autocontrast", 8, 0.9, "equalize", 2, fillcolor), SubPolicy(0.2, "shearY", 7, 0.3, "posterize", 7, fillcolor), SubPolicy(0.4, "color", 3, 0.6, "brightness", 7, fillcolor), SubPolicy(0.3, "sharpness", 9, 0.7, "brightness", 9, fillcolor), SubPolicy(0.6, "equalize", 5, 0.5, "equalize", 1, fillcolor), SubPolicy(0.6, "contrast", 7, 0.6, "sharpness", 5, fillcolor), SubPolicy(0.7, "color", 7, 0.5, "translateX", 8, fillcolor), SubPolicy(0.3, "equalize", 7, 0.4, "autocontrast", 8, fillcolor), SubPolicy(0.4, "translateY", 3, 0.2, "sharpness", 6, fillcolor), SubPolicy(0.9, "brightness", 6, 0.2, "color", 8, fillcolor), SubPolicy(0.5, "solarize", 2, 0.0, "invert", 3, fillcolor), SubPolicy(0.2, "equalize", 0, 0.6, "autocontrast", 0, fillcolor), SubPolicy(0.2, "equalize", 8, 0.6, "equalize", 4, fillcolor), SubPolicy(0.9, "color", 9, 0.6, "equalize", 6, fillcolor), SubPolicy(0.8, "autocontrast", 4, 0.2, "solarize", 8, fillcolor), SubPolicy(0.1, "brightness", 3, 0.7, "color", 0, fillcolor), SubPolicy(0.4, "solarize", 5, 0.9, "autocontrast", 3, fillcolor), SubPolicy(0.9, "translateY", 9, 0.7, "translateY", 9, fillcolor), SubPolicy(0.9, "autocontrast", 2, 0.8, "solarize", 3, fillcolor), SubPolicy(0.8, "equalize", 8, 0.1, "invert", 3, fillcolor), SubPolicy(0.7, "translateY", 9, 0.9, "autocontrast", 1, fillcolor) ] def __call__(self, img): policy_idx = random.randint(0, len(self.policies) - 1) return self.policies[policy_idx](img) def __repr__(self): return "AutoAugment CIFAR10 Policy" class SVHNPolicy(object): """ Randomly choose one of the best 25 Sub-policies on SVHN. Example: >>> policy = SVHNPolicy() >>> transformed = policy(image) Example as a PyTorch Transform: >>> transform=transforms.Compose([ >>> transforms.Resize(256), >>> SVHNPolicy(), >>> transforms.ToTensor()]) """ def __init__(self, fillcolor=(128, 128, 128)): self.policies = [ SubPolicy(0.9, "shearX", 4, 0.2, "invert", 3, fillcolor), SubPolicy(0.9, "shearY", 8, 0.7, "invert", 5, fillcolor), SubPolicy(0.6, "equalize", 5, 0.6, "solarize", 6, fillcolor), SubPolicy(0.9, "invert", 3, 0.6, "equalize", 3, fillcolor), SubPolicy(0.6, "equalize", 1, 0.9, "rotate", 3, fillcolor), SubPolicy(0.9, "shearX", 4, 0.8, "autocontrast", 3, fillcolor), SubPolicy(0.9, "shearY", 8, 0.4, "invert", 5, fillcolor), SubPolicy(0.9, "shearY", 5, 0.2, "solarize", 6, fillcolor), SubPolicy(0.9, "invert", 6, 0.8, "autocontrast", 1, fillcolor), SubPolicy(0.6, "equalize", 3, 0.9, "rotate", 3, fillcolor), SubPolicy(0.9, "shearX", 4, 0.3, "solarize", 3, fillcolor), SubPolicy(0.8, "shearY", 8, 0.7, "invert", 4, fillcolor), SubPolicy(0.9, "equalize", 5, 0.6, "translateY", 6, fillcolor), SubPolicy(0.9, "invert", 4, 0.6, "equalize", 7, fillcolor), SubPolicy(0.3, "contrast", 3, 0.8, "rotate", 4, fillcolor), SubPolicy(0.8, "invert", 5, 0.0, "translateY", 2, fillcolor), SubPolicy(0.7, "shearY", 6, 0.4, "solarize", 8, fillcolor), SubPolicy(0.6, "invert", 4, 0.8, "rotate", 4, fillcolor), SubPolicy(0.3, "shearY", 7, 0.9, "translateX", 3, fillcolor), SubPolicy(0.1, "shearX", 6, 0.6, "invert", 5, fillcolor), SubPolicy(0.7, "solarize", 2, 0.6, "translateY", 7, fillcolor), SubPolicy(0.8, "shearY", 4, 0.8, "invert", 8, fillcolor), SubPolicy(0.7, "shearX", 9, 0.8, "translateY", 3, fillcolor), SubPolicy(0.8, "shearY", 5, 0.7, "autocontrast", 3, fillcolor), SubPolicy(0.7, "shearX", 2, 0.1, "invert", 5, fillcolor) ] def __call__(self, img): policy_idx = random.randint(0, len(self.policies) - 1) return self.policies[policy_idx](img) def __repr__(self): return "AutoAugment SVHN Policy" class SubPolicy(object): def __init__(self, p1, operation1, magnitude_idx1, p2, operation2, magnitude_idx2, fillcolor=(128, 128, 128)): ranges = { "shearX": np.linspace(0, 0.3, 10), "shearY": np.linspace(0, 0.3, 10), "translateX": np.linspace(0, 150 / 331, 10), "translateY": np.linspace(0, 150 / 331, 10), "rotate": np.linspace(0, 30, 10), "color": np.linspace(0.0, 0.9, 10), "posterize": np.round(np.linspace(8, 4, 10), 0).astype(np.int), "solarize": np.linspace(256, 0, 10), "contrast": np.linspace(0.0, 0.9, 10), "sharpness": np.linspace(0.0, 0.9, 10), "brightness": np.linspace(0.0, 0.9, 10), "autocontrast": [0] * 10, "equalize": [0] * 10, "invert": [0] * 10 } # from https://stackoverflow.com/questions/5252170/specify-image-filling-color-when-rotating-in-python-with-pil-and-setting-expand def rotate_with_fill(img, magnitude): rot = img.convert("RGBA").rotate(magnitude) return Image.composite(rot, Image.new("RGBA", rot.size, (128,) * 4), rot).convert(img.mode) func = { "shearX": lambda img, magnitude: img.transform( img.size, Image.AFFINE, (1, magnitude * random.choice([-1, 1]), 0, 0, 1, 0), Image.BICUBIC, fillcolor=fillcolor), "shearY": lambda img, magnitude: img.transform( img.size, Image.AFFINE, (1, 0, 0, magnitude * random.choice([-1, 1]), 1, 0), Image.BICUBIC, fillcolor=fillcolor), "translateX": lambda img, magnitude: img.transform( img.size, Image.AFFINE, (1, 0, magnitude * img.size[0] * random.choice([-1, 1]), 0, 1, 0), fillcolor=fillcolor), "translateY": lambda img, magnitude: img.transform( img.size, Image.AFFINE, (1, 0, 0, 0, 1, magnitude * img.size[1] * random.choice([-1, 1])), fillcolor=fillcolor), "rotate": lambda img, magnitude: rotate_with_fill(img, magnitude), "color": lambda img, magnitude: ImageEnhance.Color(img).enhance(1 + magnitude * random.choice([-1, 1])), "posterize": lambda img, magnitude: ImageOps.posterize(img, magnitude), "solarize": lambda img, magnitude: ImageOps.solarize(img, magnitude), "contrast": lambda img, magnitude: ImageEnhance.Contrast(img).enhance( 1 + magnitude * random.choice([-1, 1])), "sharpness": lambda img, magnitude: ImageEnhance.Sharpness(img).enhance( 1 + magnitude * random.choice([-1, 1])), "brightness": lambda img, magnitude: ImageEnhance.Brightness(img).enhance( 1 + magnitude * random.choice([-1, 1])), "autocontrast": lambda img, magnitude: ImageOps.autocontrast(img), "equalize": lambda img, magnitude: ImageOps.equalize(img), "invert": lambda img, magnitude: ImageOps.invert(img) } self.p1 = p1 self.operation1 = func[operation1] self.magnitude1 = ranges[operation1][magnitude_idx1] self.p2 = p2 self.operation2 = func[operation2] self.magnitude2 = ranges[operation2][magnitude_idx2] def __call__(self, img): if random.random() < self.p1: img = self.operation1(img, self.magnitude1) if random.random() < self.p2: img = self.operation2(img, self.magnitude2) return img2. AutoAugment PIL 用例from autoaugment import ImageNetPolicy image = PIL.Image.open(path) policy = ImageNetPolicy() transformed = policy(image)3. PyTorch ImageNet 用例from autoaugment import ImageNetPolicy data = ImageFolder(rootdir, transform=transforms.Compose( [transforms.RandomResizedCrop(224), transforms.RandomHorizontalFlip(), ImageNetPolicy(), transforms.ToTensor(), transforms.Normalize(...)])) loader = DataLoader(data, ...)
主要集中汇总学习下 BackgroundMattingV2 中的一些关于数据集、数据增强等辅助函数.1. ImagesDatasetimport os import glob from torch.utils.data import Dataset from PIL import Image class ImagesDataset(Dataset): def __init__(self, root, mode='RGB', transforms=None): self.transforms = transforms self.mode = mode self.filenames = sorted( [*glob.glob(os.path.join(root, '**', '*.jpg'), recursive=True), *glob.glob(os.path.join(root, '**', '*.png'), recursive=True)]) def __len__(self): return len(self.filenames) def __getitem__(self, idx): with Image.open(self.filenames[idx]) as img: img = img.convert(self.mode) if self.transforms: img = self.transforms(img) return img2. VideoDatasetimport cv2 import numpy as np from torch.utils.data import Dataset from PIL import Image class VideoDataset(Dataset): def __init__(self, path: str, transforms: any = None): self.cap = cv2.VideoCapture(path) self.transforms = transforms self.width = int(self.cap.get(cv2.CAP_PROP_FRAME_WIDTH)) self.height = int(self.cap.get(cv2.CAP_PROP_FRAME_HEIGHT)) self.frame_rate = self.cap.get(cv2.CAP_PROP_FPS) self.frame_count = int(self.cap.get(cv2.CAP_PROP_FRAME_COUNT)) def __len__(self): return self.frame_count def __getitem__(self, idx): if isinstance(idx, slice): return [self[i] for i in range(*idx.indices(len(self)))] if self.cap.get(cv2.CAP_PROP_POS_FRAMES) != idx: self.cap.set(cv2.CAP_PROP_POS_FRAMES, idx) ret, img = self.cap.read() if not ret: raise IndexError(f'Idx: {idx} out of length: {len(self)}') img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) img = Image.fromarray(img) if self.transforms: img = self.transforms(img) return img def __enter__(self): return self def __exit__(self, exc_type, exc_value, exc_traceback): self.cap.release()3. ZipDatasetfrom torch.utils.data import Dataset from typing import List class ZipDataset(Dataset): def __init__(self, datasets: List[Dataset], transforms=None, assert_equal_length=False): self.datasets = datasets self.transforms = transforms if assert_equal_length: for i in range(1, len(datasets)): assert len(datasets[i]) == len(datasets[i - 1]), 'Datasets are not equal in length.' def __len__(self): return max(len(d) for d in self.datasets) def __getitem__(self, idx): x = tuple(d[idx % len(d)] for d in self.datasets) if self.transforms: x = self.transforms(*x) return x4. 数据增强augmentation.py该数据增强函数，主要有助于对成对图像的数据增强变换. 如：img1, img2, ..., imgN = transforms(img1, img2, ..., imgN)完整实现如下：import random import torch import numpy as np import math from torchvision import transforms as T from torchvision.transforms import functional as F from PIL import Image, ImageFilter """ Pair transforms are MODs of regular transforms so that it takes in multiple images and apply exact transforms on all images. This is especially useful when we want the transforms on a pair of images. Example: img1, img2, ..., imgN = transforms(img1, img2, ..., imgN) """ class PairCompose(T.Compose): def __call__(self, *x): for transform in self.transforms: x = transform(*x) return x #成对应用变换 class PairApply: def __init__(self, transforms): self.transforms = transforms def __call__(self, *x): return [self.transforms(xi) for xi in x] #成对仅对特定索引的图像进行变换 class PairApplyOnlyAtIndices: def __init__(self, indices, transforms): self.indices = indices self.transforms = transforms def __call__(self, *x): return [self.transforms(xi) if i in self.indices else xi for i, xi in enumerate(x)] #成对随机仿射变换 class PairRandomAffine(T.RandomAffine): def __init__(self, degrees, translate=None, scale=None, shear=None, resamples=None, fillcolor=0): super().__init__(degrees, translate, scale, shear, Image.NEAREST, fillcolor) self.resamples = resamples def __call__(self, *x): if not len(x): return [] param = self.get_params(self.degrees, self.translate, self.scale, self.shear, x[0].size) resamples = self.resamples or [self.resample] * len(x) return [F.affine(xi, *param, resamples[i], self.fillcolor) for i, xi in enumerate(x)] #成对随机尺寸调整和裁剪 class PairRandomResizedCrop(T.RandomResizedCrop): def __init__(self, size, scale=(0.08, 1.0), ratio=(3. / 4., 4. / 3.), interpolations=None): super().__init__(size, scale, ratio, Image.BILINEAR) self.interpolations = interpolations def __call__(self, *x): if not len(x): return [] i, j, h, w = self.get_params(x[0], self.scale, self.ratio) interpolations = self.interpolations or [self.interpolation] * len(x) return [F.resized_crop(xi, i, j, h, w, self.size, interpolations[i]) for i, xi in enumerate(x)] #成对随机水平翻转 class PairRandomHorizontalFlip(T.RandomHorizontalFlip): def __call__(self, *x): if torch.rand(1) < self.p: x = [F.hflip(xi) for xi in x] return x #随机方框模糊 class RandomBoxBlur: def __init__(self, prob, max_radius): self.prob = prob self.max_radius = max_radius def __call__(self, img): if torch.rand(1) < self.prob: fil = ImageFilter.BoxBlur(random.choice(range(self.max_radius + 1))) img = img.filter(fil) return img #成对随机方框模糊 class PairRandomBoxBlur(RandomBoxBlur): def __call__(self, *x): if torch.rand(1) < self.prob: fil = ImageFilter.BoxBlur(random.choice(range(self.max_radius + 1))) x = [xi.filter(fil) for xi in x] return x #随机锐化 class RandomSharpen: def __init__(self, prob): self.prob = prob self.filter = ImageFilter.SHARPEN def __call__(self, img): if torch.rand(1) < self.prob: img = img.filter(self.filter) return img #成对随机锐化 class PairRandomSharpen(RandomSharpen): def __call__(self, *x): if torch.rand(1) < self.prob: x = [xi.filter(self.filter) for xi in x] return x #成对随机仿射变换和尺寸调整 class PairRandomAffineAndResize: def __init__(self, size, degrees, translate, scale, shear, ratio=(3./4., 4./3.), resample=Image.BILINEAR, fillcolor=0): self.size = size self.degrees = degrees self.translate = translate self.scale = scale self.shear = shear self.ratio = ratio self.resample = resample self.fillcolor = fillcolor def __call__(self, *x): if not len(x): return [] w, h = x[0].size scale_factor = max(self.size[1] / w, self.size[0] / h) w_padded = max(w, self.size[1]) h_padded = max(h, self.size[0]) pad_h = int(math.ceil((h_padded - h) / 2)) pad_w = int(math.ceil((w_padded - w) / 2)) scale = self.scale[0] * scale_factor, self.scale[1] * scale_factor translate = self.translate[0] * scale_factor, self.translate[1] * scale_factor affine_params = T.RandomAffine.get_params(self.degrees, translate, scale, self.shear, (w, h)) def transform(img): if pad_h > 0 or pad_w > 0: img = F.pad(img, (pad_w, pad_h)) img = F.affine(img, *affine_params, self.resample, self.fillcolor) img = F.center_crop(img, self.size) return img return [transform(xi) for xi in x] #随机仿射变换和尺寸调整 class RandomAffineAndResize(PairRandomAffineAndResize): def __call__(self, img): return super().__call__(img)[0]
数据增强(Data Augmentation)，又称为数据增广，数据扩增，它是对训练集进行变换，使训练集更丰富，从而提升模型的泛化能力.torchvision transforms 数据增强主要涉及如下：[1] - 裁剪类transforms.CenterCrop - 中心裁剪transforms.RandomCrop - 随机裁剪transforms.RandomResizedCrop - 随机长宽比裁剪transforms.FiveCrop - 上下左右中心裁剪transforms.TenCrop - 上下左右中心裁剪后旋转[2] - 翻转和旋转transforms.RandomHorizontalFlip - 随机水平翻转transforms.RandomVerticalFlip - 随机垂直翻转transforms.RandomRotation - 随机旋转[3] - 图像变换transforms.Pad - 填充transforms.ColorJitter - 亮度、对比度和饱和度transforms.Grayscale - 转灰度图transforms.RandomGrayscale - 随机转灰度图transforms.RandomAffine - 随机仿射变换transforms.LinearTransformation - 线性变换transforms.RandomErasing - 随机擦除transforms.Lambda - 自定义变换transforms.Resizetransforms.ToTensor - 转为 Tensor，并归一化到[0, 1].transforms.Normalize - 标准化[4] - transforms的操作transforms.RandomChoice(transforms) - 随机选择给定 transforms 中的一种transforms.RandomApply(transforms, p=0.5) - 加上随机概率transforms.RandomOrder - 随机打乱 transforms 操作的顺序示例 - FixRes codeblockFrom FixResimport numpy as np import torch import torchvision.transforms.functional as F from torchvision import transforms class Resize(transforms.Resize): """ Resize with a ``largest=False'' argument allowing to resize to a common largest side without cropping """ def __init__(self, size, largest=False, **kwargs): super().__init__(size, **kwargs) self.largest = largest @staticmethod def target_size(w, h, size, largest=False): if h < w and largest: w, h = size, int(size * h / w) else: w, h = int(size * w / h), size size = (h, w) return size def __call__(self, img): size = self.size w, h = img.size target_size = self.target_size(w, h, size, self.largest) return F.resize(img, target_size, self.interpolation) def __repr__(self): r = super().__repr__() return r[:-1] + ', largest={})'.format(self.largest) # def get_transforms(input_size=224,test_size=224, kind='full', crop=True, need=('train', 'val')): mean, std = [0.485, 0.456, 0.406], [0.229, 0.224, 0.225] transformations = {} if 'train' in need: if kind == 'torch': transformations['train'] = transforms.Compose([ transforms.RandomResizedCrop(input_size), transforms.RandomHorizontalFlip(), transforms.ToTensor(), transforms.Normalize(mean, std), ]) elif kind == 'full': transformations['train'] = transforms.Compose([ transforms.RandomResizedCrop(input_size), transforms.RandomHorizontalFlip(), transforms.ColorJitter(0.3, 0.3, 0.3), transforms.ToTensor(), transforms.Normalize(mean, std), ]) else: raise ValueError('Transforms kind {} unknown'.format(kind)) if 'val' in need: if crop: transformations['val'] = transforms.Compose( [Resize(int((256 / 224) * test_size)), # to maintain same ratio w.r.t. 224 images transforms.CenterCrop(test_size), transforms.ToTensor(), transforms.Normalize(mean, std)]) else: transformations['val'] = transforms.Compose( [Resize(test_size, largest=True), # to maintain same ratio w.r.t. 224 images transforms.ToTensor(), transforms.Normalize(mean, std)]) return transformations transforms_list = ['torch', 'full']示例 - self-supervised learningTypical data augmentation composition (pytorch) for self-supervised learning：from torchvision import transforms ... augmentation = [ transforms.RandomResizedCrop(224, scale=(0.2, 1.)), transforms.RandomApply([ transforms.ColorJitter(0.8, 0.8, 0.8, 0.2)], p=0.8), transforms.RandomGrayscale(p=0.2), transforms.RandomApply([GaussianBlur([0.1, 2.0])], p=0.5), transforms.RandomHorizontalFlip(), transforms.ToTensor(), transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) ]参考[1] - PyTorch 学习笔记（三）：transforms的二十二个方法 - 2018.12.26[2] - pytorch torchvision transform[3] - Docs > torchvision > torchvision.transforms
From: https://github.com/PaddlePaddle/PaddleSeg/blob/release/v0.4.0/pdseg/data_aug.pyfrom __future__ import print_function import cv2 import numpy as np def resize(img, grt=None, mode=ModelPhase.TRAIN): """ 改变图像及标签图像尺寸 AUG.AUG_METHOD为unpadding，所有模式均直接resize到AUG.FIX_RESIZE_SIZE的尺寸 AUG.AUG_METHOD为stepscaling, 按比例resize，训练时比例范围AUG.MIN_SCALE_FACTOR到AUG.MAX_SCALE_FACTOR,间隔为AUG.SCALE_STEP_SIZE，其他模式返回原图 AUG.AUG_METHOD为rangescaling，长边对齐，短边按比例变化，训练时长边对齐范围AUG.MIN_RESIZE_VALUE到AUG.MAX_RESIZE_VALUE，其他模式长边对齐AUG.INF_RESIZE_VALUE Args： img(numpy.ndarray): 输入图像 grt(numpy.ndarray): 标签图像，默认为None mode(string): 模式, 默认训练模式，即ModelPhase.TRAIN Returns： resize后的图像和标签图 """ if cfg.AUG.AUG_METHOD == 'unpadding': target_size = cfg.AUG.FIX_RESIZE_SIZE img = cv2.resize(img, target_size, interpolation=cv2.INTER_LINEAR) if grt is not None: grt = cv2.resize(grt, target_size, interpolation=cv2.INTER_NEAREST) elif cfg.AUG.AUG_METHOD == 'stepscaling': if mode == ModelPhase.TRAIN: min_scale_factor = cfg.AUG.MIN_SCALE_FACTOR max_scale_factor = cfg.AUG.MAX_SCALE_FACTOR step_size = cfg.AUG.SCALE_STEP_SIZE scale_factor = get_random_scale(min_scale_factor, max_scale_factor, step_size) img, grt = randomly_scale_image_and_label( img, grt, scale=scale_factor) elif cfg.AUG.AUG_METHOD == 'rangescaling': min_resize_value = cfg.AUG.MIN_RESIZE_VALUE max_resize_value = cfg.AUG.MAX_RESIZE_VALUE if mode == ModelPhase.TRAIN: if min_resize_value == max_resize_value: random_size = min_resize_value else: random_size = int(np.random.uniform(min_resize_value, max_resize_value) + 0.5) else: random_size = cfg.AUG.INF_RESIZE_VALUE value = max(img.shape[0], img.shape[1]) scale = float(random_size) / float(value) img = cv2.resize(img, (0, 0), fx=scale, fy=scale, interpolation=cv2.INTER_LINEAR) if grt is not None: grt = cv2.resize(grt, (0, 0), fx=scale, fy=scale, interpolation=cv2.INTER_NEAREST) else: raise Exception("Unexpect data augmention method: {}".format( cfg.AUG.AUG_METHOD)) return img, grt def get_random_scale(min_scale_factor, max_scale_factor, step_size): """ 在一定范围内得到随机值，范围为min_scale_factor到max_scale_factor，间隔为step_size Args： min_scale_factor(float): 随机尺度下限，大于0 max_scale_factor(float): 随机尺度上限，不小于下限值 step_size(float): 尺度间隔，非负, 等于为0时直接返回min_scale_factor到max_scale_factor范围内任一值 Returns： 随机尺度值 """ if min_scale_factor < 0 or min_scale_factor > max_scale_factor: raise ValueError('Unexpected value of min_scale_factor.') if min_scale_factor == max_scale_factor: return min_scale_factor if step_size == 0: return np.random.uniform(min_scale_factor, max_scale_factor) num_steps = int((max_scale_factor - min_scale_factor) / step_size + 1) scale_factors = np.linspace(min_scale_factor, max_scale_factor, num_steps).tolist() np.random.shuffle(scale_factors) return scale_factors[0] def randomly_scale_image_and_label(image, label=None, scale=1.0): """ 按比例resize图像和标签图, 如果scale为1，返回原图 Args： image(numpy.ndarray): 输入图像 label(numpy.ndarray): 标签图，默认None sclae(float): 图片resize的比例，非负，默认1.0 Returns： resize后的图像和标签图 """ if scale == 1.0: return image, label height = image.shape[0] width = image.shape[1] new_height = int(height * scale + 0.5) new_width = int(width * scale + 0.5) new_image = cv2.resize(image, (new_width, new_height), interpolation=cv2.INTER_LINEAR) if label is not None: height = label.shape[0] width = label.shape[1] new_height = int(height * scale + 0.5) new_width = int(width * scale + 0.5) new_label = cv2.resize(label, (new_width, new_height), interpolation=cv2.INTER_NEAREST) return new_image, new_label def random_rotation(crop_img, crop_seg, rich_crop_max_rotation, mean_value): """ 随机旋转图像和标签图 Args： crop_img(numpy.ndarray): 输入图像 crop_seg(numpy.ndarray): 标签图 rich_crop_max_rotation(int)：旋转最大角度，0-90 mean_value(list)：均值, 对图片旋转产生的多余区域使用均值填充 Returns： 旋转后的图像和标签图 """ ignore_index = cfg.DATASET.IGNORE_INDEX if rich_crop_max_rotation > 0: (h, w) = crop_img.shape[:2] do_rotation = np.random.uniform(-rich_crop_max_rotation, rich_crop_max_rotation) pc = (w // 2, h // 2) r = cv2.getRotationMatrix2D(pc, do_rotation, 1.0) cos = np.abs(r[0, 0]) sin = np.abs(r[0, 1]) nw = int((h * sin) + (w * cos)) nh = int((h * cos) + (w * sin)) (cx, cy) = pc r[0, 2] += (nw / 2) - cx r[1, 2] += (nh / 2) - cy dsize = (nw, nh) crop_img = cv2.warpAffine( crop_img, r, dsize=dsize, flags=cv2.INTER_LINEAR, borderMode=cv2.BORDER_CONSTANT, borderValue=mean_value) crop_seg = cv2.warpAffine( crop_seg, r, dsize=dsize, flags=cv2.INTER_NEAREST, borderMode=cv2.BORDER_CONSTANT, borderValue=(ignore_index, ignore_index, ignore_index)) return crop_img, crop_seg def rand_scale_aspect(crop_img, crop_seg, rich_crop_min_scale=0, rich_crop_aspect_ratio=0): """ 从输入图像和标签图像中裁取随机宽高比的图像，并reszie回原始尺寸 Args: crop_img(numpy.ndarray): 输入图像 crop_seg(numpy.ndarray): 标签图像 rich_crop_min_scale(float)：裁取图像占原始图像的面积比，0-1，默认0返回原图 rich_crop_aspect_ratio(float): 裁取图像的宽高比范围，非负，默认0返回原图 Returns: 裁剪并resize回原始尺寸的图像和标签图像 """ if rich_crop_min_scale == 0 or rich_crop_aspect_ratio == 0: return crop_img, crop_seg else: img_height = crop_img.shape[0] img_width = crop_img.shape[1] for i in range(0, 10): area = img_height * img_width target_area = area * np.random.uniform(rich_crop_min_scale, 1.0) aspectRatio = np.random.uniform(rich_crop_aspect_ratio, 1.0 / rich_crop_aspect_ratio) dw = int(np.sqrt(target_area * 1.0 * aspectRatio)) dh = int(np.sqrt(target_area * 1.0 / aspectRatio)) if (np.random.randint(10) < 5): tmp = dw dw = dh dh = tmp if (dh < img_height and dw < img_width): h1 = np.random.randint(0, img_height - dh) w1 = np.random.randint(0, img_width - dw) crop_img = crop_img[h1:(h1 + dh), w1:(w1 + dw), :] crop_seg = crop_seg[h1:(h1 + dh), w1:(w1 + dw)] crop_img = cv2.resize( crop_img, (img_width, img_height), interpolation=cv2.INTER_LINEAR) crop_seg = cv2.resize( crop_seg, (img_width, img_height), interpolation=cv2.INTER_NEAREST) break return crop_img, crop_seg def saturation_jitter(cv_img, jitter_range): """ 调节图像饱和度 Args: cv_img(numpy.ndarray): 输入图像 jitter_range(float): 调节程度，0-1 Returns: 饱和度调整后的图像 """ greyMat = cv2.cvtColor(cv_img, cv2.COLOR_BGR2GRAY) greyMat = greyMat[:, :, None] * np.ones(3, dtype=int)[None, None, :] cv_img = cv_img.astype(np.float32) cv_img = cv_img * (1 - jitter_range) + jitter_range * greyMat cv_img = np.where(cv_img > 255, 255, cv_img) cv_img = cv_img.astype(np.uint8) return cv_img def brightness_jitter(cv_img, jitter_range): """ 调节图像亮度 Args: cv_img(numpy.ndarray): 输入图像 jitter_range(float): 调节程度，0-1 Returns: 亮度调整后的图像 """ cv_img = cv_img.astype(np.float32) cv_img = cv_img * (1.0 - jitter_range) cv_img = np.where(cv_img > 255, 255, cv_img) cv_img = cv_img.astype(np.uint8) return cv_img def contrast_jitter(cv_img, jitter_range): """ 调节图像对比度 Args: cv_img(numpy.ndarray): 输入图像 jitter_range(float): 调节程度，0-1 Returns: 对比度调整后的图像 """ greyMat = cv2.cvtColor(cv_img, cv2.COLOR_BGR2GRAY) mean = np.mean(greyMat) cv_img = cv_img.astype(np.float32) cv_img = cv_img * (1 - jitter_range) + jitter_range * mean cv_img = np.where(cv_img > 255, 255, cv_img) cv_img = cv_img.astype(np.uint8) return cv_img def random_jitter(cv_img, saturation_range, brightness_range, contrast_range): """ 图像亮度、饱和度、对比度调节，在调整范围内随机获得调节比例，并随机顺序叠加三种效果 Args: cv_img(numpy.ndarray): 输入图像 saturation_range(float): 饱和对调节范围，0-1 brightness_range(float): 亮度调节范围，0-1 contrast_range(float): 对比度调节范围，0-1 Returns: 亮度、饱和度、对比度调整后图像 """ saturation_ratio = np.random.uniform(-saturation_range, saturation_range) brightness_ratio = np.random.uniform(-brightness_range, brightness_range) contrast_ratio = np.random.uniform(-contrast_range, contrast_range) order = [0, 1, 2] np.random.shuffle(order) for i in range(3): if order[i] == 0: cv_img = saturation_jitter(cv_img, saturation_ratio) if order[i] == 1: cv_img = brightness_jitter(cv_img, brightness_ratio) if order[i] == 2: cv_img = contrast_jitter(cv_img, contrast_ratio) return cv_img def hsv_color_jitter(crop_img, brightness_jitter_ratio=0, saturation_jitter_ratio=0, contrast_jitter_ratio=0): """ 图像亮度、饱和度、对比度调节 Args: crop_img(numpy.ndarray): 输入图像 brightness_jitter_ratio(float): 亮度调节度最大值，1-0，默认0 saturation_jitter_ratio(float): 饱和度调节度最大值，1-0，默认0 contrast_jitter_ratio(float): 对比度调节度最大值，1-0，默认0 Returns： 亮度、饱和度、对比度调节后图像 """ if brightness_jitter_ratio > 0 or \ saturation_jitter_ratio > 0 or \ contrast_jitter_ratio > 0: crop_img = random_jitter(crop_img, saturation_jitter_ratio, brightness_jitter_ratio, contrast_jitter_ratio) return crop_img def rand_crop(crop_img, crop_seg, mode=ModelPhase.TRAIN): """ 随机裁剪图片和标签图, 若crop尺寸大于原始尺寸，分别使用DATASET.PADDING_VALUE值和DATASET.IGNORE_INDEX值填充再进行crop， crop尺寸与原始尺寸一致，返回原图，crop尺寸小于原始尺寸直接crop Args: crop_img(numpy.ndarray): 输入图像 crop_seg(numpy.ndarray): 标签图 mode(string): 模式, 默认训练模式，验证或预测、可视化模式时crop尺寸需大于原始图片尺寸 Returns： 裁剪后的图片和标签图 """ img_height = crop_img.shape[0] img_width = crop_img.shape[1] if ModelPhase.is_train(mode): crop_width = cfg.TRAIN_CROP_SIZE[0] crop_height = cfg.TRAIN_CROP_SIZE[1] else: crop_width = cfg.EVAL_CROP_SIZE[0] crop_height = cfg.EVAL_CROP_SIZE[1] if not ModelPhase.is_train(mode): if (crop_height < img_height or crop_width < img_width): raise Exception( "Crop size({},{}) must large than img size({},{}) when in EvalPhase." .format(crop_width, crop_height, img_width, img_height)) if img_height == crop_height and img_width == crop_width: return crop_img, crop_seg else: pad_height = max(crop_height - img_height, 0) pad_width = max(crop_width - img_width, 0) if (pad_height > 0 or pad_width > 0): crop_img = cv2.copyMakeBorder( crop_img, 0, pad_height, 0, pad_width, cv2.BORDER_CONSTANT, value=cfg.DATASET.PADDING_VALUE) if crop_seg is not None: crop_seg = cv2.copyMakeBorder( crop_seg, 0, pad_height, 0, pad_width, cv2.BORDER_CONSTANT, value=cfg.DATASET.IGNORE_INDEX) img_height = crop_img.shape[0] img_width = crop_img.shape[1] if crop_height > 0 and crop_width > 0: h_off = np.random.randint(img_height - crop_height + 1) w_off = np.random.randint(img_width - crop_width + 1) crop_img = crop_img[h_off:(crop_height + h_off), w_off:( w_off + crop_width), :] if crop_seg is not None: crop_seg = crop_seg[h_off:(crop_height + h_off), w_off:( w_off + crop_width)] return crop_img, crop_seg
Github - NVIDIA/DALIdocs - NVIDIA DALI documentationNVIDIA/DALI(NVIDIA Data Loading Library) 库包含了用于加速深度学习应用的数据预处理的高度优化的构建模块(highly optimized building blocks)和执行引擎(execution engine).深度学习应用中，往往需要复杂的、多阶段的数据预处理管道. 这些数据管道主要是在CPU上执行的数据密集型操作(compute-intensive operation),b 比如，从磁盘加载数据，解码，裁剪，随机resize，颜色和空间增强，格式转换，等等，都主要是在 CPUs 上完成的，其限制了训练和推断时的性能和可扩展性.另外，深度学习框架有很多不同的数据预处理实现，这就为网络训练和推断工作流的可移植性和代码维护带来了挑战.DALI 是高度优化后的构建模块的集合，是用于加速输入数据预处理的执行引擎. DALI 作为单独的库，为不同数据管道的加速，提供了更优的性能和灵活性. DALI 库可以很方便的整合仅不同深度学习训练和推断应用中.1. DALI Highlights[1] - 全数据管道加速 - 从磁盘读取到用于训练和推断的数据处理.[2] - 灵活性高 - 可配置图(configurable graphs)和定制操作子(custom operators).[3] - 支持图像分类和分割的数据加载.[4] - 易于整合 - 直接的框架插件和开源绑定(open source bindings).[5] - 训练工作流的便携性，支持多种输入格式：JPEG,PNG (fallback to CPU), TIFF (fallback to CPU), BMP (fallback to CPU), raw formats, LMDB, RecordIO, TFRecord.[6] - 定制化的可扩展性.2. DALI 安装NVIDIA GPU Cloud 中的 TensorFlow, PyTorch 和 MXNet containers(18.07 以后版本) 已经预安装了 DALI 库.2.1. prebuilt DALI 安装依赖项：[1] - Linux x64.[2] - NVIDIA Driver supporting CUDA 9.0 or later (i.e., 384.xx or later driver releases).[3] - 深度框架:MXNet 1.3 mxnet-cu90 or later.PyTorch 0.4 or later.TensorFlow 1.7 or later.CUDA9.0 安装：pip install --extra-index-url https://developer.download.nvidia.com/compute/redist/cuda/9.0 nvidia-daliCUDA10.0 安装(DALI 0.80 以后版本)：pip install --extra-index-url https://developer.download.nvidia.com/compute/redist/cuda/10.0 nvidia-daliDALI 0.6.1 版本以后，nvidia-dali 不再包含 DALI TensorFlow 插件的 prebuilt 版本，因此需要采用如下方式安装：pip install --extra-index-url https://developer.download.nvidia.com/compute/redist/cuda/9.0 nvidia-dali-tf-pluginDALI 0.8.0 版本以后，支持 CUDA10.0：pip install --extra-index-url https://developer.download.nvidia.com/compute/redist/cuda/10.0 nvidia-dali-tf-plugin此时，会自动安装 nvidia-dali 及其依赖项.安装指定版本：pip install --extra-index-url https://developer.download.nvidia.com/compute/redist nvidia-dali-tf-plugin==0.6.1### 2.2. Nightly builds DALI 安装在安装每日更新的 DALI 库前，建议先卸载常规 DALI.CUDA9:pip install --extra-index-url https://developer.download.nvidia.com/compute/redist/nightly/cuda/9.0 nvidia-dali-nightly pip install --extra-index-url https://developer.download.nvidia.com/compute/redist/nightly/cuda/9.0 nvidia-dali-tf-plugin-nightlyCUDA10:pip install --extra-index-url https://developer.download.nvidia.com/compute/redist/nightly/cuda/10.0 nvidia-dali-nightly pip install --extra-index-url https://developer.download.nvidia.com/compute/redist/nightly/cuda/10.0 nvidia-dali-tf-plugin-nightly2.3. Weekly builds DALI 安装每周更新版本CUDA10pip install --extra-index-url https://developer.download.nvidia.com/compute/redist/weekly/cuda/10.0 nvidia-dali-weekly pip install --extra-index-url https://developer.download.nvidia.com/compute/redist/weekly/cuda/10.0 nvidia-dali-tf-plugin-weekly2.4. DALI 源码安装Compiling DALI from source3. 简单使用深度学习框架提供的数据输入和数据增强管道，一般可分为如下两类：[1] - 快速，但不灵活 - C++ 编码，它对外保留为单一的 python 对象，仅提供了特定的操作集和顺序.[2] - 慢，但灵活 - 基于 C++ 或 Python 编码的构建模块集合，其可以任意组合数据管道，但速度变慢. 一个最大的问题是，这种数据管道会遇到 Python 的 GIL(Global Interpreter Lock) 问题. 这就需要开发者采用多进程、有效的输入管道需要复杂的设计.3.1. DALI PipelineDALI 中最重要的类型，其包含了所有必要信息以及与定义、构建、运行等相关的函数.from nvidia.dali.pipeline import Pipeline help(Pipeline)输出如：Help on class Pipeline in module nvidia.dali.pipeline: class Pipeline(__builtin__.object) | Pipeline class encapsulates all data required to define and run | DALI input pipeline. | | Parameters | ---------- | `batch_size` : int, optional, default = -1 | Batch size of the pipeline. Negative values for this parameter | are invalid - the default value may only be used with | serialized pipeline (the value stored in serialized pipeline | is used instead). | `num_threads` : int, optional, default = -1 # CPU 线程数 | Number of CPU threads used by the pipeline. | Negative values for this parameter are invalid - the default | value may only be used with serialized pipeline (the value | stored in serialized pipeline is used instead). | `device_id` : int, optional, default = -1 # GPU Id | Id of GPU used by the pipeline. | Negative values for this parameter are invalid - the default | value may only be used with serialized pipeline (the value | stored in serialized pipeline is used instead). | `seed` : int, optional, default = -1 # 随机数生成 | Seed used for random number generation. Leaving the default value | for this parameter results in random seed. | `exec_pipelined` : bool, optional, default = True | Whether to execute the pipeline in a way that enables | overlapping CPU and GPU computation, typically resulting | in faster execution speed, but larger memory consumption. | `exec_async` : bool, optional, default = True | Whether to execute the pipeline asynchronously. | This makes :meth:`nvidia.dali.pipeline.Pipeline.run` method | run asynchronously with respect to the calling Python thread. | In order to synchronize with the pipeline one needs to call | :meth:`nvidia.dali.pipeline.Pipeline.outputs` method. | `bytes_per_sample` : int, optional, default = 0 | A hint for DALI for how much memory to use for its tensors. | `set_affinity` : bool, optional, default = False | Whether to set CPU core affinity to the one closest to the | GPU being used. | `max_streams` : int, optional, default = -1 # CUDA 流数 | Limit the number of CUDA streams used by the executor. | Value of -1 does not impose a limit. | This parameter is currently unused (and behavior of | unrestricted number of streams is assumed). | `prefetch_queue_depth` : int or {"cpu_size": int, "gpu_size": int}, optional, default = 2 | Depth of the executor pipeline. Deeper pipeline makes DALI | more resistant to uneven execution time of each batch, but it | also consumes more memory for internal buffers. | Specifying a dict: | ``{ "cpu_size": x, "gpu_size": y }`` | instead of integer will cause the pipeline to use separated | queues executor, with buffer queue size `x` for cpu stage | and `y` for mixed and gpu stages. It is not supported when both `exec_async` | and `exec_pipelined` are set to `False`. | Executor will buffer cpu and gpu stages separatelly, | and will fill the buffer queues when the first :meth:`nvidia.dali.pipeline.Pipeline.run` | is issued. | | Methods defined here: | | __init__(self, batch_size=-1, num_threads=-1, device_id=-1, seed=-1, exec_pipelined=True, prefetch_queue_depth=2, exec_async=True, bytes_per_sample=0, set_affinity=False, max_streams=-1, default_cuda_stream_priority=0) | | build(self) | Build the pipeline. | | Pipeline needs to be built in order to run it standalone. | Framework-specific plugins handle this step automatically. | | define_graph(self) | This function is defined by the user to construct the | graph of operations for their pipeline. | | It returns a list of outputs created by calling DALI Operators. | | deserialize_and_build(self, serialized_pipeline) | Deserialize and build the pipeline given in serialized form. | | Parameters | ---------- | serialized_pipeline : str | Serialized pipeline. | | epoch_size(self, name=None) | Epoch size of a pipeline. | | If the `name` parameter is `None`, returns a dictionary of pairs | `(reader name, epoch size for that reader)`. | If the `name` parameter is not `None`, returns epoch size for that | reader. | | Parameters | ---------- | name : str, optional, default = None | The reader which should be used to obtain epoch size. | | feed_input(self, ref, data, layout=TensorLayout.NHWC) | Bind the NumPy array to a tensor produced by ExternalSource | operator. It is worth mentioning that `ref` should not be overriden | with other operator outputs. | | iter_setup(self) | This function can be overriden by user-defined | pipeline to perform any needed setup for each iteration. | For example, one can use this function to feed the input | data from NumPy arrays. | | outputs(self) | Returns the outputs of the pipeline and releases previous buffer. | | If the pipeline is executed asynchronously, this function blocks | until the results become available. It rises StopIteration if data set | reached its end - usually when iter_setup cannot produce any more data | | reset(self) | Resets pipeline iterator | | If pipeline iterator reached the end then reset its state to the beginning. | | run(self) | Run the pipeline and return the result. | | If the pipeline was created with `exec_pipelined` option set to `True`, | this function will also start prefetching the next iteration for | faster execution. | | save_graph_to_dot_file(self, filename) | Saves the pipeline graph to a file. | | Parameters | ---------- | filename : str | Name of the file to which the graph is written. | | serialize(self) | Serialize the pipeline to a Protobuf string. | | ---------------------------------------------------------------------- | Data descriptors defined here: | | __dict__ | dictionary for instance variables (if defined) | | __weakref__ | list of weak references to the object (if defined) | | batch_size | Batch size. | | device_id | Id of the GPU used by the pipeline. | | num_threads | Number of CPU threads used by the pipeline.3.2. 定义 DALI pipeline以猫狗分类为例.数据集路径结构如下：from __future__ import print_function import os.path import fnmatch for root, dir, files in os.walk("data/images"): depth = root.count('/') ret = "" if depth > 0: ret += " " * (depth - 1) + "|-" print (ret + root) for items in fnmatch.filter(files, "*"): print (" " * len(ret) + "|-" + items)如：images |-file_list.txt |-images/dog |-dog_4.jpg |-dog_5.jpg |-dog_9.jpg |-dog_6.jpg |-dog_3.jpg |-dog_7.jpg |-dog_10.jpg |-dog_2.jpg |-dog_8.jpg |-dog_1.jpg |-dog_11.jpg |-images/kitten |-cat_10.jpg |-cat_5.jpg |-cat_9.jpg |-cat_8.jpg |-cat_1.jpg |-cat_7.jpg |-cat_6.jpg |-cat_3.jpg |-cat_2.jpg |-cat_4.jpg这里，创建的 pipeline 将从数据集路径中读取图片，解码图片，并返回 (image, labels) 对.import nvidia.dali.ops as ops import nvidia.dali.types as types image_dir = "data/images" batch_size = 8 class SimplePipeline(Pipeline): def __init__(self, batch_size, num_threads, device_id): super(SimplePipeline, self).__init__(batch_size, num_threads, device_id, seed = 12) self.input = ops.FileReader(file_root = image_dir) # instead of path to file directory file with pairs image_name image_label_value can be provided # self.input = ops.FileReader(file_root = image_dir, file_list = image_dir + '/file_list.txt') self.decode = ops.ImageDecoder(device = 'cpu', output_type = types.RGB) def define_graph(self): jpegs, labels = self.input() images = self.decode(jpegs) return (images, labels)这里，SimplePipeline 类是 dali.pipeline.Pipeline 的子类. 此处，仅需要实现两个方法： 构建类 constructor 和 define_graph 函数.构建的 pipeline 的全局参数有：[1] - batch size[2] - 计算所用的 CPU 线程数[3] - GPU 设备Id.(SimplePipeline 没有采用到 GPU 进行计算)[4] - 随机数生成种子此外，还包含两个成员变量，采用 dali.ops 模块的操作子：[1] - FileReader - 遍历路径，并返回 (image, label) 对.[2] - ImageDecoder - 采用编码的图像作为输入，并输出解码后的 RGB 图像.3.3. 构建 DALI pipelinepipe = SimplePipeline(batch_size, 1, 0) pipe.build()3.4. 运行 DALI pipelinepipe_out = pipe.run() print(pipe_out)输出如：[<nvidia.dali.backend_impl.TensorListCPU object at 0x7ff6080bf180>, <nvidia.dali.backend_impl.TensorListCPU object at 0x7ff6080bf5e0>]数据格式为 DALI Tensor. 其可以转换为 Numpy array，但并不是所有的 DALI Tensor 都能够进行转换.images, labels = pipe_out print("Images is_dense_tensor: " + str(images.is_dense_tensor())) print("Labels is_dense_tensor: " + str(labels.is_dense_tensor()))输出如：Images is_dense_tensor: False Labels is_dense_tensor: Truelabels 的值如：import numpy as np labels_tensor = labels.as_tensor() print (labels_tensor.shape()) print (np.array(labels_tensor))输出如：[8L, 1L] [[0] [0] [0] [0] [0] [0] [0] [0]]images 的值需要对 TensorList 中的所有 tensors 进行循环，如：from __future__ import division import matplotlib.gridspec as gridspec import matplotlib.pyplot as plt def show_images(image_batch): columns = 4 rows = (batch_size + 1) // (columns) fig = plt.figure(figsize = (32,(32 // columns) * rows)) gs = gridspec.GridSpec(rows, columns) for j in range(rows*columns): plt.subplot(gs[j]) plt.axis("off") plt.imshow(image_batch.at(j)) plt.show() # show_images(images)3.5. 添加数据增强 DALI pipline3.5.1. Random shuffle随机打乱数据顺序.class ShuffledSimplePipeline(Pipeline): def __init__(self, batch_size, num_threads, device_id): super(ShuffledSimplePipeline, self).__init__(batch_size, num_threads, device_id, seed = 12) self.input = ops.FileReader(file_root = image_dir, random_shuffle = True, initial_fill = 21) self.decode = ops.ImageDecoder(device = 'cpu', output_type = types.RGB) def define_graph(self): jpegs, labels = self.input() images = self.decode(jpegs) return (images, labels) # pipe = ShuffledSimplePipeline(batch_size, 1, 0) pipe.build() pipe_out = pipe.run() images, labels = pipe_out show_images(images) 输出如：3.5.2. RotateDALI 不仅可以从磁盘读取图片，并 batch 化为 tensors；还可以对图片进行各种数据增强.图片固定角度旋转.class RotatedSimplePipeline(Pipeline): def __init__(self, batch_size, num_threads, device_id): super(RotatedSimplePipeline, self).__init__(batch_size, num_threads, device_id, seed = 12) self.input = ops.FileReader(file_root = image_dir, random_shuffle = True, initial_fill = 21) self.decode = ops.ImageDecoder(device = 'cpu', output_type = types.RGB) self.rotate = ops.Rotate(angle = 10.0) def define_graph(self): jpegs, labels = self.input() images = self.decode(jpegs) rotated_images = self.rotate(images) return (rotated_images, labels) # pipe = RotatedSimplePipeline(batch_size, 1, 0) pipe.build() pipe_out = pipe.run() images, labels = pipe_out show_images(images)输出如：3.5.3. RandomRotated随机旋转一定角度.class RandomRotatedSimplePipeline(Pipeline): def __init__(self, batch_size, num_threads, device_id): super(RandomRotatedSimplePipeline, self).__init__(batch_size, num_threads, device_id, seed = 12) self.input = ops.FileReader(file_root = image_dir, random_shuffle = True, initial_fill = 21) self.decode = ops.ImageDecoder(device = 'cpu', output_type = types.RGB) self.rotate = ops.Rotate() self.rng = ops.Uniform(range = (-10.0, 10.0)) def define_graph(self): jpegs, labels = self.input() images = self.decode(jpegs) angle = self.rng() rotated_images = self.rotate(images, angle = angle) return (rotated_images, labels) # pipe = RandomRotatedSimplePipeline(batch_size, 1, 0) pipe.build() pipe_out = pipe.run() images, labels = pipe_out show_images(images)输出如：3.6. GPU 加速DALI 库提供了 GPU 加速操作子，用于提升数据输入和数据增强的速度，而且易于扩展到 multi-GPU 系统.3.6.1. tensors 复制到 GPU以 RandomRotatedSimplePipeline Pipeline 为例：class RandomRotatedGPUPipeline(Pipeline): def __init__(self, batch_size, num_threads, device_id): super(RandomRotatedGPUPipeline, self).__init__(batch_size, num_threads, device_id, seed = 12) self.input = ops.FileReader(file_root = image_dir, random_shuffle = True, initial_fill = 21) self.decode = ops.ImageDecoder(device = 'cpu', output_type = types.RGB) self.rotate = ops.Rotate(device = "gpu") # self.rng = ops.Uniform(range = (-10.0, 10.0)) def define_graph(self): jpegs, labels = self.input() images = self.decode(jpegs) angle = self.rng() rotated_images = self.rotate(images.gpu(), angle = angle) # return (rotated_images, labels) # pipe = RandomRotatedGPUPipeline(batch_size, 1, 0) pipe.build() pipe_out = pipe.run() print(pipe_out)输出如：[<nvidia.dali.backend_impl.TensorListGPU object at 0x7ff565380ed8>, <nvidia.dali.backend_impl.TensorListCPU object at 0x7ff564060148>]TensorListGPU 即为在 GPU 上的 tensors，采用 as_cpu 可以将 GPU 数据复制到 CPU：images, labels = pipe_out show_images(images.as_cpu())输出如：注意事项： DALI 库不支持将数据在 pipeline 内部从 GPU 移动到 CPU. 在所有执行路径中，CPU运算符不能跟在GPU运算符后面.3.6.2. 混合解码(Hybrid decoding)在某些场景下，尤其是，高分辨率图片，JPEG格式存储的图片的解码是一个瓶颈. 对此，nvJPEG 库将比较有帮助，其将解码过程分为 CPU 和 GPU，显著降低解码时间.指定 ImageDecoder 的 mixed 参数，以开启 nvJPEG 支持. 其它格式的文件仍在 CPU 上进行解码.nvJPEGclass HybridPipeline(Pipeline): def __init__(self, batch_size, num_threads, device_id): super(HybridPipeline, self).__init__(batch_size, num_threads, device_id, seed = 12) self.input = ops.FileReader(file_root = image_dir, random_shuffle = True, initial_fill = 21) self.decode = ops.ImageDecoder(device = "mixed", output_type = types.RGB) def define_graph(self): jpegs, labels = self.input() images = self.decode(jpegs) # images are on the GPU return (images, labels) # pipe = HybridPipeline(batch_size, 1, 0) pipe.build() pipe_out = pipe.run() images, labels = pipe_out show_images(images.as_cpu())ImageDecoder 设置 device=mixed 参数，将使用混合方法同时在 CPU 和 GPU 上进行计算. 也就是说，其接收 CPU 输入，但返回 GPU 输出.输出如：ImageDecoder 参数设置为 cpu 和 mixed 的速度对比:from timeit import default_timer as timer test_batch_size = 64 def speedtest(pipeclass, batch, n_threads): pipe = pipeclass(batch, n_threads, 0) pipe.build() # warmup for i in range(5): pipe.run() # test n_test = 20 t_start = timer() for i in range(n_test): pipe.run() t = timer() - t_start print("Speed: {} imgs/s".format((n_test * batch)/t)) # speedtest(ShuffledSimplePipeline, test_batch_size, 4) #Speed: 2905.71010277 imgs/s # speedtest(HybridPipeline, test_batch_size, 4) #Speed: 5714.61475087 imgs/s4. 示例From: nvidia-dali GPU加速预处理 - 2019.08.17import nvidia.dali.ops as ops import nvidia.dali.types as types from nvidia.dali.pipeline import Pipeline from nvidia.dali.plugin.pytorch import DALIClassificationIterator, DALIGenericIterator class HybridTrainPipe(Pipeline): def __init__(self, batch_size, num_threads, device_id, data_dir, crop, dali_cpu=False, local_rank=0, world_size=1): super(HybridTrainPipe, self).__init__( batch_size, num_threads, device_id, seed=12 + device_id) dali_device = "gpu" self.input = ops.FileReader(file_root=data_dir, shard_id=local_rank, num_shards=world_size, random_shuffle=True) self.decode = ops.ImageDecoder(device="mixed", output_type=types.RGB) self.res = ops.RandomResizedCrop( device="gpu", size=crop, random_area=[0.08, 1.25]) self.cmnp = ops.CropMirrorNormalize( device="gpu", output_dtype=types.FLOAT, output_layout=types.NCHW, image_type=types.RGB, mean=[0.485 * 255, 0.456 * 255, 0.406 * 255], std=[0.229 * 255, 0.224 * 255, 0.225 * 255]) self.coin = ops.CoinFlip(probability=0.5) print('DALI "{0}" variant'.format(dali_device)) def define_graph(self): rng = self.coin() self.jpegs, self.labels = self.input(name="Reader") images = self.decode(self.jpegs) images = self.res(images) output = self.cmnp(images, mirror=rng) return [output, self.labels] class HybridValPipe(Pipeline): def __init__(self, batch_size, num_threads, device_id, data_dir, crop, size, local_rank=0, world_size=1): super(HybridValPipe, self).__init__( batch_size, num_threads, device_id, seed=12 + device_id) self.input = ops.FileReader(file_root=data_dir, shard_id=local_rank, num_shards=world_size, random_shuffle=False) self.decode = ops.ImageDecoder(device="mixed", output_type=types.RGB) self.res = ops.Resize( device="gpu", resize_shorter=size, interp_type=types.INTERP_TRIANGULAR) self.cmnp = ops.CropMirrorNormalize( device="gpu", output_dtype=types.FLOAT, output_layout=types.NCHW, crop=(crop, crop), image_type=types.RGB, mean=[0.485 * 255, 0.456 * 255, 0.406 * 255], std=[0.229 * 255, 0.224 * 255, 0.225 * 255]) def define_graph(self): self.jpegs, self.labels = self.input(name="Reader") images = self.decode(self.jpegs) images = self.res(images) output = self.cmnp(images) return [output, self.labels] def get_imagenet_iter_dali(type, image_dir, batch_size, num_threads, device_id, num_gpus, crop, val_size=256, world_size=1, local_rank=0): if type == 'train': pip_train = HybridTrainPipe( batch_size=batch_size, num_threads=num_threads, device_id=local_rank, data_dir=image_dir + '/train', crop=crop, world_size=world_size, local_rank=local_rank) pip_train.build() dali_iter_train = DALIClassificationIterator( pip_train, size=pip_train.epoch_size("Reader") // world_size) return dali_iter_train elif type == 'val': pip_val = HybridValPipe( batch_size=batch_size, num_threads=num_threads, device_id=local_rank, data_dir=image_dir + '/val', crop=crop, size=val_size, world_size=world_size, local_rank=local_rank) pip_val.build() dali_iter_val = DALIClassificationIterator( pip_val, size=pip_val.epoch_size("Reader") // world_size) return dali_iter_val if __name__ == '__main__': train_loader = get_imagenet_iter_dali( type='train', image_dir='/path/to/imagenet', batch_size=256, num_threads=4, crop=224, device_id=0, num_gpus=1) print('[INFO]start iterate') start = time.time() for i, data in enumerate(train_loader): images = data[0]["data"].cuda(non_blocking=True) labels = data[0]["label"].squeeze().long().cuda(non_blocking=True) end = time.time() print('[INFO]end iterate') print('[INFO]dali iterate time: %fs' % (end - start))5. 相关[1] - 英伟达DALI加速技巧：让数据预处理速度比原生PyTorch快4倍 - 2020.02.04[2] - Nvidia dali speeding up pytorch
Github 项目 - 百度语义分割库PaddleSeg - AIUAIGithub 项目 - PaddleSeg 模型训练 - AIUAI在语义分割的模型训练时，由于语义分割数据集的标注与收集难度比较大，因此往往需要借助数据增强策略进行处理.1. PaddleSeg 数据增强基本流程PaddleSeg 数据增强1.1. Resizeresize 步骤是指将输入图像按照某种规则讲图片重新缩放到某一个尺寸.PaddleSeg 支持以下3种resize方式:[1] - Unpadding 将输入图像直接resize到某一个固定大小下，送入到网络中间训练，对应参数为AUG.FIX_RESIZE_SIZE. 预测时同样操作. [2] - Step-Scaling 将输入图像按照某一个比例 resize，这个比例以某一个步长在一定范围内随机变动. 设定最小比例参数为AUG.MIN_SCALE_FACTOR, 最大比例参数AUG.MAX_SCALE_FACTOR，步长参数为AUG.SCALE_STEP_SIZE. 预测时不对输入图像做处理. [3] - Range-Scaling 固定长宽比resize，即图像长边对齐到某一个固定大小，短边随同样的比例变化. 设定最小大小参数为AUG.MIN_RESIZE_VALUE，设定最大大小参数为AUG.MAX_RESIZE_VALUE. 预测时需要将长边对齐到AUG.INF_RESIZE_VALUE所指定的大小，其中AUG.INF_RESIZE_VALUE在AUG.MIN_RESIZE_VALUE和AUG.MAX_RESIZE_VALUE范围内. Range-Scaling示意图如下：1.2. FlipPaddleSeg支持以下2种翻转方式：[1] - 左右翻转(Mirror) ：AUG.MIRROR，为True时该功能开启，为False时该功能关闭. [2] - 上下翻转(Flip) ：AUG.FLIP，为True时该功能开启，AUG.FLIP_RATIO 控制是否上下翻转的概率. 为False时该项功能关闭. 二者可独立运作，也可组合使用. 故图像翻转一共有如下4种可能的情况：1.3. Rich CropRich Crop 是 PaddleSeg 结合实际业务经验开放的一套数据增强策略，面向标注数据少，测试数据情况繁杂的分割业务场景使用的数据增强策略. 流程如下图所示:Rich crop 是指对图像进行多种变换，保证在训练过程中数据的丰富多样性，PaddleSeg支持以下几种变换. AUG.RICH_CROP.ENABLE为False时会直接跳过该步骤. [1] - blur 图像加模糊，使用开关AUG.RICH_CROP.BLUR，为False时该功能关闭. AUG.RICH_CROP.BLUR_RATIO控制加入模糊的概率. [2] - rotation 图像旋转，AUG.RICH_CROP.MAX_ROTATION控制最大旋转角度. 旋转产生的多余的区域的填充值为均值. [3] - aspect 图像长宽比调整，从图像中crop一定区域出来之后在某一长宽比内进行resize. 控制参数AUG.RICH_CROP.MIN_AREA_RATIO和AUG.RICH_CROP.ASPECT_RATIO. [4] - color jitter 图像颜色调整，控制参数AUG.RICH_CROP.BRIGHTNESS_JITTER_RATIO、AUG.RICH_CROP.SATURATION_JITTER_RATIO、AUG.RICH_CROP.CONTRAST_JITTER_RATIO.1.4. Random Crop该步骤主要是通过crop的方式使得输入到网络中的图像在某一个固定大小，控制该大小的参数为TRAIN_CROP_SIZE，类型为tuple，格式为(width, height). 当输入图像大小小于CROP_SIZE的时候会对输入图像进行padding，padding值为均值.输入图片格式原图图片格式：RGB三通道图片和RGBA四通道图片两种类型的图片进行训练，但是在一次训练过程只能存在一种格式.图片转换：灰度图片经过预处理后之后会转变成三通道图片图片参数设置：当图片为三通道图片时IMAGE_TYPE设置为rgb， 对应MEAN和STD也必须是一个长度为3的list，当图片为四通道图片时IMAGE_TYPE设置为rgba，对应的MEAN和STD必须是一个长度为4的list.标注图图片格式：标注图片必须为png格式的单通道多值图，元素值代表的是这个元素所属于的类别.图片转换：在datalayer层对label图片进行的任何resize，以及旋转的操作，都必须采用最近邻的插值方式.图片ignore：设置TRAIN.IGNORE_INDEX 参数可以选择性忽略掉属于某一个类别的所有像素点. 这个参数一般设置为2552. PaddlseSeg 数据增强定义Github 项目 - PaddleSeg 模型训练 - 数据增强## 数据增强配置 ## # 图像镜像左右翻转 cfg.AUG.MIRROR = True # 图像上下翻转开关，True/False cfg.AUG.FLIP = False # 图像启动上下翻转的概率，0-1 cfg.AUG.FLIP_RATIO = 0.5 # 图像resize的固定尺寸（宽，高），非负 cfg.AUG.FIX_RESIZE_SIZE = tuple() # 图像resize的方式有三种： # unpadding（固定尺寸），stepscaling（按比例resize），rangescaling（长边对齐） cfg.AUG.AUG_METHOD = 'rangescaling' # 图像resize方式为stepscaling，resize最小尺度，非负 cfg.AUG.MIN_SCALE_FACTOR = 0.5 # 图像resize方式为stepscaling，resize最大尺度，不小于MIN_SCALE_FACTOR cfg.AUG.MAX_SCALE_FACTOR = 2.0 # 图像resize方式为stepscaling，resize尺度范围间隔，非负 cfg.AUG.SCALE_STEP_SIZE = 0.25 # 图像resize方式为rangescaling，训练时长边resize的范围最小值，非负 cfg.AUG.MIN_RESIZE_VALUE = 400 # 图像resize方式为rangescaling，训练时长边resize的范围最大值， # 不小于MIN_RESIZE_VALUE cfg.AUG.MAX_RESIZE_VALUE = 600 # 图像resize方式为rangescaling, 测试验证可视化模式下长边resize的长度， # 在MIN_RESIZE_VALUE到MAX_RESIZE_VALUE范围内 cfg.AUG.INF_RESIZE_VALUE = 500 # RichCrop数据增广开关，用于提升模型鲁棒性 cfg.AUG.RICH_CROP.ENABLE = False # 图像旋转最大角度，0-90 cfg.AUG.RICH_CROP.MAX_ROTATION = 15 # 裁取图像与原始图像面积比，0-1 cfg.AUG.RICH_CROP.MIN_AREA_RATIO = 0.5 # 裁取图像宽高比范围，非负 cfg.AUG.RICH_CROP.ASPECT_RATIO = 0.33 # 亮度调节范围，0-1 cfg.AUG.RICH_CROP.BRIGHTNESS_JITTER_RATIO = 0.5 # 饱和度调节范围，0-1 cfg.AUG.RICH_CROP.SATURATION_JITTER_RATIO = 0.5 # 对比度调节范围，0-1 cfg.AUG.RICH_CROP.CONTRAST_JITTER_RATIO = 0.5 # 图像模糊开关，True/False cfg.AUG.RICH_CROP.BLUR = False # 图像启动模糊百分比，0-1 cfg.AUG.RICH_CROP.BLUR_RATIO = 0.13. PaddlseSeg 数据增强配置在 configs/xxx.yaml 中的使用配置，如 deeplabv3p_xception65.yaml：AUG: AUG_METHOD: "unpadding" # choice unpadding rangescaling and stepscaling FIX_RESIZE_SIZE: (800, 800) # (width, height), for unpadding INF_RESIZE_VALUE: 800 # for rangescaling MAX_RESIZE_VALUE: 1200 # for rangescaling MIN_RESIZE_VALUE: 600 # for rangescaling MAX_SCALE_FACTOR: 1.25 # for stepscaling MIN_SCALE_FACTOR: 0.75 # for stepscaling SCALE_STEP_SIZE: 0.25 # for stepscaling MIRROR: True4. PaddleSeg 数据增强实现pdseg/data_aug.py#!--*-- coding: utf8 --*-- from __future__ import print_function import cv2 import numpy as np from utils.config import cfg from models.model_builder import ModelPhase def resize(img, grt=None, mode=ModelPhase.TRAIN): """ 改变图像及标签图像尺寸 AUG.AUG_METHOD为unpadding，所有模式均直接resize到AUG.FIX_RESIZE_SIZE的尺寸 AUG.AUG_METHOD为stepscaling, 按比例resize，训练时比例范围AUG.MIN_SCALE_FACTOR到AUG.MAX_SCALE_FACTOR,间隔为AUG.SCALE_STEP_SIZE，其他模式返回原图 AUG.AUG_METHOD为rangescaling，长边对齐，短边按比例变化，训练时长边对齐范围AUG.MIN_RESIZE_VALUE到AUG.MAX_RESIZE_VALUE，其他模式长边对齐AUG.INF_RESIZE_VALUE Args： img(numpy.ndarray): 输入图像 grt(numpy.ndarray): 标签图像，默认为None mode(string): 模式, 默认训练模式，即ModelPhase.TRAIN Returns： resize后的图像和标签图 """ if cfg.AUG.AUG_METHOD == 'unpadding': target_size = cfg.AUG.FIX_RESIZE_SIZE img = cv2.resize(img, target_size, interpolation=cv2.INTER_LINEAR) if grt is not None: grt = cv2.resize(grt, target_size, interpolation=cv2.INTER_NEAREST) elif cfg.AUG.AUG_METHOD == 'stepscaling': if mode == ModelPhase.TRAIN: min_scale_factor = cfg.AUG.MIN_SCALE_FACTOR max_scale_factor = cfg.AUG.MAX_SCALE_FACTOR step_size = cfg.AUG.SCALE_STEP_SIZE scale_factor = get_random_scale( min_scale_factor, max_scale_factor, step_size) img, grt = randomly_scale_image_and_label( img, grt, scale=scale_factor) elif cfg.AUG.AUG_METHOD == 'rangescaling': min_resize_value = cfg.AUG.MIN_RESIZE_VALUE max_resize_value = cfg.AUG.MAX_RESIZE_VALUE if mode == ModelPhase.TRAIN: if min_resize_value == max_resize_value: random_size = min_resize_value else: random_size = int( np.random.uniform(min_resize_value, max_resize_value) + 0.5) else: random_size = cfg.AUG.INF_RESIZE_VALUE value = max(img.shape[0], img.shape[1]) scale = float(random_size) / float(value) img = cv2.resize(img, (0, 0), fx=scale, fy=scale, interpolation=cv2.INTER_LINEAR) if grt is not None: grt = cv2.resize( grt, (0, 0), fx=scale, fy=scale, interpolation=cv2.INTER_NEAREST) else: raise Exception("Unexpect data augmention method: {}".format( cfg.AUG.AUG_METHOD)) return img, grt def get_random_scale(min_scale_factor, max_scale_factor, step_size): """ 在一定范围内得到随机值，范围为min_scale_factor到max_scale_factor，间隔为step_size Args： min_scale_factor(float): 随机尺度下限，大于0 max_scale_factor(float): 随机尺度上限，不小于下限值 step_size(float): 尺度间隔，非负, 等于为0时直接返回min_scale_factor到max_scale_factor范围内任一值 Returns： 随机尺度值 """ if min_scale_factor < 0 or min_scale_factor > max_scale_factor: raise ValueError('Unexpected value of min_scale_factor.') if min_scale_factor == max_scale_factor: return min_scale_factor if step_size == 0: return np.random.uniform(min_scale_factor, max_scale_factor) num_steps = int((max_scale_factor - min_scale_factor) / step_size + 1) scale_factors = np.linspace(min_scale_factor, max_scale_factor, num_steps).tolist() np.random.shuffle(scale_factors) return scale_factors[0] def randomly_scale_image_and_label(image, label=None, scale=1.0): """ 按比例resize图像和标签图, 如果scale为1，返回原图 Args： image(numpy.ndarray): 输入图像 label(numpy.ndarray): 标签图，默认None sclae(float): 图片resize的比例，非负，默认1.0 Returns： resize后的图像和标签图 """ if scale == 1.0: return image, label height = image.shape[0] width = image.shape[1] new_height = int(height * scale + 0.5) new_width = int(width * scale + 0.5) new_image = cv2.resize( image, (new_width, new_height), interpolation=cv2.INTER_LINEAR) if label is not None: height = label.shape[0] width = label.shape[1] new_height = int(height * scale + 0.5) new_width = int(width * scale + 0.5) new_label = cv2.resize( label, (new_width, new_height), interpolation=cv2.INTER_NEAREST) return new_image, new_label def random_rotation(crop_img, crop_seg, rich_crop_max_rotation, mean_value): """ 随机旋转图像和标签图 Args： crop_img(numpy.ndarray): 输入图像 crop_seg(numpy.ndarray): 标签图 rich_crop_max_rotation(int)：旋转最大角度，0-90 mean_value(list)：均值, 对图片旋转产生的多余区域使用均值填充 Returns： 旋转后的图像和标签图 """ ignore_index = cfg.DATASET.IGNORE_INDEX if rich_crop_max_rotation > 0: (h, w) = crop_img.shape[:2] do_rotation = np.random.uniform(-rich_crop_max_rotation, rich_crop_max_rotation) pc = (w // 2, h // 2) r = cv2.getRotationMatrix2D(pc, do_rotation, 1.0) cos = np.abs(r[0, 0]) sin = np.abs(r[0, 1]) nw = int((h * sin) + (w * cos)) nh = int((h * cos) + (w * sin)) (cx, cy) = pc r[0, 2] += (nw / 2) - cx r[1, 2] += (nh / 2) - cy dsize = (nw, nh) crop_img = cv2.warpAffine( crop_img, r, dsize=dsize, flags=cv2.INTER_LINEAR, borderMode=cv2.BORDER_CONSTANT, borderValue=mean_value) crop_seg = cv2.warpAffine( crop_seg, r, dsize=dsize, flags=cv2.INTER_NEAREST, borderMode=cv2.BORDER_CONSTANT, borderValue=(ignore_index, ignore_index, ignore_index)) return crop_img, crop_seg def rand_scale_aspect(crop_img, crop_seg, rich_crop_min_scale=0, rich_crop_aspect_ratio=0): """ 从输入图像和标签图像中裁取随机宽高比的图像，并reszie回原始尺寸 Args: crop_img(numpy.ndarray): 输入图像 crop_seg(numpy.ndarray): 标签图像 rich_crop_min_scale(float)：裁取图像占原始图像的面积比，0-1，默认0返回原图 rich_crop_aspect_ratio(float): 裁取图像的宽高比范围，非负，默认0返回原图 Returns: 裁剪并resize回原始尺寸的图像和标签图像 """ if rich_crop_min_scale == 0 or rich_crop_aspect_ratio == 0: return crop_img, crop_seg else: img_height = crop_img.shape[0] img_width = crop_img.shape[1] for i in range(0, 10): area = img_height * img_width target_area = area * np.random.uniform(rich_crop_min_scale, 1.0) aspectRatio = np.random.uniform(rich_crop_aspect_ratio, 1.0 / rich_crop_aspect_ratio) dw = int(np.sqrt(target_area * 1.0 * aspectRatio)) dh = int(np.sqrt(target_area * 1.0 / aspectRatio)) if (np.random.randint(10) < 5): tmp = dw dw = dh dh = tmp if (dh < img_height and dw < img_width): h1 = np.random.randint(0, img_height - dh) w1 = np.random.randint(0, img_width - dw) crop_img = crop_img[h1:(h1 + dh), w1:(w1 + dw), :] crop_seg = crop_seg[h1:(h1 + dh), w1:(w1 + dw)] crop_img = cv2.resize( crop_img, (img_width, img_height), interpolation=cv2.INTER_LINEAR) crop_seg = cv2.resize( crop_seg, (img_width, img_height), interpolation=cv2.INTER_NEAREST) break return crop_img, crop_seg def saturation_jitter(cv_img, jitter_range): """ 调节图像饱和度 Args: cv_img(numpy.ndarray): 输入图像 jitter_range(float): 调节程度，0-1 Returns: 饱和度调整后的图像 """ greyMat = cv2.cvtColor(cv_img, cv2.COLOR_BGR2GRAY) greyMat = greyMat[:, :, None] * np.ones(3, dtype=int)[None, None, :] cv_img = cv_img.astype(np.float32) cv_img = cv_img * (1 - jitter_range) + jitter_range * greyMat cv_img = np.where(cv_img > 255, 255, cv_img) cv_img = cv_img.astype(np.uint8) return cv_img def brightness_jitter(cv_img, jitter_range): """ 调节图像亮度 Args: cv_img(numpy.ndarray): 输入图像 jitter_range(float): 调节程度，0-1 Returns: 亮度调整后的图像 """ cv_img = cv_img.astype(np.float32) cv_img = cv_img * (1.0 - jitter_range) cv_img = np.where(cv_img > 255, 255, cv_img) cv_img = cv_img.astype(np.uint8) return cv_img def contrast_jitter(cv_img, jitter_range): """ 调节图像对比度 Args: cv_img(numpy.ndarray): 输入图像 jitter_range(float): 调节程度，0-1 Returns: 对比度调整后的图像 """ greyMat = cv2.cvtColor(cv_img, cv2.COLOR_BGR2GRAY) mean = np.mean(greyMat) cv_img = cv_img.astype(np.float32) cv_img = cv_img * (1 - jitter_range) + jitter_range * mean cv_img = np.where(cv_img > 255, 255, cv_img) cv_img = cv_img.astype(np.uint8) return cv_img def random_jitter(cv_img, saturation_range, brightness_range, contrast_range): """ 图像亮度、饱和度、对比度调节，在调整范围内随机获得调节比例，并随机顺序叠加三种效果 Args: cv_img(numpy.ndarray): 输入图像 saturation_range(float): 饱和对调节范围，0-1 brightness_range(float): 亮度调节范围，0-1 contrast_range(float): 对比度调节范围，0-1 Returns: 亮度、饱和度、对比度调整后图像 """ saturation_ratio = np.random.uniform(-saturation_range, saturation_range) brightness_ratio = np.random.uniform(-brightness_range, brightness_range) contrast_ratio = np.random.uniform(-contrast_range, contrast_range) order = [1, 2, 3] np.random.shuffle(order) for i in range(3): if order[i] == 0: cv_img = saturation_jitter(cv_img, saturation_ratio) if order[i] == 1: cv_img = brightness_jitter(cv_img, brightness_ratio) if order[i] == 2: cv_img = contrast_jitter(cv_img, contrast_ratio) return cv_img def hsv_color_jitter(crop_img, brightness_jitter_ratio=0, saturation_jitter_ratio=0, contrast_jitter_ratio=0): """ 图像亮度、饱和度、对比度调节 Args: crop_img(numpy.ndarray): 输入图像 brightness_jitter_ratio(float): 亮度调节度最大值，1-0，默认0 saturation_jitter_ratio(float): 饱和度调节度最大值，1-0，默认0 contrast_jitter_ratio(float): 对比度调节度最大值，1-0，默认0 Returns： 亮度、饱和度、对比度调节后图像 """ if brightness_jitter_ratio > 0 or \ saturation_jitter_ratio > 0 or \ contrast_jitter_ratio > 0: crop_img = random_jitter(crop_img, saturation_jitter_ratio, brightness_jitter_ratio, contrast_jitter_ratio) return crop_img def rand_crop(crop_img, crop_seg, mode=ModelPhase.TRAIN): """ 随机裁剪图片和标签图, 若crop尺寸大于原始尺寸，分别使用均值和ignore值填充再进行crop， crop尺寸与原始尺寸一致，返回原图，crop尺寸小于原始尺寸直接crop Args: crop_img(numpy.ndarray): 输入图像 crop_seg(numpy.ndarray): 标签图 mode(string): 模式, 默认训练模式，验证或预测、可视化模式时crop尺寸需大于原始图片尺寸 Returns： 裁剪后的图片和标签图 """ img_height = crop_img.shape[0] img_width = crop_img.shape[1] if ModelPhase.is_train(mode): crop_width = cfg.TRAIN_CROP_SIZE[0] crop_height = cfg.TRAIN_CROP_SIZE[1] else: crop_width = cfg.EVAL_CROP_SIZE[0] crop_height = cfg.EVAL_CROP_SIZE[1] if not ModelPhase.is_train(mode): if (crop_height < img_height or crop_width < img_width): raise Exception( "Crop size({},{}) must large than img size({},{}) when in EvalPhase." .format(crop_width, crop_height, img_width, img_height)) if img_height == crop_height and img_width == crop_width: return crop_img, crop_seg else: pad_height = max(crop_height - img_height, 0) pad_width = max(crop_width - img_width, 0) if (pad_height > 0 or pad_width > 0): crop_img = cv2.copyMakeBorder( crop_img, 0, pad_height, 0, pad_width, cv2.BORDER_CONSTANT, value=cfg.DATASET.PADDING_VALUE) if crop_seg is not None: crop_seg = cv2.copyMakeBorder( crop_seg, 0, pad_height, 0, pad_width, cv2.BORDER_CONSTANT, value=cfg.DATASET.IGNORE_INDEX) img_height = crop_img.shape[0] img_width = crop_img.shape[1] if crop_height > 0 and crop_width > 0: h_off = np.random.randint(img_height - crop_height + 1) w_off = np.random.randint(img_width - crop_width + 1) crop_img = crop_img[h_off:(crop_height + h_off), w_off:( w_off + crop_width), :] if crop_seg is not None: crop_seg = crop_seg[h_off:(crop_height + h_off), w_off:( w_off + crop_width)] return crop_img, crop_seg5. PaddleSeg 训练的数据增强pdseg/reader.pyclass SegDataset(object): ...... def load_image(self, line, src_dir, mode=ModelPhase.TRAIN): # original image cv2.imread flag setting cv2_imread_flag = cv2.IMREAD_COLOR if cfg.DATASET.IMAGE_TYPE == "rgba": # If use RBGA 4 channel ImageType, use IMREAD_UNCHANGED flags to # reserver alpha channel cv2_imread_flag = cv2.IMREAD_UNCHANGED parts = line.strip().split(cfg.DATASET.SEPARATOR) if len(parts) != 2: if mode == ModelPhase.TRAIN or mode == ModelPhase.EVAL: raise Exception("File list format incorrect! It should be" " image_name{}label_name\\n".format( cfg.DATASET.SEPARATOR)) img_name, grt_name = parts[0], None else: img_name, grt_name = parts[0], parts[1] img_path = os.path.join(src_dir, img_name) img = cv2_imread(img_path, cv2_imread_flag) if grt_name is not None: grt_path = os.path.join(src_dir, grt_name) grt = cv2_imread(grt_path, cv2.IMREAD_GRAYSCALE) else: grt = None if img is None: raise Exception( "Empty image, src_dir: {}, img: {} & lab: {}".format( src_dir, img_path, grt_path)) img_height = img.shape[0] img_width = img.shape[1] if grt is not None: grt_height = grt.shape[0] grt_width = grt.shape[1] if img_height != grt_height or img_width != grt_width: raise Exception( "source img and label img must has the same size") else: if mode == ModelPhase.TRAIN or mode == ModelPhase.EVAL: raise Exception( "Empty image, src_dir: {}, img: {} & lab: {}".format( src_dir, img_path, grt_path)) if len(img.shape) < 3: img = cv2.cvtColor(img, cv2.COLOR_GRAY2BGR) img_channels = img.shape[2] if img_channels < 3: raise Exception("PaddleSeg only supports gray, rgb or rgba image") if img_channels != cfg.DATASET.DATA_DIM: raise Exception( "Input image channel({}) is not match cfg.DATASET.DATA_DIM({}), img_name={}" .format(img_channels, cfg.DATASET.DATADIM, img_name)) if img_channels != len(cfg.MEAN): raise Exception( "img name {}, img chns {} mean size {}, size unequal".format( img_name, img_channels, len(cfg.MEAN))) if img_channels != len(cfg.STD): raise Exception( "img name {}, img chns {} std size {}, size unequal".format( img_name, img_channels, len(cfg.STD))) return img, grt, img_name, grt_name def normalize_image(self, img): """ 像素归一化后减均值除方差 """ img = img.transpose((2, 0, 1)).astype('float32') / 255.0 img_mean = np.array(cfg.MEAN).reshape((len(cfg.MEAN), 1, 1)) img_std = np.array(cfg.STD).reshape((len(cfg.STD), 1, 1)) img -= img_mean img /= img_std return img def process_image(self, line, data_dir, mode): """ process_image """ img, grt, img_name, grt_name = self.load_image( line, data_dir, mode=mode) if mode == ModelPhase.TRAIN: #模型训练 img, grt = aug.resize(img, grt, mode) if cfg.AUG.RICH_CROP.ENABLE: if cfg.AUG.RICH_CROP.BLUR: if cfg.AUG.RICH_CROP.BLUR_RATIO <= 0: n = 0 elif cfg.AUG.RICH_CROP.BLUR_RATIO >= 1: n = 1 else: n = int(1.0 / cfg.AUG.RICH_CROP.BLUR_RATIO) if n > 0: if np.random.randint(0, n) == 0: radius = np.random.randint(3, 10) if radius % 2 != 1: radius = radius + 1 if radius > 9: radius = 9 img = cv2.GaussianBlur(img, (radius, radius), 0, 0) img, grt = aug.random_rotation( img, grt, rich_crop_max_rotation=cfg.AUG.RICH_CROP.MAX_ROTATION, mean_value=cfg.DATASET.PADDING_VALUE) img, grt = aug.rand_scale_aspect( img, grt, rich_crop_min_scale=cfg.AUG.RICH_CROP.MIN_AREA_RATIO, rich_crop_aspect_ratio=cfg.AUG.RICH_CROP.ASPECT_RATIO) img = aug.hsv_color_jitter( img, brightness_jitter_ratio=cfg.AUG.RICH_CROP. BRIGHTNESS_JITTER_RATIO, saturation_jitter_ratio=cfg.AUG.RICH_CROP. SATURATION_JITTER_RATIO, contrast_jitter_ratio=cfg.AUG.RICH_CROP. CONTRAST_JITTER_RATIO) if cfg.AUG.FLIP: if cfg.AUG.FLIP_RATIO <= 0: n = 0 elif cfg.AUG.FLIP_RATIO >= 1: n = 1 else: n = int(1.0 / cfg.AUG.FLIP_RATIO) if n > 0: if np.random.randint(0, n) == 0: img = img[::-1, :, :] grt = grt[::-1, :] if cfg.AUG.MIRROR: if np.random.randint(0, 2) == 1: img = img[:, ::-1, :] grt = grt[:, ::-1] img, grt = aug.rand_crop(img, grt, mode=mode) elif ModelPhase.is_eval(mode):#模型评估 img, grt = aug.resize(img, grt, mode=mode) img, grt = aug.rand_crop(img, grt, mode=mode) elif ModelPhase.is_visual(mode):#模型可视化 org_shape = [img.shape[0], img.shape[1]] img, grt = aug.resize(img, grt, mode=mode) valid_shape = [img.shape[0], img.shape[1]] img, grt = aug.rand_crop(img, grt, mode=mode) else: raise ValueError("Dataset mode={} Error!".format(mode)) # Normalize image img = self.normalize_image(img) if ModelPhase.is_train(mode) or ModelPhase.is_eval(mode): grt = np.expand_dims(np.array(grt).astype('int32'), axis=0) ignore = (grt != cfg.DATASET.IGNORE_INDEX).astype('int32') if ModelPhase.is_train(mode): return (img, grt, ignore) elif ModelPhase.is_eval(mode): return (img, grt, ignore) elif ModelPhase.is_visual(mode): return (img, grt, img_name, valid_shape, org_shape)
PyTorch 的 Vision 模块提供了图像变换的很多函数. vision/torchvision/transforms/functional.py# 必要的 python 模块 from __future__ import division import torch import sys import math from PIL import Image, ImageOps, ImageEnhance, PILLOW_VERSION try: import accimage except ImportError: accimage = None import numpy as np import numbers import collections import warnings import matplotlib as plt if sys.version_info < (3, 3): Sequence = collections.Sequence Iterable = collections.Iterable else: Sequence = collections.abc.Sequence Iterable = collections.abc.Iterable以下图为例：img_file = "test.jpe" img = Image.open(img_file) width, height = img.size #(750, 815) img.show()1. PyTorch 图像变换函数1.1. 判断图像数据类型# 图像格式检查，如，pil, tensor, numpy def _is_pil_image(img): if accimage is not None: return isinstance(img, (Image.Image, accimage.Image)) else: return isinstance(img, Image.Image) def _is_tensor_image(img): return torch.is_tensor(img) and img.ndimension() == 3 def _is_numpy_image(img): return isinstance(img, np.ndarray) and (img.ndim in {2, 3})如：_is_pil_image(img) # True _is_tensor_image(img) # False _is_numpy_image(img) # False _is_numpy_image(np.array(img)) # True1.2. to_tensor(pic)将 PIL Image 或 nupy.ndarray 转换为 tensor.def to_tensor(pic): """ Args: pic (PIL Image or numpy.ndarray): Image to be converted to tensor. Returns: Tensor: Converted image. """ if not(_is_pil_image(pic) or _is_numpy_image(pic)): raise TypeError('pic should be PIL Image or ndarray. Got {}'.format(type(pic))) if isinstance(pic, np.ndarray): # handle numpy array img = torch.from_numpy(pic.transpose((2, 0, 1))) # backward compatibility if isinstance(img, torch.ByteTensor): return img.float().div(255) else: return img if accimage is not None and isinstance(pic, accimage.Image): nppic = np.zeros([pic.channels, pic.height, pic.width], dtype=np.float32) pic.copyto(nppic) return torch.from_numpy(nppic) # handle PIL Image if pic.mode == 'I': img = torch.from_numpy(np.array(pic, np.int32, copy=False)) elif pic.mode == 'I;16': img = torch.from_numpy(np.array(pic, np.int16, copy=False)) elif pic.mode == 'F': img = torch.from_numpy(np.array(pic, np.float32, copy=False)) elif pic.mode == '1': img = 255 * torch.from_numpy(np.array(pic, np.uint8, copy=False)) else: img = torch.ByteTensor(torch.ByteStorage.from_buffer(pic.tobytes())) # PIL image mode: L, P, I, F, RGB, YCbCr, RGBA, CMYK if pic.mode == 'YCbCr': nchannel = 3 elif pic.mode == 'I;16': nchannel = 1 else: nchannel = len(pic.mode) img = img.view(pic.size[1], pic.size[0], nchannel) # put it from HWC to CHW format # yikes, this transpose takes 80% of the loading time/CPU img = img.transpose(0, 1).transpose(0, 2).contiguous() if isinstance(img, torch.ByteTensor): return img.float().div(255) else: return img如：img_tensor = to_tensor(img) _is_tensor_image(img_tensor) # True1.3. to_pil_image(pic, mode=None)将 tensro 或 ndarray 转换为 PIL Image.def to_pil_image(pic, mode=None): """ Args: pic (Tensor or numpy.ndarray): Image to be converted to PIL Image. mode (`PIL.Image mode`_): color space and pixel depth of input data (optional). .. _PIL.Image mode: https://pillow.readthedocs.io/en/latest/handbook/concepts.html#concept-modes Returns: PIL Image: Image converted to PIL Image. """ if not(isinstance(pic, torch.Tensor) or isinstance(pic, np.ndarray)): raise TypeError('pic should be Tensor or ndarray. Got {}.'.format(type(pic))) elif isinstance(pic, torch.Tensor): if pic.ndimension() not in {2, 3}: raise ValueError('pic should be 2/3 dimensional. Got {} '\ 'dimensions.'.format(pic.ndimension())) elif pic.ndimension() == 2: # if 2D image, add channel dimension (CHW) pic.unsqueeze_(0) elif isinstance(pic, np.ndarray): if pic.ndim not in {2, 3}: raise ValueError('pic should be 2/3 dimensional. Got {} '\ 'dimensions.'.format(pic.ndim)) elif pic.ndim == 2: # if 2D image, add channel dimension (HWC) pic = np.expand_dims(pic, 2) npimg = pic if isinstance(pic, torch.FloatTensor): pic = pic.mul(255).byte() if isinstance(pic, torch.Tensor): npimg = np.transpose(pic.numpy(), (1, 2, 0)) if not isinstance(npimg, np.ndarray): raise TypeError('Input pic must be a torch.Tensor or NumPy ndarray, ' + 'not {}'.format(type(npimg))) if npimg.shape[2] == 1: expected_mode = None npimg = npimg[:, :, 0] if npimg.dtype == np.uint8: expected_mode = 'L' elif npimg.dtype == np.int16: expected_mode = 'I;16' elif npimg.dtype == np.int32: expected_mode = 'I' elif npimg.dtype == np.float32: expected_mode = 'F' if mode is not None and mode != expected_mode: raise ValueError("Incorrect mode ({}) supplied for input type {}. Should be {}" .format(mode, np.dtype, expected_mode)) mode = expected_mode elif npimg.shape[2] == 4: permitted_4_channel_modes = ['RGBA', 'CMYK'] if mode is not None and mode not in permitted_4_channel_modes: raise ValueError("Only modes {} are supported for 4D inputs".format(permitted_4_channel_modes)) if mode is None and npimg.dtype == np.uint8: mode = 'RGBA' else: permitted_3_channel_modes = ['RGB', 'YCbCr', 'HSV'] if mode is not None and mode not in permitted_3_channel_modes: raise ValueError("Only modes {} are supported for 3D inputs".format(permitted_3_channel_modes)) if mode is None and npimg.dtype == np.uint8: mode = 'RGB' if mode is None: raise TypeError('Input type {} is not supported'.format(npimg.dtype)) return Image.fromarray(npimg, mode=mode)如：img_tensor = to_tensor(img) _is_tensor_image(img_tensor) # True img_pil = to_pil_image(img_tensor) _is_pil_image(img_pil) # True img_numpy = np.array(img) _is_numpy_image(img_numpy) # True img_pil = to_pil_image(img_numpy) _is_pil_image(img_pil) # True1.4. normalize(tensor, mean, std)归一化 tensor的图像. in-place计算.def normalize(tensor, mean, std): """ Args: tensor (Tensor): Tensor image of size (C, H, W) to be normalized. mean (sequence): Sequence of means for each channel. std (sequence): Sequence of standard deviations for each channely. Returns: Tensor: Normalized Tensor image. """ if not _is_tensor_image(tensor): raise TypeError('tensor is not a torch image.') # This is faster than using broadcasting, don't change without benchmarking for t, m, s in zip(tensor, mean, std): t.sub_(m).div_(s) return tensor如：mean = [0.485, 0.456, 0.406] std = [0.229, 0.224, 0.225] img_normalize = normalize(img_tensor, mean, std) # vis ax1 = plt.subplot(1, 2, 1) ax1.imshow(img) ax1.axis("off") ax1.set_title("orig img") ax2 = plt.subplot(1, 2, 2) ax2.imshow(to_pil_image(img_normalize)) ax2.axis("off") ax2.set_title("normalize img") plt.show()输出：1.5. resize(img, size, interpolation=Image.BILINEAR)对输入的 PIL Image 进行 resize 到给定尺寸.参数 size 为调整后的尺寸. 如果 size 是数组(h, w)，则直接调整到该 (h, w) 尺寸.如果 size 是一个 int 值，则调整后图像的最短边是该值，且保持固定的长宽比. 如，如果 height > width，则 resize 后，图像的尺寸为： $\left(\text{size} \times \frac{\text{height}}{\text{width}}, \text{size}\right)$ .def resize(img, size, interpolation=Image.BILINEAR): """ Args: img (PIL Image): Image to be resized. size (sequence or int): Desired output size. interpolation (int, optional): Desired interpolation. Default is ``PIL.Image.BILINEAR`` Returns: PIL Image: Resized image. """ if not _is_pil_image(img): raise TypeError('img should be PIL Image. Got {}'.format(type(img))) if not (isinstance(size, int) or (isinstance(size, Iterable) and len(size) == 2)): raise TypeError('Got inappropriate size arg: {}'.format(size)) if isinstance(size, int): w, h = img.size if (w <= h and w == size) or (h <= w and h == size): return img if w < h: ow = size oh = int(size * h / w) return img.resize((ow, oh), interpolation) else: oh = size ow = int(size * w / h) return img.resize((ow, oh), interpolation) else: return img.resize(size[::-1], interpolation)如：img_resize_256x256 = resize(img, (256, 256)) # (256, 256) img_resize_256 = resize(img, 256) # (256, 278) # vis ax1 = plt.subplot(1, 3, 1) ax1.imshow(img) ax1.axis("off") ax1.set_title("orig img") ax2 = plt.subplot(1, 3, 2) ax2.imshow(img_resize_256x256) ax2.axis("off") ax2.set_title("resize_256x256 img") ax3 = plt.subplot(1, 3, 3) ax3.imshow(img_resize_256) ax3.axis("off") ax3.set_title("resize_256 img") plt.show()1.6. pad(img, padding, fill=0, padding_mode='constant')根据指定的 padding 模式和填充值，对给定的 PIL Image 的所有边进行 pad 处理.参数 padding - int 或 tuple 形式.padding: 如果是 int 值 ，则对所有的边都 padding 该 int 值. 如果是长度为 2 的tuple，则对 left/right 和 top/bottom 分别进行 padding. 如果是长度为 4 的 tuple，则对 left，top，right， bottom 边分别进行 padding.参数 fill - 像素填充值，默认为 0. 如果值是长度为 3 的 tuple，则分别对 R，G，B 通道进行填充. 仅用于当 padding_mode='constant' 的情况.参数 padding_mode - 填充的类型，可选：constant，edge，reflect，symmetric. 默认为 constant. 填充常数值.constant - padding 填充常数值 fill. edge - padding 图像边缘的最后一个值. reflect - padding 图像的反射(reflection)值，（不对图像边缘的最后一个像素值进行重复） 如，[1, 2, 3, 4] 在 reflect 模式下在 两边 padding 2 个元素值，会得到： [3, 2, 1, 2, 3, 4, 3, 2] symmetric - padding 图像的反射(reflection)值，(对图像边缘的最后一个像素值进行重复). 如，[1, 2, 3, 4] 在 symmetric 模式下在 两边 padding 2 个元素值，会得到： [2, 1, 1, 2, 3, 4, 4, 3]def pad(img, padding, fill=0, padding_mode='constant'): """ Args: img (PIL Image): Image to be padded. padding (int or tuple): Padding on each border. fill: Pixel fill value for constant fill. Default is 0. padding_mode: Type of padding. Should be: constant, edge, reflect or symmetric. Default is constant. Returns: PIL Image: Padded image. """ if not _is_pil_image(img): raise TypeError('img should be PIL Image. Got {}'.format(type(img))) if not isinstance(padding, (numbers.Number, tuple)): raise TypeError('Got inappropriate padding arg') if not isinstance(fill, (numbers.Number, str, tuple)): raise TypeError('Got inappropriate fill arg') if not isinstance(padding_mode, str): raise TypeError('Got inappropriate padding_mode arg') if isinstance(padding, Sequence) and len(padding) not in [2, 4]: raise ValueError("Padding must be an int or a 2, or 4 element tuple, not a " + "{} element tuple".format(len(padding))) assert padding_mode in ['constant', 'edge', 'reflect', 'symmetric'], \ 'Padding mode should be either constant, edge, reflect or symmetric' if padding_mode == 'constant': if img.mode == 'P': palette = img.getpalette() image = ImageOps.expand(img, border=padding, fill=fill) image.putpalette(palette) return image return ImageOps.expand(img, border=padding, fill=fill) else: if isinstance(padding, int): pad_left = pad_right = pad_top = pad_bottom = padding if isinstance(padding, Sequence) and len(padding) == 2: pad_left = pad_right = padding[0] pad_top = pad_bottom = padding[1] if isinstance(padding, Sequence) and len(padding) == 4: pad_left = padding[0] pad_top = padding[1] pad_right = padding[2] pad_bottom = padding[3] if img.mode == 'P': palette = img.getpalette() img = np.asarray(img) img = np.pad(img, ((pad_top, pad_bottom), (pad_left, pad_right)), padding_mode) img = Image.fromarray(img) img.putpalette(palette) return img img = np.asarray(img) # RGB image if len(img.shape) == 3: img = np.pad(img, ((pad_top, pad_bottom), (pad_left, pad_right), (0, 0)), padding_mode) # Grayscale image if len(img.shape) == 2: img = np.pad(img, ((pad_top, pad_bottom), (pad_left, pad_right)), padding_mode) return Image.fromarray(img)如：img_padding = pad(img, (10, 20, 30 ,40), fill=128) # (750, 815) -> (790, 875) # vis ax1 = plt.subplot(1, 2, 1) ax1.imshow(img) ax1.axis("off") ax1.set_title("orig img") ax2 = plt.subplot(1, 2, 2) ax2.imshow(img_padding) ax2.axis("off") ax2.set_title("padding img") plt.show()1.7. crop(img, i, j, h, w)裁剪给定的 PIL Image.def crop(img, i, j, h, w): """ Args: img (PIL Image): Image to be cropped. i: Upper pixel coordinate. j: Left pixel coordinate. h: Height of the cropped image. w: Width of the cropped image. Returns: PIL Image: Cropped image. """ if not _is_pil_image(img): raise TypeError('img should be PIL Image. Got {}'.format(type(img))) return img.crop((j, i, j + w, i + h))如：img_crop = crop(img, 100, 100, 500, 500) # (750, 815) -> (500, 500) ax1 = plt.subplot(1, 2, 1) ax1.imshow(img) ax1.axis("off") ax1.set_title("orig img") ax2 = plt.subplot(1, 2, 2) ax2.imshow(img_crop) ax2.axis("off") ax2.set_title("crop img") plt.show()1.8. center_crop(img, output_size)def center_crop(img, output_size): if isinstance(output_size, numbers.Number): output_size = (int(output_size), int(output_size)) w, h = img.size th, tw = output_size i = int(round((h - th) / 2.)) j = int(round((w - tw) / 2.)) return crop(img, i, j, th, tw)如：img_centercrop = center_crop(img, (256, 256)) # (750, 815) -> (256, 256) ax1 = plt.subplot(1, 2, 1) ax1.imshow(img) ax1.axis("off") ax1.set_title("orig img") ax2 = plt.subplot(1, 2, 2) ax2.imshow(img_centercrop) ax2.axis("off") ax2.set_title("centercrop img") plt.show()1.9. resized_crop(img, i, j, h, w, size, interpolation=Image.BILINEAR)对给定 PIL Image 进行裁剪，并 resize 到特定尺寸.def resized_crop(img, i, j, h, w, size, interpolation=Image.BILINEAR): """ Args: img (PIL Image): Image to be cropped. i: Upper pixel coordinate. j: Left pixel coordinate. h: Height of the cropped image. w: Width of the cropped image. size (sequence or int): Desired output size. Same semantics as ``resize``. interpolation (int, optional): Desired interpolation. Default is ``PIL.Image.BILINEAR``. Returns: PIL Image: Cropped image. """ assert _is_pil_image(img), 'img should be PIL Image' img = crop(img, i, j, h, w) img = resize(img, size, interpolation) return img如，img_resizedcrop = resized_crop(img, 100, 100, 500, 500, (256, 256)) # (750, 815) -> (500, 500) -> (256, 256) ax1 = plt.subplot(1, 2, 1) ax1.imshow(img) ax1.axis("off") ax1.set_title("orig img") ax2 = plt.subplot(1, 2, 2) ax2.imshow(img_resizedcrop) ax2.axis("off") ax2.set_title("resizedcrop img") plt.show()1.10. hflip(img)水平翻转(Horizontally flip)给定的 PIL Image.def hflip(img): """ Args: img (PIL Image): Image to be flipped. Returns: PIL Image: Horizontall flipped image. """ if not _is_pil_image(img): raise TypeError('img should be PIL Image. Got {}'.format(type(img))) return img.transpose(Image.FLIP_LEFT_RIGHT)1.11. vflip(img)垂直翻转(Vertically flip)给定的 PIL Image.def vflip(img): """ Args: img (PIL Image): Image to be flipped. Returns: PIL Image: Vertically flipped image. """ if not _is_pil_image(img): raise TypeError('img should be PIL Image. Got {}'.format(type(img))) return img.transpose(Image.FLIP_TOP_BOTTOM)如：img_hflip = hflip(img) img_vflip = vflip(img) ax1 = plt.subplot(1, 3, 1) ax1.imshow(img) ax1.axis("off") ax1.set_title("orig img") ax2 = plt.subplot(1, 3, 2) ax2.imshow(img_hflip) ax2.axis("off") ax2.set_title("hflip img") ax3 = plt.subplot(1, 3, 3) ax3.imshow(img_vflip) ax3.axis("off") ax3.set_title("vflip img") plt.show()1.12. five_crop(img, size)Crop the given PIL Image into four corners and the central crop.从给定 PIL Image 的四个角和中间裁剪出五个子图像.def five_crop(img, size): """ Args: size (sequence or int): Desired output size of the crop. If size is an int instead of sequence like (h, w), a square crop (size, size) is made. Returns: tuple: tuple (tl, tr, bl, br, center) Corresponding top left, top right, bottom left, bottom right and center crop. """ if isinstance(size, numbers.Number): size = (int(size), int(size)) else: assert len(size) == 2, "Please provide only two dimensions (h, w) for size." w, h = img.size crop_h, crop_w = size if crop_w > w or crop_h > h: raise ValueError("Requested crop size {} is bigger than input size {}".format(size, (h, w))) tl = img.crop((0, 0, crop_w, crop_h)) tr = img.crop((w - crop_w, 0, w, crop_h)) bl = img.crop((0, h - crop_h, crop_w, h)) br = img.crop((w - crop_w, h - crop_h, w, h)) center = center_crop(img, (crop_h, crop_w)) return (tl, tr, bl, br, center)如：img_tl, img_tr, img_bl, img_br, img_center = five_crop(img, (400, 400)) ax1 = plt.subplot(2, 3, 1) ax1.imshow(img) ax1.axis("off") ax1.set_title("orig img") ax2 = plt.subplot(2, 3, 2) ax2.imshow(img_tl) ax2.axis("off") ax2.set_title("tl img") ax3 = plt.subplot(2, 3, 3) ax3.imshow(img_tr) ax3.axis("off") ax3.set_title("tr img") ax4 = plt.subplot(2, 3, 4) ax4.imshow(img_bl) ax4.axis("off") ax4.set_title("bl img") ax5 = plt.subplot(2, 3, 5) ax5.imshow(img_br) ax5.axis("off") ax5.set_title("br img") ax6 = plt.subplot(2, 3, 6) ax6.imshow(img_center) ax6.axis("off") ax6.set_title("center img") plt.show()1.13. ten_crop(img, size, vertical_flip=False)将给定 PIL Image 裁剪出的四个角和中间部分的五个子图像，每个子图像进行翻转处理. 默认时水平翻转.def ten_crop(img, size, vertical_flip=False): """ Args: size (sequence or int): Desired output size of the crop. If size is an int instead of sequence like (h, w), a square crop (size, size) is made. vertical_flip (bool): Use vertical flipping instead of horizontal Returns: tuple: tuple (tl, tr, bl, br, center, tl_flip, tr_flip, bl_flip, br_flip, center_flip) Corresponding top left, top right, bottom left, bottom right and center crop and same for the flipped image. """ if isinstance(size, numbers.Number): size = (int(size), int(size)) else: assert len(size) == 2, "Please provide only two dimensions (h, w) for size." first_five = five_crop(img, size) if vertical_flip: img = vflip(img) else: img = hflip(img) second_five = five_crop(img, size) return first_five + second_five1.14. adjust_brightness(img, brightness_factor)调整图像亮度.def adjust_brightness(img, brightness_factor): """ Args: img (PIL Image): PIL Image to be adjusted. brightness_factor (float): How much to adjust the brightness. Can be any non negative number. 0 gives a black image, 1 gives the original image, 2 increases the brightness by a factor of 2. Returns: PIL Image: Brightness adjusted image. """ if not _is_pil_image(img): raise TypeError('img should be PIL Image. Got {}'.format(type(img))) enhancer = ImageEnhance.Brightness(img) img = enhancer.enhance(brightness_factor) return img如：img_adjust_brightness = adjust_brightness(img, 2.5) # vis ax1 = plt.subplot(1, 2, 1) ax1.imshow(img) ax1.axis("off") ax1.set_title("orig img") ax2 = plt.subplot(1, 2, 2) ax2.imshow(img_adjust_brightness) ax2.axis("off") ax2.set_title("adjust_brightness img") plt.show()1.15. adjust_contrast(img, contrast_factor)调整对比度.def adjust_contrast(img, contrast_factor): """ Args: img (PIL Image): PIL Image to be adjusted. contrast_factor (float): How much to adjust the contrast. Can be any non negative number. 0 gives a solid gray image, 1 gives the original image, 2 increases the contrast by a factor of 2. Returns: PIL Image: Contrast adjusted image. """ if not _is_pil_image(img): raise TypeError('img should be PIL Image. Got {}'.format(type(img))) enhancer = ImageEnhance.Contrast(img) img = enhancer.enhance(contrast_factor) return img如：img_adjust_contrast = adjust_contrast(img, 2.5) # vis ax1 = plt.subplot(1, 2, 1) ax1.imshow(img) ax1.axis("off") ax1.set_title("orig img") ax2 = plt.subplot(1, 2, 2) ax2.imshow(img_adjust_contrast) ax2.axis("off") ax2.set_title("adjust_contrast img") plt.show()1.16. adjust_saturation(img, saturation_factor)调整颜色饱和度.def adjust_saturation(img, saturation_factor): """ Args: img (PIL Image): PIL Image to be adjusted. saturation_factor (float): How much to adjust the saturation. 0 will give a black and white image, 1 will give the original image while 2 will enhance the saturation by a factor of 2. Returns: PIL Image: Saturation adjusted image. """ if not _is_pil_image(img): raise TypeError('img should be PIL Image. Got {}'.format(type(img))) enhancer = ImageEnhance.Color(img) img = enhancer.enhance(saturation_factor) return img 如：img_adjust_saturation = adjust_saturation(img, 2.5) # vis ax1 = plt.subplot(1, 2, 1) ax1.imshow(img) ax1.axis("off") ax1.set_title("orig img") ax2 = plt.subplot(1, 2, 2) ax2.imshow(img_adjust_saturation) ax2.axis("off") ax2.set_title("adjust_saturation img") plt.show()1.17. adjust_hue(img, hue_factor)调整图像 HUE.通过将图像转换为 HSV 空间，并周期地移动在 hue 通道(H) 的强度，以实现图像 hue 的调整. 最后，再将结果转换回原始的图像模式.参数 hue_factor - H 通道平移的因子，其值必须在区间 [-0.5, 0.5].def adjust_hue(img, hue_factor): """ Args: img (PIL Image): PIL Image to be adjusted. hue_factor (float): How much to shift the hue channel. Should be in [-0.5, 0.5]. 0.5 and -0.5 give complete reversal of hue channel in HSV space in positive and negative direction respectively. 0 means no shift. Therefore, both -0.5 and 0.5 will give an image with complementary colors while 0 gives the original image. Returns: PIL Image: Hue adjusted image. """ if not(-0.5 <= hue_factor <= 0.5): raise ValueError('hue_factor is not in [-0.5, 0.5].'.format(hue_factor)) if not _is_pil_image(img): raise TypeError('img should be PIL Image. Got {}'.format(type(img))) input_mode = img.mode if input_mode in {'L', '1', 'I', 'F'}: return img h, s, v = img.convert('HSV').split() np_h = np.array(h, dtype=np.uint8) # uint8 addition take cares of rotation across boundaries with np.errstate(over='ignore'): np_h += np.uint8(hue_factor * 255) h = Image.fromarray(np_h, 'L') img = Image.merge('HSV', (h, s, v)).convert(input_mode) return img如：img_adjust_hue = adjust_hue(img, 0.5) # vis ax1 = plt.subplot(1, 2, 1) ax1.imshow(img) ax1.axis("off") ax1.set_title("orig img") ax2 = plt.subplot(1, 2, 2) ax2.imshow(img_adjust_hue) ax2.axis("off") ax2.set_title("adjust_hue img") plt.show()1.18. adjust_gamma(img, gamma, gain=1)对图像进行伽马校正(gamma correction). 也被叫作 Power Law Transform.RGB 模式下强度调整公式：$I_{\text{out}} = 255 \times \text{gain} \times \left(\frac{I_{\text{in}}}{255}\right)^{\gamma}$def adjust_gamma(img, gamma, gain=1): """ Args: img (PIL Image): PIL Image to be adjusted. gamma (float): Non negative real number, 如公式中的 \gamma 值. gamma larger than 1 make the shadows darker, while gamma smaller than 1 make dark regions lighter. gain (float): The constant multiplier. """ if not _is_pil_image(img): raise TypeError('img should be PIL Image. Got {}'.format(type(img))) if gamma < 0: raise ValueError('Gamma should be a non-negative real number') input_mode = img.mode img = img.convert('RGB') gamma_map = [255 * gain * pow(ele / 255., gamma) for ele in range(256)] * 3 img = img.point(gamma_map) # use PIL's point-function to accelerate this part img = img.convert(input_mode) return img如：img_adjust_gamma = adjust_gamma(img, 0.5) # vis ax1 = plt.subplot(1, 2, 1) ax1.imshow(img) ax1.axis("off") ax1.set_title("orig img") ax2 = plt.subplot(1, 2, 2) ax2.imshow(img_adjust_gamma) ax2.axis("off") ax2.set_title("adjust_gamma img") plt.show()1.19. rotate(img, angle, resample=False, expand=False, center=None)旋转图像.参数 resample - 可选值：PIL.Image.NEAREST， PIL.Image.BILINEAR， PIL.Image.BICUBIC.​ 如果参数 reshape被忽略，或图像的模式是 1 或 P，则resample=PIL.Image.NEAREST.参数 expand - 如果 expand=True，则延展输出图像，以能包含旋转后的全部图像.​ 如果 expand=False 或被忽略，则保持输出图像与输入图像的尺寸一致.​ expand 假设旋转是以中心进行旋转，且没有平移.def rotate(img, angle, resample=False, expand=False, center=None): """ Args: img (PIL Image): PIL Image to be rotated. angle (float or int): In degrees degrees counter clockwise order. resample (``PIL.Image.NEAREST`` or ``PIL.Image.BILINEAR`` or ``PIL.Image.BICUBIC``, optional): expand (bool, optional): Optional expansion flag. center (2-tuple, optional): Optional center of rotation. Origin is the upper left corner. Default is the center of the image. """ if not _is_pil_image(img): raise TypeError('img should be PIL Image. Got {}'.format(type(img))) return img.rotate(angle, resample, expand, center)如：img_rotate = rotate(img, 60) # vis ax1 = plt.subplot(1, 2, 1) ax1.imshow(img) ax1.axis("off") ax1.set_title("orig img") ax2 = plt.subplot(1, 2, 2) ax2.imshow(img_rotate) ax2.axis("off") ax2.set_title("rotate img") plt.show()1.20. affine(img, angle, translate, scale, shear, resample=0, fillcolor=None)保持图像中心不变，进行仿射变换.(Apply affine transformation on the image keeping image center invariant)def _get_inverse_affine_matrix(center, angle, translate, scale, shear): # Helper method to compute inverse matrix for affine transformation # As it is explained in PIL.Image.rotate # We need compute INVERSE of affine transformation matrix: M = T * C * RSS * C^-1 # where T is translation matrix: [1, 0, tx | 0, 1, ty | 0, 0, 1] # C is translation matrix to keep center: [1, 0, cx | 0, 1, cy | 0, 0, 1] # RSS is rotation with scale and shear matrix # RSS(a, scale, shear) = [ cos(a)*scale -sin(a + shear)*scale 0] # [ sin(a)*scale cos(a + shear)*scale 0] # [ 0 0 1] # Thus, the inverse is M^-1 = C * RSS^-1 * C^-1 * T^-1 angle = math.radians(angle) shear = math.radians(shear) scale = 1.0 / scale # Inverted rotation matrix with scale and shear d = math.cos(angle + shear) * math.cos(angle) + math.sin(angle + shear) * math.sin(angle) matrix = [ math.cos(angle + shear), math.sin(angle + shear), 0, -math.sin(angle), math.cos(angle), 0 ] matrix = [scale / d * m for m in matrix] # Apply inverse of translation and of center translation: RSS^-1 * C^-1 * T^-1 matrix[2] += matrix[0] * (-center[0] - translate[0]) + matrix[1] * (-center[1] - translate[1]) matrix[5] += matrix[3] * (-center[0] - translate[0]) + matrix[4] * (-center[1] - translate[1]) # Apply center translation: C * RSS^-1 * C^-1 * T^-1 matrix[2] += center[0] matrix[5] += center[1] return matrix def affine(img, angle, translate, scale, shear, resample=0, fillcolor=None): """ Args: img (PIL Image): PIL Image to be rotated. angle (float or int): rotation angle in degrees between -180 and 180, clockwise direction. translate (list or tuple of integers): horizontal and vertical translations (post-rotation translation) scale (float): overall scale shear (float): shear angle value in degrees between -180 to 180, clockwise direction. resample (``PIL.Image.NEAREST`` or ``PIL.Image.BILINEAR`` or ``PIL.Image.BICUBIC``, optional): fillcolor (int): Optional fill color for the area outside the transform in the output image. (Pillow>=5.0.0) """ if not _is_pil_image(img): raise TypeError('img should be PIL Image. Got {}'.format(type(img))) assert isinstance(translate, (tuple, list)) and len(translate) == 2, \ "Argument translate should be a list or tuple of length 2" assert scale > 0.0, "Argument scale should be positive" output_size = img.size center = (img.size[0] * 0.5 + 0.5, img.size[1] * 0.5 + 0.5) matrix = _get_inverse_affine_matrix(center, angle, translate, scale, shear) kwargs = {"fillcolor": fillcolor} if PILLOW_VERSION[0] == '5' else {} return img.transform(output_size, Image.AFFINE, matrix, resample, **kwargs)如：img_affine = affine(img, 60, [10, -10], 0.8, -30) # vis ax1 = plt.subplot(1, 2, 1) ax1.imshow(img) ax1.axis("off") ax1.set_title("orig img") ax2 = plt.subplot(1, 2, 2) ax2.imshow(img_affine) ax2.axis("off") ax2.set_title("affine img") plt.show()1.21. to_grayscale(img, num_output_channels=1)将图像转换为灰度图.def to_grayscale(img, num_output_channels=1): """ Args: img (PIL Image): Image to be converted to grayscale. Returns: PIL Image: Grayscale version of the image. if num_output_channels = 1 : returned image is single channel if num_output_channels = 3 : returned image is 3 channel with r = g = b """ if not _is_pil_image(img): raise TypeError('img should be PIL Image. Got {}'.format(type(img))) if num_output_channels == 1: img = img.convert('L') elif num_output_channels == 3: img = img.convert('L') np_img = np.array(img, dtype=np.uint8) np_img = np.dstack([np_img, np_img, np_img]) img = Image.fromarray(np_img, 'RGB') else: raise ValueError('num_output_channels should be either 1 or 3') return img如：img_grayscale = to_grayscale(img) # vis ax1 = plt.subplot(1, 2, 1) ax1.imshow(img) ax1.axis("off") ax1.set_title("orig img") ax2 = plt.subplot(1, 2, 2) ax2.imshow(img_grayscale) ax2.axis("off") ax2.set_title("grayscale img") plt.show()1.22. scale(args, *kwargs) (已废弃)def scale(*args, **kwargs): warnings.warn("The use of the transforms.Scale transform is deprecated, " + "please use transforms.Resize instead.") return resize(*args, **kwargs)
原文：Data Augmentation in SSD (Single Shot Detector) - 2018.06.28作者研究了 single shot 目标检测器 SSD: Single Shot MultiBox Detector 如何取得了比 Faster R-CNN 更佳的 mAP，且计算复杂度明显减少，实时运行效果更快. 这里将详细描述 SSD 所采用的数据增强策略. 根据论文所述，其数据增强策略取得了 8.8% 的 mAP 提升. 数据增强对于小目标检测的精度提升尤其重要，因为其对图像进行缩放(zoom)，使得分类器能够识别更多图片中的目标物体. 数据增强对于图像中出现被遮挡物体(occluded objects)的处理，也很重要，通过对训练数据中的图像进行裁剪，使得只有部分物体是可见的.1. 数据增强流程采用的数据增强处理流程步骤如下：光度变形(Photometric Distortions)几何变换(Geometric Distortions)图像扩展(ExpandImage)随机裁剪(RandomCrop)随机镜像(RandomMirror) 下面会详细介绍这些数据增强(包括相应的代码实现片段). 其中，每一数据变换都是概率随机的(一般 probability=0.5). 因此，不同的训练次数采用的是数据增强后的不同图像数据.Github - amdegroot/ssd.pytorch2. 光度变形(Photometric Distortions)2.1. 随机调整亮度(Random Brightness)以 probability=0.5 的 随机概率，随机对图像每个像素添加一个值，该添加值是从 [-delat, delta] 中随机选取的. 默认的 delta 值是 32.def __call__(self, image, boxes=None, labels=None): if random.randint(2): delta = random.uniform(-self.delta, self.delta) image += delta return image, boxes, labels2.2. 随机调整对比度(Contrast), 色相(Hue) 和饱和度(Saturation)亮度(Brightness)调整后，进行对比度，色相和饱和度的随机变换. 其顺序是随机的. 有两种选择，首先对比度处理，然后色相和饱和度；或者首先色相和饱和度处理，然后对比度处理. 每种选择都是随机的，probability=0.5.self.pd = [RandomContrast(), ConvertColor(transform='HSV'), RandomSaturation(), RandomHue(), ConvertColor(current='HSV', transform='BGR'), RandomContrast() ] im, boxes, labels = self.rand_brightness(im, boxes, labels) if random.randint(2): distort = Compose(self.pd[:-1]) else: distort = Compose(self.pd[1:]) im, boxes, labels = distort(im, boxes, labels)其中，对比度是在 RGB 空间处理，色相和饱和度是在 HSV 空间. 因此，在进行每一步操作前需要先作对应的颜色空间转换. 对比度，色相和饱和度的调整类似于亮度调整，都是 probability=0.5 的随机进行. 通过随机在一个上界和下界区间内选择形变的值. 例如，饱和度调整：def __call__(self, image, boxes=None, labels=None): if random.randint(2): image[:, :, 1] *= random.uniform(self.lower, self.upper) return image, boxes, labels2.3. 随机光照噪声(RandomLightingNoise)最后一种光度形变是随机光照噪声. 其包括随机颜色通道交换(color channel swap). 颜色变换定义为：self.perms = ((0, 1, 2), (0, 2, 1), (1, 0, 2), (1, 2, 0), (2, 0, 1), (2, 1, 0))对于 RGB 图像，颜色通道变换(0 2 1) 则交换 green 和 blue 通道，保持 red 通道不变.3. 几何变换光度调整只会改变图像像素，而不会改变图像的尺寸大小. 而几何变换则会对图像尺寸和维度进行调整.3.1. 随机扩展(RandomExpand)3.2. 随机裁剪(RandomCrop)随机裁剪是对 ExpandImage 得到的图像裁剪出其中的一部分图像块，并确保该图像块至少一个 groundtruth box 有重叠，至少一个 groundtruth box 的中心(centroid) 位于该图像块中. 这样就可以避免不包含明显的前景目标的图像块不用于网络训练(只有包含明显的前景目标的图像块采用于网络训练.) 同时，保证只有部分前景目标可见的图片用于网络训练. 例如，下面图像中，Groundtruth box 表示为红色，裁剪的图像块表示为绿色. 由于是随机裁剪，因此有些图像是包含扩展的画布，有些则不包含. 3.3. 随机镜像(RandomMirror)最后一步数据增强是随机镜像. 其仅是图像的左右翻转.在很多目标检测和图像分类张总均有采用. 最后，类似于其它目标检测和图像分类任务，图像为 resize 到 300x300，groundtruth 坐标进行相应调整；并对图像进行归一化和减均值处理. 完整的数据增强脚本 - augmentations.py.import torch from torchvision import transforms import cv2 import numpy as np import types from numpy import random def intersect(box_a, box_b): max_xy = np.minimum(box_a[:, 2:], box_b[2:]) min_xy = np.maximum(box_a[:, :2], box_b[:2]) inter = np.clip((max_xy - min_xy), a_min=0, a_max=np.inf) return inter[:, 0] * inter[:, 1] def jaccard_numpy(box_a, box_b): """Compute the jaccard overlap of two sets of boxes. The jaccard overlap is simply the intersection over union of two boxes. E.g.: A ∩ B / A ∪ B = A ∩ B / (area(A) + area(B) - A ∩ B) Args: box_a: Multiple bounding boxes, Shape: [num_boxes,4] box_b: Single bounding box, Shape: [4] Return: jaccard overlap: Shape: [box_a.shape[0], box_a.shape[1]] """ inter = intersect(box_a, box_b) area_a = ((box_a[:, 2]-box_a[:, 0]) * (box_a[:, 3]-box_a[:, 1])) # [A,B] area_b = ((box_b[2]-box_b[0]) * (box_b[3]-box_b[1])) # [A,B] union = area_a + area_b - inter return inter / union # [A,B] class Compose(object): """Composes several augmentations together. Args: transforms (List[Transform]): list of transforms to compose. Example: >>> augmentations.Compose([ >>> transforms.CenterCrop(10), >>> transforms.ToTensor(), >>> ]) """ def __init__(self, transforms): self.transforms = transforms def __call__(self, img, boxes=None, labels=None): for t in self.transforms: img, boxes, labels = t(img, boxes, labels) return img, boxes, labels class Lambda(object): """Applies a lambda as a transform.""" def __init__(self, lambd): assert isinstance(lambd, types.LambdaType) self.lambd = lambd def __call__(self, img, boxes=None, labels=None): return self.lambd(img, boxes, labels) class ConvertFromInts(object): def __call__(self, image, boxes=None, labels=None): return image.astype(np.float32), boxes, labels class SubtractMeans(object): def __init__(self, mean): self.mean = np.array(mean, dtype=np.float32) def __call__(self, image, boxes=None, labels=None): image = image.astype(np.float32) image -= self.mean return image.astype(np.float32), boxes, labels class ToAbsoluteCoords(object): def __call__(self, image, boxes=None, labels=None): height, width, channels = image.shape boxes[:, 0] *= width boxes[:, 2] *= width boxes[:, 1] *= height boxes[:, 3] *= height return image, boxes, labels class ToPercentCoords(object): def __call__(self, image, boxes=None, labels=None): height, width, channels = image.shape boxes[:, 0] /= width boxes[:, 2] /= width boxes[:, 1] /= height boxes[:, 3] /= height return image, boxes, labels class Resize(object): def __init__(self, size=300): self.size = size def __call__(self, image, boxes=None, labels=None): image = cv2.resize(image, (self.size, self.size)) return image, boxes, labels class RandomSaturation(object): def __init__(self, lower=0.5, upper=1.5): self.lower = lower self.upper = upper assert self.upper >= self.lower, "contrast upper must be >= lower." assert self.lower >= 0, "contrast lower must be non-negative." def __call__(self, image, boxes=None, labels=None): if random.randint(2): image[:, :, 1] *= random.uniform(self.lower, self.upper) return image, boxes, labels class RandomHue(object): def __init__(self, delta=18.0): assert delta >= 0.0 and delta <= 360.0 self.delta = delta def __call__(self, image, boxes=None, labels=None): if random.randint(2): image[:, :, 0] += random.uniform(-self.delta, self.delta) image[:, :, 0][image[:, :, 0] > 360.0] -= 360.0 image[:, :, 0][image[:, :, 0] < 0.0] += 360.0 return image, boxes, labels class RandomLightingNoise(object): def __init__(self): self.perms = ((0, 1, 2), (0, 2, 1), (1, 0, 2), (1, 2, 0), (2, 0, 1), (2, 1, 0)) def __call__(self, image, boxes=None, labels=None): if random.randint(2): swap = self.perms[random.randint(len(self.perms))] shuffle = SwapChannels(swap) # shuffle channels image = shuffle(image) return image, boxes, labels class ConvertColor(object): def __init__(self, current='BGR', transform='HSV'): self.transform = transform self.current = current def __call__(self, image, boxes=None, labels=None): if self.current == 'BGR' and self.transform == 'HSV': image = cv2.cvtColor(image, cv2.COLOR_BGR2HSV) elif self.current == 'HSV' and self.transform == 'BGR': image = cv2.cvtColor(image, cv2.COLOR_HSV2BGR) else: raise NotImplementedError return image, boxes, labels class RandomContrast(object): def __init__(self, lower=0.5, upper=1.5): self.lower = lower self.upper = upper assert self.upper >= self.lower, "contrast upper must be >= lower." assert self.lower >= 0, "contrast lower must be non-negative." # expects float image def __call__(self, image, boxes=None, labels=None): if random.randint(2): alpha = random.uniform(self.lower, self.upper) image *= alpha return image, boxes, labels class RandomBrightness(object): def __init__(self, delta=32): assert delta >= 0.0 assert delta <= 255.0 self.delta = delta def __call__(self, image, boxes=None, labels=None): if random.randint(2): delta = random.uniform(-self.delta, self.delta) image += delta return image, boxes, labels class ToCV2Image(object): def __call__(self, tensor, boxes=None, labels=None): return tensor.cpu().numpy().astype(np.float32).transpose((1, 2, 0)), boxes, labels class ToTensor(object): def __call__(self, cvimage, boxes=None, labels=None): return torch.from_numpy(cvimage.astype(np.float32)).permute(2, 0, 1), boxes, labels class RandomSampleCrop(object): """Crop Arguments: img (Image): the image being input during training boxes (Tensor): the original bounding boxes in pt form labels (Tensor): the class labels for each bbox mode (float tuple): the min and max jaccard overlaps Return: (img, boxes, classes) img (Image): the cropped image boxes (Tensor): the adjusted bounding boxes in pt form labels (Tensor): the class labels for each bbox """ def __init__(self): self.sample_options = ( # using entire original input image None, # sample a patch s.t. MIN jaccard w/ obj in .1,.3,.4,.7,.9 (0.1, None), (0.3, None), (0.7, None), (0.9, None), # randomly sample a patch (None, None), ) def __call__(self, image, boxes=None, labels=None): height, width, _ = image.shape while True: # randomly choose a mode mode = random.choice(self.sample_options) if mode is None: return image, boxes, labels min_iou, max_iou = mode if min_iou is None: min_iou = float('-inf') if max_iou is None: max_iou = float('inf') # max trails (50) for _ in range(50): current_image = image w = random.uniform(0.3 * width, width) h = random.uniform(0.3 * height, height) # aspect ratio constraint b/t .5 & 2 if h / w < 0.5 or h / w > 2: continue left = random.uniform(width - w) top = random.uniform(height - h) # convert to integer rect x1,y1,x2,y2 rect = np.array([int(left), int(top), int(left+w), int(top+h)]) # calculate IoU (jaccard overlap) b/t the cropped and gt boxes overlap = jaccard_numpy(boxes, rect) # is min and max overlap constraint satisfied? if not try again if overlap.min() < min_iou and max_iou < overlap.max(): continue # cut the crop from the image current_image = current_image[rect[1]:rect[3], rect[0]:rect[2], :] # keep overlap with gt box IF center in sampled patch centers = (boxes[:, :2] + boxes[:, 2:]) / 2.0 # mask in all gt boxes that above and to the left of centers m1 = (rect[0] < centers[:, 0]) * (rect[1] < centers[:, 1]) # mask in all gt boxes that under and to the right of centers m2 = (rect[2] > centers[:, 0]) * (rect[3] > centers[:, 1]) # mask in that both m1 and m2 are true mask = m1 * m2 # have any valid boxes? try again if not if not mask.any(): continue # take only matching gt boxes current_boxes = boxes[mask, :].copy() # take only matching gt labels current_labels = labels[mask] # should we use the box left and top corner or the crop's current_boxes[:, :2] = np.maximum(current_boxes[:, :2], rect[:2]) # adjust to crop (by substracting crop's left,top) current_boxes[:, :2] -= rect[:2] current_boxes[:, 2:] = np.minimum(current_boxes[:, 2:], rect[2:]) # adjust to crop (by substracting crop's left,top) current_boxes[:, 2:] -= rect[:2] return current_image, current_boxes, current_labels class Expand(object): def __init__(self, mean): self.mean = mean def __call__(self, image, boxes, labels): if random.randint(2): return image, boxes, labels height, width, depth = image.shape ratio = random.uniform(1, 4) left = random.uniform(0, width*ratio - width) top = random.uniform(0, height*ratio - height) expand_image = np.zeros( (int(height*ratio), int(width*ratio), depth), dtype=image.dtype) expand_image[:, :, :] = self.mean expand_image[int(top):int(top + height), int(left):int(left + width)] = image image = expand_image boxes = boxes.copy() boxes[:, :2] += (int(left), int(top)) boxes[:, 2:] += (int(left), int(top)) return image, boxes, labels class RandomMirror(object): def __call__(self, image, boxes, classes): _, width, _ = image.shape if random.randint(2): image = image[:, ::-1] boxes = boxes.copy() boxes[:, 0::2] = width - boxes[:, 2::-2] return image, boxes, classes class SwapChannels(object): """Transforms a tensorized image by swapping the channels in the order specified in the swap tuple. Args: swaps (int triple): final order of channels eg: (2, 1, 0) """ def __init__(self, swaps): self.swaps = swaps def __call__(self, image): """ Args: image (Tensor): image tensor to be transformed Return: a tensor with channels swapped according to swap """ # if torch.is_tensor(image): # image = image.data.cpu().numpy() # else: # image = np.array(image) image = image[:, :, self.swaps] return image class PhotometricDistort(object): def __init__(self): self.pd = [ RandomContrast(), ConvertColor(transform='HSV'), RandomSaturation(), RandomHue(), ConvertColor(current='HSV', transform='BGR'), RandomContrast() ] self.rand_brightness = RandomBrightness() self.rand_light_noise = RandomLightingNoise() def __call__(self, image, boxes, labels): im = image.copy() im, boxes, labels = self.rand_brightness(im, boxes, labels) if random.randint(2): distort = Compose(self.pd[:-1]) else: distort = Compose(self.pd[1:]) im, boxes, labels = distort(im, boxes, labels) return self.rand_light_noise(im, boxes, labels) class SSDAugmentation(object): def __init__(self, size=300, mean=(104, 117, 123)): self.mean = mean self.size = size self.augment = Compose([ ConvertFromInts(), ToAbsoluteCoords(), PhotometricDistort(), Expand(self.mean), RandomSampleCrop(), RandomMirror(), ToPercentCoords(), Resize(self.size), SubtractMeans(self.mean) ]) def __call__(self, img, boxes, labels): return self.augment(img, boxes, labels)