ResNetの紹介

1 簡単な要約

    Microsoft ResearchのKaiming Heを含む4人の中国人が提案したResNet (Residual Neural Network)は、ResNet Unitを使って152層のニューラルネットワークの学習に成功し、ILSVRC 2015コンペティションで、パラメータ数が少ないのに、誤差がトップ5で3.57%で優勝しました ResNetの構造はニューラルネットワークの学習を非常に高速化でき、モデルの精度向上が実現されています。ResNetの汎化は非常に優れており、InceptionNetのネットワークにそのまま使用することも可能です。

    ResNetの主なアイデアは、ネットワークに直接接続されたチャンネルを追加することで、ハイウェイネットワークのアイデアです。ResNetの考え方もHighway Networkと非常に似ていて、下図のように元の入力情報をそのまま後の層に渡すことができます。

    このように、この層のニューラルネットワークは、出力全体ではなく、前のネットワークの出力の残差を学習することができるため、ResNetと呼ばれ、残差ネットワークとも呼ばれる。

2 イノベーションのポイント

    残差学習という考え方が提案されています。従来の畳み込みネットワークや完全連結ネットワークは、情報を渡す際に多かれ少なかれ情報損失やロスがあり、また勾配消失や勾配爆発を起こすため、深いネットワークを学習させることは不可能であった。VGGNetとResNetの比較を下図に示す。ResNetの最大の違いは、入力を直接後の層に接続するためのバイパスが多く、ショートカットやスキップ接続とも呼ばれる構造であることである。

<イグ

3 ネットワーク構成

    ResNetのネットワーク構造では、2つの3*3畳み込みネットワークを連結したものを残差モジュールとし、1*1、3*3、1*1の3つの畳み込みネットワークを連結したものを残差モジュールとする2種類の残差モジュールが用いられている。下図に示す。

    ResNetには、50層、101層、152層という異なるネットワーク層があり、これらはすべて上記の残基モジュールを積み重ねることで実装されています。

4 コードの実装

#%%
# Copyright 2016 The TensorFlow Authors. All Rights Reserved.
#All Rights Reserved.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
"""

Typical use:

   from tensorflow.contrib.slim.nets import resnet_v2

ResNet-101 for image classification into 1000 classes:

   # inputs has shape [batch, 224, 224, 3]
   with slim.arg_scope(resnet_v2.resnet_arg_scope(is_training)):
      net, end_points = resnet_v2.resnet_v2_101(inputs, 1000)

ResNet-101 for semantic segmentation into 21 classes:

   # inputs has shape [batch, 513, 513, 3]
   with slim.arg_scope(resnet_v2.resnet_arg_scope(is_training)):
      net, end_points = resnet_v2.resnet_v2_101(inputs,
                                                21,
                                                global_pool=False,
                                                output_stride=16)
"""
import collections
import tensorflow as tf
slim = tf.contrib.slim


#namedtuple is a function that creates a custom tuple object and specifies the number of tuple elements
#and can refer to an element of a tuple with a property instead of an index
# is equivalent to creating a Block class, with scope, unit_fn, args properties
class Block(collections.namedtuple('Block', ['scope', 'unit_fn', 'args'])):
  """A named tuple describing a ResNet block.

  Its parts are:
    scope: The scope of the `Block`.
    unit_fn: The ResNet unit function which takes as input a `Tensor` and
      unit_fn: The ResNet unit function which takes as input a `Tensor` and returns another `Tensor` with the output of the ResNet unit.
    args: A list of length equal to the number of units in the `Block`. The list
      contains one (depth, depth_bottleneck, stride) tuple for each unit in the
      block to serve as argument to unit_fn.
  """


def subsample(inputs, factor, scope=None):
  """Subsamples the input along the spatial dimensions.

  Args:
    inputs: A `Tensor` of size [batch, height_in, width_in, channels].
    factor: The subsampling factor.
    scope: Optional variable_scope.


  }

  with slim.arg_scope(
      [slim.conv2d],
      weights_regularizer=slim.l2_regularizer(weight_decay),
      weights_initializer=slim.variance_scaling_initializer(),
      activation_fn=tf.nn.relu,
      normalizer_fn=slim.batch_norm,
      normalizer_params=batch_norm_params):
    with slim.arg_scope([slim.batch_norm], **batch_norm_params):
      # The following implies padding='SAME' for pool1, which makes feature
      # This is also used in . However the accompanying padding='SAME' for pool1, which makes feature alignment easier for dense prediction tasks.
      # However the accompanying
      # However the accompanying code of 'Deep Residual Learning for Image Recognition' uses
      However the accompanying # code of 'Deep Residual Learning for Image Recognition' uses # padding='VALID' for pool1. You can switch to that choice by setting
      # slim.arg_scope([slim.max_pool2d], padding='VALID').
      with slim.arg_scope([slim.max_pool2d], padding='SAME') as arg_sc:
        return arg_sc




@slim.add_arg_scope
def bottleneck(inputs, depth, depth_bottleneck, stride,
               outputs_collections=None, scope=None):
  """Bottleneck residual unit variant with BN before convolutions.

  This is the full preactivation residual unit variant proposed in [2]. See
  Fig. 1(b) of [2] for its definition. Note that we use here the bottleneck
  Note that we use here the bottleneck variant which has an extra bottleneck layer.

  When putting together two consecutive ResNet blocks that use this unit, one
  should use stride = 2 in the last unit of the first block.

  Args:
    inputs: A tensor of size [batch, height, width, channels].
    depth: The depth of the ResNet unit output.
    depth_bottleneck: The depth of the bottleneck layers.
    stride: The ResNet unit's stride. Determines the amount of downsampling of
      Determines the amount of downsampling of the units output compared to its input.
    rate: An integer, rate for atrous convolution.
    outputs_collections: Collection to add the ResNet unit output.
    scope: Optional variable_scope.

  Returns:
    The ResNet unit's output.
  """
  with tf.variable_scope(scope, 'bottleneck_v2', [inputs]) as sc:
    depth_in = slim.utils.last_dimension(inputs.get_shape(), min_rank=4)
    preact = slim.batch_norm(inputs, activation_fn=tf.nn.relu, scope='preact')
    if depth == depth_in:
      shortcut = subsample(inputs, stride, 'shortcut')
    else:
      shortcut = slim.conv2d(preact, depth, [1, 1], stride=stride,
                             normalizer_fn=None, activation_fn=None,
                             scope='shortcut')

    residual = slim.conv2d(preact, depth_bottleneck, [1, 1], stride=1,
                           scope='conv1')
    residual = conv2d_same(residual, depth_bottleneck, 3, stride,
                                        scope='conv2')
    residual = slim.conv2d(residual, depth, [1, 1], stride=1,
                           normalizer_fn=None, activation_fn=None,
                           scope='conv3')

    output = shortcut + residual

    return slim.utils.collect_named_outputs(outputs_collections,
                                            sc.name,
                                            output)


def resnet_v2(inputs,
              blocks,
              num_classes=None,
              global_pool=True,
              include_root_block=True,
              reuse=None,
              scope=None):
  """Generator for v2 (pre-activation) ResNet models.

  This function generates a family of ResNet v2 models. See the resnet_v2_*()
  methods for specific model instantiations, obtained by selecting different
  See the resnet_v2_*() methods for specific model instantiations, obtained by selecting different block instantiations that produce ResNets of various depths.


  Args:
    inputs: A tensor of size [bat

5 参考文献

[1]Huang WJ,Tang Y. TensorFlow in action [M](テンソルフロー イン アクション). 北京:電子工業出版社,2017.

[2] https://arxiv.org/abs/1512.03385

[3] https://github.com/tensorflow/models/blob/master/research/slim/nets/resnet_v2.py

プラグイン式。

AliCloud-City BrainグループでNLPアルゴリズムエンジニア募集！学校採用/ソーシャル採用/インターン可、可私信或联系[email protected]。