TensorFlow cnn-cifar10 サンプルコード詳細

       TensorFlowは分散対応のディープラーニングフレームワークで、Googleが牽引し、普及が進んでいます。先日、TensorFlowのチュートリアルから、CNNを使ったcifar10データセットの分類の例を学びました。ソースコードを見ながら、QueueRunnersの仕組みやTensorboardによる可視化、マルチGPUデータの並列プログラミングなど、tensorflowの構文知識のほとんどをこのサンプルがカバーしていると考えると、読後感がもっともなものでした。"実装の過程でもいくつか問題が発生したので、ここでは"know what you know"の基準に従って、プログラムの詳細な理解と発生した問題を書き留めました。 "プログラム中のコメントは、今後の参考として「なぜこのコードの行はこう書かれているか」という解釈に重点を置いています（畳込みニューラルネットワークに関する理論部分はネット上にたくさんあるので、ここではあまりやり過ぎないことにしています）。

標準的な機械学習プログラムは、データ入力、モデル自体の定義、モデル学習、モデル性能試験の4つの主要な部分を含むはずで、4つの.pyファイルに分けることができます。

(i) データ入力部 (input_dataset.py)

       概念的には、このセクションはデータパイプの構築に関するもので、データの流れは "バイナリファイル->ファイル名キュー->データキュー->読み出しデータ-バッチ" となります。このdata-batchは深層学習ネットワークの入力として情報の順伝搬に使われますが、これについてはモデル自体の定義の項で説明します。データパイプライン全体を定義する際には、TensorFlowのキュー機構を利用しますが、その詳細については、前回のブログ記事 "TensorFlow Reads Binary File Data to Queue" で解説しています。また、元のデータファイルを読み込む際には、ファイル自体の形式を組み合わせる必要があり、c言語でバイナリファイルを読み込むプログラムを書いたことがある方ならお馴染みだと思いますが、具体的には以下のようなコードになります。

# -*- coding: utf-8 -*-
import os
import tensorflow as tf
# The scale of the original image is 32*32, but common sense dictates that the information part is usually located in the center of the image, so here we define the size of the image after cropping to the center
fixed_height = 24
fixed_width = 24
# The format of the cifar10 dataset, with 50k training samples and 10k test samples, respectively
train_samples_per_epoch = 50000
test_samples_per_epoch = 10000
data_dir = '. /cifar-10-batches-bin' # Define the path to the folder where the dataset is located
batch_size=128 # Define the size of the batch to be used for each parameter update
 
def read_cifar10(filename_queue):
    # Define an empty class object, similar to the definition of a structure in the c language
    class Image(object):
        pass
    image = Image()
    image.height=32
    image.width=32
    image.depth=3
    label_bytes = 1
    image_bytes = image.height*image.width*image.depth
    Bytes_to_read = label_bytes+image_bytes
    # Define a Reader that can read a fixed number of bytes from the file at a time
    reader = tf.FixedLengthRecordReader(record_bytes=Bytes_to_read) 
    # return the (key, value) pair read from filename_queue, both key and value are tensor of string type and the filename will dequeue when a file in the queue is finished reading
    image.key, value_str = reader.read(filename_queue) 
    # Decode operation can be seen as reading a binary file, converting the bytes in the string to a vector of values, each value occupies a byte in the [0, 255] range, so out_type should be taken as uint8 type
    value = tf.decode_raw(bytes=value_str, out_type=tf.uint8) 
    # intercept a slice from a one-dimensional tensor object, similar to filtering subvectors from a one-dimensional vector, because value contains a label and feature, so the vector type tensor to 'parse' operation    
    image.label = tf.slice(input_=value, begin=[0], size=[label_bytes])# begin and size indicate the starting point and length of the fragment to be intercepted, respectively
    data_mat = tf.slice(input_=value, begin=[label_bytes], size=[image_bytes])
    data_mat = tf.reshape(data_mat, (image.depth, image.height, image.width)) # the order of the dimensions here is based on the format of the cifar binary file
    transposed_value = tf.transpose(data_mat, perm=[1, 2, 0]) # rearrange the dimensions of data_mat, the i-th dimension of the returned value corresponds to the perm[i] dimension of data_mat
    image.mat = transposed_value    
    return image    
    
def get_batch_samples(img_obj, min_samples_in_queue, batch_size, shuffle_flag):
'''
The tf.train.shuffle_batch() function is used to shuffle tensors in the queue randomly to create batches (i.e., multiple samples from the data file can be read at a time to form a batch). This function adds the following objects to the current Graph.
* a shuffling queue created to press the tensors in 'tensors' into that queue.
* a dequeue_many operation to create a batch based on the data in the queue.
* a QueueRunner object is created to start a process to press data to the queue
The capacity parameter controls the maximum length of the shuffling queue; the min_after_dequeue parameter indicates the minimum number of elements in the queue after a dequeue operation, which can be used to ensure the randomness of the elements in the batch; the num_after_dequeue parameter indicates the minimum number of elements in the queue after a dequeue operation.
The num_threads parameter is used to specify how many threads are responsible for pressing tensors into the queue; the enqueue_many parameter is used to characterize whether each tensor in the tensors represents a sample
tf.train.batch() is similar, except that it comes out of the queue sequentially (i.e., only one batch can be read from one data file at a time), with less randomness.
'''
    if shuffle_flag == False:
        image_batch, label_batch = tf.train.batch(tensors=img_obj, 
                                                  batch_size=batch_size, 
                                                  num_threads=4, 
                                                  capacity=min_samples_in_queue+3*batch_size)
    else:
        image_batch, label_batch = tf.train.shuffle_batch(tensors=img_obj, 
                                                          batch_size=batch_size, 
                                                          num_threads=4, 
                                                          min_after_dequeue=min_samples_in_queue,
                                                          capacity=min_samples_in_queue+3*batch_size)                                                    
    tf.image_summary('input_image', image_batch, max_images=6) # output the summary cache object of the preprocessed image for writing to the event file in the session                                                    
    return image_batch, tf.reshape(label_batch, shape=[batch_size])     
                                       
def preprocess_input_data():
'''This part of the program is used to perform a 'data augmentation' operation on the training dataset to prevent overfitting by increasing the size of the training set''''
    filenames = [os.path.join(data_dir, 'data_batch_%d.bin' % i) for i in range(1, 6)]
    # filenames = [os.path.join(data_dir, 'test_batch.bin')]
    for f in filenames: #check if the training dataset file exists
        if not tf.gfile.Exists(f):
            raise ValueError('fail to find file:'+f)    
    filename_queue = tf.train.string_input_producer(string_tensor=filenames) # Output filenames to the queue as the first stage of the whole data pipe
    image = read_cifar10(filename_queue) # read an image of type tensor from the filename queue
    new_img = tf.cast(image.mat, tf.float32)
    tf.image_summary('raw_input_image', tf.reshape(new_img, [1, 32, 32, 3]))# output the summary cache object of the image before preprocessing
    new_img = tf.random_crop(new_img, size=(fixed_height, fixed_width, 3)) #cut out sub-images from the original image
    new_img = tf.image.random_brightness(new_img, max_delta=63) #Randomly adjust the brightness of the image
    new_img = tf.image.random_flip_left_right(new_img) # randomly flip the image left and right
    new_img = tf.image.random_contrast(new_img, lower=0.2, upper=1.8) #Randomly adjust the image contrast
    final_img = tf.image.per_image_whitening(new_img) #whiten the image to reduce the redundancy of the input image and remove as much correlation as possible between the input features
    
    min_samples_ratio_in_queue = 0.4 # used to ensure the randomness of the samples in the read batches to cover more categories, more data files!!!
    min_samples_in_queue = int(min_samples_ratio_in_queue*train_samples_per_epoch) 
    return get_batch_samples([final_img, image.label], min_samples_in_queue, batch_size, shuffle_flag=True)

(ii) モデル本体定義部(forward_prop.py)

       この例では、ネットワーク構造は "入力層->畳み込み層->プーリング層->正規化層-> 畳み込み層->正規化層-> プーリング層->完全連結層-" ソフトマックス出力層" と定義されています。他のディープネットワークモデルと異なり、Relu(rectified linear unit)活性化関数が入力励起を[0, infinite]にマッピングするため、ここでは正規化層を導入しており、詳細コードとその解説は以下の通りです。

# -*- coding: utf-8 -*-
import tensorflow as tf
import input_dataset
# External reference to hyperparameters defined in input_dataset file
height = input_dataset.fixed_height
width = input_dataset.fixed_width
train_samples_per_epoch = input_dataset.train_samples_per_epoch
test_samples_per_epoch = input_dataset.test_samples_per_epoch
 
# Constants used to describe the training process
moving_average_decay = 0.9999 # The decay to use for the moving average.
num_epochs_per_decay = 350.0 # The decay is a step function, which controls the decay period (step width)
learning_rate_decay_factor = 0.1 # Learning rate decay factor
initial_learning_rate = 0.1 # Initial learning rate
 
def variable_on_cpu(name, shape, dtype, initializer):
    with tf.device("/cpu:0"): # A context manager, which specifies the hardware to use for the new op
        return tf.get_variable(name=name, 
                               shape=shape, 
                               initializer=initializer,
                               dtype=dtype)    
 
def variable_on_cpu_with_collection(name, shape, dtype, stddev, wd):
    with tf.device("/cpu:0"): 
        weight = tf.get_variable(name=name, 
                                 shape=shape, 
                                 initializer=tf.truncated_normal_initializer(stddev=stddev, dtype=dtype))
        if wd is not None:
            weight_decay = tf.mul(tf.nn.l2_loss(weight), wd, name='weight_loss')
            tf.add_to_collection(name='losses', value=weight_decay)         
        return weight
        
def losses_summary(total_loss):
# Maintain the sliding mean of the variables by using exponential decay. When training a model, it is beneficial to maintain a sliding mean for the training parameters. Using sliding parameters during testing than the final trained parameter values themselves.
    # will improve the actual performance (accuracy) of the model. apply() method adds shadow copies of the trained variables and adds operations to maintain the sliding mean of the variables to the shadow copies. average
    The # method gives access to the shadow variables, which is useful when creating an evaluation model.
#The sliding average is calculated by exponential decay. shadow variable is initialized with the same values as the trained variables and is updated with the following formula
# shadow_variable = decay * shadow_variable + (1 - decay) * variable
    average_op = tf.train.ExponentialMovingAverage(decay=0.9) # Create a new exponential sliding average object
    losses = tf.get_collection(key='losses') # return all variables corresponding to the keyword 'losses' from the dictionary collection, including cross-entropy losses and regular term losses
    # Create 'shadow variables', and add the operation to maintain sliding averages
    maintain_averages_op = average_op.apply(losses+[total_loss]) # maintain the sliding average of the variables, and return an action that updates the shadow variables
    for i in losses+[total_loss]:
        tf.scalar_summary(i.op.name+'_raw', i) # save the variables to a Summary cache object for writing to a file
        tf.scalar_summary(i.op.name, average_op.average(i)) #average() returns the shadow variable for a given variable.
    return maintain_averages_op #returns the update operation for the loss variable
    
def one_step_train(total_loss, step):
    batch_count = int(train_samples_per_epoch/input_dataset.batch_size) # find the number of training blocks 
    decay_step = batch_count*num_epochs_per_decay # decay lr after each decay_step
    lr = tf.train.exponential_decay(learning_rate=initial_learning_rate,
                                    global_step=step,
                                    decay_steps=decay_step,
                                    decay_rate=learning_rate_decay_factor,
                                    staircase=True)
    tf.scalar_summary('learning_rate', lr)
    losses_movingaverage_op = losses_summary(total_loss)
    # tf.control_dependencies is a context manager, which controls the order of node execution, executing the operations in control_inputs first, and then executing the operations in context
    with tf.control_dependencies(control_inputs=[losses_movingaverage_op]):
        trainer = tf.train.GradientDescentOptimizer(learning_rate=lr)
        gradient_pairs = trainer.c
    maintain_variable_average_op = variables_average_op.apply(var_list=tf.trainable_variables())# return a sliding update operation for model parameter variables
    with tf.control_dependencies(control_inputs=[gradient_update, maintain_variable_average_op]):
        gradient_update_optimizor = tf.no_op() #Does nothing. only useful as a placeholder for control edges
    return gradient_update_optimizor                    
    
def network(images):
#This part mainly calls a few common functions, which are described in detail in the previous blog 'TensorFlow Implementation of Convolutional Neural Networks', so I won't go over them here ~
    with tf.variable_scope(name_or_scope='conv1') as scope:
        weight = variable_on_cpu_with_collection(name='weight', 
                                                 shape=(5, 5, 3, 64), 
                                                 dtype=tf.float32, 
                                                 stddev=0.05,
                                                 wd = 0.0)
        bias = variable_on_cpu(name='bias', shape=(64), dtype=tf.float32, initializer=tf.constant_initializer(value=0.0))
        conv1_in = tf.nn.conv2d(input=images, filter=weight, strides=(1, 1, 1, 1), padding='SAME')
        conv1_in = tf.nn.bias_add(value=conv1_in, bias=bias)
        conv1_out = tf.nn.relu(conv1_in) 
        
    pool1 = tf.nn.max_pool(value=conv1_out, ksize=(1, 3, 3, 1), strides=(1, 2, 2, 1), padding='SAME')
    
    norm1 = tf.nn.lrn(input=pool1, depth_radius=4, bias=1.0, alpha=0.001/9.0, beta=0.75)
    
    with tf.variable_scope(name_or_scope='conv2') as scope:
        weight = variable_on_cpu_with_collection(name='weight', 
                                 shape=(5, 5, 64, 64), 
                                 dtype=tf.float32, 
                                 stddev=0.05,
                                 wd=0.0)
        bias = variable_on_cpu(name='bias', shape=(64), dtype=tf.float32, initializer=tf.constant_initializer(value=0.1))
        conv2_in = tf.nn.conv2d(norm1, weight, strides=(1, 1, 1, 1), padding='SAME')
        conv2_in = tf.nn.bias_add(conv2_in, bias)
        conv2_out = tf.nn.relu(conv2_in) 
    
    norm2 = tf.nn.lrn(input=conv2_out, depth_radius=4, bias=1.0, alpha=0.001/9.0, beta=0.75)
    
    pool2 = tf.nn.max_pool(value=norm2, ksize=(1, 3, 3, 1), strides=(1, 2, 2, 1), padding='SAME')
    # input tensor of shape `[batch, in_height, in_width, in_channels]
    reshaped_pool2 = tf.reshape(tensor=pool2, shape=(-1, 6*6*64))
    
    with tf.variable_scope(name_or_scope='fully_connected_layer1') as scope:
        weight = variable_on_cpu_with_collection(name='weight', 
                                                 shape=(6*6*64, 384), 
                                                 dtype=tf.float32,
                                                 stddev=0.04,
                                                 wd = 0.004)
        bias = variable_on_cpu(name='bias', shape=(384), dtype=tf.float32, initializer=tf.constant_initializer(value=0.1))
        fc1_in = tf.matmul(reshaped_pool2, weight)+bias
        fc1_out = tf.nn.relu(fc1_in)
    
    with tf.variable_scope(name_or_scope='fully_connected_layer2') as scope:
        weight = variable_on_cpu_with_collection(name='weight', 
                                                 shape=(384, 192), 
                                                 dtype=tf.float32,
                                                 stddev=0.04,
                                                 wd=0.004)
        bias = variable_on_cpu(name='bias', shape=(192), dtype=tf.float32, initializer=tf.constant_initializer(value=0.1))
        fc2_in = tf.matmul(fc1_out, weight)+bias
        fc2_out = tf.nn.relu(fc2_in)    
    
    wi

(iii) モデル学習部 (train.py)

       最適化アルゴリズムとしては、ミニバッチを用いた確率的勾配降下法（ミニバッチはオンライン学習との相対関係）が最も一般的で、損失関数の値を最小化するモデルパラメータを求めます。オーバーフィッティングを防ぐため、ここでの損失関数には、以下のコードで説明するように正則化項が含まれています。

# -*- coding: utf-8 -*-
import input_dataset
import forward_prop
import tensorflow as tf
import os
import numpy as np
 
max_iter_num = 100000 # Set the number of parameter iterations
checkpoint_path = '. /checkpoint' # set the path to the model parameter file
event_log_path = '. /event-log' # Set the path where the event file is located, used to store the Summary cache object periodically
 
def train():
    with tf.Graph().as_default(): #Specify the current graph as the default graph
        global_step = tf.Variable(initial_value=0, trainable=False)#set trainable=False, because it prevents the global_step variable from being sliding updated during training
        img_batch, label_batch = input_dataset.preprocess_input_data()# input image preprocessing, including brightness, contrast, image flip and other operations
        # img_batch, label_batch = input_dataset.input_data(eval_flag=False)
        logits = forward_prop.network(img_batch) # forward propagation process of image signal
        total_loss = forward_prop.loss(logits, label_batch) # calculate loss
        one_step_gradient_update = forward_prop.one_step_train(total_loss, global_step) # return one step gradient update operation
        #Create a saver object to save the parameters to a file
        saver = tf.train.Saver(var_list=tf.all_variables()) #tf.all_variables return a list of `Variable` objects        
        all_summary_obj = tf.merge_all_summaries() # return all summary objects first merge then serialize the string type tensor
        initiate_variables = tf.initialize_all_variables()        
# log_device_placement parameter can record the device used for each operation, here the operation is more, it does not need to record, so set to False
        Session(config=tf.ConfigProto(log_device_placement=False)) as sess:
            sess.run(initiate_variables) #Variable initialization           
            tf.train.start_queue_runners(sess=sess) # start all queuerunners
            Event_writer = tf.train.SummaryWriter(logdir=event_log_path, graph=sess.graph) 
            for step in range(max_iter_num):
                _, loss_value = sess.run(fetches=[one_step_gradient_update, total_loss])
                assert not np.isnan(loss_value) # used to verify that the loss_value calculated by the current iteration is reasonable
                if step%10 == 0:
                    print('step %d, the loss_value is %.2f' % (step, loss_value))
                if step%100 == 0:
                    # Add the `Summary` protocol cache to the event file, so you can't write the total_loss variable to the event file, because the total_loss here is a normal tensor type
                    all_summaries = sess.run(all_summary_obj)
                    Event_writer.add_summary(summary=all_summaries, global_step=step)
                if step%1000 == 0 or (step+1)==max_iter_num:
                    variables_save_path = os.path.join(checkpoint_path, 'model-parameters.bin') # path merge, return merged string
                    saver.save(sess, variables_save_path, global_step=step)#Save all variables (including model parameters before and after moving average) in variables_save_path path                 
if __name__ == '__main__':
    train()                

(iv) モデル性能評価部(evaluate.py)

       機械学習モデルの学習後、テストデータセットでテストしてモデルの性能を判定する必要があり、一般的な性能指標は精度、再現率などである。ちなみに、機械学習モデルの中には、学習時にデータセットを学習データセット、検証データセット、テストデータセットの3つに分けるものがあり、正則化セットの役割もオーバーフィットを防ぐことですが、ここでは、モデルパラメータを正則化することでオーバーフィットを防ぎます。そのため、このようにデータセットを分割する必要はなく、具体的なコードと説明は以下の通りです。

# -*- coding: utf-8 -*-
import tensorflow as tf
import input_dataset
import forward_prop
import train
import math
import numpy as np
 
def eval_once(summary_op, summary_writer, saver, predict_true_or_false):
    with tf.Session() as sess:
# return checkpointstate template from checkpoint file
        checkpoint_proto = tf.train.get_checkpoint_state(checkpoint_dir=train.checkpoint_path)
        if checkpoint_proto and checkpoint_proto.model_checkpoint_path:
            saver.restore(sess, checkpoint_proto.model_checkpoint_path)#restore model variables to the current session
        else:
            print('checkpoint file not found!')
            return
        # Start many threads and pass coordinator to each one    
        coord = tf.train.Coordinator() # return a coordinator class object that implements a simple mechanism for coordinating the end of many threads
        try:
            threads = [] # use coord to unify all threads
            for queue_runner in tf.get_collection(key=tf.GraphKeys.QUEUE_RUNNERS):
                threads.extend(queue_runner.create_threads(sess, coord=coord, daemon=True, start=True))
# Calculate the number of test batches, rounding up
            test_batch_num = math.ceil(input_dataset.test_samples_per_epoch/input_dataset.batch_size)
            iter_num = 0
            true_test_num = 0
# here use the number of test data blocks after rounding to calculate the total number of test samples, theoretically the total number of test samples will be large ah, not yet understood?
            total_test_num = test_batch_num*input_dataset.batch_size
            
            while iter_num<test_batch_num and not coord.should_stop():
                result_judge = sess.run([predict_true_or_false])
                true_test_num += np.sum(result_judge)
                iter_num += 1
            precision = true_test_num/total_test_num
            print("The test precision is %.3f" % precision)
        except:
            coord.request_stop()
        coord.request_stop()
        coord.join(threads)
                
def evaluate():
    with tf.Graph().as_default() as g:
        img_batch, labels = input_dataset.input_data(eval_flag=True)# read in the test dataset
        logits = forward_prop.network(img_batch)# use the model parameters before the moving average operation to calculate the model output values
# determine if targets are in the first k predictions, when k=1 is equivalent to the regular method of calculating correctness, sess.run(predict_true_or_false) will perform the symbolic calculation
        predict_true_or_false = tf.nn.in_top_k(predictions=logits, targets=labels, k=1)
        # recover the model parameters after the moving average operation
        moving_average_op = tf.train.ExponentialMovingAverage(decay=forward_prop.moving_average_decay)
# returns the mapping of names to Variables to be recovered, i.e. a map mapping. If a variable has a moving average, use the moving average variable name as the restore
# name, otherwise use the variable name
        variables_to_restore = moving_average_op.variables_to_restore()
        saver = tf.train.Saver(var_list=variables_to_restore)
        
        summary_op = tf.merge_all_summaries() # Create serialized summary object
#Create an event file for writing summary objects to a file in the logdir directory later
        summary_writer = tf.train. /event-log-test', graph=g)
        eval_once(summary_op, summary_writer, saver, predict_true_or_false)

また、プログラムそのものを書く過程で、いくつかのエラーのヒントが出てきましたので、今後の参考のためにここに記録しておきます。

(1)"SyntaxError:位置引数がキーワード引数の後にあります"。

<ブロッククオート

エラーの原因 Python でサブファンクションに引数を渡すとき、実引数はすべてキーワード引数を使うか、すべて位置引数を使います。キーワード引数と位置引数の両方を使う場合、次の例のように位置引数を先に、キーワード引数を後にする必要があります。

              def test(a, b, c):

                  return a+b+c

            そうすると、次の3つの呼び出しはすべて正しい、つまり

            test(1,2,c=3)

            テスト(1,2,3)  

            test(a=1, b=2, c=3) 

(2) tf.Graph.as_default()" と記述すると、エラー "TypeError: as_default() missing 1 required positional argument: 'self' " が発生しました。

<ブロッククオート

エラー理由：tf.Graph().as_default()とする必要があります。

(3)tensorboardを開くと "inner server error" "となる。

<ブロッククオート

エラーの原因 私のパソコンでは提灯プロキシを有効にしているため、tensorboardの起動時にサーバーエラーが発生する

(4) エラー "TypeError: int() の引数は 'Variable'" ではなく、文字列、バイトのようなオブジェクト、または数値でなければなりません。

<ブロッククオート

エラーの理由 tf.get_variable() の shape 引数はテンソル型ではありえないので、以下の2つのケースを区別することで理解できる。

ケース1：reshape = tf.reshape(pool2, [batch_size, -1])

     dim = reshape.get_shape()[1].value #reshape.get_shape()[1] は次元型のテンソルなので、そのvalueプロパティを取ればint型の値が得られる。

ケース2：reshaped_pool2 = tf.reshape(tensor=pool2, shape=(batch_size, -1))

dim = tf.shape(reshaped_pool2)[1] #dimはテンソル型です。

また、tf.get_variable(name, shape=[dim, 384], initializer=initializer, dtype=dtype) 関数の shape パラメータは ndarray 型でなければならないので、以下のようになります。

(5) 以下のように、各反復において、出力される損失値が大きすぎる。

    step 0では、loss_valueは22439.82です。

    ステップ10では、損失額は6426354679171382723219403309056.00です。

    エラーです。重み付けパラメータwを設定する際、定数に対して標準偏差が大きすぎると判断されました。

     If set weight = tf.get_variable(name=name, shape=shape, initializer=tf.truncated_normal_initializer(stddev=0.5, dtype=dtype)))

(6) このプログラムはtensorflow version 0.8.0でも動作するが、tf.train.batchとtf.train.shuffle_batch関数内のキーワードtensorsをtensor_listに変更する必要がある。

(7) エラー "UnboundLocalError: 代入前にローカル変数 'CONSTANT' を参照しました"

エラーの原因：次のコードを参照してください。

CONSTANT = 0  

def modifyConstant() :  

         print CONSTANT  

      CONSTANT += 1

このエラーは、変数 CONSTANT が関数内部で変更され、Python が CONSTANT をローカル変数とみなし、print CONSTANT が CONSTANT += 1 よりも前にある場合に発生します。

関数内でグローバル変数にアクセスし、変更する必要がある場合は、関数内でキーワード global を使用して変数 CONSTANT を宣言する必要があります。

(8) データの格納順に従ってテンソルのサイズを変更する tf.reshape() 関数と、空間的な観点から次元を回転させる tf.translate() 関数の区別に注意すること。

参考：https://www.tensorflow.org/versions/r0.11/tutorials/deep_cnn/index.html#convolutional-neural-networks