TensorFlow入門学習（機械・アルゴリズムに選択を手伝ってもらう）

0. Personal understanding
1. basic use
2. Introduction to MNIST (multiclass classification)
3. deeper into MNIST
4. convolutional neural network: classification of CIFAR-10 dataset
5. Vector Representations of Words (VOR)
6. Recurrent Neural Networks (RNN), LSTM (Long-Short Term Memory, LSTM)
7. building a chatbot with deep learning networks 

1. deep learning neural network is essentially in lean what, I think is in learn a set of parameters, or selection mode, which is often said to be a classifier, this classifier may be a high-dimensional classifier, composed of a set of parameters
2. take image captcha recognition, the parameters here means the distribution of weights in the image area (the distribution of weight pixel space is different for number 1 and number 2), if we select the pixel space of the image (e.g. 32 * 32) + RGB color channel (3) as input features (essentially this is feature engineering), these features will be treated as neurons by tensorflow and at each layer these neurons are combined and the results are calculated, while the neurons of the next layer of the neural network, the output of this layer will be combined again, the combination, according to the accuracy of the last prediction, will automatically give each combination a different weight (weight) by back propogation, this process will continue until the adjustment of a best-fit weight, which is often the pixel space weight that is closest to the real image
3. . All things in the world can be abstracted into a high-dimensional matrix, and this process will have different ways of extracting abstraction in different domains, i.e., feature engineering, and it is worth noting that having expertise in the corresponding domain is very helpful for the implementation of feature engineering
4. after we abstract the domain-specific, objects to be classified/recognized as a high-dimensional matrix, into the deep learn algorithm model becomes a neuron (node), the next thing to do in the deep learn model is called "fitting"
5. To complete the classification and recognition, the goal of deep learn is to find a fit matrix (figuratively speaking, "high-dimensional-1 classification cut "), to achieve this purpose, 3 elements are needed
　　1) the fit function (activation function): used to generate the fitted cut
　　2) error function (Loss Function): used in the process of network computation to calculate the distance of the fitted cut from the optimal value of the current fitted parameters, in order to adjust the parameters at any time
　　3) neural network structure: deep learn deep learning and ordinary neural networks is the difference between "layers ", deep neural networks often have more than 3 layers (input layer, hidden layer, output layer), when the number of layers increases, in each layer to choose how the combination of cross structure becomes a difficult thing, there is no sound theoretical support can accurately There is no sound theoretical support to precisely calculate what kind of network structure can output the best results, and the usual practice is to keep trying different network structures according to the business scenario that we do not agree with, until we "try out" a relatively good network structure, and then make parameter adjustments on top of this network structure

http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=spiral®Dataset=reg-plane&learningRate=0.03® ularizationRate=0&noise=0&networkShape=8,8,8,8,8,8,8&seed=0.33671&showTestData=false&discretize=false& percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false& cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&hideText=false

One, the neural network understands how to decouple the input space into a hierarchical set of convolutional filters
Second, the neural network understands the probabilistic mapping from a set of filters to a set of specific labels. What the neural network learns is not at all what it means to "see" in the human sense, and it certainly does not mean, in scientific terms, that we have solved the problem of computer vision

https://groups.google.com/a/tensorflow.org/forum/#!forum/discuss
https://stackoverflow.com/questions/tagged/tensorflow
http://www.tensorfly.cn/tfdoc/resources/overview.html
https://www.zhihu.com/question/41667903
http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=spiral®Dataset=reg-plane&learningRate=0.03® ularizationRate=0&noise=0&networkShape=8,8,8,8,8,8,8&seed=0.33671&showTestData=false&discretize=false& percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false& cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&hideText=false

1. Use a graph to represent a computational task. 2.
2. Execute the graph in a context called a session. 3.
3. Use tensor to represent data. 4.
4. maintain state through variables. 5.
5. use feed and fetch to assign values to or retrieve data from arbitrary operations.

# -*- coding:utf-8 -*-

import tensorflow as tf


if __name__ == "__main__":
    # Create a constant op, producing a 1x2 matrix. This op is treated as a node
    # Add to the default graph.
    #The
    # The return value of the constructor represents the return value of the constant op.
    matrix1 = tf.constant([[3., 3.]])

    # Create another constant op, producing a 2x1 matrix.
    matrix2 = tf.constant([[2.], [2.]])

    # Create a matrix multiplication matmul op , taking 'matrix1' and 'matrix2' as input.
    # The return value 'product' represents the result of the matrix multiplication.
    product = tf.matmul(matrix1, matrix2)

# -*- coding:utf-8 -*-

import tensorflow as tf

if __name__ == "__main__":
    # Create a constant op, producing a 1x2 matrix. This op is treated as a node
    # Add to the default graph.
    #The
    # The return value of the constructor represents the return value of the constant op.
    matrix1 = tf.constant([[3., 3.]])

    # Create another constant op, producing a 2x1 matrix.
    matrix2 = tf.constant([[2.], [2.]])

    # Create a matrix multiplication matmul op , taking 'matrix1' and 'matrix2' as input.
    # The return value 'product' represents the result of the matrix multiplication.
    product = tf.matmul(matrix1, matrix2)

    # The default graph now has three nodes, two constant() op, and one matmul() op. In order to actually multiply the matrices and get the result of the matrix multiplication, you must start the graph in a session.

    # Start the default graph.
    sess = tf.Session()

    # Call the 'run()' method of sess to perform the matrix multiplication op, passing 'product' as an argument to the method.
    # As mentioned above, 'product' represents the output of the matrix multiplication op, and passing it in indicates to the method that we want to retrieve
    # The output of the matrix multiplication op.
    # The entire execution is automated.
    # The entire execution is automated, and the session is responsible for passing all the input needed for the op. The op is usually executed concurrently.
    #
    # The function call 'run(product)' triggers the execution of the three op's in the diagram (two constant op's and a matrix multiplication op).
    #
    # The return value 'result' is a numpy `ndarray` object.
    result = sess.run(product)
    print result
    # ==> [[ 12.]]

    # Task complete, close the session.
    sess.close()

# -*- coding:utf-8 -*-

import tensorflow as tf

if __name__ == "__main__":
    # Create a variable, initialize to scalar 0.
    state = tf.Variable(0, name="counter")

    # Create an op, which increments state by 1
    one = tf.constant(1)
    new_value = tf.add(state, one)
    update = tf.assign(state, new_value)

    # After starting the graph, the variables must first be initialized by the `initialize` (init) op,
    # First an `initialize` op must be added to the graph.
    init_op = tf.initialize_all_variables()

    # Start the graph, run the op
    with tf.Session() as sess:
      # run 'init' op
      sess.run(init_op)
      # Print the initial value of 'state'
      print sess.run(state)
      # Run the op, update 'state', and print 'state'
      for _ in range(3):
        sess.run(update)
        print sess.run(state)

# -*- coding:utf-8 -*-

import tensorflow as tf

if __name__ == "__main__":
    # Start the default graph.
    sess = tf.Session()

    input1 = tf.constant(3.0)
    input2 = tf.constant(2.0)
    input3 = tf.constant(5.0)
    intermed = tf.add(input2, input3)
    mul = tf.multiply(input1, intermed)

    with tf.Session():
      result = sess.run([mul, intermed])
      print result

# -*- coding:utf-8 -*-

import tensorflow as tf

if __name__ == "__main__":
    input1 = tf.placeholder(tf.types.float32)
    input2 = tf.placeholder(tf.types.float32)
    output = tf.multiply(input1, input2)

    with tf.Session() as sess:
        print sess.run([output])
        print sess.run([output], feed_dict={input1:[7.], input2:[2.]})

1. the first, traverse the entire data set to calculate the loss function once, and then calculate the gradient of the function for each parameter, update the gradient. This method looks at all the samples in the dataset for each parameter update, which is computationally overhead, slow, and does not support online learning, which is called Batch gradient descent.
This is called stochastic gradient descent, which is faster, but the convergence performance is not so good, and it may wander around the optimal point and not hit the optimal point. The two parameter updates may also cancel each other out, causing the target function to oscillate more violently 

http://www.tensorfly.cn/tfdoc/get_started/os_setup.html
http://keras-cn.readthedocs.io/en/latest/getting_started/concepts/#batch

https://tensorflow.googlesource.com/tensorflow/+/master/tensorflow/examples/tutorials/mnist/input_data.py#

y = tf.nn.softmax(tf.matmul(x,W) + b)

TensorFlow has a graph that describes your individual computational units, and it can automatically use the backpropagation algorithm to efficiently determine how your variables affect the value of that cost you want to minimize. TensorFlow then uses the optimization algorithm you choose to continuously modify the variables to reduce the cost.

train_step = tf.train.GradientDescentOptimizer(0.01).minimize(cross_entropy)

correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))

accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))

print sess.run(accuracy, feed_dict={x: mnist.test.images, y_: mnist.test.labels})

# Copyright 2015 The TensorFlow Authors.
#All Rights Reserved.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for