Why does TensorFlow start learning from the 1800th step?

dmitrii_fediuk · May 21, 2019, 3:20am

dmitrii_fediuk · May 21, 2019, 3:22am

It could be because of this:

Typically with quantization, a model will train with full precision for a certain number of steps before switching to quantized training.
The delay number above tells ML Engine to begin quantizing our weights and activations after 1800 training steps.

medium.com/tensorflow/training-and-serving-a-realtime-mobile-object-detector-in-30-minutes-with-cloud-tpus-b78971cf1193

But I have never set it to 1800 explicitly, and default it 500000:

github.com

tensorflow/models/blob/v1.13.0/research/object_detection/protos/graph_rewriter.proto#L5-L14


      
          // Message to configure graph rewriter for the tf graph.
          message GraphRewriter {
            optional Quantization quantization = 1;
          }
          
          // Message for quantization options. See
          // tensorflow/contrib/quantize/python/quantize.py for details.
          message Quantization {
            // Number of steps to delay before quantization takes effect during training.
            optional int32 delay = 1 [default = 500000];