How to suspend / resume the `model_main.py` based training in TensorFlow?

dmitrii_fediuk · May 20, 2019, 2:06am

Step 1

Stop the model_main.py process by CTRL+C.

Step 2

Go to model_dir and rename the files of the last checkpoint:

model.ckpt-<step>.meta => model.ckpt.meta
model.ckpt-<step>.index => model.ckpt.index
model.ckpt-<step>.data-00000-of-00001 => model.ckpt.data-00000-of-00001

Step 3

Set the fine_tune_checkpoint parameter to the directory where your last checkpoint is located (e.g., model_dir).

github.com

dmitry-ai/tf-tutorial-pascal-voc/blob/0.0.7/faster_rcnn_resnet101_voc07.pbtxt#L104-L104


	fine_tune_checkpoint: "/w/tutorials/tensorflow/pascal-voc/run/2019-05-17/model.ckpt"

from_detection_checkpoint should be set to true:

Step 4

Run the model_main.py process again with a new model_dir:

github.com

dmitry-ai/tf-tutorial-pascal-voc/blob/0.0.7/training.sh#L2-L8


objectDetection='/w/tools/tensorflow/models/research/object_detection/'
python ${objectDetection}model_main.py \
	--pipeline_config_path=faster_rcnn_resnet101_voc07.pbtxt \
	--model_dir=../run/2019-05-20 \
	--num_train_steps=10000 \
	--sample_1_of_n_eval_examples=1 \
	--alsologtostderr