How to suspend / resume the `` based training in TensorFlow?

(Dmitry Fedyuk) #1

Step 1

Stop the process by CTRL+C.

Step 2

Go to model_dir and rename the files of the last checkpoint:

  • model.ckpt-<step>.meta => model.ckpt.meta
  • model.ckpt-<step>.index => model.ckpt.index
  • model.ckpt-<step>.data-00000-of-00001 =>

Step 3

Set the fine_tune_checkpoint parameter to the directory where your last checkpoint is located (e.g., model_dir).

from_detection_checkpoint should be set to true:

Step 4

Run the process again with a new model_dir: