ClientError: канал поезда не указан с помощью AWS object_detection_augmented_manifest_training с использованием наземных истинных изображений

Я завершил работу по маркировке в AWS и начал работать над шаблоном записной книжки для обнаружения объектов.

У меня есть 2 манифеста, в которых есть 293 помеченных изображения для птиц в поезде и набор проверки следующим образом:

{"source-ref":"s3://XXXXXXX/Train/Blackbird_1.JPG","Bird-Label-Train":{"workerId":XXXXXXXX,"imageSource":{"s3Uri":"s3://XXXXXXX/Train/Blackbird_1.JPG"},"boxesInfo":{"annotatedResult":{"boundingBoxes":[{"width":1612,"top":841,"label":"Blackbird","left":1276,"height":757}],"inputImageProperties":{"width":3872,"height":2592}}}},"Bird-Label-Train-metadata":{"type":"groundtruth/custom","job-name":"bird-label-train","human-annotated":"yes","creation-date":"2019-01-16T17:28:23+0000"}}

Ниже приведены параметры, которые я использую для экземпляра ноутбука:

training_params = \
{
    "AlgorithmSpecification": {
        "TrainingImage": training_image, # NB. This is one of the named constants defined in the first cell.
        "TrainingInputMode": "Pipe"
    },
    "RoleArn": role,
    "OutputDataConfig": {
        "S3OutputPath": s3_output_path
    },
    "ResourceConfig": {
        "InstanceCount": 1,   
        "InstanceType": "ml.p3.2xlarge",
        "VolumeSizeInGB": 5
    },
    "TrainingJobName": job_name,
    "HyperParameters": { # NB. These hyperparameters are at the user's discretion and are beyond the scope of this demo.
         "base_network": "resnet-50",
         "use_pretrained_model": "1",
         "num_classes": "1",
         "mini_batch_size": "16",
         "epochs": "5",
         "learning_rate": "0.001",
         "lr_scheduler_step": "3,6",
         "lr_scheduler_factor": "0.1",
         "optimizer": "rmsprop",
         "momentum": "0.9",
         "weight_decay": "0.0005",
         "overlap_threshold": "0.5",
         "nms_threshold": "0.45",
         "image_shape": "300",
         "label_width": "350",
         "num_training_samples": str(num_training_samples)
    },
    "StoppingCondition": {
        "MaxRuntimeInSeconds": 86400
    },
 "InputDataConfig": [
    {
        "ChannelName": "train",
        "DataSource": {
            "S3DataSource": {
                "S3DataType": "AugmentedManifestFile", # NB. Augmented Manifest
                "S3Uri": s3_train_data_path,
                "S3DataDistributionType": "FullyReplicated",
                "AttributeNames": ["source-ref","Bird-Label-Train"] # NB. This must correspond to the JSON field names in your augmented manifest.
            }
        },
        "ContentType": "image/jpeg",
        "RecordWrapperType": "None",
        "CompressionType": "None"
    },
    {
        "ChannelName": "validation",
        "DataSource": {
            "S3DataSource": {
                "S3DataType": "AugmentedManifestFile", # NB. Augmented Manifest
                "S3Uri": s3_validation_data_path,
                "S3DataDistributionType": "FullyReplicated",
                "AttributeNames": ["source-ref","Bird-Label"] # NB. This must correspond to the JSON field names in your augmented manifest.
            }
        },
        "ContentType": "image/jpeg",
        "RecordWrapperType": "None",
        "CompressionType": "None"
    }
]

Я бы напечатал это после запуска моего экземпляра ml.p3.2xlarge:

InProgress Starting
InProgress Starting
InProgress Starting
InProgress Training
Failed Failed

Затем следует это сообщение об ошибке: «ClientError: канал поезда не указан».

Есть ли у кого-нибудь мысли о том, как я могу запустить это без ошибок? Любая помощь очень ценится!

Успешный запуск. Ниже приведены параметры, которые использовались вместе с объектами JSON расширенного манифеста для успешного выполнения.

training_params = \
{
    "AlgorithmSpecification": {
        "TrainingImage": training_image, # NB. This is one of the named constants defined in the first cell.
        "TrainingInputMode": "Pipe"
    },
    "RoleArn": role,
    "OutputDataConfig": {
        "S3OutputPath": s3_output_path
    },
    "ResourceConfig": {
        "InstanceCount": 1,   
        "InstanceType": "ml.p3.2xlarge",
        "VolumeSizeInGB": 50
    },
    "TrainingJobName": job_name,
    "HyperParameters": { # NB. These hyperparameters are at the user's discretion and are beyond the scope of this demo.
         "base_network": "resnet-50",
         "use_pretrained_model": "1",
         "num_classes": "3",
         "mini_batch_size": "1",
         "epochs": "5",
         "learning_rate": "0.001",
         "lr_scheduler_step": "3,6",
         "lr_scheduler_factor": "0.1",
         "optimizer": "rmsprop",
         "momentum": "0.9",
         "weight_decay": "0.0005",
         "overlap_threshold": "0.5",
         "nms_threshold": "0.45",
         "image_shape": "300",
         "label_width": "350",
         "num_training_samples": str(num_training_samples)
    },
    "StoppingCondition": {
        "MaxRuntimeInSeconds": 86400
    },
    "InputDataConfig": [
        {
            "ChannelName": "train",
            "DataSource": {
                "S3DataSource": {
                    "S3DataType": "AugmentedManifestFile", # NB. Augmented Manifest
                    "S3Uri": s3_train_data_path,
                    "S3DataDistributionType": "FullyReplicated",
                    "AttributeNames": attribute_names # NB. This must correspond to the JSON field names in your **TRAIN** augmented manifest.
                }
            },
            "ContentType": "application/x-recordio",
            "RecordWrapperType": "RecordIO",
            "CompressionType": "None"
        },
        {
            "ChannelName": "validation",
            "DataSource": {
                "S3DataSource": {
                    "S3DataType": "AugmentedManifestFile", # NB. Augmented Manifest
                    "S3Uri": s3_validation_data_path,
                    "S3DataDistributionType": "FullyReplicated",
                    "AttributeNames": ["source-ref","ValidateBird"] # NB. This must correspond to the JSON field names in your **VALIDATION** augmented manifest.
                }
            },
            "ContentType": "application/x-recordio",
            "RecordWrapperType": "RecordIO",
            "CompressionType": "None"
        }
    ]
}

Расширенный файл манифеста обучения, созданный во время выполнения задания на обучение

Line 1
{"source-ref":"s3://XXXXX/Train/Blackbird_1.JPG","TrainBird":{"annotations":[{"class_id":0,"width":1613,"top":840,"height":766,"left":1293}],"image_size":[{"width":3872,"depth":3,"height":2592}]},"TrainBird-metadata":{"job-name":"labeling-job/trainbird","class-map":{"0":"Blackbird"},"human-annotated":"yes","objects":[{"confidence":0.09}],"creation-date":"2019-02-09T14:21:29.829003","type":"groundtruth/object-detection"}}


Line 2
{"source-ref":"s3://xxxxx/Train/Blackbird_2.JPG","TrainBird":{"annotations":[{"class_id":0,"width":897,"top":665,"height":1601,"left":1598}],"image_size":[{"width":3872,"depth":3,"height":2592}]},"TrainBird-metadata":{"job-name":"labeling-job/trainbird","class-map":{"0":"Blackbird"},"human-annotated":"yes","objects":[{"confidence":0.09}],"creation-date":"2019-02-09T14:22:34.502274","type":"groundtruth/object-detection"}}


Line 3
{"source-ref":"s3://XXXXX/Train/Blackbird_3.JPG","TrainBird":{"annotations":[{"class_id":0,"width":1040,"top":509,"height":1695,"left":1548}],"image_size":[{"width":3872,"depth":3,"height":2592}]},"TrainBird-metadata":{"job-name":"labeling-job/trainbird","class-map":{"0":"Blackbird"},"human-annotated":"yes","objects":[{"confidence":0.09}],"creation-date":"2019-02-09T14:20:26.660164","type":"groundtruth/object-detection"}}

Затем я распаковываю файл model.tar, чтобы получить следующие файлы: hyperparams.JSON, model_algo_1-0000.params и model_algo_1-symbol.

hyperparams.JSON выглядит так:

{"label_width": "350", "early_stopping_min_epochs": "10", "epochs": "5", "overlap_threshold": "0.5", "lr_scheduler_factor": "0.1", "_num_kv_servers": "auto", "weight_decay": "0.0005", "mini_batch_size": "1", "use_pretrained_model": "1", "freeze_layer_pattern": "", "lr_scheduler_step": "3,6", "early_stopping": "False", "early_stopping_patience": "5", "momentum": "0.9", "num_training_samples": "11", "optimizer": "rmsprop", "_tuning_objective_metric": "", "early_stopping_tolerance": "0.0", "learning_rate": "0.001", "kv_store": "device", "nms_threshold": "0.45", "num_classes": "1", "base_network": "resnet-50", "nms_topk": "400", "_kvstore": "device", "image_shape": "300"}