docs: update docs/dyn (#1096)
This PR was generated using Autosynth. :rainbow:
Synth log will be available here:
https://source.cloud.google.com/results/invocations/6f0f288a-a1e8-4b2d-a85f-00b1c6150185/targets
- [ ] To automatically regenerate this PR, check this box.
Source-Link: https://github.com/googleapis/synthtool/commit/39b7149da4026765385403632db3c6f63db96b2c
Source-Link: https://github.com/googleapis/synthtool/commit/9a7d9fbb7045c34c9d3d22c1ff766eeae51f04c9
Source-Link: https://github.com/googleapis/synthtool/commit/dc9903a8c30c3662b6098f0e4a97f221d67268b2
Source-Link: https://github.com/googleapis/synthtool/commit/7fcc405a579d5d53a726ff3da1b7c8c08f0f2d58
Source-Link: https://github.com/googleapis/synthtool/commit/d5fc0bcf9ea9789c5b0e3154a9e3b29e5cea6116
Source-Link: https://github.com/googleapis/synthtool/commit/e89175cf074dccc4babb4eca66ae913696e47a71
Source-Link: https://github.com/googleapis/synthtool/commit/7d652819519dfa24da9e14548232e4aaba71a11c
Source-Link: https://github.com/googleapis/synthtool/commit/7db8a6c5ffb12a6e4c2f799c18f00f7f3d60e279
Source-Link: https://github.com/googleapis/synthtool/commit/1f1148d3c7a7a52f0c98077f976bd9b3c948ee2b
Source-Link: https://github.com/googleapis/synthtool/commit/2c8aecedd55b0480fb4e123b6e07fa5b12953862
Source-Link: https://github.com/googleapis/synthtool/commit/3d3e94c4e02370f307a9a200b0c743c3d8d19f29
Source-Link: https://github.com/googleapis/synthtool/commit/c7824ea48ff6d4d42dfae0849aec8a85acd90bd9
Source-Link: https://github.com/googleapis/synthtool/commit/ba9918cd22874245b55734f57470c719b577e591
Source-Link: https://github.com/googleapis/synthtool/commit/b19b401571e77192f8dd38eab5fb2300a0de9324
Source-Link: https://github.com/googleapis/synthtool/commit/6542bd723403513626f61642fc02ddca528409aa
diff --git a/docs/dyn/datalabeling_v1beta1.projects.evaluationJobs.html b/docs/dyn/datalabeling_v1beta1.projects.evaluationJobs.html
index f40dacd..12aab21 100644
--- a/docs/dyn/datalabeling_v1beta1.projects.evaluationJobs.html
+++ b/docs/dyn/datalabeling_v1beta1.projects.evaluationJobs.html
@@ -87,7 +87,7 @@
<code><a href="#get">get(name, x__xgafv=None)</a></code></p>
<p class="firstline">Gets an evaluation job by resource name.</p>
<p class="toc_element">
- <code><a href="#list">list(parent, pageSize=None, filter=None, pageToken=None, x__xgafv=None)</a></code></p>
+ <code><a href="#list">list(parent, pageToken=None, pageSize=None, filter=None, x__xgafv=None)</a></code></p>
<p class="firstline">Lists all evaluation jobs within a project with possible filters. Pagination is supported.</p>
<p class="toc_element">
<code><a href="#list_next">list_next(previous_request, previous_response)</a></code></p>
@@ -118,76 +118,15 @@
{ # Request message for CreateEvaluationJob.
"job": { # Defines an evaluation job that runs periodically to generate Evaluations. [Creating an evaluation job](/ml-engine/docs/continuous-evaluation/create-job) is the starting point for using continuous evaluation. # Required. The evaluation job to create.
- "annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
- "evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
- "exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
- "exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
- "evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
- "email": "A String", # Required. An email address to send alerts to.
- "minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
- },
- "evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
- "boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
- "iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
- },
- },
- "boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
- "instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- },
- "humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
- "questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
- "annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
- "labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
- "userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
- "contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
- "A String",
- ],
- "annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
- "languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
- "replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
- "instruction": "A String", # Required. Instruction resource name.
- },
- "textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
- "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- "sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
- "enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
- },
- },
- "imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
- "answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
- "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- },
- "bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
- "a_key": "A String",
- },
- "inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
- "gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
- "mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
- "inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
- },
- "bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
- "inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
- },
- "annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
- "classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
- "isMultiLabel": True or False, # Whether the classification task is multi-label or not.
- },
- "textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
- "languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
- },
- "dataType": "A String", # Required. Data type must be specifed when user tries to import data.
- },
- },
- "createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
+ "modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
"name": "A String", # Output only. After you create a job, Data Labeling Service assigns a name to the job with the following format: "projects/{project_id}/evaluationJobs/ {evaluation_job_id}"
+ "createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
"schedule": "A String", # Required. Describes the interval at which the job runs. This interval must be at least 1 day, and it is rounded to the nearest day. For example, if you specify a 50-hour interval, the job runs every 2 days. You can provide the schedule in [crontab format](/scheduler/docs/configuring/cron-job-schedules) or in an [English-like format](/appengine/docs/standard/python/config/cronref#schedule_format). Regardless of what you specify, the job will run at 10:00 AM UTC. Only the interval from this schedule is used, not the specific time of day.
"state": "A String", # Output only. Describes the current state of the job.
"labelMissingGroundTruth": True or False, # Required. Whether you want Data Labeling Service to provide ground truth labels for prediction input. If you want the service to assign human labelers to annotate your data, set this to `true`. If you want to provide your own ground truth labels in the evaluation job's BigQuery table, set this to `false`.
"attempts": [ # Output only. Every time the evaluation job runs and an error occurs, the failed attempt is appended to this array.
{ # Records a failed evaluation job run.
+ "attemptTime": "A String",
"partialFailures": [ # Details of errors that occurred.
{ # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors).
"details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
@@ -199,11 +138,72 @@
"code": 42, # The status code, which should be an enum value of google.rpc.Code.
},
],
- "attemptTime": "A String",
},
],
+ "annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
+ "evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
+ "inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
+ "gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
+ "inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
+ "mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
+ },
+ "bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
+ "inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
+ },
+ "annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
+ "textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
+ "languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
+ },
+ "classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
+ "isMultiLabel": True or False, # Whether the classification task is multi-label or not.
+ },
+ "dataType": "A String", # Required. Data type must be specifed when user tries to import data.
+ },
+ "exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
+ "textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
+ "sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
+ "enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
+ },
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
+ },
+ "boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
+ },
+ "evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
+ "boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
+ "iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
+ },
+ },
+ "humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
+ "questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
+ "languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
+ "userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
+ "instruction": "A String", # Required. Instruction resource name.
+ "contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
+ "A String",
+ ],
+ "labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
+ "replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
+ "annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
+ "annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
+ },
+ "exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
+ "imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
+ "answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
+ },
+ "evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
+ "email": "A String", # Required. An email address to send alerts to.
+ "minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
+ },
+ "bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
+ "a_key": "A String",
+ },
+ },
"description": "A String", # Required. Description of the job. The description can be up to 25,000 characters long.
- "modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
},
}
@@ -216,76 +216,15 @@
An object of the form:
{ # Defines an evaluation job that runs periodically to generate Evaluations. [Creating an evaluation job](/ml-engine/docs/continuous-evaluation/create-job) is the starting point for using continuous evaluation.
- "annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
- "evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
- "exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
- "exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
- "evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
- "email": "A String", # Required. An email address to send alerts to.
- "minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
- },
- "evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
- "boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
- "iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
- },
- },
- "boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
- "instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- },
- "humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
- "questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
- "annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
- "labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
- "userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
- "contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
- "A String",
- ],
- "annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
- "languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
- "replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
- "instruction": "A String", # Required. Instruction resource name.
- },
- "textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
- "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- "sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
- "enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
- },
- },
- "imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
- "answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
- "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- },
- "bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
- "a_key": "A String",
- },
- "inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
- "gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
- "mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
- "inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
- },
- "bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
- "inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
- },
- "annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
- "classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
- "isMultiLabel": True or False, # Whether the classification task is multi-label or not.
- },
- "textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
- "languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
- },
- "dataType": "A String", # Required. Data type must be specifed when user tries to import data.
- },
- },
- "createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
+ "modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
"name": "A String", # Output only. After you create a job, Data Labeling Service assigns a name to the job with the following format: "projects/{project_id}/evaluationJobs/ {evaluation_job_id}"
+ "createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
"schedule": "A String", # Required. Describes the interval at which the job runs. This interval must be at least 1 day, and it is rounded to the nearest day. For example, if you specify a 50-hour interval, the job runs every 2 days. You can provide the schedule in [crontab format](/scheduler/docs/configuring/cron-job-schedules) or in an [English-like format](/appengine/docs/standard/python/config/cronref#schedule_format). Regardless of what you specify, the job will run at 10:00 AM UTC. Only the interval from this schedule is used, not the specific time of day.
"state": "A String", # Output only. Describes the current state of the job.
"labelMissingGroundTruth": True or False, # Required. Whether you want Data Labeling Service to provide ground truth labels for prediction input. If you want the service to assign human labelers to annotate your data, set this to `true`. If you want to provide your own ground truth labels in the evaluation job's BigQuery table, set this to `false`.
"attempts": [ # Output only. Every time the evaluation job runs and an error occurs, the failed attempt is appended to this array.
{ # Records a failed evaluation job run.
+ "attemptTime": "A String",
"partialFailures": [ # Details of errors that occurred.
{ # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors).
"details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
@@ -297,11 +236,72 @@
"code": 42, # The status code, which should be an enum value of google.rpc.Code.
},
],
- "attemptTime": "A String",
},
],
+ "annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
+ "evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
+ "inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
+ "gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
+ "inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
+ "mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
+ },
+ "bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
+ "inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
+ },
+ "annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
+ "textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
+ "languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
+ },
+ "classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
+ "isMultiLabel": True or False, # Whether the classification task is multi-label or not.
+ },
+ "dataType": "A String", # Required. Data type must be specifed when user tries to import data.
+ },
+ "exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
+ "textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
+ "sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
+ "enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
+ },
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
+ },
+ "boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
+ },
+ "evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
+ "boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
+ "iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
+ },
+ },
+ "humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
+ "questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
+ "languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
+ "userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
+ "instruction": "A String", # Required. Instruction resource name.
+ "contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
+ "A String",
+ ],
+ "labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
+ "replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
+ "annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
+ "annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
+ },
+ "exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
+ "imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
+ "answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
+ },
+ "evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
+ "email": "A String", # Required. An email address to send alerts to.
+ "minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
+ },
+ "bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
+ "a_key": "A String",
+ },
+ },
"description": "A String", # Required. Description of the job. The description can be up to 25,000 characters long.
- "modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
}</pre>
</div>
@@ -338,76 +338,15 @@
An object of the form:
{ # Defines an evaluation job that runs periodically to generate Evaluations. [Creating an evaluation job](/ml-engine/docs/continuous-evaluation/create-job) is the starting point for using continuous evaluation.
- "annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
- "evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
- "exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
- "exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
- "evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
- "email": "A String", # Required. An email address to send alerts to.
- "minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
- },
- "evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
- "boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
- "iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
- },
- },
- "boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
- "instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- },
- "humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
- "questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
- "annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
- "labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
- "userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
- "contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
- "A String",
- ],
- "annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
- "languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
- "replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
- "instruction": "A String", # Required. Instruction resource name.
- },
- "textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
- "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- "sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
- "enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
- },
- },
- "imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
- "answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
- "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- },
- "bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
- "a_key": "A String",
- },
- "inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
- "gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
- "mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
- "inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
- },
- "bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
- "inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
- },
- "annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
- "classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
- "isMultiLabel": True or False, # Whether the classification task is multi-label or not.
- },
- "textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
- "languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
- },
- "dataType": "A String", # Required. Data type must be specifed when user tries to import data.
- },
- },
- "createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
+ "modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
"name": "A String", # Output only. After you create a job, Data Labeling Service assigns a name to the job with the following format: "projects/{project_id}/evaluationJobs/ {evaluation_job_id}"
+ "createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
"schedule": "A String", # Required. Describes the interval at which the job runs. This interval must be at least 1 day, and it is rounded to the nearest day. For example, if you specify a 50-hour interval, the job runs every 2 days. You can provide the schedule in [crontab format](/scheduler/docs/configuring/cron-job-schedules) or in an [English-like format](/appengine/docs/standard/python/config/cronref#schedule_format). Regardless of what you specify, the job will run at 10:00 AM UTC. Only the interval from this schedule is used, not the specific time of day.
"state": "A String", # Output only. Describes the current state of the job.
"labelMissingGroundTruth": True or False, # Required. Whether you want Data Labeling Service to provide ground truth labels for prediction input. If you want the service to assign human labelers to annotate your data, set this to `true`. If you want to provide your own ground truth labels in the evaluation job's BigQuery table, set this to `false`.
"attempts": [ # Output only. Every time the evaluation job runs and an error occurs, the failed attempt is appended to this array.
{ # Records a failed evaluation job run.
+ "attemptTime": "A String",
"partialFailures": [ # Details of errors that occurred.
{ # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors).
"details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
@@ -419,23 +358,84 @@
"code": 42, # The status code, which should be an enum value of google.rpc.Code.
},
],
- "attemptTime": "A String",
},
],
+ "annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
+ "evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
+ "inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
+ "gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
+ "inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
+ "mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
+ },
+ "bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
+ "inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
+ },
+ "annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
+ "textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
+ "languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
+ },
+ "classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
+ "isMultiLabel": True or False, # Whether the classification task is multi-label or not.
+ },
+ "dataType": "A String", # Required. Data type must be specifed when user tries to import data.
+ },
+ "exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
+ "textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
+ "sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
+ "enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
+ },
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
+ },
+ "boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
+ },
+ "evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
+ "boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
+ "iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
+ },
+ },
+ "humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
+ "questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
+ "languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
+ "userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
+ "instruction": "A String", # Required. Instruction resource name.
+ "contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
+ "A String",
+ ],
+ "labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
+ "replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
+ "annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
+ "annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
+ },
+ "exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
+ "imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
+ "answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
+ },
+ "evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
+ "email": "A String", # Required. An email address to send alerts to.
+ "minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
+ },
+ "bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
+ "a_key": "A String",
+ },
+ },
"description": "A String", # Required. Description of the job. The description can be up to 25,000 characters long.
- "modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
}</pre>
</div>
<div class="method">
- <code class="details" id="list">list(parent, pageSize=None, filter=None, pageToken=None, x__xgafv=None)</code>
+ <code class="details" id="list">list(parent, pageToken=None, pageSize=None, filter=None, x__xgafv=None)</code>
<pre>Lists all evaluation jobs within a project with possible filters. Pagination is supported.
Args:
parent: string, Required. Evaluation job resource parent. Format: "projects/{project_id}" (required)
+ pageToken: string, Optional. A token identifying a page of results for the server to return. Typically obtained by the nextPageToken in the response to the previous request. The request returns the first page if this is empty.
pageSize: integer, Optional. Requested page size. Server may return fewer results than requested. Default value is 100.
filter: string, Optional. You can filter the jobs to list by model_id (also known as model_name, as described in EvaluationJob.modelVersion) or by evaluation job state (as described in EvaluationJob.state). To filter by both criteria, use the `AND` operator or the `OR` operator. For example, you can use the following string for your filter: "evaluation_job.model_id = {model_name} AND evaluation_job.state = {evaluation_job_state}"
- pageToken: string, Optional. A token identifying a page of results for the server to return. Typically obtained by the nextPageToken in the response to the previous request. The request returns the first page if this is empty.
x__xgafv: string, V1 error format.
Allowed values
1 - v1 error format
@@ -445,79 +445,17 @@
An object of the form:
{ # Results for listing evaluation jobs.
- "nextPageToken": "A String", # A token to retrieve next page of results.
"evaluationJobs": [ # The list of evaluation jobs to return.
{ # Defines an evaluation job that runs periodically to generate Evaluations. [Creating an evaluation job](/ml-engine/docs/continuous-evaluation/create-job) is the starting point for using continuous evaluation.
- "annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
- "evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
- "exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
- "exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
- "evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
- "email": "A String", # Required. An email address to send alerts to.
- "minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
- },
- "evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
- "boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
- "iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
- },
- },
- "boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
- "instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- },
- "humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
- "questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
- "annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
- "labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
- "userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
- "contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
- "A String",
- ],
- "annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
- "languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
- "replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
- "instruction": "A String", # Required. Instruction resource name.
- },
- "textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
- "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- "sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
- "enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
- },
- },
- "imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
- "answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
- "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- },
- "bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
- "a_key": "A String",
- },
- "inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
- "gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
- "mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
- "inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
- },
- "bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
- "inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
- },
- "annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
- "classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
- "isMultiLabel": True or False, # Whether the classification task is multi-label or not.
- },
- "textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
- "languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
- },
- "dataType": "A String", # Required. Data type must be specifed when user tries to import data.
- },
- },
- "createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
+ "modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
"name": "A String", # Output only. After you create a job, Data Labeling Service assigns a name to the job with the following format: "projects/{project_id}/evaluationJobs/ {evaluation_job_id}"
+ "createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
"schedule": "A String", # Required. Describes the interval at which the job runs. This interval must be at least 1 day, and it is rounded to the nearest day. For example, if you specify a 50-hour interval, the job runs every 2 days. You can provide the schedule in [crontab format](/scheduler/docs/configuring/cron-job-schedules) or in an [English-like format](/appengine/docs/standard/python/config/cronref#schedule_format). Regardless of what you specify, the job will run at 10:00 AM UTC. Only the interval from this schedule is used, not the specific time of day.
"state": "A String", # Output only. Describes the current state of the job.
"labelMissingGroundTruth": True or False, # Required. Whether you want Data Labeling Service to provide ground truth labels for prediction input. If you want the service to assign human labelers to annotate your data, set this to `true`. If you want to provide your own ground truth labels in the evaluation job's BigQuery table, set this to `false`.
"attempts": [ # Output only. Every time the evaluation job runs and an error occurs, the failed attempt is appended to this array.
{ # Records a failed evaluation job run.
+ "attemptTime": "A String",
"partialFailures": [ # Details of errors that occurred.
{ # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors).
"details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
@@ -529,13 +467,75 @@
"code": 42, # The status code, which should be an enum value of google.rpc.Code.
},
],
- "attemptTime": "A String",
},
],
+ "annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
+ "evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
+ "inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
+ "gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
+ "inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
+ "mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
+ },
+ "bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
+ "inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
+ },
+ "annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
+ "textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
+ "languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
+ },
+ "classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
+ "isMultiLabel": True or False, # Whether the classification task is multi-label or not.
+ },
+ "dataType": "A String", # Required. Data type must be specifed when user tries to import data.
+ },
+ "exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
+ "textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
+ "sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
+ "enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
+ },
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
+ },
+ "boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
+ },
+ "evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
+ "boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
+ "iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
+ },
+ },
+ "humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
+ "questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
+ "languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
+ "userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
+ "instruction": "A String", # Required. Instruction resource name.
+ "contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
+ "A String",
+ ],
+ "labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
+ "replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
+ "annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
+ "annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
+ },
+ "exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
+ "imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
+ "answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
+ },
+ "evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
+ "email": "A String", # Required. An email address to send alerts to.
+ "minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
+ },
+ "bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
+ "a_key": "A String",
+ },
+ },
"description": "A String", # Required. Description of the job. The description can be up to 25,000 characters long.
- "modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
},
],
+ "nextPageToken": "A String", # A token to retrieve next page of results.
}</pre>
</div>
@@ -563,76 +563,15 @@
The object takes the form of:
{ # Defines an evaluation job that runs periodically to generate Evaluations. [Creating an evaluation job](/ml-engine/docs/continuous-evaluation/create-job) is the starting point for using continuous evaluation.
- "annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
- "evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
- "exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
- "exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
- "evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
- "email": "A String", # Required. An email address to send alerts to.
- "minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
- },
- "evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
- "boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
- "iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
- },
- },
- "boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
- "instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- },
- "humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
- "questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
- "annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
- "labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
- "userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
- "contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
- "A String",
- ],
- "annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
- "languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
- "replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
- "instruction": "A String", # Required. Instruction resource name.
- },
- "textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
- "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- "sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
- "enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
- },
- },
- "imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
- "answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
- "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- },
- "bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
- "a_key": "A String",
- },
- "inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
- "gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
- "mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
- "inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
- },
- "bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
- "inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
- },
- "annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
- "classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
- "isMultiLabel": True or False, # Whether the classification task is multi-label or not.
- },
- "textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
- "languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
- },
- "dataType": "A String", # Required. Data type must be specifed when user tries to import data.
- },
- },
- "createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
+ "modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
"name": "A String", # Output only. After you create a job, Data Labeling Service assigns a name to the job with the following format: "projects/{project_id}/evaluationJobs/ {evaluation_job_id}"
+ "createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
"schedule": "A String", # Required. Describes the interval at which the job runs. This interval must be at least 1 day, and it is rounded to the nearest day. For example, if you specify a 50-hour interval, the job runs every 2 days. You can provide the schedule in [crontab format](/scheduler/docs/configuring/cron-job-schedules) or in an [English-like format](/appengine/docs/standard/python/config/cronref#schedule_format). Regardless of what you specify, the job will run at 10:00 AM UTC. Only the interval from this schedule is used, not the specific time of day.
"state": "A String", # Output only. Describes the current state of the job.
"labelMissingGroundTruth": True or False, # Required. Whether you want Data Labeling Service to provide ground truth labels for prediction input. If you want the service to assign human labelers to annotate your data, set this to `true`. If you want to provide your own ground truth labels in the evaluation job's BigQuery table, set this to `false`.
"attempts": [ # Output only. Every time the evaluation job runs and an error occurs, the failed attempt is appended to this array.
{ # Records a failed evaluation job run.
+ "attemptTime": "A String",
"partialFailures": [ # Details of errors that occurred.
{ # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors).
"details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
@@ -644,11 +583,72 @@
"code": 42, # The status code, which should be an enum value of google.rpc.Code.
},
],
- "attemptTime": "A String",
},
],
+ "annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
+ "evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
+ "inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
+ "gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
+ "inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
+ "mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
+ },
+ "bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
+ "inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
+ },
+ "annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
+ "textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
+ "languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
+ },
+ "classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
+ "isMultiLabel": True or False, # Whether the classification task is multi-label or not.
+ },
+ "dataType": "A String", # Required. Data type must be specifed when user tries to import data.
+ },
+ "exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
+ "textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
+ "sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
+ "enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
+ },
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
+ },
+ "boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
+ },
+ "evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
+ "boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
+ "iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
+ },
+ },
+ "humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
+ "questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
+ "languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
+ "userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
+ "instruction": "A String", # Required. Instruction resource name.
+ "contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
+ "A String",
+ ],
+ "labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
+ "replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
+ "annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
+ "annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
+ },
+ "exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
+ "imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
+ "answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
+ },
+ "evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
+ "email": "A String", # Required. An email address to send alerts to.
+ "minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
+ },
+ "bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
+ "a_key": "A String",
+ },
+ },
"description": "A String", # Required. Description of the job. The description can be up to 25,000 characters long.
- "modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
}
updateMask: string, Optional. Mask for which fields to update. You can only provide the following fields: * `evaluationJobConfig.humanAnnotationConfig.instruction` * `evaluationJobConfig.exampleCount` * `evaluationJobConfig.exampleSamplePercentage` You can provide more than one of these fields by separating them with commas.
@@ -661,76 +661,15 @@
An object of the form:
{ # Defines an evaluation job that runs periodically to generate Evaluations. [Creating an evaluation job](/ml-engine/docs/continuous-evaluation/create-job) is the starting point for using continuous evaluation.
- "annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
- "evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
- "exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
- "exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
- "evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
- "email": "A String", # Required. An email address to send alerts to.
- "minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
- },
- "evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
- "boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
- "iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
- },
- },
- "boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
- "instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- },
- "humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
- "questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
- "annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
- "labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
- "userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
- "contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
- "A String",
- ],
- "annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
- "languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
- "replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
- "instruction": "A String", # Required. Instruction resource name.
- },
- "textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
- "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- "sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
- "enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
- },
- },
- "imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
- "answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
- "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- },
- "bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
- "a_key": "A String",
- },
- "inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
- "gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
- "mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
- "inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
- },
- "bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
- "inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
- },
- "annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
- "classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
- "isMultiLabel": True or False, # Whether the classification task is multi-label or not.
- },
- "textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
- "languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
- },
- "dataType": "A String", # Required. Data type must be specifed when user tries to import data.
- },
- },
- "createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
+ "modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
"name": "A String", # Output only. After you create a job, Data Labeling Service assigns a name to the job with the following format: "projects/{project_id}/evaluationJobs/ {evaluation_job_id}"
+ "createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
"schedule": "A String", # Required. Describes the interval at which the job runs. This interval must be at least 1 day, and it is rounded to the nearest day. For example, if you specify a 50-hour interval, the job runs every 2 days. You can provide the schedule in [crontab format](/scheduler/docs/configuring/cron-job-schedules) or in an [English-like format](/appengine/docs/standard/python/config/cronref#schedule_format). Regardless of what you specify, the job will run at 10:00 AM UTC. Only the interval from this schedule is used, not the specific time of day.
"state": "A String", # Output only. Describes the current state of the job.
"labelMissingGroundTruth": True or False, # Required. Whether you want Data Labeling Service to provide ground truth labels for prediction input. If you want the service to assign human labelers to annotate your data, set this to `true`. If you want to provide your own ground truth labels in the evaluation job's BigQuery table, set this to `false`.
"attempts": [ # Output only. Every time the evaluation job runs and an error occurs, the failed attempt is appended to this array.
{ # Records a failed evaluation job run.
+ "attemptTime": "A String",
"partialFailures": [ # Details of errors that occurred.
{ # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors).
"details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
@@ -742,11 +681,72 @@
"code": 42, # The status code, which should be an enum value of google.rpc.Code.
},
],
- "attemptTime": "A String",
},
],
+ "annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
+ "evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
+ "inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
+ "gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
+ "inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
+ "mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
+ },
+ "bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
+ "inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
+ },
+ "annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
+ "textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
+ "languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
+ },
+ "classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
+ "isMultiLabel": True or False, # Whether the classification task is multi-label or not.
+ },
+ "dataType": "A String", # Required. Data type must be specifed when user tries to import data.
+ },
+ "exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
+ "textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
+ "sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
+ "enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
+ },
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
+ },
+ "boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
+ },
+ "evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
+ "boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
+ "iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
+ },
+ },
+ "humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
+ "questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
+ "languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
+ "userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
+ "instruction": "A String", # Required. Instruction resource name.
+ "contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
+ "A String",
+ ],
+ "labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
+ "replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
+ "annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
+ "annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
+ },
+ "exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
+ "imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
+ "answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
+ },
+ "evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
+ "email": "A String", # Required. An email address to send alerts to.
+ "minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
+ },
+ "bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
+ "a_key": "A String",
+ },
+ },
"description": "A String", # Required. Description of the job. The description can be up to 25,000 characters long.
- "modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
}</pre>
</div>