chore: update docs/dyn (#1162)
This PR was generated using Autosynth. :rainbow:
Synth log will be available here:
https://source.cloud.google.com/results/invocations/b5e48daa-1759-436b-9fe7-ffce1482b520/targets
- [ ] To automatically regenerate this PR, check this box.
diff --git a/docs/dyn/datalabeling_v1beta1.projects.evaluationJobs.html b/docs/dyn/datalabeling_v1beta1.projects.evaluationJobs.html
index ab60652..2f6625d 100644
--- a/docs/dyn/datalabeling_v1beta1.projects.evaluationJobs.html
+++ b/docs/dyn/datalabeling_v1beta1.projects.evaluationJobs.html
@@ -87,7 +87,7 @@
<code><a href="#get">get(name, x__xgafv=None)</a></code></p>
<p class="firstline">Gets an evaluation job by resource name.</p>
<p class="toc_element">
- <code><a href="#list">list(parent, pageToken=None, pageSize=None, filter=None, x__xgafv=None)</a></code></p>
+ <code><a href="#list">list(parent, filter=None, pageSize=None, pageToken=None, x__xgafv=None)</a></code></p>
<p class="firstline">Lists all evaluation jobs within a project with possible filters. Pagination is supported.</p>
<p class="toc_element">
<code><a href="#list_next">list_next(previous_request, previous_response)</a></code></p>
@@ -117,95 +117,95 @@
The object takes the form of:
{ # Request message for CreateEvaluationJob.
- "job": { # Defines an evaluation job that runs periodically to generate Evaluations. [Creating an evaluation job](/ml-engine/docs/continuous-evaluation/create-job) is the starting point for using continuous evaluation. # Required. The evaluation job to create.
- "attempts": [ # Output only. Every time the evaluation job runs and an error occurs, the failed attempt is appended to this array.
- { # Records a failed evaluation job run.
- "attemptTime": "A String",
- "partialFailures": [ # Details of errors that occurred.
- { # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors).
- "message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
- "code": 42, # The status code, which should be an enum value of google.rpc.Code.
- "details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
- {
- "a_key": "", # Properties of the object. Contains field @type with type URL.
- },
- ],
+ "job": { # Defines an evaluation job that runs periodically to generate Evaluations. [Creating an evaluation job](/ml-engine/docs/continuous-evaluation/create-job) is the starting point for using continuous evaluation. # Required. The evaluation job to create.
+ "annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
+ "attempts": [ # Output only. Every time the evaluation job runs and an error occurs, the failed attempt is appended to this array.
+ { # Records a failed evaluation job run.
+ "attemptTime": "A String",
+ "partialFailures": [ # Details of errors that occurred.
+ { # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors).
+ "code": 42, # The status code, which should be an enum value of google.rpc.Code.
+ "details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
+ {
+ "a_key": "", # Properties of the object. Contains field @type with type URL.
},
],
+ "message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
},
],
- "createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
- "state": "A String", # Output only. Describes the current state of the job.
- "name": "A String", # Output only. After you create a job, Data Labeling Service assigns a name to the job with the following format: "projects/{project_id}/evaluationJobs/ {evaluation_job_id}"
- "modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
- "annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
- "description": "A String", # Required. Description of the job. The description can be up to 25,000 characters long.
- "labelMissingGroundTruth": True or False, # Required. Whether you want Data Labeling Service to provide ground truth labels for prediction input. If you want the service to assign human labelers to annotate your data, set this to `true`. If you want to provide your own ground truth labels in the evaluation job's BigQuery table, set this to `false`.
- "schedule": "A String", # Required. Describes the interval at which the job runs. This interval must be at least 1 day, and it is rounded to the nearest day. For example, if you specify a 50-hour interval, the job runs every 2 days. You can provide the schedule in [crontab format](/scheduler/docs/configuring/cron-job-schedules) or in an [English-like format](/appengine/docs/standard/python/config/cronref#schedule_format). Regardless of what you specify, the job will run at 10:00 AM UTC. Only the interval from this schedule is used, not the specific time of day.
- "evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
- "textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
- "sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
- "enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
- },
- },
- "evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
- "email": "A String", # Required. An email address to send alerts to.
- "minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
- },
- "bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
- "a_key": "A String",
- },
- "evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
- "boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
- "iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
- },
- },
- "humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
- "instruction": "A String", # Required. Instruction resource name.
- "replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
- "contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
- "A String",
- ],
- "annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
- "questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
- "userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
- "annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
- "languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
- "labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
- },
- "inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
- "classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
- "isMultiLabel": True or False, # Whether the classification task is multi-label or not.
- },
- "annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
- "bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
- "inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
- },
- "textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
- "languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
- },
- "gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
- "mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
- "inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
- },
- "dataType": "A String", # Required. Data type must be specifed when user tries to import data.
- },
- "exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
- "imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
- "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- "answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
- },
- "boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- "instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
- },
- "exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
+ },
+ ],
+ "createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
+ "description": "A String", # Required. Description of the job. The description can be up to 25,000 characters long.
+ "evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
+ "bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
+ "a_key": "A String",
+ },
+ "boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
+ },
+ "evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
+ "boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
+ "iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
},
},
- }
+ "evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
+ "email": "A String", # Required. An email address to send alerts to.
+ "minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
+ },
+ "exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
+ "exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
+ "humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
+ "annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
+ "annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
+ "contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
+ "A String",
+ ],
+ "instruction": "A String", # Required. Instruction resource name.
+ "labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
+ "languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
+ "questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
+ "replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
+ "userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
+ },
+ "imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
+ "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
+ },
+ "inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
+ "annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
+ "bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
+ "inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
+ },
+ "classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
+ "isMultiLabel": True or False, # Whether the classification task is multi-label or not.
+ },
+ "dataType": "A String", # Required. Data type must be specifed when user tries to import data.
+ "gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
+ "inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
+ "mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
+ },
+ "textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
+ "languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
+ },
+ },
+ "textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
+ "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
+ "enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
+ },
+ },
+ },
+ "labelMissingGroundTruth": True or False, # Required. Whether you want Data Labeling Service to provide ground truth labels for prediction input. If you want the service to assign human labelers to annotate your data, set this to `true`. If you want to provide your own ground truth labels in the evaluation job's BigQuery table, set this to `false`.
+ "modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
+ "name": "A String", # Output only. After you create a job, Data Labeling Service assigns a name to the job with the following format: "projects/{project_id}/evaluationJobs/ {evaluation_job_id}"
+ "schedule": "A String", # Required. Describes the interval at which the job runs. This interval must be at least 1 day, and it is rounded to the nearest day. For example, if you specify a 50-hour interval, the job runs every 2 days. You can provide the schedule in [crontab format](/scheduler/docs/configuring/cron-job-schedules) or in an [English-like format](/appengine/docs/standard/python/config/cronref#schedule_format). Regardless of what you specify, the job will run at 10:00 AM UTC. Only the interval from this schedule is used, not the specific time of day.
+ "state": "A String", # Output only. Describes the current state of the job.
+ },
+}
x__xgafv: string, V1 error format.
Allowed values
@@ -216,93 +216,93 @@
An object of the form:
{ # Defines an evaluation job that runs periodically to generate Evaluations. [Creating an evaluation job](/ml-engine/docs/continuous-evaluation/create-job) is the starting point for using continuous evaluation.
- "attempts": [ # Output only. Every time the evaluation job runs and an error occurs, the failed attempt is appended to this array.
- { # Records a failed evaluation job run.
- "attemptTime": "A String",
- "partialFailures": [ # Details of errors that occurred.
- { # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors).
- "message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
- "code": 42, # The status code, which should be an enum value of google.rpc.Code.
- "details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
- {
- "a_key": "", # Properties of the object. Contains field @type with type URL.
- },
- ],
+ "annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
+ "attempts": [ # Output only. Every time the evaluation job runs and an error occurs, the failed attempt is appended to this array.
+ { # Records a failed evaluation job run.
+ "attemptTime": "A String",
+ "partialFailures": [ # Details of errors that occurred.
+ { # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors).
+ "code": 42, # The status code, which should be an enum value of google.rpc.Code.
+ "details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
+ {
+ "a_key": "", # Properties of the object. Contains field @type with type URL.
},
],
+ "message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
},
],
- "createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
- "state": "A String", # Output only. Describes the current state of the job.
- "name": "A String", # Output only. After you create a job, Data Labeling Service assigns a name to the job with the following format: "projects/{project_id}/evaluationJobs/ {evaluation_job_id}"
- "modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
- "annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
- "description": "A String", # Required. Description of the job. The description can be up to 25,000 characters long.
- "labelMissingGroundTruth": True or False, # Required. Whether you want Data Labeling Service to provide ground truth labels for prediction input. If you want the service to assign human labelers to annotate your data, set this to `true`. If you want to provide your own ground truth labels in the evaluation job's BigQuery table, set this to `false`.
- "schedule": "A String", # Required. Describes the interval at which the job runs. This interval must be at least 1 day, and it is rounded to the nearest day. For example, if you specify a 50-hour interval, the job runs every 2 days. You can provide the schedule in [crontab format](/scheduler/docs/configuring/cron-job-schedules) or in an [English-like format](/appengine/docs/standard/python/config/cronref#schedule_format). Regardless of what you specify, the job will run at 10:00 AM UTC. Only the interval from this schedule is used, not the specific time of day.
- "evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
- "textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
- "sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
- "enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
- },
- },
- "evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
- "email": "A String", # Required. An email address to send alerts to.
- "minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
- },
- "bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
- "a_key": "A String",
- },
- "evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
- "boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
- "iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
- },
- },
- "humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
- "instruction": "A String", # Required. Instruction resource name.
- "replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
- "contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
- "A String",
- ],
- "annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
- "questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
- "userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
- "annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
- "languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
- "labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
- },
- "inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
- "classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
- "isMultiLabel": True or False, # Whether the classification task is multi-label or not.
- },
- "annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
- "bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
- "inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
- },
- "textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
- "languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
- },
- "gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
- "mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
- "inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
- },
- "dataType": "A String", # Required. Data type must be specifed when user tries to import data.
- },
- "exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
- "imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
- "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- "answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
- },
- "boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- "instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
- },
- "exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
+ },
+ ],
+ "createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
+ "description": "A String", # Required. Description of the job. The description can be up to 25,000 characters long.
+ "evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
+ "bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
+ "a_key": "A String",
+ },
+ "boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
+ },
+ "evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
+ "boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
+ "iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
},
- }</pre>
+ },
+ "evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
+ "email": "A String", # Required. An email address to send alerts to.
+ "minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
+ },
+ "exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
+ "exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
+ "humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
+ "annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
+ "annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
+ "contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
+ "A String",
+ ],
+ "instruction": "A String", # Required. Instruction resource name.
+ "labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
+ "languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
+ "questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
+ "replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
+ "userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
+ },
+ "imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
+ "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
+ },
+ "inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
+ "annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
+ "bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
+ "inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
+ },
+ "classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
+ "isMultiLabel": True or False, # Whether the classification task is multi-label or not.
+ },
+ "dataType": "A String", # Required. Data type must be specifed when user tries to import data.
+ "gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
+ "inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
+ "mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
+ },
+ "textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
+ "languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
+ },
+ },
+ "textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
+ "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
+ "enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
+ },
+ },
+ },
+ "labelMissingGroundTruth": True or False, # Required. Whether you want Data Labeling Service to provide ground truth labels for prediction input. If you want the service to assign human labelers to annotate your data, set this to `true`. If you want to provide your own ground truth labels in the evaluation job's BigQuery table, set this to `false`.
+ "modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
+ "name": "A String", # Output only. After you create a job, Data Labeling Service assigns a name to the job with the following format: "projects/{project_id}/evaluationJobs/ {evaluation_job_id}"
+ "schedule": "A String", # Required. Describes the interval at which the job runs. This interval must be at least 1 day, and it is rounded to the nearest day. For example, if you specify a 50-hour interval, the job runs every 2 days. You can provide the schedule in [crontab format](/scheduler/docs/configuring/cron-job-schedules) or in an [English-like format](/appengine/docs/standard/python/config/cronref#schedule_format). Regardless of what you specify, the job will run at 10:00 AM UTC. Only the interval from this schedule is used, not the specific time of day.
+ "state": "A String", # Output only. Describes the current state of the job.
+}</pre>
</div>
<div class="method">
@@ -320,7 +320,7 @@
An object of the form:
{ # A generic empty message that you can re-use to avoid defining duplicated empty messages in your APIs. A typical example is to use it as the request or the response type of an API method. For instance: service Foo { rpc Bar(google.protobuf.Empty) returns (google.protobuf.Empty); } The JSON representation for `Empty` is empty JSON object `{}`.
- }</pre>
+}</pre>
</div>
<div class="method">
@@ -338,104 +338,104 @@
An object of the form:
{ # Defines an evaluation job that runs periodically to generate Evaluations. [Creating an evaluation job](/ml-engine/docs/continuous-evaluation/create-job) is the starting point for using continuous evaluation.
- "attempts": [ # Output only. Every time the evaluation job runs and an error occurs, the failed attempt is appended to this array.
- { # Records a failed evaluation job run.
- "attemptTime": "A String",
- "partialFailures": [ # Details of errors that occurred.
- { # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors).
- "message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
- "code": 42, # The status code, which should be an enum value of google.rpc.Code.
- "details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
- {
- "a_key": "", # Properties of the object. Contains field @type with type URL.
- },
- ],
+ "annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
+ "attempts": [ # Output only. Every time the evaluation job runs and an error occurs, the failed attempt is appended to this array.
+ { # Records a failed evaluation job run.
+ "attemptTime": "A String",
+ "partialFailures": [ # Details of errors that occurred.
+ { # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors).
+ "code": 42, # The status code, which should be an enum value of google.rpc.Code.
+ "details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
+ {
+ "a_key": "", # Properties of the object. Contains field @type with type URL.
},
],
+ "message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
},
],
- "createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
- "state": "A String", # Output only. Describes the current state of the job.
- "name": "A String", # Output only. After you create a job, Data Labeling Service assigns a name to the job with the following format: "projects/{project_id}/evaluationJobs/ {evaluation_job_id}"
- "modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
- "annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
- "description": "A String", # Required. Description of the job. The description can be up to 25,000 characters long.
- "labelMissingGroundTruth": True or False, # Required. Whether you want Data Labeling Service to provide ground truth labels for prediction input. If you want the service to assign human labelers to annotate your data, set this to `true`. If you want to provide your own ground truth labels in the evaluation job's BigQuery table, set this to `false`.
- "schedule": "A String", # Required. Describes the interval at which the job runs. This interval must be at least 1 day, and it is rounded to the nearest day. For example, if you specify a 50-hour interval, the job runs every 2 days. You can provide the schedule in [crontab format](/scheduler/docs/configuring/cron-job-schedules) or in an [English-like format](/appengine/docs/standard/python/config/cronref#schedule_format). Regardless of what you specify, the job will run at 10:00 AM UTC. Only the interval from this schedule is used, not the specific time of day.
- "evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
- "textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
- "sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
- "enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
- },
- },
- "evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
- "email": "A String", # Required. An email address to send alerts to.
- "minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
- },
- "bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
- "a_key": "A String",
- },
- "evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
- "boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
- "iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
- },
- },
- "humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
- "instruction": "A String", # Required. Instruction resource name.
- "replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
- "contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
- "A String",
- ],
- "annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
- "questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
- "userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
- "annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
- "languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
- "labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
- },
- "inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
- "classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
- "isMultiLabel": True or False, # Whether the classification task is multi-label or not.
- },
- "annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
- "bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
- "inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
- },
- "textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
- "languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
- },
- "gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
- "mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
- "inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
- },
- "dataType": "A String", # Required. Data type must be specifed when user tries to import data.
- },
- "exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
- "imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
- "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- "answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
- },
- "boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- "instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
- },
- "exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
+ },
+ ],
+ "createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
+ "description": "A String", # Required. Description of the job. The description can be up to 25,000 characters long.
+ "evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
+ "bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
+ "a_key": "A String",
+ },
+ "boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
+ },
+ "evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
+ "boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
+ "iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
},
- }</pre>
+ },
+ "evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
+ "email": "A String", # Required. An email address to send alerts to.
+ "minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
+ },
+ "exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
+ "exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
+ "humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
+ "annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
+ "annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
+ "contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
+ "A String",
+ ],
+ "instruction": "A String", # Required. Instruction resource name.
+ "labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
+ "languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
+ "questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
+ "replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
+ "userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
+ },
+ "imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
+ "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
+ },
+ "inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
+ "annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
+ "bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
+ "inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
+ },
+ "classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
+ "isMultiLabel": True or False, # Whether the classification task is multi-label or not.
+ },
+ "dataType": "A String", # Required. Data type must be specifed when user tries to import data.
+ "gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
+ "inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
+ "mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
+ },
+ "textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
+ "languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
+ },
+ },
+ "textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
+ "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
+ "enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
+ },
+ },
+ },
+ "labelMissingGroundTruth": True or False, # Required. Whether you want Data Labeling Service to provide ground truth labels for prediction input. If you want the service to assign human labelers to annotate your data, set this to `true`. If you want to provide your own ground truth labels in the evaluation job's BigQuery table, set this to `false`.
+ "modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
+ "name": "A String", # Output only. After you create a job, Data Labeling Service assigns a name to the job with the following format: "projects/{project_id}/evaluationJobs/ {evaluation_job_id}"
+ "schedule": "A String", # Required. Describes the interval at which the job runs. This interval must be at least 1 day, and it is rounded to the nearest day. For example, if you specify a 50-hour interval, the job runs every 2 days. You can provide the schedule in [crontab format](/scheduler/docs/configuring/cron-job-schedules) or in an [English-like format](/appengine/docs/standard/python/config/cronref#schedule_format). Regardless of what you specify, the job will run at 10:00 AM UTC. Only the interval from this schedule is used, not the specific time of day.
+ "state": "A String", # Output only. Describes the current state of the job.
+}</pre>
</div>
<div class="method">
- <code class="details" id="list">list(parent, pageToken=None, pageSize=None, filter=None, x__xgafv=None)</code>
+ <code class="details" id="list">list(parent, filter=None, pageSize=None, pageToken=None, x__xgafv=None)</code>
<pre>Lists all evaluation jobs within a project with possible filters. Pagination is supported.
Args:
parent: string, Required. Evaluation job resource parent. Format: "projects/{project_id}" (required)
- pageToken: string, Optional. A token identifying a page of results for the server to return. Typically obtained by the nextPageToken in the response to the previous request. The request returns the first page if this is empty.
- pageSize: integer, Optional. Requested page size. Server may return fewer results than requested. Default value is 100.
filter: string, Optional. You can filter the jobs to list by model_id (also known as model_name, as described in EvaluationJob.modelVersion) or by evaluation job state (as described in EvaluationJob.state). To filter by both criteria, use the `AND` operator or the `OR` operator. For example, you can use the following string for your filter: "evaluation_job.model_id = {model_name} AND evaluation_job.state = {evaluation_job_state}"
+ pageSize: integer, Optional. Requested page size. Server may return fewer results than requested. Default value is 100.
+ pageToken: string, Optional. A token identifying a page of results for the server to return. Typically obtained by the nextPageToken in the response to the previous request. The request returns the first page if this is empty.
x__xgafv: string, V1 error format.
Allowed values
1 - v1 error format
@@ -445,98 +445,98 @@
An object of the form:
{ # Results for listing evaluation jobs.
- "nextPageToken": "A String", # A token to retrieve next page of results.
- "evaluationJobs": [ # The list of evaluation jobs to return.
- { # Defines an evaluation job that runs periodically to generate Evaluations. [Creating an evaluation job](/ml-engine/docs/continuous-evaluation/create-job) is the starting point for using continuous evaluation.
- "attempts": [ # Output only. Every time the evaluation job runs and an error occurs, the failed attempt is appended to this array.
- { # Records a failed evaluation job run.
- "attemptTime": "A String",
- "partialFailures": [ # Details of errors that occurred.
- { # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors).
- "message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
- "code": 42, # The status code, which should be an enum value of google.rpc.Code.
- "details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
- {
- "a_key": "", # Properties of the object. Contains field @type with type URL.
- },
- ],
+ "evaluationJobs": [ # The list of evaluation jobs to return.
+ { # Defines an evaluation job that runs periodically to generate Evaluations. [Creating an evaluation job](/ml-engine/docs/continuous-evaluation/create-job) is the starting point for using continuous evaluation.
+ "annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
+ "attempts": [ # Output only. Every time the evaluation job runs and an error occurs, the failed attempt is appended to this array.
+ { # Records a failed evaluation job run.
+ "attemptTime": "A String",
+ "partialFailures": [ # Details of errors that occurred.
+ { # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors).
+ "code": 42, # The status code, which should be an enum value of google.rpc.Code.
+ "details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
+ {
+ "a_key": "", # Properties of the object. Contains field @type with type URL.
},
],
+ "message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
},
],
- "createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
- "state": "A String", # Output only. Describes the current state of the job.
- "name": "A String", # Output only. After you create a job, Data Labeling Service assigns a name to the job with the following format: "projects/{project_id}/evaluationJobs/ {evaluation_job_id}"
- "modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
- "annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
- "description": "A String", # Required. Description of the job. The description can be up to 25,000 characters long.
- "labelMissingGroundTruth": True or False, # Required. Whether you want Data Labeling Service to provide ground truth labels for prediction input. If you want the service to assign human labelers to annotate your data, set this to `true`. If you want to provide your own ground truth labels in the evaluation job's BigQuery table, set this to `false`.
- "schedule": "A String", # Required. Describes the interval at which the job runs. This interval must be at least 1 day, and it is rounded to the nearest day. For example, if you specify a 50-hour interval, the job runs every 2 days. You can provide the schedule in [crontab format](/scheduler/docs/configuring/cron-job-schedules) or in an [English-like format](/appengine/docs/standard/python/config/cronref#schedule_format). Regardless of what you specify, the job will run at 10:00 AM UTC. Only the interval from this schedule is used, not the specific time of day.
- "evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
- "textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
- "sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
- "enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
- },
- },
- "evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
- "email": "A String", # Required. An email address to send alerts to.
- "minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
- },
- "bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
- "a_key": "A String",
- },
- "evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
- "boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
- "iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
- },
- },
- "humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
- "instruction": "A String", # Required. Instruction resource name.
- "replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
- "contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
- "A String",
- ],
- "annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
- "questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
- "userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
- "annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
- "languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
- "labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
- },
- "inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
- "classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
- "isMultiLabel": True or False, # Whether the classification task is multi-label or not.
- },
- "annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
- "bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
- "inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
- },
- "textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
- "languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
- },
- "gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
- "mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
- "inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
- },
- "dataType": "A String", # Required. Data type must be specifed when user tries to import data.
- },
- "exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
- "imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
- "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- "answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
- },
- "boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- "instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
- },
- "exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
+ },
+ ],
+ "createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
+ "description": "A String", # Required. Description of the job. The description can be up to 25,000 characters long.
+ "evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
+ "bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
+ "a_key": "A String",
+ },
+ "boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
+ },
+ "evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
+ "boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
+ "iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
},
},
- ],
- }</pre>
+ "evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
+ "email": "A String", # Required. An email address to send alerts to.
+ "minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
+ },
+ "exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
+ "exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
+ "humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
+ "annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
+ "annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
+ "contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
+ "A String",
+ ],
+ "instruction": "A String", # Required. Instruction resource name.
+ "labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
+ "languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
+ "questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
+ "replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
+ "userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
+ },
+ "imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
+ "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
+ },
+ "inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
+ "annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
+ "bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
+ "inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
+ },
+ "classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
+ "isMultiLabel": True or False, # Whether the classification task is multi-label or not.
+ },
+ "dataType": "A String", # Required. Data type must be specifed when user tries to import data.
+ "gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
+ "inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
+ "mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
+ },
+ "textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
+ "languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
+ },
+ },
+ "textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
+ "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
+ "enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
+ },
+ },
+ },
+ "labelMissingGroundTruth": True or False, # Required. Whether you want Data Labeling Service to provide ground truth labels for prediction input. If you want the service to assign human labelers to annotate your data, set this to `true`. If you want to provide your own ground truth labels in the evaluation job's BigQuery table, set this to `false`.
+ "modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
+ "name": "A String", # Output only. After you create a job, Data Labeling Service assigns a name to the job with the following format: "projects/{project_id}/evaluationJobs/ {evaluation_job_id}"
+ "schedule": "A String", # Required. Describes the interval at which the job runs. This interval must be at least 1 day, and it is rounded to the nearest day. For example, if you specify a 50-hour interval, the job runs every 2 days. You can provide the schedule in [crontab format](/scheduler/docs/configuring/cron-job-schedules) or in an [English-like format](/appengine/docs/standard/python/config/cronref#schedule_format). Regardless of what you specify, the job will run at 10:00 AM UTC. Only the interval from this schedule is used, not the specific time of day.
+ "state": "A String", # Output only. Describes the current state of the job.
+ },
+ ],
+ "nextPageToken": "A String", # A token to retrieve next page of results.
+}</pre>
</div>
<div class="method">
@@ -563,93 +563,93 @@
The object takes the form of:
{ # Defines an evaluation job that runs periodically to generate Evaluations. [Creating an evaluation job](/ml-engine/docs/continuous-evaluation/create-job) is the starting point for using continuous evaluation.
- "attempts": [ # Output only. Every time the evaluation job runs and an error occurs, the failed attempt is appended to this array.
- { # Records a failed evaluation job run.
- "attemptTime": "A String",
- "partialFailures": [ # Details of errors that occurred.
- { # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors).
- "message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
- "code": 42, # The status code, which should be an enum value of google.rpc.Code.
- "details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
- {
- "a_key": "", # Properties of the object. Contains field @type with type URL.
- },
- ],
- },
- ],
- },
- ],
- "createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
- "state": "A String", # Output only. Describes the current state of the job.
- "name": "A String", # Output only. After you create a job, Data Labeling Service assigns a name to the job with the following format: "projects/{project_id}/evaluationJobs/ {evaluation_job_id}"
- "modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
- "annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
- "description": "A String", # Required. Description of the job. The description can be up to 25,000 characters long.
- "labelMissingGroundTruth": True or False, # Required. Whether you want Data Labeling Service to provide ground truth labels for prediction input. If you want the service to assign human labelers to annotate your data, set this to `true`. If you want to provide your own ground truth labels in the evaluation job's BigQuery table, set this to `false`.
- "schedule": "A String", # Required. Describes the interval at which the job runs. This interval must be at least 1 day, and it is rounded to the nearest day. For example, if you specify a 50-hour interval, the job runs every 2 days. You can provide the schedule in [crontab format](/scheduler/docs/configuring/cron-job-schedules) or in an [English-like format](/appengine/docs/standard/python/config/cronref#schedule_format). Regardless of what you specify, the job will run at 10:00 AM UTC. Only the interval from this schedule is used, not the specific time of day.
- "evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
- "textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
- "sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
- "enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
+ "annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
+ "attempts": [ # Output only. Every time the evaluation job runs and an error occurs, the failed attempt is appended to this array.
+ { # Records a failed evaluation job run.
+ "attemptTime": "A String",
+ "partialFailures": [ # Details of errors that occurred.
+ { # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors).
+ "code": 42, # The status code, which should be an enum value of google.rpc.Code.
+ "details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
+ {
+ "a_key": "", # Properties of the object. Contains field @type with type URL.
+ },
+ ],
+ "message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
},
- },
- "evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
- "email": "A String", # Required. An email address to send alerts to.
- "minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
- },
- "bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
- "a_key": "A String",
- },
- "evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
- "boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
- "iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
- },
- },
- "humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
- "instruction": "A String", # Required. Instruction resource name.
- "replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
- "contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
- "A String",
- ],
- "annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
- "questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
- "userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
- "annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
- "languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
- "labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
- },
- "inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
- "classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
- "isMultiLabel": True or False, # Whether the classification task is multi-label or not.
- },
- "annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
- "bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
- "inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
- },
- "textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
- "languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
- },
- "gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
- "mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
- "inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
- },
- "dataType": "A String", # Required. Data type must be specifed when user tries to import data.
- },
- "exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
- "imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
- "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- "answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
- },
- "boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- "instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
- },
- "exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
+ ],
},
- }
+ ],
+ "createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
+ "description": "A String", # Required. Description of the job. The description can be up to 25,000 characters long.
+ "evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
+ "bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
+ "a_key": "A String",
+ },
+ "boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
+ },
+ "evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
+ "boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
+ "iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
+ },
+ },
+ "evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
+ "email": "A String", # Required. An email address to send alerts to.
+ "minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
+ },
+ "exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
+ "exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
+ "humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
+ "annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
+ "annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
+ "contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
+ "A String",
+ ],
+ "instruction": "A String", # Required. Instruction resource name.
+ "labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
+ "languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
+ "questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
+ "replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
+ "userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
+ },
+ "imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
+ "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
+ },
+ "inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
+ "annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
+ "bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
+ "inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
+ },
+ "classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
+ "isMultiLabel": True or False, # Whether the classification task is multi-label or not.
+ },
+ "dataType": "A String", # Required. Data type must be specifed when user tries to import data.
+ "gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
+ "inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
+ "mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
+ },
+ "textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
+ "languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
+ },
+ },
+ "textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
+ "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
+ "enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
+ },
+ },
+ },
+ "labelMissingGroundTruth": True or False, # Required. Whether you want Data Labeling Service to provide ground truth labels for prediction input. If you want the service to assign human labelers to annotate your data, set this to `true`. If you want to provide your own ground truth labels in the evaluation job's BigQuery table, set this to `false`.
+ "modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
+ "name": "A String", # Output only. After you create a job, Data Labeling Service assigns a name to the job with the following format: "projects/{project_id}/evaluationJobs/ {evaluation_job_id}"
+ "schedule": "A String", # Required. Describes the interval at which the job runs. This interval must be at least 1 day, and it is rounded to the nearest day. For example, if you specify a 50-hour interval, the job runs every 2 days. You can provide the schedule in [crontab format](/scheduler/docs/configuring/cron-job-schedules) or in an [English-like format](/appengine/docs/standard/python/config/cronref#schedule_format). Regardless of what you specify, the job will run at 10:00 AM UTC. Only the interval from this schedule is used, not the specific time of day.
+ "state": "A String", # Output only. Describes the current state of the job.
+}
updateMask: string, Optional. Mask for which fields to update. You can only provide the following fields: * `evaluationJobConfig.humanAnnotationConfig.instruction` * `evaluationJobConfig.exampleCount` * `evaluationJobConfig.exampleSamplePercentage` You can provide more than one of these fields by separating them with commas.
x__xgafv: string, V1 error format.
@@ -661,93 +661,93 @@
An object of the form:
{ # Defines an evaluation job that runs periodically to generate Evaluations. [Creating an evaluation job](/ml-engine/docs/continuous-evaluation/create-job) is the starting point for using continuous evaluation.
- "attempts": [ # Output only. Every time the evaluation job runs and an error occurs, the failed attempt is appended to this array.
- { # Records a failed evaluation job run.
- "attemptTime": "A String",
- "partialFailures": [ # Details of errors that occurred.
- { # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors).
- "message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
- "code": 42, # The status code, which should be an enum value of google.rpc.Code.
- "details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
- {
- "a_key": "", # Properties of the object. Contains field @type with type URL.
- },
- ],
+ "annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
+ "attempts": [ # Output only. Every time the evaluation job runs and an error occurs, the failed attempt is appended to this array.
+ { # Records a failed evaluation job run.
+ "attemptTime": "A String",
+ "partialFailures": [ # Details of errors that occurred.
+ { # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors).
+ "code": 42, # The status code, which should be an enum value of google.rpc.Code.
+ "details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
+ {
+ "a_key": "", # Properties of the object. Contains field @type with type URL.
},
],
+ "message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
},
],
- "createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
- "state": "A String", # Output only. Describes the current state of the job.
- "name": "A String", # Output only. After you create a job, Data Labeling Service assigns a name to the job with the following format: "projects/{project_id}/evaluationJobs/ {evaluation_job_id}"
- "modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
- "annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
- "description": "A String", # Required. Description of the job. The description can be up to 25,000 characters long.
- "labelMissingGroundTruth": True or False, # Required. Whether you want Data Labeling Service to provide ground truth labels for prediction input. If you want the service to assign human labelers to annotate your data, set this to `true`. If you want to provide your own ground truth labels in the evaluation job's BigQuery table, set this to `false`.
- "schedule": "A String", # Required. Describes the interval at which the job runs. This interval must be at least 1 day, and it is rounded to the nearest day. For example, if you specify a 50-hour interval, the job runs every 2 days. You can provide the schedule in [crontab format](/scheduler/docs/configuring/cron-job-schedules) or in an [English-like format](/appengine/docs/standard/python/config/cronref#schedule_format). Regardless of what you specify, the job will run at 10:00 AM UTC. Only the interval from this schedule is used, not the specific time of day.
- "evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
- "textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
- "sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
- "enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
- },
- },
- "evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
- "email": "A String", # Required. An email address to send alerts to.
- "minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
- },
- "bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
- "a_key": "A String",
- },
- "evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
- "boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
- "iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
- },
- },
- "humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
- "instruction": "A String", # Required. Instruction resource name.
- "replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
- "contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
- "A String",
- ],
- "annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
- "questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
- "userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
- "annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
- "languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
- "labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
- },
- "inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
- "classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
- "isMultiLabel": True or False, # Whether the classification task is multi-label or not.
- },
- "annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
- "bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
- "inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
- },
- "textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
- "languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
- },
- "gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
- "mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
- "inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
- },
- "dataType": "A String", # Required. Data type must be specifed when user tries to import data.
- },
- "exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
- "imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
- "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- "answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
- },
- "boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
- "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
- "instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
- },
- "exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
+ },
+ ],
+ "createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
+ "description": "A String", # Required. Description of the job. The description can be up to 25,000 characters long.
+ "evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
+ "bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
+ "a_key": "A String",
+ },
+ "boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
+ },
+ "evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
+ "boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
+ "iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
},
- }</pre>
+ },
+ "evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
+ "email": "A String", # Required. An email address to send alerts to.
+ "minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
+ },
+ "exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
+ "exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
+ "humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
+ "annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
+ "annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
+ "contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
+ "A String",
+ ],
+ "instruction": "A String", # Required. Instruction resource name.
+ "labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
+ "languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
+ "questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
+ "replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
+ "userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
+ },
+ "imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
+ "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
+ },
+ "inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
+ "annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
+ "bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
+ "inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
+ },
+ "classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
+ "isMultiLabel": True or False, # Whether the classification task is multi-label or not.
+ },
+ "dataType": "A String", # Required. Data type must be specifed when user tries to import data.
+ "gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
+ "inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
+ "mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
+ },
+ "textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
+ "languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
+ },
+ },
+ "textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
+ "allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
+ "annotationSpecSet": "A String", # Required. Annotation spec set resource name.
+ "sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
+ "enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
+ },
+ },
+ },
+ "labelMissingGroundTruth": True or False, # Required. Whether you want Data Labeling Service to provide ground truth labels for prediction input. If you want the service to assign human labelers to annotate your data, set this to `true`. If you want to provide your own ground truth labels in the evaluation job's BigQuery table, set this to `false`.
+ "modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
+ "name": "A String", # Output only. After you create a job, Data Labeling Service assigns a name to the job with the following format: "projects/{project_id}/evaluationJobs/ {evaluation_job_id}"
+ "schedule": "A String", # Required. Describes the interval at which the job runs. This interval must be at least 1 day, and it is rounded to the nearest day. For example, if you specify a 50-hour interval, the job runs every 2 days. You can provide the schedule in [crontab format](/scheduler/docs/configuring/cron-job-schedules) or in an [English-like format](/appengine/docs/standard/python/config/cronref#schedule_format). Regardless of what you specify, the job will run at 10:00 AM UTC. Only the interval from this schedule is used, not the specific time of day.
+ "state": "A String", # Output only. Describes the current state of the job.
+}</pre>
</div>
<div class="method">
@@ -760,7 +760,7 @@
The object takes the form of:
{ # Request message for PauseEvaluationJob.
- }
+}
x__xgafv: string, V1 error format.
Allowed values
@@ -771,7 +771,7 @@
An object of the form:
{ # A generic empty message that you can re-use to avoid defining duplicated empty messages in your APIs. A typical example is to use it as the request or the response type of an API method. For instance: service Foo { rpc Bar(google.protobuf.Empty) returns (google.protobuf.Empty); } The JSON representation for `Empty` is empty JSON object `{}`.
- }</pre>
+}</pre>
</div>
<div class="method">
@@ -784,7 +784,7 @@
The object takes the form of:
{ # Request message ResumeEvaluationJob.
- }
+}
x__xgafv: string, V1 error format.
Allowed values
@@ -795,7 +795,7 @@
An object of the form:
{ # A generic empty message that you can re-use to avoid defining duplicated empty messages in your APIs. A typical example is to use it as the request or the response type of an API method. For instance: service Foo { rpc Bar(google.protobuf.Empty) returns (google.protobuf.Empty); } The JSON representation for `Empty` is empty JSON object `{}`.
- }</pre>
+}</pre>
</div>
</body></html>
\ No newline at end of file