Blame - docs/dyn/datalabeling_v1beta1.projects.evaluationJobs.html - platform/external/python/google-api-python-client

blob: 2f6625d65e9f944d53caa663bbc1c002050ae6b7 [file] [log] [blame]

Bu Sun Kim	673ec5c	2020-11-16 11:05:03 -0700	[diff] [blame]	1	<html><body>
				2	<style>
				3
				4	body, h1, h2, h3, div, span, p, pre, a {
				5	margin: 0;
				6	padding: 0;
				7	border: 0;
				8	font-weight: inherit;
				9	font-style: inherit;
				10	font-size: 100%;
				11	font-family: inherit;
				12	vertical-align: baseline;
				13	}
				14
				15	body {
				16	font-size: 13px;
				17	padding: 1em;
				18	}
				19
				20	h1 {
				21	font-size: 26px;
				22	margin-bottom: 1em;
				23	}
				24
				25	h2 {
				26	font-size: 24px;
				27	margin-bottom: 1em;
				28	}
				29
				30	h3 {
				31	font-size: 20px;
				32	margin-bottom: 1em;
				33	margin-top: 1em;
				34	}
				35
				36	pre, code {
				37	line-height: 1.5;
				38	font-family: Monaco, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console', monospace;
				39	}
				40
				41	pre {
				42	margin-top: 0.5em;
				43	}
				44
				45	h1, h2, h3, p {
				46	font-family: Arial, sans serif;
				47	}
				48
				49	h1, h2, h3 {
				50	border-bottom: solid #CCC 1px;
				51	}
				52
				53	.toc_element {
				54	margin-top: 0.5em;
				55	}
				56
				57	.firstline {
				58	margin-left: 2 em;
				59	}
				60
				61	.method {
				62	margin-top: 1em;
				63	border: solid 1px #CCC;
				64	padding: 1em;
				65	background: #EEE;
				66	}
				67
				68	.details {
				69	font-weight: bold;
				70	font-size: 14px;
				71	}
				72
				73	</style>
				74
				75	<h1><a href="datalabeling_v1beta1.html">Data Labeling API</a> . <a href="datalabeling_v1beta1.projects.html">projects</a> . <a href="datalabeling_v1beta1.projects.evaluationJobs.html">evaluationJobs</a></h1>
				76	<h2>Instance Methods</h2>
				77	<p class="toc_element">
				78	<code><a href="#close">close()</a></code></p>
				79	<p class="firstline">Close httplib2 connections.</p>
				80	<p class="toc_element">
				81	<code><a href="#create">create(parent, body=None, x__xgafv=None)</a></code></p>
				82	<p class="firstline">Creates an evaluation job.</p>
				83	<p class="toc_element">
				84	<code><a href="#delete">delete(name, x__xgafv=None)</a></code></p>
				85	<p class="firstline">Stops and deletes an evaluation job.</p>
				86	<p class="toc_element">
				87	<code><a href="#get">get(name, x__xgafv=None)</a></code></p>
				88	<p class="firstline">Gets an evaluation job by resource name.</p>
				89	<p class="toc_element">
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	90	<code><a href="#list">list(parent, filter=None, pageSize=None, pageToken=None, x__xgafv=None)</a></code></p>
Bu Sun Kim	673ec5c	2020-11-16 11:05:03 -0700	[diff] [blame]	91	<p class="firstline">Lists all evaluation jobs within a project with possible filters. Pagination is supported.</p>
				92	<p class="toc_element">
				93	<code><a href="#list_next">list_next(previous_request, previous_response)</a></code></p>
				94	<p class="firstline">Retrieves the next page of results.</p>
				95	<p class="toc_element">
				96	<code><a href="#patch">patch(name, body=None, updateMask=None, x__xgafv=None)</a></code></p>
				97	<p class="firstline">Updates an evaluation job. You can only update certain fields of the job's EvaluationJobConfig: `humanAnnotationConfig.instruction`, `exampleCount`, and `exampleSamplePercentage`. If you want to change any other aspect of the evaluation job, you must delete the job and create a new one.</p>
				98	<p class="toc_element">
				99	<code><a href="#pause">pause(name, body=None, x__xgafv=None)</a></code></p>
				100	<p class="firstline">Pauses an evaluation job. Pausing an evaluation job that is already in a `PAUSED` state is a no-op.</p>
				101	<p class="toc_element">
				102	<code><a href="#resume">resume(name, body=None, x__xgafv=None)</a></code></p>
				103	<p class="firstline">Resumes a paused evaluation job. A deleted evaluation job can't be resumed. Resuming a running or scheduled evaluation job is a no-op.</p>
				104	<h3>Method Details</h3>
				105	<div class="method">
				106	<code class="details" id="close">close()</code>
				107	<pre>Close httplib2 connections.</pre>
				108	</div>
				109
				110	<div class="method">
				111	<code class="details" id="create">create(parent, body=None, x__xgafv=None)</code>
				112	<pre>Creates an evaluation job.
				113
				114	Args:
				115	parent: string, Required. Evaluation job resource parent. Format: "projects/{project_id}" (required)
				116	body: object, The request body.
				117	The object takes the form of:
				118
				119	{ # Request message for CreateEvaluationJob.
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	120	"job": { # Defines an evaluation job that runs periodically to generate Evaluations. [Creating an evaluation job](/ml-engine/docs/continuous-evaluation/create-job) is the starting point for using continuous evaluation. # Required. The evaluation job to create.
				121	"annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
				122	"attempts": [ # Output only. Every time the evaluation job runs and an error occurs, the failed attempt is appended to this array.
				123	{ # Records a failed evaluation job run.
				124	"attemptTime": "A String",
				125	"partialFailures": [ # Details of errors that occurred.
				126	{ # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors).
				127	"code": 42, # The status code, which should be an enum value of google.rpc.Code.
				128	"details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
				129	{
				130	"a_key": "", # Properties of the object. Contains field @type with type URL.
Yoshi Automation Bot	0bf565c	2020-12-09 08:56:03 -0800	[diff] [blame]	131	},
				132	],
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	133	"message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
Yoshi Automation Bot	0bf565c	2020-12-09 08:56:03 -0800	[diff] [blame]	134	},
				135	],
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	136	},
				137	],
				138	"createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
				139	"description": "A String", # Required. Description of the job. The description can be up to 25,000 characters long.
				140	"evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
				141	"bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
				142	"a_key": "A String",
				143	},
				144	"boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
				145	"annotationSpecSet": "A String", # Required. Annotation spec set resource name.
				146	"instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
				147	},
				148	"evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
				149	"boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
				150	"iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
Yoshi Automation Bot	c2228be	2020-11-24 15:48:03 -0800	[diff] [blame]	151	},
				152	},
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	153	"evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
				154	"email": "A String", # Required. An email address to send alerts to.
				155	"minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
				156	},
				157	"exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
				158	"exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
				159	"humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
				160	"annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
				161	"annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
				162	"contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
				163	"A String",
				164	],
				165	"instruction": "A String", # Required. Instruction resource name.
				166	"labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
				167	"languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
				168	"questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
				169	"replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
				170	"userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
				171	},
				172	"imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
				173	"allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
				174	"annotationSpecSet": "A String", # Required. Annotation spec set resource name.
				175	"answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
				176	},
				177	"inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
				178	"annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
				179	"bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
				180	"inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
				181	},
				182	"classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
				183	"isMultiLabel": True or False, # Whether the classification task is multi-label or not.
				184	},
				185	"dataType": "A String", # Required. Data type must be specifed when user tries to import data.
				186	"gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
				187	"inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
				188	"mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
				189	},
				190	"textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
				191	"languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
				192	},
				193	},
				194	"textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
				195	"allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
				196	"annotationSpecSet": "A String", # Required. Annotation spec set resource name.
				197	"sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
				198	"enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
				199	},
				200	},
				201	},
				202	"labelMissingGroundTruth": True or False, # Required. Whether you want Data Labeling Service to provide ground truth labels for prediction input. If you want the service to assign human labelers to annotate your data, set this to `true`. If you want to provide your own ground truth labels in the evaluation job's BigQuery table, set this to `false`.
				203	"modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
				204	"name": "A String", # Output only. After you create a job, Data Labeling Service assigns a name to the job with the following format: "projects/{project_id}/evaluationJobs/ {evaluation_job_id}"
				205	"schedule": "A String", # Required. Describes the interval at which the job runs. This interval must be at least 1 day, and it is rounded to the nearest day. For example, if you specify a 50-hour interval, the job runs every 2 days. You can provide the schedule in [crontab format](/scheduler/docs/configuring/cron-job-schedules) or in an [English-like format](/appengine/docs/standard/python/config/cronref#schedule_format). Regardless of what you specify, the job will run at 10:00 AM UTC. Only the interval from this schedule is used, not the specific time of day.
				206	"state": "A String", # Output only. Describes the current state of the job.
				207	},
				208	}
Bu Sun Kim	673ec5c	2020-11-16 11:05:03 -0700	[diff] [blame]	209
				210	x__xgafv: string, V1 error format.
				211	Allowed values
				212	1 - v1 error format
				213	2 - v2 error format
				214
				215	Returns:
				216	An object of the form:
				217
				218	{ # Defines an evaluation job that runs periodically to generate Evaluations. [Creating an evaluation job](/ml-engine/docs/continuous-evaluation/create-job) is the starting point for using continuous evaluation.
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	219	"annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
				220	"attempts": [ # Output only. Every time the evaluation job runs and an error occurs, the failed attempt is appended to this array.
				221	{ # Records a failed evaluation job run.
				222	"attemptTime": "A String",
				223	"partialFailures": [ # Details of errors that occurred.
				224	{ # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors).
				225	"code": 42, # The status code, which should be an enum value of google.rpc.Code.
				226	"details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
				227	{
				228	"a_key": "", # Properties of the object. Contains field @type with type URL.
Yoshi Automation Bot	0bf565c	2020-12-09 08:56:03 -0800	[diff] [blame]	229	},
				230	],
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	231	"message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
Yoshi Automation Bot	0bf565c	2020-12-09 08:56:03 -0800	[diff] [blame]	232	},
				233	],
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	234	},
				235	],
				236	"createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
				237	"description": "A String", # Required. Description of the job. The description can be up to 25,000 characters long.
				238	"evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
				239	"bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
				240	"a_key": "A String",
				241	},
				242	"boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
				243	"annotationSpecSet": "A String", # Required. Annotation spec set resource name.
				244	"instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
				245	},
				246	"evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
				247	"boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
				248	"iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
Yoshi Automation Bot	c2228be	2020-11-24 15:48:03 -0800	[diff] [blame]	249	},
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	250	},
				251	"evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
				252	"email": "A String", # Required. An email address to send alerts to.
				253	"minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
				254	},
				255	"exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
				256	"exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
				257	"humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
				258	"annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
				259	"annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
				260	"contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
				261	"A String",
				262	],
				263	"instruction": "A String", # Required. Instruction resource name.
				264	"labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
				265	"languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
				266	"questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
				267	"replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
				268	"userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
				269	},
				270	"imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
				271	"allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
				272	"annotationSpecSet": "A String", # Required. Annotation spec set resource name.
				273	"answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
				274	},
				275	"inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
				276	"annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
				277	"bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
				278	"inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
				279	},
				280	"classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
				281	"isMultiLabel": True or False, # Whether the classification task is multi-label or not.
				282	},
				283	"dataType": "A String", # Required. Data type must be specifed when user tries to import data.
				284	"gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
				285	"inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
				286	"mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
				287	},
				288	"textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
				289	"languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
				290	},
				291	},
				292	"textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
				293	"allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
				294	"annotationSpecSet": "A String", # Required. Annotation spec set resource name.
				295	"sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
				296	"enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
				297	},
				298	},
				299	},
				300	"labelMissingGroundTruth": True or False, # Required. Whether you want Data Labeling Service to provide ground truth labels for prediction input. If you want the service to assign human labelers to annotate your data, set this to `true`. If you want to provide your own ground truth labels in the evaluation job's BigQuery table, set this to `false`.
				301	"modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
				302	"name": "A String", # Output only. After you create a job, Data Labeling Service assigns a name to the job with the following format: "projects/{project_id}/evaluationJobs/ {evaluation_job_id}"
				303	"schedule": "A String", # Required. Describes the interval at which the job runs. This interval must be at least 1 day, and it is rounded to the nearest day. For example, if you specify a 50-hour interval, the job runs every 2 days. You can provide the schedule in [crontab format](/scheduler/docs/configuring/cron-job-schedules) or in an [English-like format](/appengine/docs/standard/python/config/cronref#schedule_format). Regardless of what you specify, the job will run at 10:00 AM UTC. Only the interval from this schedule is used, not the specific time of day.
				304	"state": "A String", # Output only. Describes the current state of the job.
				305	}</pre>
Bu Sun Kim	673ec5c	2020-11-16 11:05:03 -0700	[diff] [blame]	306	</div>
				307
				308	<div class="method">
				309	<code class="details" id="delete">delete(name, x__xgafv=None)</code>
				310	<pre>Stops and deletes an evaluation job.
				311
				312	Args:
				313	name: string, Required. Name of the evaluation job that is going to be deleted. Format: "projects/{project_id}/evaluationJobs/{evaluation_job_id}" (required)
				314	x__xgafv: string, V1 error format.
				315	Allowed values
				316	1 - v1 error format
				317	2 - v2 error format
				318
				319	Returns:
				320	An object of the form:
				321
				322	{ # A generic empty message that you can re-use to avoid defining duplicated empty messages in your APIs. A typical example is to use it as the request or the response type of an API method. For instance: service Foo { rpc Bar(google.protobuf.Empty) returns (google.protobuf.Empty); } The JSON representation for `Empty` is empty JSON object `{}`.
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	323	}</pre>
Bu Sun Kim	673ec5c	2020-11-16 11:05:03 -0700	[diff] [blame]	324	</div>
				325
				326	<div class="method">
				327	<code class="details" id="get">get(name, x__xgafv=None)</code>
				328	<pre>Gets an evaluation job by resource name.
				329
				330	Args:
				331	name: string, Required. Name of the evaluation job. Format: "projects/{project_id} /evaluationJobs/{evaluation_job_id}" (required)
				332	x__xgafv: string, V1 error format.
				333	Allowed values
				334	1 - v1 error format
				335	2 - v2 error format
				336
				337	Returns:
				338	An object of the form:
				339
				340	{ # Defines an evaluation job that runs periodically to generate Evaluations. [Creating an evaluation job](/ml-engine/docs/continuous-evaluation/create-job) is the starting point for using continuous evaluation.
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	341	"annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
				342	"attempts": [ # Output only. Every time the evaluation job runs and an error occurs, the failed attempt is appended to this array.
				343	{ # Records a failed evaluation job run.
				344	"attemptTime": "A String",
				345	"partialFailures": [ # Details of errors that occurred.
				346	{ # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors).
				347	"code": 42, # The status code, which should be an enum value of google.rpc.Code.
				348	"details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
				349	{
				350	"a_key": "", # Properties of the object. Contains field @type with type URL.
Yoshi Automation Bot	0bf565c	2020-12-09 08:56:03 -0800	[diff] [blame]	351	},
				352	],
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	353	"message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
Yoshi Automation Bot	0bf565c	2020-12-09 08:56:03 -0800	[diff] [blame]	354	},
				355	],
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	356	},
				357	],
				358	"createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
				359	"description": "A String", # Required. Description of the job. The description can be up to 25,000 characters long.
				360	"evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
				361	"bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
				362	"a_key": "A String",
				363	},
				364	"boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
				365	"annotationSpecSet": "A String", # Required. Annotation spec set resource name.
				366	"instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
				367	},
				368	"evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
				369	"boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
				370	"iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
Yoshi Automation Bot	c2228be	2020-11-24 15:48:03 -0800	[diff] [blame]	371	},
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	372	},
				373	"evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
				374	"email": "A String", # Required. An email address to send alerts to.
				375	"minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
				376	},
				377	"exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
				378	"exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
				379	"humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
				380	"annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
				381	"annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
				382	"contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
				383	"A String",
				384	],
				385	"instruction": "A String", # Required. Instruction resource name.
				386	"labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
				387	"languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
				388	"questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
				389	"replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
				390	"userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
				391	},
				392	"imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
				393	"allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
				394	"annotationSpecSet": "A String", # Required. Annotation spec set resource name.
				395	"answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
				396	},
				397	"inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
				398	"annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
				399	"bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
				400	"inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
				401	},
				402	"classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
				403	"isMultiLabel": True or False, # Whether the classification task is multi-label or not.
				404	},
				405	"dataType": "A String", # Required. Data type must be specifed when user tries to import data.
				406	"gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
				407	"inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
				408	"mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
				409	},
				410	"textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
				411	"languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
				412	},
				413	},
				414	"textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
				415	"allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
				416	"annotationSpecSet": "A String", # Required. Annotation spec set resource name.
				417	"sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
				418	"enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
				419	},
				420	},
				421	},
				422	"labelMissingGroundTruth": True or False, # Required. Whether you want Data Labeling Service to provide ground truth labels for prediction input. If you want the service to assign human labelers to annotate your data, set this to `true`. If you want to provide your own ground truth labels in the evaluation job's BigQuery table, set this to `false`.
				423	"modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
				424	"name": "A String", # Output only. After you create a job, Data Labeling Service assigns a name to the job with the following format: "projects/{project_id}/evaluationJobs/ {evaluation_job_id}"
				425	"schedule": "A String", # Required. Describes the interval at which the job runs. This interval must be at least 1 day, and it is rounded to the nearest day. For example, if you specify a 50-hour interval, the job runs every 2 days. You can provide the schedule in [crontab format](/scheduler/docs/configuring/cron-job-schedules) or in an [English-like format](/appengine/docs/standard/python/config/cronref#schedule_format). Regardless of what you specify, the job will run at 10:00 AM UTC. Only the interval from this schedule is used, not the specific time of day.
				426	"state": "A String", # Output only. Describes the current state of the job.
				427	}</pre>
Bu Sun Kim	673ec5c	2020-11-16 11:05:03 -0700	[diff] [blame]	428	</div>
				429
				430	<div class="method">
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	431	<code class="details" id="list">list(parent, filter=None, pageSize=None, pageToken=None, x__xgafv=None)</code>
Bu Sun Kim	673ec5c	2020-11-16 11:05:03 -0700	[diff] [blame]	432	<pre>Lists all evaluation jobs within a project with possible filters. Pagination is supported.
				433
				434	Args:
				435	parent: string, Required. Evaluation job resource parent. Format: "projects/{project_id}" (required)
Yoshi Automation Bot	b6971b0	2020-11-26 17:16:03 -0800	[diff] [blame]	436	filter: string, Optional. You can filter the jobs to list by model_id (also known as model_name, as described in EvaluationJob.modelVersion) or by evaluation job state (as described in EvaluationJob.state). To filter by both criteria, use the `AND` operator or the `OR` operator. For example, you can use the following string for your filter: "evaluation_job.model_id = {model_name} AND evaluation_job.state = {evaluation_job_state}"
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	437	pageSize: integer, Optional. Requested page size. Server may return fewer results than requested. Default value is 100.
				438	pageToken: string, Optional. A token identifying a page of results for the server to return. Typically obtained by the nextPageToken in the response to the previous request. The request returns the first page if this is empty.
Bu Sun Kim	673ec5c	2020-11-16 11:05:03 -0700	[diff] [blame]	439	x__xgafv: string, V1 error format.
				440	Allowed values
				441	1 - v1 error format
				442	2 - v2 error format
				443
				444	Returns:
				445	An object of the form:
				446
				447	{ # Results for listing evaluation jobs.
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	448	"evaluationJobs": [ # The list of evaluation jobs to return.
				449	{ # Defines an evaluation job that runs periodically to generate Evaluations. [Creating an evaluation job](/ml-engine/docs/continuous-evaluation/create-job) is the starting point for using continuous evaluation.
				450	"annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
				451	"attempts": [ # Output only. Every time the evaluation job runs and an error occurs, the failed attempt is appended to this array.
				452	{ # Records a failed evaluation job run.
				453	"attemptTime": "A String",
				454	"partialFailures": [ # Details of errors that occurred.
				455	{ # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors).
				456	"code": 42, # The status code, which should be an enum value of google.rpc.Code.
				457	"details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
				458	{
				459	"a_key": "", # Properties of the object. Contains field @type with type URL.
Yoshi Automation Bot	0bf565c	2020-12-09 08:56:03 -0800	[diff] [blame]	460	},
				461	],
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	462	"message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
Yoshi Automation Bot	0bf565c	2020-12-09 08:56:03 -0800	[diff] [blame]	463	},
				464	],
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	465	},
				466	],
				467	"createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
				468	"description": "A String", # Required. Description of the job. The description can be up to 25,000 characters long.
				469	"evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
				470	"bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
				471	"a_key": "A String",
				472	},
				473	"boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
				474	"annotationSpecSet": "A String", # Required. Annotation spec set resource name.
				475	"instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
				476	},
				477	"evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
				478	"boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
				479	"iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
Yoshi Automation Bot	c2228be	2020-11-24 15:48:03 -0800	[diff] [blame]	480	},
				481	},
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	482	"evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
				483	"email": "A String", # Required. An email address to send alerts to.
				484	"minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
				485	},
				486	"exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
				487	"exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
				488	"humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
				489	"annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
				490	"annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
				491	"contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
				492	"A String",
				493	],
				494	"instruction": "A String", # Required. Instruction resource name.
				495	"labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
				496	"languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
				497	"questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
				498	"replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
				499	"userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
				500	},
				501	"imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
				502	"allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
				503	"annotationSpecSet": "A String", # Required. Annotation spec set resource name.
				504	"answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
				505	},
				506	"inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
				507	"annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
				508	"bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
				509	"inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
				510	},
				511	"classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
				512	"isMultiLabel": True or False, # Whether the classification task is multi-label or not.
				513	},
				514	"dataType": "A String", # Required. Data type must be specifed when user tries to import data.
				515	"gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
				516	"inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
				517	"mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
				518	},
				519	"textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
				520	"languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
				521	},
				522	},
				523	"textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
				524	"allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
				525	"annotationSpecSet": "A String", # Required. Annotation spec set resource name.
				526	"sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
				527	"enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
				528	},
				529	},
				530	},
				531	"labelMissingGroundTruth": True or False, # Required. Whether you want Data Labeling Service to provide ground truth labels for prediction input. If you want the service to assign human labelers to annotate your data, set this to `true`. If you want to provide your own ground truth labels in the evaluation job's BigQuery table, set this to `false`.
				532	"modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
				533	"name": "A String", # Output only. After you create a job, Data Labeling Service assigns a name to the job with the following format: "projects/{project_id}/evaluationJobs/ {evaluation_job_id}"
				534	"schedule": "A String", # Required. Describes the interval at which the job runs. This interval must be at least 1 day, and it is rounded to the nearest day. For example, if you specify a 50-hour interval, the job runs every 2 days. You can provide the schedule in [crontab format](/scheduler/docs/configuring/cron-job-schedules) or in an [English-like format](/appengine/docs/standard/python/config/cronref#schedule_format). Regardless of what you specify, the job will run at 10:00 AM UTC. Only the interval from this schedule is used, not the specific time of day.
				535	"state": "A String", # Output only. Describes the current state of the job.
				536	},
				537	],
				538	"nextPageToken": "A String", # A token to retrieve next page of results.
				539	}</pre>
Bu Sun Kim	673ec5c	2020-11-16 11:05:03 -0700	[diff] [blame]	540	</div>
				541
				542	<div class="method">
				543	<code class="details" id="list_next">list_next(previous_request, previous_response)</code>
				544	<pre>Retrieves the next page of results.
				545
				546	Args:
				547	previous_request: The request for the previous page. (required)
				548	previous_response: The response from the request for the previous page. (required)
				549
				550	Returns:
				551	A request object that you can call 'execute()' on to request the next
				552	page. Returns None if there are no more items in the collection.
				553	</pre>
				554	</div>
				555
				556	<div class="method">
				557	<code class="details" id="patch">patch(name, body=None, updateMask=None, x__xgafv=None)</code>
				558	<pre>Updates an evaluation job. You can only update certain fields of the job's EvaluationJobConfig: `humanAnnotationConfig.instruction`, `exampleCount`, and `exampleSamplePercentage`. If you want to change any other aspect of the evaluation job, you must delete the job and create a new one.
				559
				560	Args:
				561	name: string, Output only. After you create a job, Data Labeling Service assigns a name to the job with the following format: "projects/{project_id}/evaluationJobs/ {evaluation_job_id}" (required)
				562	body: object, The request body.
				563	The object takes the form of:
				564
				565	{ # Defines an evaluation job that runs periodically to generate Evaluations. [Creating an evaluation job](/ml-engine/docs/continuous-evaluation/create-job) is the starting point for using continuous evaluation.
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	566	"annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
				567	"attempts": [ # Output only. Every time the evaluation job runs and an error occurs, the failed attempt is appended to this array.
				568	{ # Records a failed evaluation job run.
				569	"attemptTime": "A String",
				570	"partialFailures": [ # Details of errors that occurred.
				571	{ # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors).
				572	"code": 42, # The status code, which should be an enum value of google.rpc.Code.
				573	"details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
				574	{
				575	"a_key": "", # Properties of the object. Contains field @type with type URL.
				576	},
				577	],
				578	"message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
Bu Sun Kim	673ec5c	2020-11-16 11:05:03 -0700	[diff] [blame]	579	},
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	580	],
Yoshi Automation Bot	c2228be	2020-11-24 15:48:03 -0800	[diff] [blame]	581	},
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	582	],
				583	"createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
				584	"description": "A String", # Required. Description of the job. The description can be up to 25,000 characters long.
				585	"evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
				586	"bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
				587	"a_key": "A String",
				588	},
				589	"boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
				590	"annotationSpecSet": "A String", # Required. Annotation spec set resource name.
				591	"instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
				592	},
				593	"evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
				594	"boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
				595	"iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
				596	},
				597	},
				598	"evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
				599	"email": "A String", # Required. An email address to send alerts to.
				600	"minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
				601	},
				602	"exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
				603	"exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
				604	"humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
				605	"annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
				606	"annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
				607	"contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
				608	"A String",
				609	],
				610	"instruction": "A String", # Required. Instruction resource name.
				611	"labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
				612	"languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
				613	"questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
				614	"replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
				615	"userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
				616	},
				617	"imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
				618	"allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
				619	"annotationSpecSet": "A String", # Required. Annotation spec set resource name.
				620	"answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
				621	},
				622	"inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
				623	"annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
				624	"bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
				625	"inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
				626	},
				627	"classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
				628	"isMultiLabel": True or False, # Whether the classification task is multi-label or not.
				629	},
				630	"dataType": "A String", # Required. Data type must be specifed when user tries to import data.
				631	"gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
				632	"inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
				633	"mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
				634	},
				635	"textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
				636	"languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
				637	},
				638	},
				639	"textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
				640	"allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
				641	"annotationSpecSet": "A String", # Required. Annotation spec set resource name.
				642	"sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
				643	"enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
				644	},
				645	},
				646	},
				647	"labelMissingGroundTruth": True or False, # Required. Whether you want Data Labeling Service to provide ground truth labels for prediction input. If you want the service to assign human labelers to annotate your data, set this to `true`. If you want to provide your own ground truth labels in the evaluation job's BigQuery table, set this to `false`.
				648	"modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
				649	"name": "A String", # Output only. After you create a job, Data Labeling Service assigns a name to the job with the following format: "projects/{project_id}/evaluationJobs/ {evaluation_job_id}"
				650	"schedule": "A String", # Required. Describes the interval at which the job runs. This interval must be at least 1 day, and it is rounded to the nearest day. For example, if you specify a 50-hour interval, the job runs every 2 days. You can provide the schedule in [crontab format](/scheduler/docs/configuring/cron-job-schedules) or in an [English-like format](/appengine/docs/standard/python/config/cronref#schedule_format). Regardless of what you specify, the job will run at 10:00 AM UTC. Only the interval from this schedule is used, not the specific time of day.
				651	"state": "A String", # Output only. Describes the current state of the job.
				652	}
Bu Sun Kim	673ec5c	2020-11-16 11:05:03 -0700	[diff] [blame]	653
				654	updateMask: string, Optional. Mask for which fields to update. You can only provide the following fields: * `evaluationJobConfig.humanAnnotationConfig.instruction` * `evaluationJobConfig.exampleCount` * `evaluationJobConfig.exampleSamplePercentage` You can provide more than one of these fields by separating them with commas.
				655	x__xgafv: string, V1 error format.
				656	Allowed values
				657	1 - v1 error format
				658	2 - v2 error format
				659
				660	Returns:
				661	An object of the form:
				662
				663	{ # Defines an evaluation job that runs periodically to generate Evaluations. [Creating an evaluation job](/ml-engine/docs/continuous-evaluation/create-job) is the starting point for using continuous evaluation.
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	664	"annotationSpecSet": "A String", # Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
				665	"attempts": [ # Output only. Every time the evaluation job runs and an error occurs, the failed attempt is appended to this array.
				666	{ # Records a failed evaluation job run.
				667	"attemptTime": "A String",
				668	"partialFailures": [ # Details of errors that occurred.
				669	{ # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors).
				670	"code": 42, # The status code, which should be an enum value of google.rpc.Code.
				671	"details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
				672	{
				673	"a_key": "", # Properties of the object. Contains field @type with type URL.
Yoshi Automation Bot	0bf565c	2020-12-09 08:56:03 -0800	[diff] [blame]	674	},
				675	],
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	676	"message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
Yoshi Automation Bot	0bf565c	2020-12-09 08:56:03 -0800	[diff] [blame]	677	},
				678	],
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	679	},
				680	],
				681	"createTime": "A String", # Output only. Timestamp of when this evaluation job was created.
				682	"description": "A String", # Required. Description of the job. The description can be up to 25,000 characters long.
				683	"evaluationJobConfig": { # Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob. # Required. Configuration details for the evaluation job.
				684	"bigqueryImportKeys": { # Required. Prediction keys that tell Data Labeling Service where to find the data for evaluation in your BigQuery table. When the service samples prediction input and output from your model version and saves it to BigQuery, the data gets stored as JSON strings in the BigQuery table. These keys tell Data Labeling Service how to parse the JSON. You can provide the following entries in this field: * `data_json_key`: the data key for prediction input. You must provide either this key or `reference_json_key`. * `reference_json_key`: the data reference key for prediction input. You must provide either this key or `data_json_key`. * `label_json_key`: the label key for prediction output. Required. * `label_score_json_key`: the score key for prediction output. Required. * `bounding_box_json_key`: the bounding box key for prediction output. Required if your model version perform image object detection. Learn [how to configure prediction keys](/ml-engine/docs/continuous-evaluation/create-job#prediction-keys).
				685	"a_key": "A String",
				686	},
				687	"boundingPolyConfig": { # Config for image bounding poly (and bounding box) human labeling task. # Specify this field if your model version performs image object detection (bounding box detection). `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet.
				688	"annotationSpecSet": "A String", # Required. Annotation spec set resource name.
				689	"instructionMessage": "A String", # Optional. Instruction message showed on contributors UI.
				690	},
				691	"evaluationConfig": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Required. Details for calculating evaluation metrics and creating Evaulations. If your model version performs image object detection, you must specify the `boundingBoxEvaluationOptions` field within this configuration. Otherwise, provide an empty object for this configuration.
				692	"boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.
				693	"iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.
Yoshi Automation Bot	c2228be	2020-11-24 15:48:03 -0800	[diff] [blame]	694	},
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	695	},
				696	"evaluationJobAlertConfig": { # Provides details for how an evaluation job sends email alerts based on the results of a run. # Optional. Configuration details for evaluation job alerts. Specify this field if you want to receive email alerts if the evaluation job finds that your predictions have low mean average precision during a run.
				697	"email": "A String", # Required. An email address to send alerts to.
				698	"minAcceptableMeanAveragePrecision": 3.14, # Required. A number between 0 and 1 that describes a minimum mean average precision threshold. When the evaluation job runs, if it calculates that your model version's predictions from the recent interval have meanAveragePrecision below this threshold, then it sends an alert to your specified email.
				699	},
				700	"exampleCount": 42, # Required. The maximum number of predictions to sample and save to BigQuery during each evaluation interval. This limit overrides `example_sample_percentage`: even if the service has not sampled enough predictions to fulfill `example_sample_perecentage` during an interval, it stops sampling predictions when it meets this limit.
				701	"exampleSamplePercentage": 3.14, # Required. Fraction of predictions to sample and save to BigQuery during each evaluation interval. For example, 0.1 means 10% of predictions served by your model version get saved to BigQuery.
				702	"humanAnnotationConfig": { # Configuration for how human labeling task should be done. # Optional. Details for human annotation of your data. If you set labelMissingGroundTruth to `true` for this evaluation job, then you must specify this field. If you plan to provide your own ground truth labels, then omit this field. Note that you must create an Instruction resource before you can specify this field. Provide the name of the instruction resource in the `instruction` field within this configuration.
				703	"annotatedDatasetDescription": "A String", # Optional. A human-readable description for AnnotatedDataset. The description can be up to 10000 characters long.
				704	"annotatedDatasetDisplayName": "A String", # Required. A human-readable name for AnnotatedDataset defined by users. Maximum of 64 characters .
				705	"contributorEmails": [ # Optional. If you want your own labeling contributors to manage and work on this labeling request, you can set these contributors here. We will give them access to the question types in crowdcompute. Note that these emails must be registered in crowdcompute worker UI: https://crowd-compute.appspot.com/
				706	"A String",
				707	],
				708	"instruction": "A String", # Required. Instruction resource name.
				709	"labelGroup": "A String", # Optional. A human-readable label used to logically group labeling tasks. This string must match the regular expression `[a-zA-Z\\d_-]{0,128}`.
				710	"languageCode": "A String", # Optional. The Language of this question, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US. Only need to set this when task is language related. For example, French text classification.
				711	"questionDuration": "A String", # Optional. Maximum duration for contributors to answer a question. Maximum is 3600 seconds. Default is 3600 seconds.
				712	"replicaCount": 42, # Optional. Replication of questions. Each question will be sent to up to this number of contributors to label. Aggregated answers will be returned. Default is set to 1. For image related labeling, valid values are 1, 3, 5.
				713	"userEmailAddress": "A String", # Email of the user who started the labeling task and should be notified by email. If empty no notification will be sent.
				714	},
				715	"imageClassificationConfig": { # Config for image classification human labeling task. # Specify this field if your model version performs image classification or general classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
				716	"allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one image.
				717	"annotationSpecSet": "A String", # Required. Annotation spec set resource name.
				718	"answerAggregationType": "A String", # Optional. The type of how to aggregate answers.
				719	},
				720	"inputConfig": { # The configuration of input data, including data type, location, etc. # Rquired. Details for the sampled prediction input. Within this configuration, there are requirements for several fields: * `dataType` must be one of `IMAGE`, `TEXT`, or `GENERAL_DATA`. * `annotationType` must be one of `IMAGE_CLASSIFICATION_ANNOTATION`, `TEXT_CLASSIFICATION_ANNOTATION`, `GENERAL_CLASSIFICATION_ANNOTATION`, or `IMAGE_BOUNDING_BOX_ANNOTATION` (image object detection). * If your machine learning model performs classification, you must specify `classificationMetadata.isMultiLabel`. * You must specify `bigquerySource` (not `gcsSource`).
				721	"annotationType": "A String", # Optional. The type of annotation to be performed on this data. You must specify this field if you are using this InputConfig in an EvaluationJob.
				722	"bigquerySource": { # The BigQuery location for input data. If used in an EvaluationJob, this is where the service saves the prediction input and output sampled from the model version. # Source located in BigQuery. You must specify this field if you are using this InputConfig in an EvaluationJob.
				723	"inputUri": "A String", # Required. BigQuery URI to a table, up to 2,000 characters long. If you specify the URI of a table that does not exist, Data Labeling Service creates a table at the URI with the correct schema when you create your EvaluationJob. If you specify the URI of a table that already exists, it must have the [correct schema](/ml-engine/docs/continuous-evaluation/create-job#table-schema). Provide the table URI in the following format: "bq://{your_project_id}/ {your_dataset_name}/{your_table_name}" [Learn more](/ml-engine/docs/continuous-evaluation/create-job#table-schema).
				724	},
				725	"classificationMetadata": { # Metadata for classification annotations. # Optional. Metadata about annotations for the input. You must specify this field if you are using this InputConfig in an EvaluationJob for a model version that performs classification.
				726	"isMultiLabel": True or False, # Whether the classification task is multi-label or not.
				727	},
				728	"dataType": "A String", # Required. Data type must be specifed when user tries to import data.
				729	"gcsSource": { # Source of the Cloud Storage file to be imported. # Source located in Cloud Storage.
				730	"inputUri": "A String", # Required. The input URI of source file. This must be a Cloud Storage path (`gs://...`).
				731	"mimeType": "A String", # Required. The format of the source file. Only "text/csv" is supported.
				732	},
				733	"textMetadata": { # Metadata for the text. # Required for text import, as language code must be specified.
				734	"languageCode": "A String", # The language of this text, as a [BCP-47](https://www.rfc-editor.org/rfc/bcp/bcp47.txt). Default value is en-US.
				735	},
				736	},
				737	"textClassificationConfig": { # Config for text classification human labeling task. # Specify this field if your model version performs text classification. `annotationSpecSet` in this configuration must match EvaluationJob.annotationSpecSet. `allowMultiLabel` in this configuration must match `classificationMetadata.isMultiLabel` in input_config.
				738	"allowMultiLabel": True or False, # Optional. If allow_multi_label is true, contributors are able to choose multiple labels for one text segment.
				739	"annotationSpecSet": "A String", # Required. Annotation spec set resource name.
				740	"sentimentConfig": { # Config for setting up sentiments. # Optional. Configs for sentiment selection.
				741	"enableLabelSentimentSelection": True or False, # If set to true, contributors will have the option to select sentiment of the label they selected, to mark it as negative or positive label. Default is false.
				742	},
				743	},
				744	},
				745	"labelMissingGroundTruth": True or False, # Required. Whether you want Data Labeling Service to provide ground truth labels for prediction input. If you want the service to assign human labelers to annotate your data, set this to `true`. If you want to provide your own ground truth labels in the evaluation job's BigQuery table, set this to `false`.
				746	"modelVersion": "A String", # Required. The [AI Platform Prediction model version](/ml-engine/docs/prediction-overview) to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
				747	"name": "A String", # Output only. After you create a job, Data Labeling Service assigns a name to the job with the following format: "projects/{project_id}/evaluationJobs/ {evaluation_job_id}"
				748	"schedule": "A String", # Required. Describes the interval at which the job runs. This interval must be at least 1 day, and it is rounded to the nearest day. For example, if you specify a 50-hour interval, the job runs every 2 days. You can provide the schedule in [crontab format](/scheduler/docs/configuring/cron-job-schedules) or in an [English-like format](/appengine/docs/standard/python/config/cronref#schedule_format). Regardless of what you specify, the job will run at 10:00 AM UTC. Only the interval from this schedule is used, not the specific time of day.
				749	"state": "A String", # Output only. Describes the current state of the job.
				750	}</pre>
Bu Sun Kim	673ec5c	2020-11-16 11:05:03 -0700	[diff] [blame]	751	</div>
				752
				753	<div class="method">
				754	<code class="details" id="pause">pause(name, body=None, x__xgafv=None)</code>
				755	<pre>Pauses an evaluation job. Pausing an evaluation job that is already in a `PAUSED` state is a no-op.
				756
				757	Args:
				758	name: string, Required. Name of the evaluation job that is going to be paused. Format: "projects/{project_id}/evaluationJobs/{evaluation_job_id}" (required)
				759	body: object, The request body.
				760	The object takes the form of:
				761
				762	{ # Request message for PauseEvaluationJob.
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	763	}
Bu Sun Kim	673ec5c	2020-11-16 11:05:03 -0700	[diff] [blame]	764
				765	x__xgafv: string, V1 error format.
				766	Allowed values
				767	1 - v1 error format
				768	2 - v2 error format
				769
				770	Returns:
				771	An object of the form:
				772
				773	{ # A generic empty message that you can re-use to avoid defining duplicated empty messages in your APIs. A typical example is to use it as the request or the response type of an API method. For instance: service Foo { rpc Bar(google.protobuf.Empty) returns (google.protobuf.Empty); } The JSON representation for `Empty` is empty JSON object `{}`.
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	774	}</pre>
Bu Sun Kim	673ec5c	2020-11-16 11:05:03 -0700	[diff] [blame]	775	</div>
				776
				777	<div class="method">
				778	<code class="details" id="resume">resume(name, body=None, x__xgafv=None)</code>
				779	<pre>Resumes a paused evaluation job. A deleted evaluation job can't be resumed. Resuming a running or scheduled evaluation job is a no-op.
				780
				781	Args:
				782	name: string, Required. Name of the evaluation job that is going to be resumed. Format: "projects/{project_id}/evaluationJobs/{evaluation_job_id}" (required)
				783	body: object, The request body.
				784	The object takes the form of:
				785
				786	{ # Request message ResumeEvaluationJob.
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	787	}
Bu Sun Kim	673ec5c	2020-11-16 11:05:03 -0700	[diff] [blame]	788
				789	x__xgafv: string, V1 error format.
				790	Allowed values
				791	1 - v1 error format
				792	2 - v2 error format
				793
				794	Returns:
				795	An object of the form:
				796
				797	{ # A generic empty message that you can re-use to avoid defining duplicated empty messages in your APIs. A typical example is to use it as the request or the response type of an API method. For instance: service Foo { rpc Bar(google.protobuf.Empty) returns (google.protobuf.Empty); } The JSON representation for `Empty` is empty JSON object `{}`.
Yoshi Automation Bot	cc94ec8	2021-01-15 07:10:04 -0800	[diff] [blame^]	798	}</pre>
Bu Sun Kim	673ec5c	2020-11-16 11:05:03 -0700	[diff] [blame]	799	</div>
				800
				801	</body></html>