Blame - docs/dyn/datalabeling_v1beta1.projects.evaluations.html - platform/external/python/google-api-python-client

<h1><a href="datalabeling_v1beta1.html">Data Labeling API</a> . <a href="datalabeling_v1beta1.projects.html">projects</a> . <a href="datalabeling_v1beta1.projects.evaluations.html">evaluations</a></h1>

76

<h2>Instance Methods</h2>

77

78

<code><a href="#close">close()</a></code></p>

79

<p class="firstline">Close httplib2 connections.</p>

80

Yoshi Automation Bot

c2228be

2020-11-24 15:48:03 -0800

[diff] [blame^]

81

<code><a href="#search">search(parent, pageSize=None, pageToken=None, filter=None, x__xgafv=None)</a></code></p>

Bu Sun Kim

673ec5c

2020-11-16 11:05:03 -0700

[diff] [blame]

82

<p class="firstline">Searches evaluations within a project.</p>

83

84

<code><a href="#search_next">search_next(previous_request, previous_response)</a></code></p>

85

<p class="firstline">Retrieves the next page of results.</p>

86

<h3>Method Details</h3>

87

88

<code class="details" id="close">close()</code>

89

<pre>Close httplib2 connections.</pre>

</div>

Yoshi Automation Bot

2020-11-24 15:48:03 -0800

[diff] [blame^]

93

<code class="details" id="search">search(parent, pageSize=None, pageToken=None, filter=None, x__xgafv=None)</code>

Bu Sun Kim

673ec5c

2020-11-16 11:05:03 -0700

[diff] [blame]

94

<pre>Searches evaluations within a project.

95

96

Args:

97

parent: string, Required. Evaluation search parent (project ID). Format: "projects/ {project_id}" (required)

Bu Sun Kim

673ec5c

2020-11-16 11:05:03 -0700

[diff] [blame]

98

pageSize: integer, Optional. Requested page size. Server may return fewer results than requested. Default value is 100.

99

pageToken: string, Optional. A token identifying a page of results for the server to return. Typically obtained by the nextPageToken of the response to a previous search request. If you don't specify this field, the API call requests the first page of the search.

Yoshi Automation Bot

c2228be

2020-11-24 15:48:03 -0800

[diff] [blame^]

100

filter: string, Optional. To search evaluations, you can filter by the following: * evaluation_job.evaluation_job_id (the last part of EvaluationJob.name) * evaluation_job.model_id (the {model_name} portion of EvaluationJob.modelVersion) * evaluation_job.evaluation_job_run_time_start (Minimum threshold for the evaluationJobRunTime that created the evaluation) * evaluation_job.evaluation_job_run_time_end (Maximum threshold for the evaluationJobRunTime that created the evaluation) * evaluation_job.job_state (EvaluationJob.state) * annotation_spec.display_name (the Evaluation contains a metric for the annotation spec with this displayName) To filter by multiple critiera, use the `AND` operator or the `OR` operator. The following examples shows a string that filters by several critiera: "evaluation_job.evaluation_job_id = {evaluation_job_id} AND evaluation_job.model_id = {model_name} AND evaluation_job.evaluation_job_run_time_start = {timestamp_1} AND evaluation_job.evaluation_job_run_time_end = {timestamp_2} AND annotation_spec.display_name = {display_name}"

Bu Sun Kim

673ec5c

2020-11-16 11:05:03 -0700

[diff] [blame]

101

x__xgafv: string, V1 error format.

Allowed values

1 - v1 error format

2 - v2 error format

Returns:

An object of the form:

108

109

{ # Results of searching evaluations.

110

"nextPageToken": "A String", # A token to retrieve next page of results.

111

"evaluations": [ # The list of evaluations matching the search.

112

{ # Describes an evaluation between a machine learning model's predictions and ground truth labels. Created when an EvaluationJob runs successfully.

Bu Sun Kim

673ec5c

2020-11-16 11:05:03 -0700

[diff] [blame]

113

"annotationType": "A String", # Output only. Type of task that the model version being evaluated performs, as defined in the evaluationJobConfig.inputConfig.annotationType field of the evaluation job that created this evaluation.

Yoshi Automation Bot

c2228be

2020-11-24 15:48:03 -0800

[diff] [blame^]

114

"name": "A String", # Output only. Resource name of an evaluation. The name has the following format: "projects/{project_id}/datasets/{dataset_id}/evaluations/ {evaluation_id}'

115

"evaluationJobRunTime": "A String", # Output only. Timestamp for when the evaluation job that created this evaluation ran.

Bu Sun Kim

673ec5c

2020-11-16 11:05:03 -0700

[diff] [blame]

116

"config": { # Configuration details used for calculating evaluation metrics and creating an Evaluation. # Output only. Options used in the evaluation job that created this evaluation.

117

"boundingBoxEvaluationOptions": { # Options regarding evaluation between bounding boxes. # Only specify this field if the related model performs image object detection (`IMAGE_BOUNDING_BOX_ANNOTATION`). Describes how to evaluate bounding boxes.

118

"iouThreshold": 3.14, # Minimum [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) required for 2 bounding boxes to be considered a match. This must be a number between 0 and 1.

119

},

120

},

121

"createTime": "A String", # Output only. Timestamp for when this evaluation was created.

Yoshi Automation Bot

c2228be

2020-11-24 15:48:03 -0800

[diff] [blame^]

122

"evaluationMetrics": { # Output only. Metrics comparing predictions to ground truth labels.

123

"objectDetectionMetrics": { # Metrics calculated for an image object detection (bounding box) model.

124

"prCurve": { # Precision-recall curve.

125

"meanAveragePrecision": 3.14, # Mean average prcision of this curve.

126

"annotationSpec": { # Container of information related to one possible annotation that can be used in a labeling task. For example, an image classification task where images are labeled as `dog` or `cat` must reference an AnnotationSpec for `dog` and an AnnotationSpec for `cat`. # The annotation spec of the label for which the precision-recall curve calculated. If this field is empty, that means the precision-recall curve is an aggregate curve for all labels.

127

"description": "A String", # Optional. User-provided description of the annotation specification. The description can be up to 10,000 characters long.

128

"displayName": "A String", # Required. The display name of the AnnotationSpec. Maximum of 64 characters.

129

"index": 42, # Output only. This is the integer index of the AnnotationSpec. The index for the whole AnnotationSpecSet is sequential starting from 0. For example, an AnnotationSpecSet with classes `dog` and `cat`, might contain one AnnotationSpec with `{ display_name: "dog", index: 0 }` and one AnnotationSpec with `{ display_name: "cat", index: 1 }`. This is especially useful for model training as it encodes the string labels into numeric values.

130

},

131

"areaUnderCurve": 3.14, # Area under the precision-recall curve. Not to be confused with area under a receiver operating characteristic (ROC) curve.

132

"confidenceMetricsEntries": [ # Entries that make up the precision-recall graph. Each entry is a "point" on the graph drawn for a different `confidence_threshold`.

133

{

134

"recallAt1": 3.14, # Recall value for entries with label that has highest score.

135

"f1Score": 3.14, # Harmonic mean of recall and precision.

136

"precision": 3.14, # Precision value.

137

"precisionAt1": 3.14, # Precision value for entries with label that has highest score.

138

"f1ScoreAt1": 3.14, # The harmonic mean of recall_at1 and precision_at1.

139

"confidenceThreshold": 3.14, # Threshold used for this entry. For classification tasks, this is a classification threshold: a predicted label is categorized as positive or negative (in the context of this point on the PR curve) based on whether the label's score meets this threshold. For image object detection (bounding box) tasks, this is the [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) threshold for the context of this point on the PR curve.

140

"precisionAt5": 3.14, # Precision value for entries with label that has highest 5 scores.

141

"f1ScoreAt5": 3.14, # The harmonic mean of recall_at5 and precision_at5.

142

"recallAt5": 3.14, # Recall value for entries with label that has highest 5 scores.

143

"recall": 3.14, # Recall value.

},

],

},

},

"classificationMetrics": { # Metrics calculated for a classification model.

149

"prCurve": { # Precision-recall curve based on ground truth labels, predicted labels, and scores for the predicted labels.

150

"meanAveragePrecision": 3.14, # Mean average prcision of this curve.

151

"annotationSpec": { # Container of information related to one possible annotation that can be used in a labeling task. For example, an image classification task where images are labeled as `dog` or `cat` must reference an AnnotationSpec for `dog` and an AnnotationSpec for `cat`. # The annotation spec of the label for which the precision-recall curve calculated. If this field is empty, that means the precision-recall curve is an aggregate curve for all labels.

152

"description": "A String", # Optional. User-provided description of the annotation specification. The description can be up to 10,000 characters long.

153

"displayName": "A String", # Required. The display name of the AnnotationSpec. Maximum of 64 characters.

154

"index": 42, # Output only. This is the integer index of the AnnotationSpec. The index for the whole AnnotationSpecSet is sequential starting from 0. For example, an AnnotationSpecSet with classes `dog` and `cat`, might contain one AnnotationSpec with `{ display_name: "dog", index: 0 }` and one AnnotationSpec with `{ display_name: "cat", index: 1 }`. This is especially useful for model training as it encodes the string labels into numeric values.

155

},

156

"areaUnderCurve": 3.14, # Area under the precision-recall curve. Not to be confused with area under a receiver operating characteristic (ROC) curve.

157

"confidenceMetricsEntries": [ # Entries that make up the precision-recall graph. Each entry is a "point" on the graph drawn for a different `confidence_threshold`.

158

{

159

"recallAt1": 3.14, # Recall value for entries with label that has highest score.

160

"f1Score": 3.14, # Harmonic mean of recall and precision.

161

"precision": 3.14, # Precision value.

162

"precisionAt1": 3.14, # Precision value for entries with label that has highest score.

163

"f1ScoreAt1": 3.14, # The harmonic mean of recall_at1 and precision_at1.

164

"confidenceThreshold": 3.14, # Threshold used for this entry. For classification tasks, this is a classification threshold: a predicted label is categorized as positive or negative (in the context of this point on the PR curve) based on whether the label's score meets this threshold. For image object detection (bounding box) tasks, this is the [intersection-over-union (IOU)](/vision/automl/object-detection/docs/evaluate#intersection-over-union) threshold for the context of this point on the PR curve.

165

"precisionAt5": 3.14, # Precision value for entries with label that has highest 5 scores.

166

"f1ScoreAt5": 3.14, # The harmonic mean of recall_at5 and precision_at5.

167

"recallAt5": 3.14, # Recall value for entries with label that has highest 5 scores.

168

"recall": 3.14, # Recall value.

},

],

},

"confusionMatrix": { # Confusion matrix of the model running the classification. Only applicable when the metrics entry aggregates multiple labels. Not applicable when the entry is for a single label. # Confusion matrix of predicted labels vs. ground truth labels.

173

"row": [

174

{ # A row in the confusion matrix. Each entry in this row has the same ground truth label.

175

"entries": [ # A list of the confusion matrix entries. One entry for each possible predicted label.

176

{

177

"annotationSpec": { # Container of information related to one possible annotation that can be used in a labeling task. For example, an image classification task where images are labeled as `dog` or `cat` must reference an AnnotationSpec for `dog` and an AnnotationSpec for `cat`. # The annotation spec of a predicted label.

178

"description": "A String", # Optional. User-provided description of the annotation specification. The description can be up to 10,000 characters long.

179

"displayName": "A String", # Required. The display name of the AnnotationSpec. Maximum of 64 characters.

180

"index": 42, # Output only. This is the integer index of the AnnotationSpec. The index for the whole AnnotationSpecSet is sequential starting from 0. For example, an AnnotationSpecSet with classes `dog` and `cat`, might contain one AnnotationSpec with `{ display_name: "dog", index: 0 }` and one AnnotationSpec with `{ display_name: "cat", index: 1 }`. This is especially useful for model training as it encodes the string labels into numeric values.

181

},

182

"itemCount": 42, # Number of items predicted to have this label. (The ground truth label for these items is the `Row.annotationSpec` of this entry's parent.)

183

},

184

],

185

"annotationSpec": { # Container of information related to one possible annotation that can be used in a labeling task. For example, an image classification task where images are labeled as `dog` or `cat` must reference an AnnotationSpec for `dog` and an AnnotationSpec for `cat`. # The annotation spec of the ground truth label for this row.

186

"description": "A String", # Optional. User-provided description of the annotation specification. The description can be up to 10,000 characters long.

187

"displayName": "A String", # Required. The display name of the AnnotationSpec. Maximum of 64 characters.

188

"index": 42, # Output only. This is the integer index of the AnnotationSpec. The index for the whole AnnotationSpecSet is sequential starting from 0. For example, an AnnotationSpecSet with classes `dog` and `cat`, might contain one AnnotationSpec with `{ display_name: "dog", index: 0 }` and one AnnotationSpec with `{ display_name: "cat", index: 1 }`. This is especially useful for model training as it encodes the string labels into numeric values.

},

},

],

},

},

},

Bu Sun Kim

673ec5c

2020-11-16 11:05:03 -0700

[diff] [blame]

195

"evaluatedItemCount": "A String", # Output only. The number of items in the ground truth dataset that were used for this evaluation. Only populated when the evaulation is for certain AnnotationTypes.

},

],

}</pre>

</div>

<code class="details" id="search_next">search_next(previous_request, previous_response)</code>