blob: 94d3277601cf7be01f76400b5caa8a15aa7c9444 [file] [log] [blame]
Jon Wayne Parrott692617a2017-01-06 09:58:29 -08001<html><body>
2<style>
3
4body, h1, h2, h3, div, span, p, pre, a {
5 margin: 0;
6 padding: 0;
7 border: 0;
8 font-weight: inherit;
9 font-style: inherit;
10 font-size: 100%;
11 font-family: inherit;
12 vertical-align: baseline;
13}
14
15body {
16 font-size: 13px;
17 padding: 1em;
18}
19
20h1 {
21 font-size: 26px;
22 margin-bottom: 1em;
23}
24
25h2 {
26 font-size: 24px;
27 margin-bottom: 1em;
28}
29
30h3 {
31 font-size: 20px;
32 margin-bottom: 1em;
33 margin-top: 1em;
34}
35
36pre, code {
37 line-height: 1.5;
38 font-family: Monaco, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console', monospace;
39}
40
41pre {
42 margin-top: 0.5em;
43}
44
45h1, h2, h3, p {
46 font-family: Arial, sans serif;
47}
48
49h1, h2, h3 {
50 border-bottom: solid #CCC 1px;
51}
52
53.toc_element {
54 margin-top: 0.5em;
55}
56
57.firstline {
58 margin-left: 2 em;
59}
60
61.method {
62 margin-top: 1em;
63 border: solid 1px #CCC;
64 padding: 1em;
65 background: #EEE;
66}
67
68.details {
69 font-weight: bold;
70 font-size: 14px;
71}
72
73</style>
74
Bu Sun Kim715bd7f2019-06-14 16:50:42 -070075<h1><a href="dataflow_v1b3.html">Dataflow API</a> . <a href="dataflow_v1b3.projects.html">projects</a> . <a href="dataflow_v1b3.projects.locations.html">locations</a></h1>
Jon Wayne Parrott692617a2017-01-06 09:58:29 -080076<h2>Instance Methods</h2>
77<p class="toc_element">
Dan O'Mearadd494642020-05-01 07:42:23 -070078 <code><a href="dataflow_v1b3.projects.locations.flexTemplates.html">flexTemplates()</a></code>
79</p>
80<p class="firstline">Returns the flexTemplates Resource.</p>
81
82<p class="toc_element">
Jon Wayne Parrott692617a2017-01-06 09:58:29 -080083 <code><a href="dataflow_v1b3.projects.locations.jobs.html">jobs()</a></code>
84</p>
85<p class="firstline">Returns the jobs Resource.</p>
86
Sai Cheemalapati4ba8c232017-06-06 18:46:08 -040087<p class="toc_element">
Bu Sun Kim715bd7f2019-06-14 16:50:42 -070088 <code><a href="dataflow_v1b3.projects.locations.snapshots.html">snapshots()</a></code>
89</p>
90<p class="firstline">Returns the snapshots Resource.</p>
91
92<p class="toc_element">
93 <code><a href="dataflow_v1b3.projects.locations.sql.html">sql()</a></code>
94</p>
95<p class="firstline">Returns the sql Resource.</p>
96
97<p class="toc_element">
Sai Cheemalapati4ba8c232017-06-06 18:46:08 -040098 <code><a href="dataflow_v1b3.projects.locations.templates.html">templates()</a></code>
99</p>
100<p class="firstline">Returns the templates Resource.</p>
101
102<p class="toc_element">
Dan O'Mearadd494642020-05-01 07:42:23 -0700103 <code><a href="#workerMessages">workerMessages(projectId, location, body=None, x__xgafv=None)</a></code></p>
Sai Cheemalapati4ba8c232017-06-06 18:46:08 -0400104<p class="firstline">Send a worker_message to the service.</p>
105<h3>Method Details</h3>
106<div class="method">
Dan O'Mearadd494642020-05-01 07:42:23 -0700107 <code class="details" id="workerMessages">workerMessages(projectId, location, body=None, x__xgafv=None)</code>
Sai Cheemalapati4ba8c232017-06-06 18:46:08 -0400108 <pre>Send a worker_message to the service.
109
110Args:
111 projectId: string, The project to send the WorkerMessages to. (required)
Bu Sun Kim715bd7f2019-06-14 16:50:42 -0700112 location: string, The [regional endpoint]
113(https://cloud.google.com/dataflow/docs/concepts/regional-endpoints) that
114contains the job. (required)
Dan O'Mearadd494642020-05-01 07:42:23 -0700115 body: object, The request body.
Sai Cheemalapati4ba8c232017-06-06 18:46:08 -0400116 The object takes the form of:
117
118{ # A request for sending worker messages to the service.
Bu Sun Kim65020912020-05-20 12:08:20 -0700119 &quot;location&quot;: &quot;A String&quot;, # The [regional endpoint]
120 # (https://cloud.google.com/dataflow/docs/concepts/regional-endpoints) that
121 # contains the job.
122 &quot;workerMessages&quot;: [ # The WorkerMessages to send.
Sai Cheemalapati4ba8c232017-06-06 18:46:08 -0400123 { # WorkerMessage provides information to the backend about a worker.
Bu Sun Kim65020912020-05-20 12:08:20 -0700124 &quot;workerMetrics&quot;: { # Worker metrics exported from workers. This contains resource utilization # Resource metrics reported by workers.
125 # metrics accumulated from a variety of sources. For more information, see
126 # go/df-resource-signals.
Bu Sun Kim65020912020-05-20 12:08:20 -0700127 &quot;containers&quot;: { # Per container information.
128 # Key: container name.
129 &quot;a_key&quot;: # Object with schema name: ResourceUtilizationReport
130 },
Bu Sun Kim4ed7d3f2020-05-27 12:20:54 -0700131 &quot;memoryInfo&quot;: [ # Memory utilization samples.
132 { # Information about the memory usage of a worker or a container within a
133 # worker.
134 &quot;timestamp&quot;: &quot;A String&quot;, # Timestamp of the measurement.
135 &quot;currentLimitBytes&quot;: &quot;A String&quot;, # Instantenous memory limit in bytes.
136 &quot;currentRssBytes&quot;: &quot;A String&quot;, # Instantenous memory (RSS) size in bytes.
137 &quot;totalGbMs&quot;: &quot;A String&quot;, # Total memory (RSS) usage since start up in GB * ms.
138 },
139 ],
Bu Sun Kim65020912020-05-20 12:08:20 -0700140 &quot;cpuTime&quot;: [ # CPU utilization samples.
141 { # Modeled after information exposed by /proc/stat.
Bu Sun Kim65020912020-05-20 12:08:20 -0700142 &quot;rate&quot;: 3.14, # Average CPU utilization rate (% non-idle cpu / second) since previous
143 # sample.
144 &quot;timestamp&quot;: &quot;A String&quot;, # Timestamp of the measurement.
Bu Sun Kim4ed7d3f2020-05-27 12:20:54 -0700145 &quot;totalMs&quot;: &quot;A String&quot;, # Total active CPU time across all cores (ie., non-idle) in milliseconds
146 # since start-up.
Bu Sun Kim65020912020-05-20 12:08:20 -0700147 },
148 ],
149 },
150 &quot;workerMessageCode&quot;: { # A message code is used to report status and error messages to the service. # A worker message code.
Sai Cheemalapati4ba8c232017-06-06 18:46:08 -0400151 # The message codes are intended to be machine readable. The service will
152 # take care of translating these into user understandable messages if
153 # necessary.
154 #
155 # Example use cases:
156 # 1. Worker processes reporting successful startup.
157 # 2. Worker processes reporting specific errors (e.g. package staging
158 # failure).
Bu Sun Kim4ed7d3f2020-05-27 12:20:54 -0700159 &quot;code&quot;: &quot;A String&quot;, # The code is a string intended for consumption by a machine that identifies
160 # the type of message being sent.
161 # Examples:
162 # 1. &quot;HARNESS_STARTED&quot; might be used to indicate the worker harness has
163 # started.
164 # 2. &quot;GCS_DOWNLOAD_ERROR&quot; might be used to indicate an error downloading
165 # a GCS file as part of the boot process of one of the worker containers.
166 #
167 # This is a string and not an enum to make it easy to add new codes without
168 # waiting for an API change.
Bu Sun Kim65020912020-05-20 12:08:20 -0700169 &quot;parameters&quot;: { # Parameters contains specific information about the code.
Sai Cheemalapati4ba8c232017-06-06 18:46:08 -0400170 #
171 # This is a struct to allow parameters of different types.
172 #
173 # Examples:
Bu Sun Kim65020912020-05-20 12:08:20 -0700174 # 1. For a &quot;HARNESS_STARTED&quot; message parameters might provide the name
Sai Cheemalapati4ba8c232017-06-06 18:46:08 -0400175 # of the worker and additional data like timing information.
Bu Sun Kim65020912020-05-20 12:08:20 -0700176 # 2. For a &quot;GCS_DOWNLOAD_ERROR&quot; parameters might contain fields listing
Sai Cheemalapati4ba8c232017-06-06 18:46:08 -0400177 # the GCS objects being downloaded and fields containing errors.
178 #
179 # In general complex data structures should be avoided. If a worker
180 # needs to send a specific and complicated data structure then please
181 # consider defining a new proto and adding it to the data oneof in
182 # WorkerMessageResponse.
183 #
184 # Conventions:
Bu Sun Kim65020912020-05-20 12:08:20 -0700185 # Parameters should only be used for information that isn&#x27;t typically passed
Sai Cheemalapati4ba8c232017-06-06 18:46:08 -0400186 # as a label.
187 # hostname and other worker identifiers should almost always be passed
188 # as labels since they will be included on most messages.
Bu Sun Kim65020912020-05-20 12:08:20 -0700189 &quot;a_key&quot;: &quot;&quot;, # Properties of the object.
Sai Cheemalapati4ba8c232017-06-06 18:46:08 -0400190 },
Bu Sun Kim4ed7d3f2020-05-27 12:20:54 -0700191 },
192 &quot;labels&quot;: { # Labels are used to group WorkerMessages.
193 # For example, a worker_message about a particular container
194 # might have the labels:
195 # { &quot;JOB_ID&quot;: &quot;2015-04-22&quot;,
196 # &quot;WORKER_ID&quot;: &quot;wordcount-vm-2015…&quot;
197 # &quot;CONTAINER_TYPE&quot;: &quot;worker&quot;,
198 # &quot;CONTAINER_ID&quot;: &quot;ac1234def&quot;}
199 # Label tags typically correspond to Label enum values. However, for ease
200 # of development other strings can be used as tags. LABEL_UNSPECIFIED should
201 # not be used here.
202 &quot;a_key&quot;: &quot;A String&quot;,
203 },
204 &quot;time&quot;: &quot;A String&quot;, # The timestamp of the worker_message.
205 &quot;workerLifecycleEvent&quot;: { # A report of an event in a worker&#x27;s lifecycle. # Record of worker lifecycle events.
206 # The proto contains one event, because the worker is expected to
207 # asynchronously send each message immediately after the event.
208 # Due to this asynchrony, messages may arrive out of order (or missing), and it
209 # is up to the consumer to interpret.
210 # The timestamp of the event is in the enclosing WorkerMessage proto.
211 &quot;containerStartTime&quot;: &quot;A String&quot;, # The start time of this container. All events will report this so that
212 # events can be grouped together across container/VM restarts.
213 &quot;event&quot;: &quot;A String&quot;, # The event being reported.
214 &quot;metadata&quot;: { # Other stats that can accompany an event. E.g.
215 # { &quot;downloaded_bytes&quot; : &quot;123456&quot; }
216 &quot;a_key&quot;: &quot;A String&quot;,
217 },
218 },
219 &quot;workerShutdownNotice&quot;: { # Shutdown notification from workers. This is to be sent by the shutdown # Shutdown notice by workers.
220 # script of the worker VM so that the backend knows that the VM is being
221 # shut down.
222 &quot;reason&quot;: &quot;A String&quot;, # The reason for the worker shutdown.
223 # Current possible values are:
224 # &quot;UNKNOWN&quot;: shutdown reason is unknown.
225 # &quot;PREEMPTION&quot;: shutdown reason is preemption.
226 # Other possible reasons may be added in the future.
227 },
228 &quot;workerHealthReport&quot;: { # WorkerHealthReport contains information about the health of a worker. # The health of a worker.
229 #
230 # The VM should be identified by the labels attached to the WorkerMessage that
231 # this health ping belongs to.
232 &quot;pods&quot;: [ # The pods running on the worker. See:
233 # http://kubernetes.io/v1.1/docs/api-reference/v1/definitions.html#_v1_pod
Bu Sun Kim715bd7f2019-06-14 16:50:42 -0700234 #
Bu Sun Kim4ed7d3f2020-05-27 12:20:54 -0700235 # This field is used by the worker to send the status of the indvidual
236 # containers running on each worker.
237 {
238 &quot;a_key&quot;: &quot;&quot;, # Properties of the object.
239 },
240 ],
241 &quot;reportInterval&quot;: &quot;A String&quot;, # The interval at which the worker is sending health reports.
242 # The default value of 0 should be interpreted as the field is not being
243 # explicitly set by the worker.
244 &quot;vmIsHealthy&quot;: True or False, # Whether the VM is currently healthy.
245 &quot;vmIsBroken&quot;: True or False, # Whether the VM is in a permanently broken state.
246 # Broken VMs should be abandoned or deleted ASAP to avoid assigning or
247 # completing any work.
248 &quot;vmStartupTime&quot;: &quot;A String&quot;, # The time the VM was booted.
249 &quot;msg&quot;: &quot;A String&quot;, # A message describing any unusual health reports.
Bu Sun Kim715bd7f2019-06-14 16:50:42 -0700250 },
Sai Cheemalapati4ba8c232017-06-06 18:46:08 -0400251 },
252 ],
Sai Cheemalapati4ba8c232017-06-06 18:46:08 -0400253 }
254
255 x__xgafv: string, V1 error format.
256 Allowed values
257 1 - v1 error format
258 2 - v2 error format
259
260Returns:
261 An object of the form:
262
263 { # The response to the worker messages.
Bu Sun Kim65020912020-05-20 12:08:20 -0700264 &quot;workerMessageResponses&quot;: [ # The servers response to the worker messages.
Sai Cheemalapati4ba8c232017-06-06 18:46:08 -0400265 { # A worker_message response allows the server to pass information to the
266 # sender.
Bu Sun Kim65020912020-05-20 12:08:20 -0700267 &quot;workerHealthReportResponse&quot;: { # WorkerHealthReportResponse contains information returned to the worker # The service&#x27;s response to a worker&#x27;s health report.
Sai Cheemalapati4ba8c232017-06-06 18:46:08 -0400268 # in response to a health ping.
Bu Sun Kim65020912020-05-20 12:08:20 -0700269 &quot;reportInterval&quot;: &quot;A String&quot;, # A positive value indicates the worker should change its reporting interval
Sai Cheemalapati4ba8c232017-06-06 18:46:08 -0400270 # to the specified value.
271 #
272 # The default value of zero means no change in report rate is requested by
273 # the server.
274 },
Bu Sun Kim4ed7d3f2020-05-27 12:20:54 -0700275 &quot;workerShutdownNoticeResponse&quot;: { # Service-side response to WorkerMessage issuing shutdown notice. # Service&#x27;s response to shutdown notice (currently empty).
276 },
277 &quot;workerMetricsResponse&quot;: { # Service-side response to WorkerMessage reporting resource utilization. # Service&#x27;s response to reporting worker metrics (currently empty).
278 },
Sai Cheemalapati4ba8c232017-06-06 18:46:08 -0400279 },
280 ],
281 }</pre>
282</div>
283
Jon Wayne Parrott692617a2017-01-06 09:58:29 -0800284</body></html>