openstack
/
ironic
Code Issues Proposed changes

Implementation of inspect wait state 

This patch provides implementations to feature of adding inspect wait state.

Changes covered in this patch:

* Added state and transitions, state diagram regenerated.
* inspector and oneview inspect interface now return INSPECTWAIT instead of
  INSPECTING. Move node to inspect wait if inspect interface returns
  INSPECTING or INSPECTWAIT.
* Add a timeout option to conductor, and a periodic task to check timeout
  in the inspect wait state.

Story: #1725211
Task: #10630

Partial-Bug: #1725211
Change-Id: Ie76bfdad5966014a4dae826919ff5705462c743b
This commit is contained in:
Kaifeng Wang 2018-03-22 17:35:00 +08:00
parent c03dfbf2eb
commit 6df82ee2bc
20 changed files with 602 additions and 312 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
doc/source/contributor/states.rst
View File

@ -68,8 +68,15 @@ manageable (stable state)
inspecting
``inspecting`` will utilize node introspection to update hardware-derived
node properties to reflect the current state of the hardware. If
introspection fails, the node will transition to ``inspect failed``.
node properties to reflect the current state of the hardware. Typically,
the node will transition to ``manageable`` if inspection is synchronous,
or ``inspect wait`` if asynchronous. The node will transition to
``inspect failed`` if error occurred.
inspect wait
This is the provision state used when an asynchronous inspection is in
progress. A successfully inspected node shall transition to ``manageable``
state.
inspect failed
This is the state a node will move into when inspection of the node fails. From

6
doc/source/contributor/webapi-version-history.rst
View File

@ -2,6 +2,12 @@
REST API Version History
========================
1.39 (Rocky, master)
--------------------
Added ``inspect wait`` to available provision states. A node is shown as
``inspect wait`` instead of ``inspecting`` during asynchronous inspection.
1.38 (Queens, 10.1.0)
---------------------

527
doc/source/images/states.svg
View File

@ -4,479 +4,502 @@
<!-- Generated by graphviz version 2.38.0 (20140413.2041)
-->
<!-- Title: Ironic states Pages: 1 -->
<svg width="3138pt" height="777pt"
viewBox="0.00 0.00 3137.57 776.58" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 772.576)">
<svg width="2609pt" height="851pt"
viewBox="0.00 0.00 2609.29 851.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 847)">
<title>Ironic states</title>
<polygon fill="white" stroke="none" points="-4,4 -4,-772.576 3133.57,-772.576 3133.57,4 -4,4"/>
<polygon fill="white" stroke="none" points="-4,4 -4,-847 2605.29,-847 2605.29,4 -4,4"/>
<!-- enroll -->
<g id="node1" class="node"><title>enroll</title>
<ellipse fill="none" stroke="black" stroke-width="1.7" cx="30.6682" cy="-177.576" rx="30.8374" ry="18"/>
<text text-anchor="middle" x="30.6682" y="-174.776" font-family="Times,serif" font-size="11.00">enroll</text>
<ellipse fill="none" stroke="black" stroke-width="1.7" cx="27" cy="-233" rx="27" ry="18"/>
<text text-anchor="middle" x="27" y="-230.2" font-family="Times,serif" font-size="11.00">enroll</text>
</g>
<!-- verifying -->
<g id="node2" class="node"><title>verifying</title>
<ellipse fill="none" stroke="black" cx="244.227" cy="-177.576" rx="40.7822" ry="18"/>
<text text-anchor="middle" x="244.227" y="-174.776" font-family="Times,serif" font-size="11.00" fill="gray">verifying</text>
<ellipse fill="none" stroke="black" cx="208.675" cy="-233" rx="33.8507" ry="18"/>
<text text-anchor="middle" x="208.675" y="-230.2" font-family="Times,serif" font-size="11.00" fill="gray">verifying</text>
</g>
<!-- enroll&#45;&gt;verifying -->
<g id="edge1" class="edge"><title>enroll&#45;&gt;verifying</title>
<path fill="none" stroke="black" d="M61.5308,-177.576C95.6561,-177.576 152.061,-177.576 192.985,-177.576"/>
<polygon fill="black" stroke="black" points="193.033,-181.076 203.033,-177.576 193.033,-174.076 193.033,-181.076"/>
<text text-anchor="middle" x="132.336" y="-180.976" font-family="Times,serif" font-size="12.00">manage (via API)</text>
<path fill="none" stroke="black" d="M54.1263,-233C83.0573,-233 130.15,-233 164.565,-233"/>
<polygon fill="black" stroke="black" points="164.805,-236.5 174.805,-233 164.805,-229.5 164.805,-236.5"/>
<text text-anchor="middle" x="114.5" y="-236.4" font-family="Times,serif" font-size="12.00">manage (via API)</text>
</g>
<!-- verifying&#45;&gt;enroll -->
<g id="edge17" class="edge"><title>verifying&#45;&gt;enroll</title>
<path fill="none" stroke="black" d="M211.739,-166.561C203.273,-164.089 194.046,-161.828 185.336,-160.576 138.705,-153.87 125.758,-152.546 79.3365,-160.576 74.8317,-161.355 70.189,-162.52 65.6505,-163.881"/>
<polygon fill="black" stroke="black" points="64.3578,-160.622 55.9753,-167.102 66.5689,-167.264 64.3578,-160.622"/>
<text text-anchor="middle" x="132.336" y="-163.976" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
<path fill="none" stroke="black" d="M181.26,-222.345C173.566,-219.736 165.059,-217.316 157,-216 119.716,-209.913 109.138,-209.075 72,-216 67.7433,-216.794 63.368,-217.985 59.1046,-219.375"/>
<polygon fill="black" stroke="black" points="57.6647,-216.175 49.4683,-222.888 60.0622,-222.751 57.6647,-216.175"/>
<text text-anchor="middle" x="114.5" y="-219.4" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
</g>
<!-- manageable -->
<g id="node3" class="node"><title>manageable</title>
<ellipse fill="none" stroke="black" stroke-width="1.7" cx="407.239" cy="-177.576" rx="54.2411" ry="18"/>
<text text-anchor="middle" x="407.239" y="-174.776" font-family="Times,serif" font-size="11.00">manageable</text>
<ellipse fill="none" stroke="black" stroke-width="1.7" cx="344.443" cy="-233" rx="42.1875" ry="18"/>
<text text-anchor="middle" x="344.443" y="-230.2" font-family="Times,serif" font-size="11.00">manageable</text>
</g>
<!-- verifying&#45;&gt;manageable -->
<g id="edge16" class="edge"><title>verifying&#45;&gt;manageable</title>
<path fill="none" stroke="black" d="M285.442,-177.576C302.733,-177.576 323.366,-177.576 342.643,-177.576"/>
<polygon fill="black" stroke="black" points="342.879,-181.076 352.879,-177.576 342.879,-174.076 342.879,-181.076"/>
<text text-anchor="middle" x="319.118" y="-180.976" font-family="Times,serif" font-size="12.00" fill="gray">done</text>
<path fill="none" stroke="black" d="M242.398,-233C257.259,-233 275.195,-233 291.805,-233"/>
<polygon fill="black" stroke="black" points="291.937,-236.5 301.937,-233 291.937,-229.5 291.937,-236.5"/>
<text text-anchor="middle" x="272.35" y="-236.4" font-family="Times,serif" font-size="12.00" fill="gray">done</text>
</g>
<!-- cleaning -->
<g id="node4" class="node"><title>cleaning</title>
<ellipse fill="none" stroke="black" cx="654.359" cy="-211.576" rx="40.7822" ry="18"/>
<text text-anchor="middle" x="654.359" y="-208.776" font-family="Times,serif" font-size="11.00" fill="gray">cleaning</text>
<ellipse fill="none" stroke="black" cx="549.037" cy="-286" rx="32.4445" ry="18"/>
<text text-anchor="middle" x="549.037" y="-283.2" font-family="Times,serif" font-size="11.00" fill="gray">cleaning</text>
</g>
<!-- manageable&#45;&gt;cleaning -->
<g id="edge2" class="edge"><title>manageable&#45;&gt;cleaning</title>
<path fill="none" stroke="black" d="M435.697,-193.121C448.535,-199.541 464.287,-206.25 479.359,-209.576 520.843,-218.728 569.253,-218.429 604.464,-216.301"/>
<polygon fill="black" stroke="black" points="604.744,-219.79 614.486,-215.624 604.273,-212.806 604.744,-219.79"/>
<text text-anchor="middle" x="531.359" y="-220.976" font-family="Times,serif" font-size="12.00">provide (via API)</text>
<path fill="none" stroke="black" d="M359.543,-249.899C370.458,-261.604 386.688,-276.249 404.537,-283 437.687,-295.537 478.297,-295.069 507.845,-292.178"/>
<polygon fill="black" stroke="black" points="508.252,-295.654 517.805,-291.071 507.479,-288.697 508.252,-295.654"/>
<text text-anchor="middle" x="446.537" y="-296.4" font-family="Times,serif" font-size="12.00">provide (via API)</text>
</g>
<!-- manageable&#45;&gt;cleaning -->
<g id="edge3" class="edge"><title>manageable&#45;&gt;cleaning</title>
<path fill="none" stroke="black" d="M461.051,-179.626C495.97,-181.587 542.646,-185.367 583.359,-192.576 592.278,-194.155 601.711,-196.347 610.666,-198.682"/>
<polygon fill="black" stroke="black" points="609.952,-202.115 620.519,-201.355 611.784,-195.359 609.952,-202.115"/>
<text text-anchor="middle" x="531.359" y="-195.976" font-family="Times,serif" font-size="12.00">clean (via API)</text>
<path fill="none" stroke="black" d="M381.682,-241.647C389.232,-243.431 397.142,-245.288 404.537,-247 441.827,-255.634 451.682,-255.667 488.537,-266 496.052,-268.107 504.019,-270.6 511.597,-273.099"/>
<polygon fill="black" stroke="black" points="510.67,-276.48 521.264,-276.355 512.904,-269.846 510.67,-276.48"/>
<text text-anchor="middle" x="446.537" y="-269.4" font-family="Times,serif" font-size="12.00">clean (via API)</text>
</g>
<!-- inspecting -->
<g id="node5" class="node"><title>inspecting</title>
<ellipse fill="none" stroke="black" cx="654.359" cy="-28.5757" rx="46.8089" ry="18"/>
<text text-anchor="middle" x="654.359" y="-25.7757" font-family="Times,serif" font-size="11.00" fill="gray">inspecting</text>
<ellipse fill="none" stroke="black" cx="549.037" cy="-98" rx="37.0671" ry="18"/>
<text text-anchor="middle" x="549.037" y="-95.2" font-family="Times,serif" font-size="11.00" fill="gray">inspecting</text>
</g>
<!-- manageable&#45;&gt;inspecting -->
<g id="edge4" class="edge"><title>manageable&#45;&gt;inspecting</title>
<path fill="none" stroke="black" d="M413.247,-159.638C421.876,-132.333 442.291,-80.8522 479.359,-55.5757 498.182,-42.7405 554.071,-35.6986 597.483,-32.0549"/>
<polygon fill="black" stroke="black" points="597.968,-35.5275 607.658,-31.2429 597.411,-28.5496 597.968,-35.5275"/>
<text text-anchor="middle" x="531.359" y="-58.9757" font-family="Times,serif" font-size="12.00">inspect (via API)</text>
<path fill="none" stroke="black" d="M351.504,-214.864C359.877,-192.636 377.133,-155.415 404.537,-135 408.707,-131.894 464.841,-117.955 505.643,-108.104"/>
<polygon fill="black" stroke="black" points="506.482,-111.502 515.385,-105.759 504.844,-104.696 506.482,-111.502"/>
<text text-anchor="middle" x="446.537" y="-138.4" font-family="Times,serif" font-size="12.00">inspect (via API)</text>
</g>
<!-- adopting -->
<g id="node6" class="node"><title>adopting</title>
<ellipse fill="none" stroke="black" cx="654.359" cy="-429.576" rx="42.1875" ry="18"/>
<text text-anchor="middle" x="654.359" y="-426.776" font-family="Times,serif" font-size="11.00" fill="gray">adopting</text>
<ellipse fill="none" stroke="black" cx="549.037" cy="-434" rx="32.4445" ry="18"/>
<text text-anchor="middle" x="549.037" y="-431.2" font-family="Times,serif" font-size="11.00" fill="gray">adopting</text>
</g>
<!-- manageable&#45;&gt;adopting -->
<g id="edge5" class="edge"><title>manageable&#45;&gt;adopting</title>
<path fill="none" stroke="black" d="M408.623,-195.573C410.526,-236.7 421.202,-338.055 479.359,-390.576 512.541,-420.541 563.662,-429.109 601.92,-430.885"/>
<polygon fill="black" stroke="black" points="602.238,-434.397 612.344,-431.218 602.461,-427.401 602.238,-434.397"/>
<text text-anchor="middle" x="531.359" y="-431.976" font-family="Times,serif" font-size="12.00">adopt (via API)</text>
<path fill="none" stroke="black" d="M346.454,-251.026C349.374,-285.184 360.7,-359.451 404.537,-399 432.164,-423.924 474.573,-431.839 506.132,-434.018"/>
<polygon fill="black" stroke="black" points="506.213,-437.527 516.382,-434.554 506.578,-430.537 506.213,-437.527"/>
<text text-anchor="middle" x="446.537" y="-435.4" font-family="Times,serif" font-size="12.00">adopt (via API)</text>
</g>
<!-- cleaning&#45;&gt;manageable -->
<g id="edge30" class="edge"><title>cleaning&#45;&gt;manageable</title>
<path fill="none" stroke="black" d="M635.626,-195.284C622.333,-184.236 603.103,-170.589 583.359,-164.576 543.013,-152.287 495.131,-157.077 459.642,-164.075"/>
<polygon fill="black" stroke="black" points="458.583,-160.721 449.513,-166.198 460.019,-167.572 458.583,-160.721"/>
<text text-anchor="middle" x="531.359" y="-167.976" font-family="Times,serif" font-size="12.00" fill="gray">manage</text>
<path fill="none" stroke="black" d="M536.055,-269.182C525.345,-255.722 508.317,-237.844 488.537,-230 457.988,-217.885 420.811,-219.089 392.009,-223.083"/>
<polygon fill="black" stroke="black" points="391.152,-219.674 381.804,-224.661 392.222,-226.592 391.152,-219.674"/>
<text text-anchor="middle" x="446.537" y="-233.4" font-family="Times,serif" font-size="12.00" fill="gray">manage</text>
</g>
<!-- available -->
<g id="node7" class="node"><title>available</title>
<ellipse fill="none" stroke="black" stroke-width="1.7" cx="907.689" cy="-327.576" rx="42.8909" ry="18"/>
<text text-anchor="middle" x="907.689" y="-324.776" font-family="Times,serif" font-size="11.00">available</text>
<ellipse fill="none" stroke="black" stroke-width="1.7" cx="755.232" cy="-457" rx="34.054" ry="18"/>
<text text-anchor="middle" x="755.232" y="-454.2" font-family="Times,serif" font-size="11.00">available</text>
</g>
<!-- cleaning&#45;&gt;available -->
<g id="edge27" class="edge"><title>cleaning&#45;&gt;available</title>
<path fill="none" stroke="black" d="M668.753,-228.655C681.624,-243.86 702.454,-265.51 725.359,-277.576 768.011,-300.043 785.015,-287.304 831.359,-300.576 842.825,-303.859 855.073,-307.977 866.288,-312"/>
<polygon fill="black" stroke="black" points="865.239,-315.343 875.833,-315.488 867.642,-308.768 865.239,-315.343"/>
<text text-anchor="middle" x="778.359" y="-303.976" font-family="Times,serif" font-size="12.00" fill="gray">done</text>
<path fill="none" stroke="black" d="M562.458,-302.567C573.694,-316.818 591.239,-337.335 609.537,-352 643.392,-379.132 659.691,-374.154 694.537,-400 708.003,-409.988 721.629,-422.752 732.456,-433.669"/>
<polygon fill="black" stroke="black" points="729.986,-436.149 739.468,-440.875 735.003,-431.267 729.986,-436.149"/>
<text text-anchor="middle" x="652.037" y="-403.4" font-family="Times,serif" font-size="12.00" fill="gray">done</text>
</g>
<!-- clean failed -->
<g id="node17" class="node"><title>clean failed</title>
<ellipse fill="none" stroke="black" cx="1147.73" cy="-177.576" rx="51.931" ry="18"/>
<text text-anchor="middle" x="1147.73" y="-174.776" font-family="Times,serif" font-size="11.00" fill="red">clean failed</text>
<ellipse fill="none" stroke="black" cx="956.23" cy="-253" rx="41.4846" ry="18"/>
<text text-anchor="middle" x="956.23" y="-250.2" font-family="Times,serif" font-size="11.00" fill="red">clean failed</text>
</g>
<!-- cleaning&#45;&gt;clean failed -->
<g id="edge28" class="edge"><title>cleaning&#45;&gt;clean failed</title>
<path fill="none" stroke="black" d="M695.26,-212.735C770.566,-214.293 938.605,-214.761 1078.02,-193.576 1083.84,-192.691 1089.89,-191.576 1095.89,-190.346"/>
<polygon fill="black" stroke="black" points="1096.84,-193.722 1105.88,-188.191 1095.36,-186.879 1096.84,-193.722"/>
<text text-anchor="middle" x="907.689" y="-215.976" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
<path fill="none" stroke="black" d="M581.627,-286.825C610.914,-287.395 655.693,-287.767 694.537,-286 782.539,-281.996 804.985,-282.191 891.927,-268 897.854,-267.033 904.05,-265.82 910.147,-264.506"/>
<polygon fill="black" stroke="black" points="911.268,-267.841 920.251,-262.223 909.725,-261.013 911.268,-267.841"/>
<text text-anchor="middle" x="755.232" y="-288.4" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
</g>
<!-- clean wait -->
<g id="node18" class="node"><title>clean wait</title>
<ellipse fill="none" stroke="black" cx="907.689" cy="-171.576" rx="46.1069" ry="18"/>
<text text-anchor="middle" x="907.689" y="-168.776" font-family="Times,serif" font-size="11.00" fill="gray">clean wait</text>
<ellipse fill="none" stroke="black" cx="755.232" cy="-246" rx="37.7689" ry="18"/>
<text text-anchor="middle" x="755.232" y="-243.2" font-family="Times,serif" font-size="11.00" fill="gray">clean wait</text>
</g>
<!-- cleaning&#45;&gt;clean wait -->
<g id="edge29" class="edge"><title>cleaning&#45;&gt;clean wait</title>
<path fill="none" stroke="black" d="M692.008,-204.514C702.743,-202.515 714.509,-200.39 725.359,-198.576 768.359,-191.384 817.258,-184.158 853.432,-178.994"/>
<polygon fill="black" stroke="black" points="854.347,-182.399 863.755,-177.527 853.362,-175.469 854.347,-182.399"/>
<text text-anchor="middle" x="778.359" y="-201.976" font-family="Times,serif" font-size="12.00" fill="gray">wait</text>
<path fill="none" stroke="black" d="M579.355,-279.408C588.96,-277.298 599.692,-274.994 609.537,-273 642.885,-266.247 680.686,-259.215 709.287,-254.021"/>
<polygon fill="black" stroke="black" points="710.165,-257.419 719.382,-252.194 708.919,-250.53 710.165,-257.419"/>
<text text-anchor="middle" x="652.037" y="-276.4" font-family="Times,serif" font-size="12.00" fill="gray">wait</text>
</g>
<!-- inspecting&#45;&gt;manageable -->
<g id="edge37" class="edge"><title>inspecting&#45;&gt;manageable</title>
<path fill="none" stroke="black" d="M621.586,-15.6814C584.332,-2.81929 522.236,11.3578 479.359,-16.5757 434.152,-46.0274 417.472,-111.443 411.474,-149.202"/>
<polygon fill="black" stroke="black" points="407.95,-149.118 409.993,-159.514 414.879,-150.113 407.95,-149.118"/>
<text text-anchor="middle" x="531.359" y="-19.9757" font-family="Times,serif" font-size="12.00" fill="gray">done</text>
<path fill="none" stroke="black" d="M517.555,-88.3444C486.468,-80.4713 438.156,-73.8882 404.537,-96 367.983,-120.042 353.94,-171.998 348.616,-204.567"/>
<polygon fill="black" stroke="black" points="345.103,-204.401 347.125,-214.801 352.03,-205.41 345.103,-204.401"/>
<text text-anchor="middle" x="446.537" y="-99.4" font-family="Times,serif" font-size="12.00" fill="gray">done</text>
</g>
<!-- inspect failed -->
<g id="node19" class="node"><title>inspect failed</title>
<ellipse fill="none" stroke="black" cx="907.689" cy="-56.5757" rx="58.16" ry="18"/>
<text text-anchor="middle" x="907.689" y="-53.7757" font-family="Times,serif" font-size="11.00" fill="red">inspect failed</text>
<ellipse fill="none" stroke="black" cx="956.23" cy="-93" rx="46.1069" ry="18"/>
<text text-anchor="middle" x="956.23" y="-90.2" font-family="Times,serif" font-size="11.00" fill="red">inspect failed</text>
</g>
<!-- inspecting&#45;&gt;inspect failed -->
<g id="edge38" class="edge"><title>inspecting&#45;&gt;inspect failed</title>
<path fill="none" stroke="black" d="M692.898,-38.9911C703.334,-41.5308 714.724,-43.9746 725.359,-45.5757 762.667,-51.1919 804.702,-53.9613 838.901,-55.3181"/>
<polygon fill="black" stroke="black" points="839.128,-58.8286 849.249,-55.6954 839.383,-51.8332 839.128,-58.8286"/>
<text text-anchor="middle" x="778.359" y="-58.9757" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
<path fill="none" stroke="black" d="M586.352,-97.5519C657.397,-96.6752 815.176,-94.7282 899.518,-93.6875"/>
<polygon fill="black" stroke="black" points="899.721,-97.1853 909.677,-93.5621 899.635,-90.1859 899.721,-97.1853"/>
<text text-anchor="middle" x="755.232" y="-99.4" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
</g>
<!-- inspect wait -->
<g id="node20" class="node"><title>inspect wait</title>
<ellipse fill="none" stroke="black" cx="755.232" cy="-18" rx="42.8909" ry="18"/>
<text text-anchor="middle" x="755.232" y="-15.2" font-family="Times,serif" font-size="11.00" fill="gray">inspect wait</text>
</g>
<!-- inspecting&#45;&gt;inspect wait -->
<g id="edge39" class="edge"><title>inspecting&#45;&gt;inspect wait</title>
<path fill="none" stroke="black" d="M563.969,-81.4809C575.062,-69.4465 591.653,-53.7737 609.537,-45 625.929,-36.9587 669.104,-29.4821 704.055,-24.4452"/>
<polygon fill="black" stroke="black" points="704.716,-27.8866 714.131,-23.0267 703.741,-20.955 704.716,-27.8866"/>
<text text-anchor="middle" x="652.037" y="-48.4" font-family="Times,serif" font-size="12.00" fill="gray">wait</text>
</g>
<!-- active -->
<g id="node9" class="node"><title>active</title>
<ellipse fill="none" stroke="black" stroke-width="1.7" cx="1396.38" cy="-471.576" rx="31.0408" ry="18"/>
<text text-anchor="middle" x="1396.38" y="-468.776" font-family="Times,serif" font-size="11.00">active</text>
<ellipse fill="none" stroke="black" stroke-width="1.7" cx="1167.64" cy="-555" rx="27" ry="18"/>
<text text-anchor="middle" x="1167.64" y="-552.2" font-family="Times,serif" font-size="11.00">active</text>
</g>
<!-- adopting&#45;&gt;active -->
<g id="edge41" class="edge"><title>adopting&#45;&gt;active</title>
<path fill="none" stroke="black" d="M693.34,-422.627C767.471,-410.046 936.558,-386.16 1078.02,-401.576 1180.93,-412.791 1299.35,-443.889 1358.19,-460.607"/>
<polygon fill="black" stroke="black" points="1357.54,-464.064 1368.12,-463.455 1359.47,-457.335 1357.54,-464.064"/>
<text text-anchor="middle" x="1031.02" y="-404.976" font-family="Times,serif" font-size="12.00" fill="gray">done</text>
<g id="edge44" class="edge"><title>adopting&#45;&gt;active</title>
<path fill="none" stroke="black" d="M581.489,-432.996C614.22,-432.006 666.965,-430.561 712.537,-430 750.485,-429.533 760.482,-423.825 797.927,-430 927.7,-451.399 1073.22,-512.489 1135.57,-540.566"/>
<polygon fill="black" stroke="black" points="1134.3,-543.831 1144.85,-544.778 1137.19,-537.457 1134.3,-543.831"/>
<text text-anchor="middle" x="853.927" y="-453.4" font-family="Times,serif" font-size="12.00" fill="gray">done</text>
</g>
<!-- adopt failed -->
<g id="node20" class="node"><title>adopt failed</title>
<ellipse fill="none" stroke="black" cx="907.689" cy="-447.576" rx="53.5381" ry="18"/>
<text text-anchor="middle" x="907.689" y="-444.776" font-family="Times,serif" font-size="11.00" fill="red">adopt failed</text>
<g id="node21" class="node"><title>adopt failed</title>
<ellipse fill="none" stroke="black" cx="755.232" cy="-511" rx="41.4846" ry="18"/>
<text text-anchor="middle" x="755.232" y="-508.2" font-family="Times,serif" font-size="11.00" fill="red">adopt failed</text>
</g>
<!-- adopting&#45;&gt;adopt failed -->
<g id="edge42" class="edge"><title>adopting&#45;&gt;adopt failed</title>
<path fill="none" stroke="black" d="M688.107,-440.39C699.734,-443.714 713.002,-446.928 725.359,-448.576 764.98,-453.857 809.903,-453.654 845.123,-452.107"/>
<polygon fill="black" stroke="black" points="845.437,-455.596 855.255,-451.614 845.097,-448.604 845.437,-455.596"/>
<text text-anchor="middle" x="778.359" y="-455.976" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
<g id="edge45" class="edge"><title>adopting&#45;&gt;adopt failed</title>
<path fill="none" stroke="black" d="M568.577,-448.632C582.662,-459.509 601.016,-473.109 609.537,-477 639.747,-490.793 676.151,-499.31 704.886,-504.371"/>
<polygon fill="black" stroke="black" points="704.659,-507.882 715.1,-506.082 705.816,-500.978 704.659,-507.882"/>
<text text-anchor="middle" x="652.037" y="-505.4" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
</g>
<!-- available&#45;&gt;manageable -->
<g id="edge7" class="edge"><title>available&#45;&gt;manageable</title>
<path fill="none" stroke="black" d="M864.906,-327.11C827.911,-326.046 772.608,-322.854 725.359,-313.576 612.678,-291.449 580.507,-288.941 479.359,-234.576 462.738,-225.642 446.17,-212.663 433.211,-201.346"/>
<polygon fill="black" stroke="black" points="435.425,-198.63 425.641,-194.565 430.755,-203.844 435.425,-198.63"/>
<text text-anchor="middle" x="654.359" y="-312.976" font-family="Times,serif" font-size="12.00">manage (via API)</text>
<path fill="none" stroke="black" d="M726.023,-466.944C660.612,-488.018 497.13,-528.998 404.537,-449 376.12,-424.448 356.939,-313.941 349.238,-261.193"/>
<polygon fill="black" stroke="black" points="352.684,-260.568 347.81,-251.161 345.754,-261.554 352.684,-260.568"/>
<text text-anchor="middle" x="549.037" y="-499.4" font-family="Times,serif" font-size="12.00">manage (via API)</text>
</g>
<!-- deploying -->
<g id="node8" class="node"><title>deploying</title>
<ellipse fill="none" stroke="black" cx="1147.73" cy="-610.576" rx="44.498" ry="18"/>
<text text-anchor="middle" x="1147.73" y="-607.776" font-family="Times,serif" font-size="11.00" fill="gray">deploying</text>
<ellipse fill="none" stroke="black" cx="956.23" cy="-674" rx="35.4579" ry="18"/>
<text text-anchor="middle" x="956.23" y="-671.2" font-family="Times,serif" font-size="11.00" fill="gray">deploying</text>
</g>
<!-- available&#45;&gt;deploying -->
<g id="edge6" class="edge"><title>available&#45;&gt;deploying</title>
<path fill="none" stroke="black" d="M922.792,-344.732C937.883,-363.063 962.553,-392.948 984.019,-418.576 1034.62,-478.989 1094.69,-549.578 1125.76,-586.017"/>
<polygon fill="black" stroke="black" points="1123.15,-588.344 1132.3,-593.68 1128.47,-583.801 1123.15,-588.344"/>
<text text-anchor="middle" x="1031.02" y="-530.976" font-family="Times,serif" font-size="12.00">active (via API)</text>
<path fill="none" stroke="black" d="M779.208,-470.106C785.558,-474.19 792.257,-478.956 797.927,-484 856.136,-535.775 911.05,-609.91 937.958,-648.482"/>
<polygon fill="black" stroke="black" points="935.311,-650.808 943.877,-657.043 941.069,-646.827 935.311,-650.808"/>
<text text-anchor="middle" x="853.927" y="-585.4" font-family="Times,serif" font-size="12.00">active (via API)</text>
</g>
<!-- deploying&#45;&gt;active -->
<g id="edge20" class="edge"><title>deploying&#45;&gt;active</title>
<path fill="none" stroke="black" d="M1152.42,-592.617C1159.43,-563.575 1177.68,-506.995 1217.45,-481.576 1259.15,-454.921 1318.21,-457.672 1356.65,-463.488"/>
<polygon fill="black" stroke="black" points="1356.48,-467.007 1366.91,-465.187 1357.62,-460.101 1356.48,-467.007"/>
<text text-anchor="middle" x="1268.45" y="-484.976" font-family="Times,serif" font-size="12.00" fill="gray">done</text>
<path fill="none" stroke="black" d="M961.095,-655.96C967.906,-629.163 984.846,-579.678 1020.53,-558 1054.24,-537.527 1100.87,-540.808 1132.31,-546.546"/>
<polygon fill="black" stroke="black" points="1131.86,-550.025 1142.35,-548.555 1133.24,-543.161 1131.86,-550.025"/>
<text text-anchor="middle" x="1061.03" y="-561.4" font-family="Times,serif" font-size="12.00" fill="gray">done</text>
</g>
<!-- deploy failed -->
<g id="node15" class="node"><title>deploy failed</title>
<ellipse fill="none" stroke="black" cx="1646.24" cy="-643.576" rx="55.8489" ry="18"/>
<text text-anchor="middle" x="1646.24" y="-640.776" font-family="Times,serif" font-size="11.00" fill="red">deploy failed</text>
<ellipse fill="none" stroke="black" cx="1374.25" cy="-723" rx="44.498" ry="18"/>
<text text-anchor="middle" x="1374.25" y="-720.2" font-family="Times,serif" font-size="11.00" fill="red">deploy failed</text>
</g>
<!-- deploying&#45;&gt;deploy failed -->
<g id="edge18" class="edge"><title>deploying&#45;&gt;deploy failed</title>
<path fill="none" stroke="black" d="M1186.09,-619.902C1259.85,-637.168 1429.42,-671.168 1572.31,-657.576 1577.85,-657.049 1583.58,-656.303 1589.3,-655.426"/>
<polygon fill="black" stroke="black" points="1590.23,-658.819 1599.52,-653.724 1589.08,-651.914 1590.23,-658.819"/>
<text text-anchor="middle" x="1396.38" y="-661.976" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
<path fill="none" stroke="black" d="M966.444,-691.258C976.924,-708.883 995.833,-735.27 1020.53,-747 1123.58,-795.935 1262.91,-761.165 1331.53,-738.438"/>
<polygon fill="black" stroke="black" points="1332.78,-741.71 1341.13,-735.184 1330.53,-735.08 1332.78,-741.71"/>
<text text-anchor="middle" x="1167.64" y="-774.4" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
</g>
<!-- wait call&#45;back -->
<g id="node16" class="node"><title>wait call&#45;back</title>
<ellipse fill="none" stroke="black" cx="1396.38" cy="-708.576" rx="58.8623" ry="18"/>
<text text-anchor="middle" x="1396.38" y="-705.776" font-family="Times,serif" font-size="11.00" fill="gray">wait call&#45;back</text>
<ellipse fill="none" stroke="black" cx="1167.64" cy="-723" rx="48.2143" ry="18"/>
<text text-anchor="middle" x="1167.64" y="-720.2" font-family="Times,serif" font-size="11.00" fill="gray">wait call&#45;back</text>
</g>
<!-- deploying&#45;&gt;wait call&#45;back -->
<g id="edge19" class="edge"><title>deploying&#45;&gt;wait call&#45;back</title>
<path fill="none" stroke="black" d="M1155.7,-628.531C1165.59,-651.196 1186.04,-689.197 1217.45,-705.576 1254.13,-724.707 1301.12,-724.644 1337.6,-720.233"/>
<polygon fill="black" stroke="black" points="1338.15,-723.691 1347.59,-718.886 1337.22,-716.754 1338.15,-723.691"/>
<text text-anchor="middle" x="1268.45" y="-725.976" font-family="Times,serif" font-size="12.00" fill="gray">wait</text>
<path fill="none" stroke="black" d="M972.756,-689.955C984.658,-700.992 1002.12,-714.772 1020.53,-721 1049.75,-730.882 1084.16,-731.829 1112.25,-730.099"/>
<polygon fill="black" stroke="black" points="1112.58,-733.585 1122.3,-729.352 1112.06,-726.604 1112.58,-733.585"/>
<text text-anchor="middle" x="1061.03" y="-733.4" font-family="Times,serif" font-size="12.00" fill="gray">wait</text>
</g>
<!-- active&#45;&gt;deploying -->
<g id="edge8" class="edge"><title>active&#45;&gt;deploying</title>
<path fill="none" stroke="black" d="M1369.18,-480.474C1321.31,-496.783 1224.01,-530.202 1217.45,-534.576 1197.28,-548.019 1179.12,-568.681 1166.54,-585.069"/>
<polygon fill="black" stroke="black" points="1163.65,-583.079 1160.49,-593.189 1169.27,-587.262 1163.65,-583.079"/>
<text text-anchor="middle" x="1268.45" y="-537.976" font-family="Times,serif" font-size="12.00">rebuild (via API)</text>
<path fill="none" stroke="black" d="M1142.64,-562.015C1101.35,-574.092 1021.19,-597.569 1020.53,-598 1001.04,-610.808 984.407,-631.423 973.091,-647.954"/>
<polygon fill="black" stroke="black" points="969.946,-646.365 967.367,-656.641 975.791,-650.217 969.946,-646.365"/>
<text text-anchor="middle" x="1061.03" y="-601.4" font-family="Times,serif" font-size="12.00">rebuild (via API)</text>
</g>
<!-- deleting -->
<g id="node10" class="node"><title>deleting</title>
<ellipse fill="none" stroke="black" cx="2893.96" cy="-534.576" rx="39.1741" ry="18"/>
<text text-anchor="middle" x="2893.96" y="-531.776" font-family="Times,serif" font-size="11.00" fill="gray">deleting</text>
<ellipse fill="none" stroke="black" cx="2398.02" cy="-686" rx="31.0408" ry="18"/>
<text text-anchor="middle" x="2398.02" y="-683.2" font-family="Times,serif" font-size="11.00" fill="gray">deleting</text>
</g>
<!-- active&#45;&gt;deleting -->
<g id="edge9" class="edge"><title>active&#45;&gt;deleting</title>
<path fill="none" stroke="black" d="M1419.57,-483.817C1470.69,-511.225 1601.44,-576.895 1720.16,-599.576 1776.66,-610.37 2263.87,-644.811 2836.87,-550.576 2841.74,-549.776 2846.78,-548.701 2851.76,-547.486"/>
<polygon fill="black" stroke="black" points="2853.03,-550.77 2861.8,-544.835 2851.24,-544.002 2853.03,-550.77"/>
<text text-anchor="middle" x="2121.48" y="-617.976" font-family="Times,serif" font-size="12.00">deleted (via API)</text>
<path fill="none" stroke="black" d="M1188.2,-567.071C1231.19,-592.846 1338,-652.725 1436.75,-675 1526.73,-695.299 2174.51,-692.944 2266.75,-694 2303.2,-694.417 2312.43,-696.99 2348.75,-694 2351.83,-693.747 2355,-693.404 2358.19,-693.003"/>
<polygon fill="black" stroke="black" points="2358.88,-696.441 2368.28,-691.559 2357.88,-689.511 2358.88,-696.441"/>
<text text-anchor="middle" x="1767.53" y="-694.4" font-family="Times,serif" font-size="12.00">deleted (via API)</text>
</g>
<!-- rescuing -->
<g id="node11" class="node"><title>rescuing</title>
<ellipse fill="none" stroke="black" cx="1646.24" cy="-466.576" rx="40.7822" ry="18"/>
<text text-anchor="middle" x="1646.24" y="-463.776" font-family="Times,serif" font-size="11.00" fill="gray">rescuing</text>
<ellipse fill="none" stroke="black" cx="1374.25" cy="-554" rx="32.4445" ry="18"/>
<text text-anchor="middle" x="1374.25" y="-551.2" font-family="Times,serif" font-size="11.00" fill="gray">rescuing</text>
</g>
<!-- active&#45;&gt;rescuing -->
<g id="edge10" class="edge"><title>active&#45;&gt;rescuing</title>
<path fill="none" stroke="black" d="M1427.77,-470.963C1469.41,-470.123 1544.72,-468.603 1594.86,-467.592"/>
<polygon fill="black" stroke="black" points="1595.23,-471.085 1605.15,-467.384 1595.09,-464.087 1595.23,-471.085"/>
<text text-anchor="middle" x="1522.81" y="-472.976" font-family="Times,serif" font-size="12.00">rescue (via API)</text>
<path fill="none" stroke="black" d="M1194.97,-554.871C1229.52,-554.702 1290.56,-554.404 1331.46,-554.204"/>
<polygon fill="black" stroke="black" points="1331.67,-557.703 1341.65,-554.154 1331.63,-550.703 1331.67,-557.703"/>
<text text-anchor="middle" x="1272.75" y="-557.4" font-family="Times,serif" font-size="12.00">rescue (via API)</text>
</g>
<!-- deleting&#45;&gt;cleaning -->
<g id="edge36" class="edge"><title>deleting&#45;&gt;cleaning</title>
<path fill="none" stroke="black" d="M2859.68,-525.664C2822.97,-514.079 2764.88,-490.228 2732.87,-448.576 2711.72,-421.051 2740.49,-395.997 2714.87,-372.576 2403.2,-87.6439 2192.95,-266.576 1770.66,-266.576 906.689,-266.576 906.689,-266.576 906.689,-266.576 831.054,-266.576 745.514,-242.511 696.265,-226.218"/>
<polygon fill="black" stroke="black" points="697.212,-222.844 686.618,-222.972 694.979,-229.478 697.212,-222.844"/>
<text text-anchor="middle" x="1769.66" y="-269.976" font-family="Times,serif" font-size="12.00" fill="gray">clean</text>
<path fill="none" stroke="black" d="M2381.76,-670.467C2354.15,-641.411 2296.06,-575.723 2266.75,-509 2250,-470.853 2279.58,-445.028 2248.75,-417 1993.65,-185.069 1821.52,-341 1476.75,-341 754.232,-341 754.232,-341 754.232,-341 692.502,-341 623.776,-317.275 583.872,-300.989"/>
<polygon fill="black" stroke="black" points="585.186,-297.745 574.609,-297.13 582.494,-304.207 585.186,-297.745"/>
<text text-anchor="middle" x="1475.75" y="-344.4" font-family="Times,serif" font-size="12.00" fill="gray">clean</text>
</g>
<!-- error -->
<g id="node12" class="node"><title>error</title>
<ellipse fill="none" stroke="black" stroke-width="1.7" cx="3101.31" cy="-570.576" rx="28.0277" ry="18"/>
<text text-anchor="middle" x="3101.31" y="-567.776" font-family="Times,serif" font-size="11.00" fill="red">error</text>
<ellipse fill="none" stroke="black" stroke-width="1.7" cx="2574.29" cy="-722" rx="27" ry="18"/>
<text text-anchor="middle" x="2574.29" y="-719.2" font-family="Times,serif" font-size="11.00" fill="red">error</text>
</g>
<!-- deleting&#45;&gt;error -->
<g id="edge35" class="edge"><title>deleting&#45;&gt;error</title>
<path fill="none" stroke="black" d="M2929.59,-526.815C2962.87,-520.981 3014,-516.333 3055.04,-531.576 3063.99,-534.898 3072.35,-540.869 3079.39,-547.157"/>
<polygon fill="black" stroke="black" points="3077.4,-550.094 3087.01,-554.543 3082.27,-545.068 3077.4,-550.094"/>
<text text-anchor="middle" x="3003.04" y="-534.976" font-family="Times,serif" font-size="12.00" fill="gray">error</text>
<path fill="none" stroke="black" d="M2427.24,-679.396C2454.35,-674.41 2495.98,-670.361 2529.29,-683 2538.21,-686.383 2546.48,-692.492 2553.39,-698.887"/>
<polygon fill="black" stroke="black" points="2551.3,-701.748 2560.83,-706.378 2556.26,-696.816 2551.3,-701.748"/>
<text text-anchor="middle" x="2488.29" y="-686.4" font-family="Times,serif" font-size="12.00" fill="gray">error</text>
</g>
<!-- rescue -->
<g id="node13" class="node"><title>rescue</title>
<ellipse fill="none" stroke="black" stroke-width="1.7" cx="2000.48" cy="-558.576" rx="33.8507" ry="18"/>
<text text-anchor="middle" x="2000.48" y="-555.776" font-family="Times,serif" font-size="11.00">rescue</text>
<ellipse fill="none" stroke="black" stroke-width="1.7" cx="1668.53" cy="-638" rx="27.824" ry="18"/>
<text text-anchor="middle" x="1668.53" y="-635.2" font-family="Times,serif" font-size="11.00">rescue</text>
</g>
<!-- rescuing&#45;&gt;rescue -->
<g id="edge45" class="edge"><title>rescuing&#45;&gt;rescue</title>
<path fill="none" stroke="black" d="M1664.32,-482.786C1678.23,-494.988 1698.97,-511.024 1720.16,-519.576 1762.18,-536.535 1889.46,-549.359 1956.74,-555.161"/>
<polygon fill="black" stroke="black" points="1956.74,-558.673 1967,-556.031 1957.33,-551.698 1956.74,-558.673"/>
<text text-anchor="middle" x="1769.66" y="-542.976" font-family="Times,serif" font-size="12.00" fill="gray">done</text>
<g id="edge48" class="edge"><title>rescuing&#45;&gt;rescue</title>
<path fill="none" stroke="black" d="M1394.85,-568.16C1406.51,-575.964 1421.88,-585.214 1436.75,-591 1502.5,-616.598 1584.46,-629.041 1630.91,-634.434"/>
<polygon fill="black" stroke="black" points="1630.77,-637.939 1641.09,-635.568 1631.54,-630.982 1630.77,-637.939"/>
<text text-anchor="middle" x="1475.75" y="-616.4" font-family="Times,serif" font-size="12.00" fill="gray">done</text>
</g>
<!-- rescue wait -->
<g id="node21" class="node"><title>rescue wait</title>
<ellipse fill="none" stroke="black" cx="1887.07" cy="-410.576" rx="49.8222" ry="18"/>
<text text-anchor="middle" x="1887.07" y="-407.776" font-family="Times,serif" font-size="11.00" fill="gray">rescue wait</text>
<g id="node22" class="node"><title>rescue wait</title>
<ellipse fill="none" stroke="black" cx="1573.64" cy="-523" rx="40.7822" ry="18"/>
<text text-anchor="middle" x="1573.64" y="-520.2" font-family="Times,serif" font-size="11.00" fill="gray">rescue wait</text>
</g>
<!-- rescuing&#45;&gt;rescue wait -->
<g id="edge46" class="edge"><title>rescuing&#45;&gt;rescue wait</title>
<path fill="none" stroke="black" d="M1675.14,-453.824C1688.53,-448.139 1704.93,-441.79 1720.16,-437.576 1755.69,-427.745 1796.56,-421.042 1828.84,-416.774"/>
<polygon fill="black" stroke="black" points="1829.53,-420.214 1839,-415.472 1828.64,-413.271 1829.53,-420.214"/>
<text text-anchor="middle" x="1769.66" y="-440.976" font-family="Times,serif" font-size="12.00" fill="gray">wait</text>
<g id="edge49" class="edge"><title>rescuing&#45;&gt;rescue wait</title>
<path fill="none" stroke="black" d="M1405.66,-549.224C1437.39,-544.241 1487.59,-536.357 1524.67,-530.533"/>
<polygon fill="black" stroke="black" points="1525.42,-533.959 1534.75,-528.949 1524.33,-527.043 1525.42,-533.959"/>
<text text-anchor="middle" x="1475.75" y="-546.4" font-family="Times,serif" font-size="12.00" fill="gray">wait</text>
</g>
<!-- rescue failed -->
<g id="node22" class="node"><title>rescue failed</title>
<ellipse fill="none" stroke="black" cx="2121.48" cy="-365.576" rx="55.8489" ry="18"/>
<text text-anchor="middle" x="2121.48" y="-362.776" font-family="Times,serif" font-size="11.00" fill="red">rescue failed</text>
<g id="node23" class="node"><title>rescue failed</title>
<ellipse fill="none" stroke="black" cx="1767.53" cy="-511" rx="44.498" ry="18"/>
<text text-anchor="middle" x="1767.53" y="-508.2" font-family="Times,serif" font-size="11.00" fill="red">rescue failed</text>
</g>
<!-- rescuing&#45;&gt;rescue failed -->
<g id="edge47" class="edge"><title>rescuing&#45;&gt;rescue failed</title>
<path fill="none" stroke="black" d="M1686.95,-468.766C1760.76,-471.356 1922.76,-470.014 2045.98,-418.576 2063.21,-411.385 2080.33,-399.617 2093.8,-388.995"/>
<polygon fill="black" stroke="black" points="2096.14,-391.604 2101.69,-382.58 2091.73,-386.173 2096.14,-391.604"/>
<text text-anchor="middle" x="1887.07" y="-467.976" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
<g id="edge50" class="edge"><title>rescuing&#45;&gt;rescue failed</title>
<path fill="none" stroke="black" d="M1406.17,-557.61C1415.94,-558.591 1426.78,-559.515 1436.75,-560 1555.95,-565.801 1589.64,-572.303 1704.53,-540 1713.6,-537.449 1723,-533.695 1731.66,-529.729"/>
<polygon fill="black" stroke="black" points="1733.21,-532.868 1740.71,-525.388 1730.18,-526.557 1733.21,-532.868"/>
<text text-anchor="middle" x="1573.64" y="-567.4" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
</g>
<!-- error&#45;&gt;deploying -->
<g id="edge11" class="edge"><title>error&#45;&gt;deploying</title>
<path fill="none" stroke="black" d="M3091.88,-587.648C3067.95,-633.447 2994.77,-755.576 2894.96,-755.576 1395.38,-755.576 1395.38,-755.576 1395.38,-755.576 1315.98,-755.576 1283.62,-783.454 1217.45,-739.576 1182.45,-716.369 1163.99,-669.259 1155.3,-638.847"/>
<polygon fill="black" stroke="black" points="1158.6,-637.609 1152.63,-628.853 1151.83,-639.418 1158.6,-637.609"/>
<text text-anchor="middle" x="2121.48" y="-758.976" font-family="Times,serif" font-size="12.00">rebuild (via API)</text>
<path fill="none" stroke="black" d="M2560.15,-737.474C2533.22,-767.337 2468.43,-830 2399.02,-830 1166.64,-830 1166.64,-830 1166.64,-830 1096.77,-830 1074.52,-816.342 1020.53,-772 997.081,-752.739 979.397,-722.455 968.762,-700.649"/>
<polygon fill="black" stroke="black" points="971.926,-699.152 964.508,-691.588 965.59,-702.127 971.926,-699.152"/>
<text text-anchor="middle" x="1767.53" y="-833.4" font-family="Times,serif" font-size="12.00">rebuild (via API)</text>
</g>
<!-- error&#45;&gt;deleting -->
<g id="edge12" class="edge"><title>error&#45;&gt;deleting</title>
<path fill="none" stroke="black" d="M3073.48,-567.428C3043.49,-563.724 2993.57,-556.996 2951.04,-548.576 2946.84,-547.742 2942.47,-546.797 2938.11,-545.8"/>
<polygon fill="black" stroke="black" points="2938.59,-542.316 2928.05,-543.409 2936.97,-549.127 2938.59,-542.316"/>
<text text-anchor="middle" x="3003.04" y="-567.976" font-family="Times,serif" font-size="12.00">deleted (via API)</text>
<path fill="none" stroke="black" d="M2547.55,-718.349C2521.97,-714.523 2481.69,-707.966 2447.29,-700 2443.48,-699.117 2439.52,-698.114 2435.59,-697.06"/>
<polygon fill="black" stroke="black" points="2436.52,-693.686 2425.95,-694.374 2434.64,-700.43 2436.52,-693.686"/>
<text text-anchor="middle" x="2488.29" y="-718.4" font-family="Times,serif" font-size="12.00">deleted (via API)</text>
</g>
<!-- rescue&#45;&gt;deleting -->
<g id="edge14" class="edge"><title>rescue&#45;&gt;deleting</title>
<path fill="none" stroke="black" d="M2034.27,-557.693C2169.44,-554.054 2683.92,-540.203 2844.67,-535.876"/>
<polygon fill="black" stroke="black" points="2844.84,-539.373 2854.74,-535.605 2844.65,-532.375 2844.84,-539.373"/>
<text text-anchor="middle" x="2381.98" y="-552.976" font-family="Times,serif" font-size="12.00">deleted (via API)</text>
<path fill="none" stroke="black" d="M1696.28,-639.765C1806.35,-647.028 2224.63,-674.625 2356.76,-683.344"/>
<polygon fill="black" stroke="black" points="2356.65,-686.844 2366.86,-684.01 2357.11,-679.859 2356.65,-686.844"/>
<text text-anchor="middle" x="1979.53" y="-664.4" font-family="Times,serif" font-size="12.00">deleted (via API)</text>
</g>
<!-- rescue&#45;&gt;rescuing -->
<g id="edge13" class="edge"><title>rescue&#45;&gt;rescuing</title>
<path fill="none" stroke="black" d="M1970.3,-566.677C1917.34,-579.707 1803.23,-599.999 1720.16,-559.576 1692.07,-545.907 1671.6,-515.602 1659.62,-493.284"/>
<polygon fill="black" stroke="black" points="1662.68,-491.585 1655.01,-484.28 1656.45,-494.776 1662.68,-491.585"/>
<text text-anchor="middle" x="1769.66" y="-585.976" font-family="Times,serif" font-size="12.00">rescue (via API)</text>
<path fill="none" stroke="black" d="M1643.31,-645.61C1598.14,-658.211 1499.91,-677.922 1436.75,-634 1418.72,-621.467 1431.86,-605.607 1418.75,-588 1414.84,-582.754 1409.92,-577.888 1404.81,-573.573"/>
<polygon fill="black" stroke="black" points="1406.78,-570.667 1396.74,-567.27 1402.47,-576.184 1406.78,-570.667"/>
<text text-anchor="middle" x="1475.75" y="-663.4" font-family="Times,serif" font-size="12.00">rescue (via API)</text>
</g>
<!-- unrescuing -->
<g id="node14" class="node"><title>unrescuing</title>
<ellipse fill="none" stroke="black" cx="2381.98" cy="-365.576" rx="49.8222" ry="18"/>
<text text-anchor="middle" x="2381.98" y="-362.776" font-family="Times,serif" font-size="11.00" fill="gray">unrescuing</text>
<ellipse fill="none" stroke="black" cx="1979.53" cy="-473" rx="39.1741" ry="18"/>
<text text-anchor="middle" x="1979.53" y="-470.2" font-family="Times,serif" font-size="11.00" fill="gray">unrescuing</text>
</g>
<!-- rescue&#45;&gt;unrescuing -->
<g id="edge15" class="edge"><title>rescue&#45;&gt;unrescuing</title>
<path fill="none" stroke="black" d="M2009.83,-541.024C2020,-521.413 2038.98,-490.005 2063.98,-471.576 2107.06,-439.815 2252.65,-398.788 2330.65,-378.34"/>
<polygon fill="black" stroke="black" points="2331.94,-381.621 2340.74,-375.713 2330.18,-374.847 2331.94,-381.621"/>
<text text-anchor="middle" x="2121.48" y="-474.976" font-family="Times,serif" font-size="12.00">unrescue (via API)</text>
<path fill="none" stroke="black" d="M1690.87,-627.307C1734.41,-605.398 1836.45,-553.544 1920.53,-507 1928.93,-502.35 1937.92,-497.178 1946.25,-492.3"/>
<polygon fill="black" stroke="black" points="1948.13,-495.252 1954.97,-487.161 1944.58,-489.22 1948.13,-495.252"/>
<text text-anchor="middle" x="1767.53" y="-614.4" font-family="Times,serif" font-size="12.00">unrescue (via API)</text>
</g>
<!-- unrescuing&#45;&gt;active -->
<g id="edge55" class="edge"><title>unrescuing&#45;&gt;active</title>
<path fill="none" stroke="black" d="M2345.07,-353.416C2295.58,-337.769 2203.34,-312.576 2122.48,-312.576 1645.24,-312.576 1645.24,-312.576 1645.24,-312.576 1546.57,-312.576 1455.81,-402.936 1417.1,-447.419"/>
<polygon fill="black" stroke="black" points="1414.33,-445.282 1410.49,-455.159 1419.65,-449.828 1414.33,-445.282"/>
<text text-anchor="middle" x="1887.07" y="-315.976" font-family="Times,serif" font-size="12.00" fill="gray">done</text>
<g id="edge58" class="edge"><title>unrescuing&#45;&gt;active</title>
<path fill="none" stroke="black" d="M1952.92,-459.812C1921.01,-444.131 1864.35,-418.736 1812.53,-408 1648.96,-374.115 1595.85,-363.143 1436.75,-414 1342.62,-444.085 1242.24,-505.962 1195.49,-536.758"/>
<polygon fill="black" stroke="black" points="1193.53,-533.859 1187.13,-542.308 1197.4,-539.69 1193.53,-533.859"/>
<text text-anchor="middle" x="1573.64" y="-390.4" font-family="Times,serif" font-size="12.00" fill="gray">done</text>
</g>
<!-- unrescue failed -->
<g id="node23" class="node"><title>unrescue failed</title>
<ellipse fill="none" stroke="black" cx="2649.93" cy="-399.576" rx="64.889" ry="18"/>
<text text-anchor="middle" x="2649.93" y="-396.776" font-family="Times,serif" font-size="11.00" fill="red">unrescue failed</text>
<g id="node24" class="node"><title>unrescue failed</title>
<ellipse fill="none" stroke="black" cx="2197.64" cy="-444" rx="51.2277" ry="18"/>
<text text-anchor="middle" x="2197.64" y="-441.2" font-family="Times,serif" font-size="11.00" fill="red">unrescue failed</text>
</g>
<!-- unrescuing&#45;&gt;unrescue failed -->
<g id="edge56" class="edge"><title>unrescuing&#45;&gt;unrescue failed</title>
<path fill="none" stroke="black" d="M2425.96,-374.222C2434.56,-375.797 2443.54,-377.332 2451.98,-378.576 2493.34,-384.667 2539.83,-389.708 2577.25,-393.324"/>
<polygon fill="black" stroke="black" points="2577.27,-396.841 2587.56,-394.305 2577.93,-389.873 2577.27,-396.841"/>
<text text-anchor="middle" x="2509.48" y="-394.976" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
<g id="edge59" class="edge"><title>unrescuing&#45;&gt;unrescue failed</title>
<path fill="none" stroke="black" d="M2017.19,-468.081C2050.68,-463.586 2100.75,-456.868 2139.49,-451.669"/>
<polygon fill="black" stroke="black" points="2140.14,-455.113 2149.59,-450.314 2139.21,-448.175 2140.14,-455.113"/>
<text text-anchor="middle" x="2083.53" y="-467.4" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
</g>
<!-- deploy failed&#45;&gt;deploying -->
<g id="edge24" class="edge"><title>deploy failed&#45;&gt;deploying</title>
<path fill="none" stroke="black" d="M1591.39,-639.997C1495.7,-633.637 1297.13,-620.439 1201.91,-614.11"/>
<polygon fill="black" stroke="black" points="1202.14,-610.617 1191.93,-613.446 1201.67,-617.602 1202.14,-610.617"/>
<text text-anchor="middle" x="1396.38" y="-633.976" font-family="Times,serif" font-size="12.00">rebuild (via API)</text>
<path fill="none" stroke="black" d="M1338.95,-712.026C1307.33,-702.452 1258.9,-689.158 1215.75,-683 1141.3,-672.375 1053.63,-671.799 1001.89,-672.672"/>
<polygon fill="black" stroke="black" points="1001.66,-669.176 991.726,-672.87 1001.79,-676.175 1001.66,-669.176"/>
<text text-anchor="middle" x="1167.64" y="-686.4" font-family="Times,serif" font-size="12.00">rebuild (via API)</text>
</g>
<!-- deploy failed&#45;&gt;deploying -->
<g id="edge25" class="edge"><title>deploy failed&#45;&gt;deploying</title>
<path fill="none" stroke="black" d="M1599.77,-633.378C1590.68,-631.403 1581.2,-629.386 1572.31,-627.576 1520.46,-617.016 1507.99,-610.486 1455.31,-605.576 1403.15,-600.715 1389.83,-604.882 1337.45,-605.576 1291.92,-606.179 1240.22,-607.6 1202.46,-608.774"/>
<polygon fill="black" stroke="black" points="1202.09,-605.284 1192.2,-609.097 1202.31,-612.28 1202.09,-605.284"/>
<text text-anchor="middle" x="1396.38" y="-608.976" font-family="Times,serif" font-size="12.00">active (via API)</text>
<path fill="none" stroke="black" d="M1353.19,-707.136C1341.63,-698.649 1326.5,-688.613 1311.75,-682 1271.51,-663.966 1259.48,-662.601 1215.75,-657 1140.2,-647.325 1051.49,-657.887 1000.08,-666.147"/>
<polygon fill="black" stroke="black" points="999.297,-662.729 990.002,-667.815 1000.44,-669.635 999.297,-662.729"/>
<text text-anchor="middle" x="1167.64" y="-660.4" font-family="Times,serif" font-size="12.00">active (via API)</text>
</g>
<!-- deploy failed&#45;&gt;deleting -->
<g id="edge26" class="edge"><title>deploy failed&#45;&gt;deleting</title>
<path fill="none" stroke="black" d="M1701.77,-646.021C1798.28,-649.644 2005.39,-654.254 2178.98,-636.576 2361.78,-617.96 2402.88,-583.085 2584.98,-558.576 2676.43,-546.268 2784.06,-539.663 2844.82,-536.655"/>
<polygon fill="black" stroke="black" points="2845.06,-540.148 2854.88,-536.169 2844.73,-533.156 2845.06,-540.148"/>
<text text-anchor="middle" x="2254.48" y="-636.976" font-family="Times,serif" font-size="12.00">deleted (via API)</text>
<path fill="none" stroke="black" d="M1418.69,-721.813C1497.28,-719.664 1668.29,-714.983 1812.53,-711 1860.53,-709.675 1872.54,-709.774 1920.53,-708 2082.06,-702.028 2274.29,-692.373 2356.49,-688.123"/>
<polygon fill="black" stroke="black" points="2356.84,-691.609 2366.65,-687.596 2356.48,-684.619 2356.84,-691.609"/>
<text text-anchor="middle" x="1875.53" y="-713.4" font-family="Times,serif" font-size="12.00">deleted (via API)</text>
</g>
<!-- wait call&#45;back&#45;&gt;deploying -->
<g id="edge21" class="edge"><title>wait call&#45;back&#45;&gt;deploying</title>
<path fill="none" stroke="black" d="M1352.15,-696.602C1341.5,-693.814 1330.09,-690.965 1319.45,-688.576 1274.46,-678.474 1259.03,-689.501 1217.45,-669.576 1199.95,-661.192 1183.36,-647.183 1170.86,-634.92"/>
<polygon fill="black" stroke="black" points="1173.15,-632.251 1163.64,-627.575 1168.16,-637.159 1173.15,-632.251"/>
<text text-anchor="middle" x="1268.45" y="-691.976" font-family="Times,serif" font-size="12.00" fill="gray">resume</text>
<path fill="none" stroke="black" d="M1129.94,-711.739C1120.7,-709.058 1110.79,-706.308 1101.53,-704 1067.59,-695.533 1028.75,-687.554 999.918,-681.956"/>
<polygon fill="black" stroke="black" points="1000.26,-678.457 989.777,-680.005 998.936,-685.331 1000.26,-678.457"/>
<text text-anchor="middle" x="1061.03" y="-707.4" font-family="Times,serif" font-size="12.00" fill="gray">resume</text>
</g>
<!-- wait call&#45;back&#45;&gt;deleting -->
<g id="edge23" class="edge"><title>wait call&#45;back&#45;&gt;deleting</title>
<path fill="none" stroke="black" d="M1455.47,-707.842C1505.7,-707.271 1580.26,-706.576 1645.24,-706.576 1645.24,-706.576 1645.24,-706.576 2001.48,-706.576 2056.73,-706.576 2672.7,-596.316 2836.87,-553.576 2842.22,-552.183 2847.79,-550.538 2853.26,-548.808"/>
<polygon fill="black" stroke="black" points="2854.52,-552.08 2862.93,-545.637 2852.33,-545.429 2854.52,-552.08"/>
<text text-anchor="middle" x="2121.48" y="-701.976" font-family="Times,serif" font-size="12.00">deleted (via API)</text>
<path fill="none" stroke="black" d="M1198.4,-737.04C1237.57,-754.205 1309,-781 1373.25,-781 1373.25,-781 1373.25,-781 1669.53,-781 1973.71,-781 2049.15,-749.526 2348.75,-697 2352.16,-696.403 2355.69,-695.722 2359.21,-694.999"/>
<polygon fill="black" stroke="black" points="2360.01,-698.408 2369.04,-692.88 2358.53,-691.565 2360.01,-698.408"/>
<text text-anchor="middle" x="1767.53" y="-783.4" font-family="Times,serif" font-size="12.00">deleted (via API)</text>
</g>
<!-- wait call&#45;back&#45;&gt;deploy failed -->
<g id="edge22" class="edge"><title>wait call&#45;back&#45;&gt;deploy failed</title>
<path fill="none" stroke="black" d="M1431.76,-694.121C1444.64,-689.059 1459.5,-683.631 1473.31,-679.576 1510.3,-668.713 1552.69,-659.8 1586.12,-653.56"/>
<polygon fill="black" stroke="black" points="1586.98,-656.959 1596.19,-651.71 1585.72,-650.074 1586.98,-656.959"/>
<text text-anchor="middle" x="1522.81" y="-682.976" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
<path fill="none" stroke="black" d="M1215.84,-723C1246.72,-723 1287.15,-723 1319.31,-723"/>
<polygon fill="black" stroke="black" points="1319.43,-726.5 1329.43,-723 1319.43,-719.5 1319.43,-726.5"/>
<text text-anchor="middle" x="1272.75" y="-726.4" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
</g>
<!-- clean failed&#45;&gt;manageable -->
<g id="edge34" class="edge"><title>clean failed&#45;&gt;manageable</title>
<path fill="none" stroke="black" d="M1117.01,-162.866C1105.17,-157.702 1091.23,-152.472 1078.02,-149.576 1029.35,-138.907 1015.82,-146.242 966.019,-144.576 749.81,-137.342 691.914,-113.336 479.359,-153.576 470.428,-155.266 461.085,-157.817 452.216,-160.634"/>
<polygon fill="black" stroke="black" points="450.828,-157.408 442.452,-163.896 453.046,-164.047 450.828,-157.408"/>
<text text-anchor="middle" x="778.359" y="-140.976" font-family="Times,serif" font-size="12.00">manage (via API)</text>
<path fill="none" stroke="black" d="M929.475,-239.134C918.26,-233.755 904.767,-228.143 891.927,-225 851.265,-215.045 839.745,-220.941 797.927,-219 623.273,-210.894 577.797,-193.529 404.537,-217 399.338,-217.704 393.947,-218.732 388.632,-219.932"/>
<polygon fill="black" stroke="black" points="387.444,-216.62 378.573,-222.412 389.12,-223.416 387.444,-216.62"/>
<text text-anchor="middle" x="652.037" y="-216.4" font-family="Times,serif" font-size="12.00">manage (via API)</text>
</g>
<!-- clean wait&#45;&gt;cleaning -->
<g id="edge33" class="edge"><title>clean wait&#45;&gt;cleaning</title>
<path fill="none" stroke="black" d="M866.156,-163.432C828.79,-157.555 772.195,-152.82 725.359,-166.576 709.704,-171.174 694.238,-180.473 681.841,-189.43"/>
<polygon fill="black" stroke="black" points="679.655,-186.695 673.785,-195.515 683.874,-192.281 679.655,-186.695"/>
<text text-anchor="middle" x="778.359" y="-169.976" font-family="Times,serif" font-size="12.00" fill="gray">resume</text>
<path fill="none" stroke="black" d="M721.076,-237.942C691.098,-232.2 646.179,-227.488 609.537,-240 595.752,-244.707 582.682,-254.048 572.322,-263.089"/>
<polygon fill="black" stroke="black" points="569.949,-260.517 564.96,-269.864 574.689,-265.668 569.949,-260.517"/>
<text text-anchor="middle" x="652.037" y="-243.4" font-family="Times,serif" font-size="12.00" fill="gray">resume</text>
</g>
<!-- clean wait&#45;&gt;clean failed -->
<g id="edge31" class="edge"><title>clean wait&#45;&gt;clean failed</title>
<path fill="none" stroke="black" d="M954.1,-172.796C963.963,-173.057 974.341,-173.328 984.019,-173.576 1017.65,-174.435 1055.18,-175.361 1085.72,-176.104"/>
<polygon fill="black" stroke="black" points="1085.72,-179.605 1095.8,-176.349 1085.89,-172.607 1085.72,-179.605"/>
<text text-anchor="middle" x="1031.02" y="-179.976" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
<path fill="none" stroke="black" d="M793.165,-247.98C800.695,-248.349 808.565,-248.709 815.927,-249 845.273,-250.159 878.109,-251.118 904.528,-251.806"/>
<polygon fill="black" stroke="black" points="904.6,-255.309 914.686,-252.066 904.779,-248.311 904.6,-255.309"/>
<text text-anchor="middle" x="853.927" y="-254.4" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
</g>
<!-- clean wait&#45;&gt;clean failed -->
<g id="edge32" class="edge"><title>clean wait&#45;&gt;clean failed</title>
<path fill="none" stroke="black" d="M947.568,-162.319C959.219,-159.957 972.075,-157.753 984.019,-156.576 1025.59,-152.476 1036.7,-150.403 1078.02,-156.576 1085.83,-157.742 1093.97,-159.615 1101.81,-161.774"/>
<polygon fill="black" stroke="black" points="1100.95,-165.169 1111.53,-164.63 1102.92,-158.452 1100.95,-165.169"/>
<text text-anchor="middle" x="1031.02" y="-159.976" font-family="Times,serif" font-size="12.00">abort (via API)</text>
<path fill="none" stroke="black" d="M788.212,-237.07C797.112,-234.977 806.833,-233.052 815.927,-232 849.481,-228.118 858.582,-226.613 891.927,-232 899.723,-233.259 907.859,-235.338 915.599,-237.711"/>
<polygon fill="black" stroke="black" points="914.544,-241.048 925.137,-240.833 916.722,-234.395 914.544,-241.048"/>
<text text-anchor="middle" x="853.927" y="-235.4" font-family="Times,serif" font-size="12.00">abort (via API)</text>
</g>
<!-- inspect failed&#45;&gt;manageable -->
<g id="edge39" class="edge"><title>inspect failed&#45;&gt;manageable</title>
<path fill="none" stroke="black" d="M850.036,-59.343C749.254,-65.0049 543.392,-80.1205 479.359,-111.576 459.714,-121.226 441.689,-138.083 428.7,-152.345"/>
<polygon fill="black" stroke="black" points="425.998,-150.118 422.024,-159.939 431.256,-154.74 425.998,-150.118"/>
<text text-anchor="middle" x="654.359" y="-85.9757" font-family="Times,serif" font-size="12.00">manage (via API)</text>
<g id="edge40" class="edge"><title>inspect failed&#45;&gt;manageable</title>
<path fill="none" stroke="black" d="M911.165,-97.2979C816.144,-107.546 585.898,-137.675 404.537,-204 396.437,-206.962 387.977,-210.698 380.074,-214.494"/>
<polygon fill="black" stroke="black" points="378.514,-211.361 371.106,-218.936 381.62,-217.634 378.514,-211.361"/>
<text text-anchor="middle" x="652.037" y="-149.4" font-family="Times,serif" font-size="12.00">manage (via API)</text>
</g>
<!-- inspect failed&#45;&gt;inspecting -->
<g id="edge40" class="edge"><title>inspect failed&#45;&gt;inspecting</title>
<path fill="none" stroke="black" d="M873.23,-41.8394C860.397,-36.9231 845.45,-32.066 831.359,-29.5757 791.238,-22.4852 745.165,-22.5096 710.224,-24.1608"/>
<polygon fill="black" stroke="black" points="710.019,-20.6667 700.219,-24.6921 710.39,-27.6569 710.019,-20.6667"/>
<text text-anchor="middle" x="778.359" y="-32.9757" font-family="Times,serif" font-size="12.00">inspect (via API)</text>
<g id="edge41" class="edge"><title>inspect failed&#45;&gt;inspecting</title>
<path fill="none" stroke="black" d="M910.157,-90.4617C878.716,-88.8216 835.779,-86.8418 797.927,-86 759.985,-85.1562 750.466,-84.7126 712.537,-86 666.68,-87.5565 655.289,-89.5232 609.537,-93 605.115,-93.336 600.509,-93.6994 595.905,-94.0716"/>
<polygon fill="black" stroke="black" points="595.578,-90.5865 585.898,-94.8932 596.151,-97.563 595.578,-90.5865"/>
<text text-anchor="middle" x="755.232" y="-89.4" font-family="Times,serif" font-size="12.00">inspect (via API)</text>
</g>
<!-- inspect wait&#45;&gt;manageable -->
<g id="edge42" class="edge"><title>inspect wait&#45;&gt;manageable</title>
<path fill="none" stroke="black" d="M714.456,-12.2744C642.961,-4.36884 492.051,1.09013 404.537,-77 367.231,-110.289 353.367,-169.626 348.293,-204.739"/>
<polygon fill="black" stroke="black" points="344.804,-204.435 346.985,-214.803 351.745,-205.337 344.804,-204.435"/>
<text text-anchor="middle" x="549.037" y="-26.4" font-family="Times,serif" font-size="12.00" fill="gray">done</text>
</g>
<!-- inspect wait&#45;&gt;inspect failed -->
<g id="edge43" class="edge"><title>inspect wait&#45;&gt;inspect failed</title>
<path fill="none" stroke="black" d="M797.48,-21.3299C825.163,-24.7203 861.915,-31.504 891.927,-45 905.56,-51.1302 918.995,-60.6725 929.956,-69.6393"/>
<polygon fill="black" stroke="black" points="927.931,-72.5116 937.819,-76.318 932.463,-67.1765 927.931,-72.5116"/>
<text text-anchor="middle" x="853.927" y="-48.4" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
</g>
<!-- adopt failed&#45;&gt;manageable -->
<g id="edge44" class="edge"><title>adopt failed&#45;&gt;manageable</title>
<path fill="none" stroke="black" d="M868.877,-435.057C776.709,-404.07 543.338,-322.609 479.359,-271.576 455.808,-252.79 435.824,-224.525 423.093,-203.895"/>
<polygon fill="black" stroke="black" points="426.078,-202.066 417.93,-195.294 420.076,-205.669 426.078,-202.066"/>
<text text-anchor="middle" x="654.359" y="-378.976" font-family="Times,serif" font-size="12.00">manage (via API)</text>
<g id="edge47" class="edge"><title>adopt failed&#45;&gt;manageable</title>
<path fill="none" stroke="black" d="M727.614,-497.184C722.123,-493.451 716.741,-489.029 712.537,-484 699.249,-468.105 709.749,-455.066 694.537,-441 665.334,-413.996 643.124,-434.307 609.537,-413 599.715,-406.769 601.023,-400.733 591.537,-394 517.241,-341.261 475.997,-366.522 404.537,-310 386.806,-295.976 371.43,-275.504 360.811,-259.194"/>
<polygon fill="black" stroke="black" points="363.706,-257.224 355.415,-250.628 357.783,-260.955 363.706,-257.224"/>
<text text-anchor="middle" x="549.037" y="-397.4" font-family="Times,serif" font-size="12.00">manage (via API)</text>
</g>
<!-- adopt failed&#45;&gt;adopting -->
<g id="edge43" class="edge"><title>adopt failed&#45;&gt;adopting</title>
<path fill="none" stroke="black" d="M863.905,-436.956C853.333,-434.757 841.99,-432.753 831.359,-431.576 789.48,-426.936 741.654,-426.715 706.438,-427.497"/>
<polygon fill="black" stroke="black" points="706.302,-424 696.396,-427.755 706.482,-430.997 706.302,-424"/>
<text text-anchor="middle" x="778.359" y="-434.976" font-family="Times,serif" font-size="12.00">adopt (via API)</text>
<g id="edge46" class="edge"><title>adopt failed&#45;&gt;adopting</title>
<path fill="none" stroke="black" d="M728.923,-496.649C723.246,-492.889 717.454,-488.59 712.537,-484 702.79,-474.902 705.886,-466.999 694.537,-460 677.654,-449.587 627.891,-442.226 591.467,-438.054"/>
<polygon fill="black" stroke="black" points="591.438,-434.53 581.115,-436.912 590.67,-441.487 591.438,-434.53"/>
<text text-anchor="middle" x="652.037" y="-463.4" font-family="Times,serif" font-size="12.00">adopt (via API)</text>
</g>
<!-- rescue wait&#45;&gt;deleting -->
<g id="edge51" class="edge"><title>rescue wait&#45;&gt;deleting</title>
<path fill="none" stroke="black" d="M1934.09,-416.616C1941.08,-417.377 1948.22,-418.066 1954.98,-418.576 2054.31,-426.051 2079.76,-416.889 2178.98,-425.576 2473.28,-451.341 2546.99,-463.618 2836.87,-520.576 2841.08,-521.403 2845.45,-522.344 2849.8,-523.338"/>
<polygon fill="black" stroke="black" points="2849.33,-526.822 2859.87,-525.726 2850.94,-520.012 2849.33,-526.822"/>
<text text-anchor="middle" x="2381.98" y="-453.976" font-family="Times,serif" font-size="12.00">deleted (via API)</text>
<g id="edge54" class="edge"><title>rescue wait&#45;&gt;deleting</title>
<path fill="none" stroke="black" d="M1607.54,-533.303C1615.67,-535.669 1624.38,-538.057 1632.53,-540 1711.24,-558.758 2270.01,-653.373 2348.75,-672 2352.56,-672.901 2356.52,-673.915 2360.45,-674.977"/>
<polygon fill="black" stroke="black" points="2359.52,-678.349 2370.09,-677.672 2361.4,-671.608 2359.52,-678.349"/>
<text text-anchor="middle" x="1979.53" y="-614.4" font-family="Times,serif" font-size="12.00">deleted (via API)</text>
</g>
<!-- rescue wait&#45;&gt;rescuing -->
<g id="edge48" class="edge"><title>rescue wait&#45;&gt;rescuing</title>
<path fill="none" stroke="black" d="M1847.79,-399.468C1813.24,-391.462 1761.63,-384.597 1720.16,-400.576 1698.78,-408.813 1679.78,-426.542 1666.54,-441.552"/>
<polygon fill="black" stroke="black" points="1663.58,-439.637 1659.8,-449.534 1668.93,-444.154 1663.58,-439.637"/>
<text text-anchor="middle" x="1769.66" y="-403.976" font-family="Times,serif" font-size="12.00" fill="gray">resume</text>
<g id="edge51" class="edge"><title>rescue wait&#45;&gt;rescuing</title>
<path fill="none" stroke="black" d="M1537.28,-514.509C1509.42,-509.334 1469.88,-505.379 1436.75,-515 1424.03,-518.69 1411.44,-525.885 1400.98,-533.1"/>
<polygon fill="black" stroke="black" points="1398.78,-530.375 1392.75,-539.089 1402.9,-536.035 1398.78,-530.375"/>
<text text-anchor="middle" x="1475.75" y="-518.4" font-family="Times,serif" font-size="12.00" fill="gray">resume</text>
</g>
<!-- rescue wait&#45;&gt;rescue failed -->
<g id="edge49" class="edge"><title>rescue wait&#45;&gt;rescue failed</title>
<path fill="none" stroke="black" d="M1934.64,-405.04C1966.24,-400.908 2008.85,-394.599 2045.98,-386.576 2054.01,-384.841 2062.46,-382.736 2070.66,-380.538"/>
<polygon fill="black" stroke="black" points="2071.79,-383.858 2080.5,-377.829 2069.93,-377.109 2071.79,-383.858"/>
<text text-anchor="middle" x="2000.48" y="-404.976" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
<g id="edge52" class="edge"><title>rescue wait&#45;&gt;rescue failed</title>
<path fill="none" stroke="black" d="M1614.51,-523.873C1640.24,-524.07 1674.42,-523.646 1704.53,-521 1708.45,-520.655 1712.49,-520.209 1716.56,-519.697"/>
<polygon fill="black" stroke="black" points="1717.18,-523.143 1726.61,-518.308 1716.23,-516.209 1717.18,-523.143"/>
<text text-anchor="middle" x="1668.53" y="-526.4" font-family="Times,serif" font-size="12.00" fill="gray">fail</text>
</g>
<!-- rescue wait&#45;&gt;rescue failed -->
<g id="edge50" class="edge"><title>rescue wait&#45;&gt;rescue failed</title>
<path fill="none" stroke="black" d="M1912.23,-394.924C1929.41,-383.941 1950.25,-370.973 1954.98,-369.576 1987.92,-359.855 2026.17,-358.338 2057.59,-359.483"/>
<polygon fill="black" stroke="black" points="2057.82,-362.997 2067.97,-359.966 2058.14,-356.005 2057.82,-362.997"/>
<text text-anchor="middle" x="2000.48" y="-372.976" font-family="Times,serif" font-size="12.00">abort (via API)</text>
<g id="edge53" class="edge"><title>rescue wait&#45;&gt;rescue failed</title>
<path fill="none" stroke="black" d="M1604.53,-511.19C1613.4,-508.236 1623.23,-505.487 1632.53,-504 1659.82,-499.636 1690.6,-500.821 1715.77,-503.348"/>
<polygon fill="black" stroke="black" points="1715.51,-506.841 1725.83,-504.459 1716.28,-499.883 1715.51,-506.841"/>
<text text-anchor="middle" x="1668.53" y="-507.4" font-family="Times,serif" font-size="12.00">abort (via API)</text>
</g>
<!-- rescue failed&#45;&gt;deleting -->
<g id="edge54" class="edge"><title>rescue failed&#45;&gt;deleting</title>
<path fill="none" stroke="black" d="M2166.87,-376.352C2176.75,-378.569 2187.2,-380.774 2196.98,-382.576 2267.59,-395.574 2773.44,-438.939 2836.87,-472.576 2852.86,-481.054 2866.58,-495.982 2876.45,-509.065"/>
<polygon fill="black" stroke="black" points="2873.69,-511.224 2882.36,-517.31 2879.38,-507.144 2873.69,-511.224"/>
<text text-anchor="middle" x="2509.48" y="-427.976" font-family="Times,serif" font-size="12.00">deleted (via API)</text>
<g id="edge57" class="edge"><title>rescue failed&#45;&gt;deleting</title>
<path fill="none" stroke="black" d="M1811.94,-508.266C1931.91,-501.602 2264.67,-489.417 2348.75,-553 2381.35,-577.651 2391.97,-626.44 2395.4,-657.578"/>
<polygon fill="black" stroke="black" points="2391.94,-658.15 2396.34,-667.787 2398.91,-657.508 2391.94,-658.15"/>
<text text-anchor="middle" x="2083.53" y="-508.4" font-family="Times,serif" font-size="12.00">deleted (via API)</text>
</g>
<!-- rescue failed&#45;&gt;rescuing -->
<g id="edge52" class="edge"><title>rescue failed&#45;&gt;rescuing</title>
<path fill="none" stroke="black" d="M2073.71,-356.084C1995.44,-342.695 1835.22,-326.124 1720.16,-386.576 1696.43,-399.045 1676.5,-422.46 1663.51,-440.749"/>
<polygon fill="black" stroke="black" points="1660.45,-439.014 1657.7,-449.245 1666.23,-442.965 1660.45,-439.014"/>
<text text-anchor="middle" x="1887.07" y="-352.976" font-family="Times,serif" font-size="12.00">rescue (via API)</text>
<g id="edge55" class="edge"><title>rescue failed&#45;&gt;rescuing</title>
<path fill="none" stroke="black" d="M1729.7,-501.286C1683.65,-490.364 1602.41,-475.12 1532.75,-483 1489.25,-487.92 1476.18,-485.999 1436.75,-505 1423.02,-511.611 1409.58,-521.766 1398.78,-531.12"/>
<polygon fill="black" stroke="black" points="1396.18,-528.758 1391.08,-538.045 1400.86,-533.964 1396.18,-528.758"/>
<text text-anchor="middle" x="1573.64" y="-486.4" font-family="Times,serif" font-size="12.00">rescue (via API)</text>
</g>
<!-- rescue failed&#45;&gt;unrescuing -->
<g id="edge53" class="edge"><title>rescue failed&#45;&gt;unrescuing</title>
<path fill="none" stroke="black" d="M2177.62,-365.576C2220.04,-365.576 2278.54,-365.576 2321.91,-365.576"/>
<polygon fill="black" stroke="black" points="2322.03,-369.076 2332.03,-365.576 2322.03,-362.076 2322.03,-369.076"/>
<text text-anchor="middle" x="2254.48" y="-368.976" font-family="Times,serif" font-size="12.00">unrescue (via API)</text>
<g id="edge56" class="edge"><title>rescue failed&#45;&gt;unrescuing</title>
<path fill="none" stroke="black" d="M1800.34,-498.783C1809.92,-495.496 1820.55,-492.237 1830.53,-490 1863.34,-482.645 1901.01,-478.376 1930.13,-475.955"/>
<polygon fill="black" stroke="black" points="1930.75,-479.417 1940.44,-475.145 1930.2,-472.439 1930.75,-479.417"/>
<text text-anchor="middle" x="1875.53" y="-493.4" font-family="Times,serif" font-size="12.00">unrescue (via API)</text>
</g>
<!-- unrescue failed&#45;&gt;deleting -->
<g id="edge59" class="edge"><title>unrescue failed&#45;&gt;deleting</title>
<path fill="none" stroke="black" d="M2714.36,-397.093C2753.01,-398.627 2801.58,-406.312 2836.87,-431.576 2862.22,-449.724 2877.31,-483.013 2885.35,-506.836"/>
<polygon fill="black" stroke="black" points="2882.02,-507.915 2888.37,-516.395 2888.69,-505.806 2882.02,-507.915"/>
<text text-anchor="middle" x="2784.87" y="-434.976" font-family="Times,serif" font-size="12.00">deleted (via API)</text>
<g id="edge62" class="edge"><title>unrescue failed&#45;&gt;deleting</title>
<path fill="none" stroke="black" d="M2248.11,-447.572C2281.04,-452.559 2322.85,-464.339 2348.75,-492 2371.14,-515.906 2386.75,-610 2393.41,-657.803"/>
<polygon fill="black" stroke="black" points="2389.96,-658.365 2394.77,-667.804 2396.89,-657.424 2389.96,-658.365"/>
<text text-anchor="middle" x="2307.75" y="-495.4" font-family="Times,serif" font-size="12.00">deleted (via API)</text>
</g>
<!-- unrescue failed&#45;&gt;rescuing -->
<g id="edge57" class="edge"><title>unrescue failed&#45;&gt;rescuing</title>
<path fill="none" stroke="black" d="M2585.58,-396.578C2546.78,-395.801 2496.25,-396.687 2451.98,-403.576 2326.93,-423.035 2304.23,-466.387 2178.98,-484.576 1977.17,-513.883 1922.95,-503.11 1720.16,-481.576 1711,-480.603 1701.3,-479.011 1692.05,-477.218"/>
<polygon fill="black" stroke="black" points="1692.75,-473.789 1682.26,-475.215 1691.35,-480.647 1692.75,-473.789"/>
<text text-anchor="middle" x="2121.48" y="-500.976" font-family="Times,serif" font-size="12.00">rescue (via API)</text>
<g id="edge60" class="edge"><title>unrescue failed&#45;&gt;rescuing</title>
<path fill="none" stroke="black" d="M2146.17,-443.498C2094.13,-443.191 2010.62,-443.281 1938.53,-446 1715.16,-454.425 1635.78,-378.265 1436.75,-480 1416.29,-490.456 1400.06,-511.035 1389.42,-527.843"/>
<polygon fill="black" stroke="black" points="1386.25,-526.33 1384.1,-536.704 1392.25,-529.936 1386.25,-526.33"/>
<text text-anchor="middle" x="1767.53" y="-445.4" font-family="Times,serif" font-size="12.00">rescue (via API)</text>
</g>
<!-- unrescue failed&#45;&gt;unrescuing -->
<g id="edge58" class="edge"><title>unrescue failed&#45;&gt;unrescuing</title>
<path fill="none" stroke="black" d="M2620.75,-383.264C2605.4,-375.316 2585.75,-366.617 2566.98,-362.576 2524.73,-353.479 2475.75,-354.834 2438.95,-358.135"/>
<polygon fill="black" stroke="black" points="2438.53,-354.66 2428.91,-359.111 2439.21,-361.627 2438.53,-354.66"/>
<text text-anchor="middle" x="2509.48" y="-365.976" font-family="Times,serif" font-size="12.00">unrescue (via API)</text>
<g id="edge61" class="edge"><title>unrescue failed&#45;&gt;unrescuing</title>
<path fill="none" stroke="black" d="M2154.15,-434.38C2121.8,-428.83 2076.52,-424.913 2038.53,-436 2027.58,-439.195 2016.73,-445.154 2007.43,-451.37"/>
<polygon fill="black" stroke="black" points="2005.16,-448.687 1999.03,-457.328 2009.21,-454.396 2005.16,-448.687"/>
<text text-anchor="middle" x="2083.53" y="-439.4" font-family="Times,serif" font-size="12.00">unrescue (via API)</text>
</g>
</g>
</svg>

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 37 KiB

After

Width:  |  Height:  |  Size: 39 KiB

11
ironic/api/controllers/v1/node.py
View File

@ -173,6 +173,10 @@ def update_state_in_older_versions(obj):
if (pecan.request.version.minor < versions.MINOR_2_AVAILABLE_STATE and
obj.provision_state == ir_states.AVAILABLE):
obj.provision_state = ir_states.NOSTATE
# if requested version < 1.39, convert INSPECTWAIT to INSPECTING
if (not api_utils.allow_inspect_wait_state() and
obj.provision_state == ir_states.INSPECTWAIT):
obj.provision_state = ir_states.INSPECTING
class BootDeviceController(rest.RestController):
@ -1928,6 +1932,13 @@ class NodesController(rest.RestController):
"is in progress.")
raise wsme.exc.ClientSideError(
msg % node_ident, status_code=http_client.CONFLICT)
elif (rpc_node.provision_state == ir_states.INSPECTING and
api_utils.allow_inspect_wait_state()):
msg = _('Cannot update node "%(node)s" while it is in state '
'"%(state)s".') % {'node': rpc_node.uuid,
'state': ir_states.INSPECTING}
raise wsme.exc.ClientSideError(msg,
status_code=http_client.CONFLICT)
names = api_utils.get_patch_values(patch, '/name')
if len(names):

10
ironic/api/controllers/v1/port.py
View File

@ -34,6 +34,7 @@ from ironic.api import expose
from ironic.common import exception
from ironic.common.i18n import _
from ironic.common import policy
from ironic.common import states as ir_states
from ironic import objects
METRICS = metrics_utils.get_metrics_logger(__name__)
@ -668,6 +669,15 @@ class PortsController(rest.RestController):
rpc_port[field] = patch_val
rpc_node = objects.Node.get_by_id(context, rpc_port.node_id)
if (rpc_node.provision_state == ir_states.INSPECTING and
api_utils.allow_inspect_wait_state()):
msg = _('Cannot update port "%(port)s" on "%(node)s" while it is '
'in state "%(state)s".') % {'port': rpc_port.uuid,
'node': rpc_node.uuid,
'state': ir_states.INSPECTING}
raise wsme.exc.ClientSideError(msg,
status_code=http_client.CONFLICT)
notify_extra = {'node_uuid': rpc_node.uuid,
'portgroup_uuid': port.portgroup_uuid}
notify.emit_start_notification(context, rpc_port, 'update',

9
ironic/api/controllers/v1/portgroup.py
View File

@ -30,6 +30,7 @@ from ironic.api import expose
from ironic.common import exception
from ironic.common.i18n import _
from ironic.common import policy
from ironic.common import states as ir_states
from ironic import objects
METRICS = metrics_utils.get_metrics_logger(__name__)
@ -543,6 +544,14 @@ class PortgroupsController(pecan.rest.RestController):
rpc_portgroup[field] = patch_val
rpc_node = objects.Node.get_by_id(context, rpc_portgroup.node_id)
if (rpc_node.provision_state == ir_states.INSPECTING and
api_utils.allow_inspect_wait_state()):
msg = _('Cannot update portgroup "%(portgroup)s" on node '
'"%(node)s" while it is in state "%(state)s".') % {
'portgroup': rpc_portgroup.uuid, 'node': rpc_node.uuid,
'state': ir_states.INSPECTING}
raise wsme.exc.ClientSideError(msg,
status_code=http_client.CONFLICT)
notify.emit_start_notification(context, rpc_portgroup, 'update',
node_uuid=rpc_node.uuid)

9
ironic/api/controllers/v1/utils.py
View File

@ -721,6 +721,15 @@ def allow_traits():
return pecan.request.version.minor >= versions.MINOR_37_NODE_TRAITS
def allow_inspect_wait_state():
"""Check if inspect wait is allowed for the node.
Version 1.39 of the API adds 'inspect wait' state to substitute
'inspecting' state during asynchronous hardware inspection.
"""
return pecan.request.version.minor >= versions.MINOR_39_INSPECT_WAIT
def handle_post_port_like_extra_vif(p_dict):
"""Handle a Post request that sets .extra['vif_port_id'].

5
ironic/api/controllers/v1/versions.py
View File

@ -75,7 +75,7 @@ BASE_VERSION = 1
# v1.36: Add Ironic Python Agent version support.
# v1.37: Add node traits.
# v1.38: Add rescue and unrescue provision states
# Add rescue_interface to the node object
# v1.39: Add inspect wait provision state.
MINOR_0_JUNO = 0
MINOR_1_INITIAL_VERSION = 1
@ -116,6 +116,7 @@ MINOR_35_REBUILD_CONFIG_DRIVE = 35
MINOR_36_AGENT_VERSION_HEARTBEAT = 36
MINOR_37_NODE_TRAITS = 37
MINOR_38_RESCUE_INTERFACE = 38
MINOR_39_INSPECT_WAIT = 39
# When adding another version, update:
# - MINOR_MAX_VERSION
@ -123,7 +124,7 @@ MINOR_38_RESCUE_INTERFACE = 38
# explanation of what changed in the new version
# - common/release_mappings.py, RELEASE_MAPPING['master']['api']
MINOR_MAX_VERSION = MINOR_38_RESCUE_INTERFACE
MINOR_MAX_VERSION = MINOR_39_INSPECT_WAIT
# String representations of the minor and maximum versions
_MIN_VERSION_STRING = '{}.{}'.format(BASE_VERSION, MINOR_1_INITIAL_VERSION)

2
ironic/common/release_mappings.py
View File

@ -100,7 +100,7 @@ RELEASE_MAPPING = {
}
},
'master': {
'api': '1.38',
'api': '1.39',
'rpc': '1.44',
'objects': {
'Node': ['1.23'],

30
ironic/common/states.py
View File

@ -164,13 +164,20 @@ INSPECTING = 'inspecting'
""" Node is under inspection.
This is the provision state used when inspection is started. A successfully
inspected node shall transition to MANAGEABLE state.
inspected node shall transition to MANAGEABLE state. For asynchronous
inspection, node shall transition to INSPECTWAIT state.
"""
INSPECTFAIL = 'inspect failed'
""" Node inspection failed. """
INSPECTWAIT = 'inspect wait'
""" Node is under inspection.
This is the provision state used when an asynchronous inspection is in
progress. A successfully inspected node shall transition to MANAGEABLE state.
"""
ADOPTING = 'adopting'
""" Node is being adopted.
@ -209,8 +216,9 @@ UNRESCUEFAIL = 'unrescue failed'
UNRESCUING = 'unrescuing'
""" Node is being restored from rescue mode (to active state). """
UPDATE_ALLOWED_STATES = (DEPLOYFAIL, INSPECTING, INSPECTFAIL, CLEANFAIL, ERROR,
VERIFYING, ADOPTFAIL, RESCUEFAIL, UNRESCUEFAIL)
UPDATE_ALLOWED_STATES = (DEPLOYFAIL, INSPECTING, INSPECTFAIL, INSPECTWAIT,
CLEANFAIL, ERROR, VERIFYING, ADOPTFAIL, RESCUEFAIL,
UNRESCUEFAIL)
"""Transitional states in which we allow updating a node."""
DELETE_ALLOWED_STATES = (AVAILABLE, MANAGEABLE, ENROLL, ADOPTFAIL)
@ -220,8 +228,8 @@ STABLE_STATES = (ENROLL, MANAGEABLE, AVAILABLE, ACTIVE, ERROR, RESCUE)
"""States that will not transition unless receiving a request."""
UNSTABLE_STATES = (DEPLOYING, DEPLOYWAIT, CLEANING, CLEANWAIT, VERIFYING,
DELETING, INSPECTING, ADOPTING, RESCUING, RESCUEWAIT,
UNRESCUING)
DELETING, INSPECTING, INSPECTWAIT, ADOPTING, RESCUING,
RESCUEWAIT, UNRESCUING)
"""States that can be changed without external request."""
##############
@ -292,6 +300,7 @@ machine.add_transition(AVAILABLE, DEPLOYING, 'deploy')
# Add inspect* states.
machine.add_state(INSPECTING, target=MANAGEABLE, **watchers)
machine.add_state(INSPECTFAIL, target=MANAGEABLE, **watchers)
machine.add_state(INSPECTWAIT, target=MANAGEABLE, **watchers)
# Add adopt* states
machine.add_state(ADOPTING, target=ACTIVE, **watchers)
@ -392,6 +401,15 @@ machine.add_transition(INSPECTING, MANAGEABLE, 'done')
# Inspection may fail.
machine.add_transition(INSPECTING, INSPECTFAIL, 'fail')
# Transition for asynchronous inspection
machine.add_transition(INSPECTING, INSPECTWAIT, 'wait')
# Inspection is done
machine.add_transition(INSPECTWAIT, MANAGEABLE, 'done')
# Inspection failed.
machine.add_transition(INSPECTWAIT, INSPECTFAIL, 'fail')
# Move the node to manageable state for any other
# action.
machine.add_transition(INSPECTFAIL, MANAGEABLE, 'manage')

44
ironic/conductor/manager.py
View File

@ -170,7 +170,8 @@ class ConductorManager(base_manager.BaseConductorManager):
# TODO(dtantsur): reconsider allowing changing some (but not all)
# interfaces for active nodes in the future.
allowed_update_states = [states.ENROLL, states.INSPECTING,
states.MANAGEABLE, states.AVAILABLE]
states.INSPECTWAIT, states.MANAGEABLE,
states.AVAILABLE]
action = _("Node %(node)s can not have %(field)s "
"updated unless it is in one of allowed "
"(%(allowed)s) states or in maintenance mode.")
@ -2231,7 +2232,7 @@ class ConductorManager(base_manager.BaseConductorManager):
registered on another port already.
:raises: InvalidState if port connectivity attributes
are updated while node not in a MANAGEABLE or ENROLL or
INSPECTING state or not in MAINTENANCE mode.
INSPECTING or INSPECTWAIT state or not in MAINTENANCE mode.
:raises: Conflict if trying to set extra/vif_port_id or
pxe_enabled=True on port which is a member of portgroup with
standalone_ports_supported=False.
@ -2268,6 +2269,7 @@ class ConductorManager(base_manager.BaseConductorManager):
'physical_network'}
allowed_update_states = [states.ENROLL,
states.INSPECTING,
states.INSPECTWAIT,
states.MANAGEABLE]
if (set(port_obj.obj_what_changed()) & connectivity_attr
and not (node.provision_state in allowed_update_states
@ -2312,8 +2314,8 @@ class ConductorManager(base_manager.BaseConductorManager):
:raises: PortgroupMACAlreadyExists if the update is setting a MAC which
is registered on another portgroup already.
:raises: InvalidState if portgroup-node association is updated while
node not in a MANAGEABLE or ENROLL or INSPECTING state or not
in MAINTENANCE mode.
node not in a MANAGEABLE or ENROLL or INSPECTING or
INSPECTWAIT state or not in MAINTENANCE mode.
:raises: PortgroupNotEmpty if there are ports associated with this
portgroup.
:raises: Conflict when trying to set standalone_ports_supported=False
@ -2332,10 +2334,11 @@ class ConductorManager(base_manager.BaseConductorManager):
if 'node_id' in portgroup_obj.obj_what_changed():
# NOTE(zhenguo): If portgroup update is modifying the
# portgroup-node association then node should be in
# MANAGEABLE/INSPECTING/ENROLL provisioning state or in
# maintenance mode, otherwise InvalidState is raised.
# MANAGEABLE/INSPECTING/INSPECTWAIT/ENROLL provisioning state
# or in maintenance mode, otherwise InvalidState is raised.
allowed_update_states = [states.ENROLL,
states.INSPECTING,
states.INSPECTWAIT,
states.MANAGEABLE]
if (node.provision_state not in allowed_update_states
and not node.maintenance):
@ -2771,24 +2774,24 @@ class ConductorManager(base_manager.BaseConductorManager):
action='inspect', node=task.node.uuid,
state=task.node.provision_state)
@METRICS.timer('ConductorManager._check_inspect_timeouts')
@METRICS.timer('ConductorManager._check_inspect_wait_timeouts')
@periodics.periodic(spacing=CONF.conductor.check_provision_state_interval)
def _check_inspect_timeouts(self, context):
"""Periodically checks inspect_timeout and fails upon reaching it.
def _check_inspect_wait_timeouts(self, context):
"""Periodically checks inspect_wait_timeout and fails upon reaching it.
:param: context: request context
"""
callback_timeout = CONF.conductor.inspect_timeout
callback_timeout = CONF.conductor.inspect_wait_timeout
if not callback_timeout:
return
filters = {'reserved': False,
'provision_state': states.INSPECTING,
'provision_state': states.INSPECTWAIT,
'inspection_started_before': callback_timeout}
sort_key = 'inspection_started_at'
last_error = _("timeout reached while inspecting the node")
self._fail_if_in_state(context, filters, states.INSPECTING,
self._fail_if_in_state(context, filters, states.INSPECTWAIT,
sort_key, last_error=last_error)
@METRICS.timer('ConductorManager.set_target_raid_config')
@ -3484,7 +3487,7 @@ def _do_inspect_hardware(task):
:param: task: a TaskManager instance with an exclusive lock
on its node.
:raises: HardwareInspectionFailure if driver doesn't
return the state as states.MANAGEABLE or
return the state as states.MANAGEABLE, states.INSPECTWAIT or
states.INSPECTING.
"""
@ -3512,7 +3515,20 @@ def _do_inspect_hardware(task):
task.process_event('done')
LOG.info('Successfully inspected node %(node)s',
{'node': node.uuid})
elif new_state != states.INSPECTING:
# TODO(kaifeng): remove INSPECTING support during S* cycle.
elif new_state in (states.INSPECTING, states.INSPECTWAIT):
task.process_event('wait')
if new_state == states.INSPECTING:
inspect_intf_name = task.driver.inspect.__class__.__name__
LOG.warning('Received INSPECTING state from %(intf)s. Returning '
'INSPECTING from InspectInterface.inspect_hardware '
'is deprecated, and will cause node be moved to '
'INSPECTFAIL state after deprecation period. Please '
'return INSPECTWAIT instead if the inspection process '
'is asynchronous.', {'intf': inspect_intf_name})
LOG.info('Successfully started introspection on node %(node)s',
{'node': node.uuid})
else:
error = (_("During inspection, driver returned unexpected "
"state %(state)s") % {'state': new_state})
handle_failure(error)

3
ironic/conf/conductor.py
View File

@ -122,8 +122,9 @@ opts = [
help=_('Name of the Swift container to store config drive '
'data. Used when configdrive_use_object_store is '
'True.')),
cfg.IntOpt('inspect_timeout',
cfg.IntOpt('inspect_wait_timeout',
default=1800,
deprecated_name='inspect_timeout',
help=_('Timeout (seconds) for waiting for node inspection. '
'0 - unlimited.')),
cfg.BoolOpt('automated_clean',

8
ironic/drivers/modules/inspector.py
View File

@ -119,7 +119,7 @@ class Inspector(base.InspectInterface):
ironic-inspector. Results will be checked in a periodic task.
:param task: a task from TaskManager.
:returns: states.INSPECTING
:returns: states.INSPECTWAIT
"""
LOG.debug('Starting inspection for node %(uuid)s using '
'ironic-inspector', {'uuid': task.node.uuid})
@ -128,13 +128,13 @@ class Inspector(base.InspectInterface):
# we can release a lock as soon as possible and allow ironic-inspector
# to operate on a node.
eventlet.spawn_n(_start_inspection, task.node.uuid, task.context)
return states.INSPECTING
return states.INSPECTWAIT
@periodics.periodic(spacing=CONF.inspector.status_check_period,
enabled=CONF.inspector.enabled)
def _periodic_check_result(self, manager, context):
"""Periodic task checking results of inspection."""
filters = {'provision_state': states.INSPECTING}
filters = {'provision_state': states.INSPECTWAIT}
node_iter = manager.iter_nodes(filters=filters)
for node_uuid, driver in node_iter:
@ -171,7 +171,7 @@ def _start_inspection(node_uuid, context):
def _check_status(task):
"""Check inspection status for node given by a task."""
node = task.node
if node.provision_state != states.INSPECTING:
if node.provision_state != states.INSPECTWAIT:
return
if not isinstance(task.driver.inspect, Inspector):
return

4
ironic/drivers/modules/oneview/inspect.py
View File

@ -66,7 +66,7 @@ class OneViewInspect(inspector.Inspector):
@periodics.periodic(spacing=CONF.inspector.status_check_period,
enabled=CONF.inspector.enabled)
def _periodic_check_result(self, manager, context):
filters = {'provision_state': states.INSPECTING}
filters = {'provision_state': states.INSPECTWAIT}
node_iter = manager.iter_nodes(filters=filters)
for node_uuid, driver in node_iter:
@ -87,7 +87,7 @@ class OneViewInspect(inspector.Inspector):
state_after = task.node.provision_state
# inspection finished
if state_before == states.INSPECTING and state_after in [
if state_before == states.INSPECTWAIT and state_after in [
states.MANAGEABLE, states.INSPECTFAIL
]:
deploy_utils.deallocate_server_hardware_from_ironic(task.node)

37
ironic/tests/unit/api/controllers/v1/test_node.py
View File

@ -227,6 +227,20 @@ class TestListNodes(test_api_base.BaseApiTest):
headers={api_base.Version.string: '1.36'})
self.assertNotIn('traits', data)
def test_node_inspect_wait_state_between_api_versions(self):
node = obj_utils.create_test_node(self.context,
provision_state='inspect wait')
lower_version_data = self.get_json(
'/nodes/%s' % node.uuid,
headers={api_base.Version.string: '1.38'})
self.assertEqual('inspecting', lower_version_data['provision_state'])
higher_version_data = self.get_json(
'/nodes/%s' % node.uuid,
headers={api_base.Version.string: '1.39'})
self.assertEqual('inspect wait',
higher_version_data['provision_state'])
def test_get_one_custom_fields(self):
node = obj_utils.create_test_node(self.context,
chassis_id=self.chassis.id)
@ -1641,6 +1655,29 @@ class TestPatch(test_api_base.BaseApiTest):
'op': 'remove'}])
self.assertEqual(http_client.OK, response.status_code)
def test_update_in_inspecting_not_allowed(self):
node = obj_utils.create_test_node(self.context,
uuid=uuidutils.generate_uuid(),
provision_state=states.INSPECTING)
self.mock_update_node.return_value = node
response = self.patch_json('/nodes/%s' % node.uuid,
[{'path': '/instance_uuid',
'op': 'remove'}],
headers={api_base.Version.string: "1.39"},
expect_errors=True)
self.assertEqual(http_client.CONFLICT, response.status_code)
def test_update_in_inspecting_allowed(self):
node = obj_utils.create_test_node(self.context,
uuid=uuidutils.generate_uuid(),
provision_state=states.INSPECTING)
self.mock_update_node.return_value = node
response = self.patch_json('/nodes/%s' % node.uuid,
[{'path': '/instance_uuid',
'op': 'remove'}],
headers={api_base.Version.string: "1.38"})
self.assertEqual(http_client.OK, response.status_code)
def test_add_state_in_deployfail(self):
node = obj_utils.create_test_node(self.context,
uuid=uuidutils.generate_uuid(),

26
ironic/tests/unit/api/controllers/v1/test_port.py
View File

@ -33,6 +33,7 @@ from ironic.api.controllers.v1 import port as api_port
from ironic.api.controllers.v1 import utils as api_utils
from ironic.api.controllers.v1 import versions
from ironic.common import exception
from ironic.common import states
from ironic.common import utils as common_utils
from ironic.conductor import rpcapi
from ironic import objects
@ -1504,6 +1505,31 @@ class TestPatch(test_api_base.BaseApiTest):
self.assertEqual(internal_info, response.json['internal_info'])
self.assertTrue(mock_warn.called)
def test_update_in_inspecting_not_allowed(self, mock_upd):
self.node.provision_state = states.INSPECTING
self.node.save()
response = self.patch_json('/ports/%s' % self.port.uuid,
[{'path': '/extra/foo',
'value': 'bar',
'op': 'add'}],
headers={api_base.Version.string: "1.39"},
expect_errors=True)
self.assertEqual(http_client.CONFLICT, response.status_code)
self.assertFalse(mock_upd.called)
def test_update_in_inspecting_allowed(self, mock_upd):
self.node.provision_state = states.INSPECTING
self.node.save()
extra = {'foo': 'bar'}
response = self.patch_json('/ports/%s' % self.port.uuid,
[{'path': '/extra/foo',
'value': 'bar',
'op': 'add'}],
headers={api_base.Version.string: "1.38"})
self.assertEqual(http_client.OK, response.status_code)
self.assertEqual(extra, response.json['extra'])
self.assertTrue(mock_upd.called)
@mock.patch.object(rpcapi.ConductorAPI, 'create_port', autospec=True,
side_effect=_rpcapi_create_port)

32
ironic/tests/unit/api/controllers/v1/test_portgroup.py
View File

@ -31,6 +31,7 @@ from ironic.api.controllers.v1 import notification_utils
from ironic.api.controllers.v1 import portgroup as api_portgroup
from ironic.api.controllers.v1 import utils as api_utils
from ironic.common import exception
from ironic.common import states
from ironic.common import utils as common_utils
from ironic.conductor import rpcapi
from ironic import objects
@ -974,6 +975,37 @@ class TestPatch(test_api_base.BaseApiTest):
self.assertTrue(response.json['error_message'])
self.assertFalse(mock_upd.called)
def test_update_in_inspecting_not_allowed(self, mock_upd):
self.node.provision_state = states.INSPECTING
self.node.save()
address = 'AA:BB:CC:DD:EE:FF'
response = self.patch_json('/portgroups/%s' % self.portgroup.uuid,
[{'path': '/address',
'value': address,
'op': 'replace'}],
expect_errors=True,
headers={api_base.Version.string: "1.39"})
self.assertEqual(http_client.CONFLICT, response.status_code)
self.assertFalse(mock_upd.called)
def test_update_in_inspecting_allowed(self, mock_upd):
self.node.provision_state = states.INSPECTING
self.node.save()
address = 'AA:BB:CC:DD:EE:FF'
mock_upd.return_value = self.portgroup
mock_upd.return_value.address = address.lower()
response = self.patch_json('/portgroups/%s' % self.portgroup.uuid,
[{'path': '/address',
'value': address,
'op': 'replace'}],
expect_errors=True,
headers={api_base.Version.string: "1.38"})
self.assertEqual(http_client.OK, response.status_int)
self.assertEqual(address.lower(), response.json['address'])
self.assertTrue(mock_upd.called)
kargs = mock_upd.call_args[0][1]
self.assertEqual(address.lower(), kargs.address)
@mock.patch.object(rpcapi.ConductorAPI, 'update_portgroup', autospec=True,
side_effect=_rpcapi_update_portgroup)

108
ironic/tests/unit/conductor/test_manager.py
View File

@ -613,7 +613,7 @@ class UpdateNodeTestCase(mgr_utils.ServiceSetUpMixin, db_base.DbTestCase):
def test_update_node_interface_in_allowed_state(self):
for state in [states.ENROLL, states.MANAGEABLE, states.INSPECTING,
states.AVAILABLE]:
states.INSPECTWAIT, states.AVAILABLE]:
self._test_update_node_interface_in_allowed_state(state)
def test_update_node_interface_in_maintenance(self):
@ -4011,6 +4011,22 @@ class UpdatePortTestCase(mgr_utils.ServiceSetUpMixin, db_base.DbTestCase):
mock_val.assert_called_once_with(mock.ANY, mock.ANY)
mock_pc.assert_called_once_with(mock.ANY, mock.ANY, port)
@mock.patch.object(n_flat.FlatNetwork, 'port_changed', autospec=True)
@mock.patch.object(n_flat.FlatNetwork, 'validate', autospec=True)
def test_update_port_to_node_in_inspect_wait_state(self, mock_val,
mock_pc):
node = obj_utils.create_test_node(self.context, driver='fake',
provision_state=states.INSPECTWAIT)
port = obj_utils.create_test_port(self.context,
node_id=node.id,
extra={'foo': 'bar'})
port.pxe_enabled = True
self.service.update_port(self.context, port)
port.refresh()
self.assertEqual(True, port.pxe_enabled)
mock_val.assert_called_once_with(mock.ANY, mock.ANY)
mock_pc.assert_called_once_with(mock.ANY, mock.ANY, port)
@mock.patch.object(n_flat.FlatNetwork, 'port_changed', autospec=True)
@mock.patch.object(n_flat.FlatNetwork, 'validate', autospec=True)
def test_update_port_node_active_state_and_maintenance(self, mock_val,
@ -4637,6 +4653,32 @@ class UpdatePortgroupTestCase(mgr_utils.ServiceSetUpMixin, db_base.DbTestCase):
mock_val.assert_called_once_with(mock.ANY, mock.ANY)
mock_pgc.assert_called_once_with(mock.ANY, mock.ANY, portgroup)
@mock.patch.object(dbapi.IMPL, 'get_ports_by_portgroup_id')
@mock.patch.object(n_flat.FlatNetwork, 'portgroup_changed', autospec=True)
@mock.patch.object(n_flat.FlatNetwork, 'validate', autospec=True)
def test_update_portgroup_to_node_in_inspect_wait_state(self, mock_val,
mock_pgc,
mock_get_ports):
node = obj_utils.create_test_node(self.context, driver='fake')
portgroup = obj_utils.create_test_portgroup(self.context,
node_id=node.id,
extra={'foo': 'bar'})
update_node = obj_utils.create_test_node(
self.context, driver='fake',
provision_state=states.INSPECTWAIT,
uuid=uuidutils.generate_uuid())
mock_get_ports.return_value = []
self._start_service()
portgroup.node_id = update_node.id
self.service.update_portgroup(self.context, portgroup)
portgroup.refresh()
self.assertEqual(update_node.id, portgroup.node_id)
mock_get_ports.assert_called_once_with(portgroup.uuid)
mock_val.assert_called_once_with(mock.ANY, mock.ANY)
mock_pgc.assert_called_once_with(mock.ANY, mock.ANY, portgroup)
@mock.patch.object(dbapi.IMPL, 'get_ports_by_portgroup_id')
@mock.patch.object(n_flat.FlatNetwork, 'portgroup_changed', autospec=True)
@mock.patch.object(n_flat.FlatNetwork, 'validate', autospec=True)
@ -5893,8 +5935,22 @@ class NodeInspectHardware(mgr_utils.ServiceSetUpMixin, db_base.DbTestCase):
mock_inspect.return_value = states.INSPECTING
manager._do_inspect_hardware(task)
node.refresh()
self.assertEqual(states.INSPECTING, node.provision_state)
self.assertEqual(states.NOSTATE, node.target_provision_state)
self.assertEqual(states.INSPECTWAIT, node.provision_state)
self.assertEqual(states.MANAGEABLE, node.target_provision_state)
self.assertIsNone(node.last_error)
mock_inspect.assert_called_once_with(mock.ANY)
@mock.patch('ironic.drivers.modules.fake.FakeInspect.inspect_hardware')
def test_inspect_hardware_return_inspect_wait(self, mock_inspect):
self._start_service()
node = obj_utils.create_test_node(self.context, driver='fake',
provision_state=states.INSPECTING)
task = task_manager.TaskManager(self.context, node.uuid)
mock_inspect.return_value = states.INSPECTWAIT
manager._do_inspect_hardware(task)
node.refresh()
self.assertEqual(states.INSPECTWAIT, node.provision_state)
self.assertEqual(states.MANAGEABLE, node.target_provision_state)
self.assertIsNone(node.last_error)
mock_inspect.assert_called_once_with(mock.ANY)
@ -5915,17 +5971,17 @@ class NodeInspectHardware(mgr_utils.ServiceSetUpMixin, db_base.DbTestCase):
mock_inspect.assert_called_once_with(mock.ANY)
self.assertTrue(log_mock.error.called)
def test__check_inspect_timeouts(self):
def test__check_inspect_wait_timeouts(self):
self._start_service()
CONF.set_override('inspect_timeout', 1, group='conductor')
CONF.set_override('inspect_wait_timeout', 1, group='conductor')
node = obj_utils.create_test_node(
self.context, driver='fake',
provision_state=states.INSPECTING,
provision_state=states.INSPECTWAIT,
target_provision_state=states.MANAGEABLE,
provision_updated_at=datetime.datetime(2000, 1, 1, 0, 0),
inspection_started_at=datetime.datetime(2000, 1, 1, 0, 0))
self.service._check_inspect_timeouts(self.context)
self.service._check_inspect_wait_timeouts(self.context)
self._stop_service()
node.refresh()
self.assertEqual(states.INSPECTFAIL, node.provision_state)
@ -6030,26 +6086,26 @@ class NodeInspectHardware(mgr_utils.ServiceSetUpMixin, db_base.DbTestCase):
@mock.patch.object(task_manager, 'acquire')
@mock.patch.object(manager.ConductorManager, '_mapped_to_this_conductor')
@mock.patch.object(dbapi.IMPL, 'get_nodeinfo_list')
class ManagerCheckInspectTimeoutsTestCase(mgr_utils.CommonMixIn,
db_base.DbTestCase):
class ManagerCheckInspectWaitTimeoutsTestCase(mgr_utils.CommonMixIn,
db_base.DbTestCase):
def setUp(self):
super(ManagerCheckInspectTimeoutsTestCase, self).setUp()
self.config(inspect_timeout=300, group='conductor')
super(ManagerCheckInspectWaitTimeoutsTestCase, self).setUp()
self.config(inspect_wait_timeout=300, group='conductor')
self.service = manager.ConductorManager('hostname', 'test-topic')
self.service.dbapi = self.dbapi
self.node = self._create_node(provision_state=states.INSPECTING,
self.node = self._create_node(provision_state=states.INSPECTWAIT,
target_provision_state=states.MANAGEABLE)
self.task = self._create_task(node=self.node)
self.node2 = self._create_node(
provision_state=states.INSPECTING,
provision_state=states.INSPECTWAIT,
target_provision_state=states.MANAGEABLE)
self.task2 = self._create_task(node=self.node2)
self.filters = {'reserved': False,
'inspection_started_before': 300,
'provision_state': states.INSPECTING}
'provision_state': states.INSPECTWAIT}
self.columns = ['uuid', 'driver']
def _assert_get_nodeinfo_args(self, get_nodeinfo_mock):
@ -6059,9 +6115,9 @@ class ManagerCheckInspectTimeoutsTestCase(mgr_utils.CommonMixIn,
def test__check_inspect_timeouts_disabled(self, get_nodeinfo_mock,
mapped_mock, acquire_mock):
self.config(inspect_timeout=0, group='conductor')
self.config(inspect_wait_timeout=0, group='conductor')
self.service._check_inspect_timeouts(self.context)
self.service._check_inspect_wait_timeouts(self.context)
self.assertFalse(get_nodeinfo_mock.called)
self.assertFalse(mapped_mock.called)
@ -6072,7 +6128,7 @@ class ManagerCheckInspectTimeoutsTestCase(mgr_utils.CommonMixIn,
get_nodeinfo_mock.return_value = self._get_nodeinfo_list_response()
mapped_mock.return_value = False
self.service._check_inspect_timeouts(self.context)
self.service._check_inspect_wait_timeouts(self.context)
self._assert_get_nodeinfo_args(get_nodeinfo_mock)
mapped_mock.assert_called_once_with(self.node.uuid, self.node.driver)
@ -6084,7 +6140,7 @@ class ManagerCheckInspectTimeoutsTestCase(mgr_utils.CommonMixIn,
mapped_mock.return_value = True
acquire_mock.side_effect = self._get_acquire_side_effect(self.task)
self.service._check_inspect_timeouts(self.context)
self.service._check_inspect_wait_timeouts(self.context)
self._assert_get_nodeinfo_args(get_nodeinfo_mock)
mapped_mock.assert_called_once_with(self.node.uuid, self.node.driver)
@ -6101,7 +6157,7 @@ class ManagerCheckInspectTimeoutsTestCase(mgr_utils.CommonMixIn,
acquire_mock.side_effect = exception.NodeNotFound(node='fake')
# Exception eaten
self.service._check_inspect_timeouts(self.context)
self.service._check_inspect_wait_timeouts(self.context)
self._assert_get_nodeinfo_args(get_nodeinfo_mock)
mapped_mock.assert_called_once_with(self.node.uuid,
@ -6121,7 +6177,7 @@ class ManagerCheckInspectTimeoutsTestCase(mgr_utils.CommonMixIn,
host='fake')
# Exception eaten
self.service._check_inspect_timeouts(self.context)
self.service._check_inspect_wait_timeouts(self.context)
self._assert_get_nodeinfo_args(get_nodeinfo_mock)
mapped_mock.assert_called_once_with(self.node.uuid,
@ -6142,7 +6198,7 @@ class ManagerCheckInspectTimeoutsTestCase(mgr_utils.CommonMixIn,
mapped_mock.return_value = True
acquire_mock.side_effect = self._get_acquire_side_effect(task)
self.service._check_inspect_timeouts(self.context)
self.service._check_inspect_wait_timeouts(self.context)
self._assert_get_nodeinfo_args(get_nodeinfo_mock)
mapped_mock.assert_called_once_with(
@ -6155,7 +6211,7 @@ class ManagerCheckInspectTimeoutsTestCase(mgr_utils.CommonMixIn,
def test__check_inspect_timeouts_to_maintenance_after_lock(
self, get_nodeinfo_mock, mapped_mock, acquire_mock):
task = self._create_task(
node_attrs=dict(provision_state=states.INSPECTING,
node_attrs=dict(provision_state=states.INSPECTWAIT,
target_provision_state=states.MANAGEABLE,
maintenance=True,
uuid=self.node.uuid))
@ -6165,7 +6221,7 @@ class ManagerCheckInspectTimeoutsTestCase(mgr_utils.CommonMixIn,
acquire_mock.side_effect = (
self._get_acquire_side_effect([task, self.task2]))
self.service._check_inspect_timeouts(self.context)
self.service._check_inspect_wait_timeouts(self.context)
self._assert_get_nodeinfo_args(get_nodeinfo_mock)
self.assertEqual([mock.call(self.node.uuid, task.node.driver),
@ -6190,7 +6246,7 @@ class ManagerCheckInspectTimeoutsTestCase(mgr_utils.CommonMixIn,
[(self.task, exception.NoFreeConductorWorker()), self.task2])
# Exception should be nuked
self.service._check_inspect_timeouts(self.context)
self.service._check_inspect_wait_timeouts(self.context)
self._assert_get_nodeinfo_args(get_nodeinfo_mock)
# mapped should be only called for the first node as we should
@ -6212,7 +6268,7 @@ class ManagerCheckInspectTimeoutsTestCase(mgr_utils.CommonMixIn,
# Should re-raise
self.assertRaises(exception.IronicException,
self.service._check_inspect_timeouts,
self.service._check_inspect_wait_timeouts,
self.context)
self._assert_get_nodeinfo_args(get_nodeinfo_mock)
@ -6238,7 +6294,7 @@ class ManagerCheckInspectTimeoutsTestCase(mgr_utils.CommonMixIn,
acquire_mock.side_effect = (
self._get_acquire_side_effect([self.task] * 3))
self.service._check_inspect_timeouts(self.context)
self.service._check_inspect_wait_timeouts(self.context)
# Should only have ran 2.
self.assertEqual([mock.call(self.node.uuid, self.node.driver)] * 2,

10
ironic/tests/unit/drivers/modules/test_inspector.py
View File

@ -150,7 +150,7 @@ class CommonFunctionsTestCase(BaseTestCase):
class InspectHardwareTestCase(BaseTestCase):
def test_ok(self, mock_client):
mock_introspect = mock_client.return_value.introspect
self.assertEqual(states.INSPECTING,
self.assertEqual(states.INSPECTWAIT,
self.driver.inspect.inspect_hardware(self.task))
mock_introspect.assert_called_once_with(self.node.uuid)
@ -169,7 +169,7 @@ class InspectHardwareTestCase(BaseTestCase):
class CheckStatusTestCase(BaseTestCase):
def setUp(self):
super(CheckStatusTestCase, self).setUp()
self.node.provision_state = states.INSPECTING
self.node.provision_state = states.INSPECTWAIT
def test_not_inspecting(self, mock_client):
mock_get = mock_client.return_value.get_status
@ -177,6 +177,12 @@ class CheckStatusTestCase(BaseTestCase):
inspector._check_status(self.task)
self.assertFalse(mock_get.called)
def test_not_check_inspecting(self, mock_client):
mock_get = mock_client.return_value.get_status
self.node.provision_state = states.INSPECTING
inspector._check_status(self.task)
self.assertFalse(mock_get.called)
def test_not_inspector(self, mock_client):
mock_get = mock_client.return_value.get_status
self.task.driver.inspect = object()

22
releasenotes/notes/add-inspect-wait-state-948f83dfe342897b.yaml Normal file
View File

@ -0,0 +1,22 @@
---
upgrade:
- |
Adds an ``inspect wait`` state to handle asynchronous hardware
introspection. Caution should be taken due to the timeout monitoring
is shifted from ``inspecting`` to ``inspect wait``, please stop all
running asynchronous hardware inspection or wait until it is finished
before upgrading to the Rocky release. Otherwise nodes in asynchronous
inspection will be left at ``inspecting`` state forever unless the
database is manually updated.
deprecations:
- |
Adds an ``inspect wait`` state to handle asynchronous hardware
introspection. The ``[conductor]inspect_timeout`` configuration option
is deprecated for removal, please use ``[conductor]inspect_wait_timeout``
instead to specify the timeout of inspection process.
other:
- |
Adds an ``inspect wait`` state to handle asynchronous hardware
introspection. Returning ``INSPECTING`` from the ``inspect_hardware``
method of inspect interface is deprecated, ``INSPECTWAIT`` should be
returned instead.