<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="generator" content="Asciidoctor 2.0.22">
<meta name="author" content="Perforce Professional Services">
<title>SDP Migration and Upgrade Guide</title>
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:300,300italic,400,400italic,600,600italic%7CNoto+Serif:400,400italic,700,700italic%7CDroid+Sans+Mono:400,700">
<style>
/*! Asciidoctor default stylesheet | MIT License | https://asciidoctor.org */
/* Uncomment the following line when using as a custom stylesheet */
/* @import "https://fonts.googleapis.com/css?family=Open+Sans:300,300italic,400,400italic,600,600italic%7CNoto+Serif:400,400italic,700,700italic%7CDroid+Sans+Mono:400,700"; */
html{font-family:sans-serif;-webkit-text-size-adjust:100%}
a{background:none}
a:focus{outline:thin dotted}
a:active,a:hover{outline:0}
h1{font-size:2em;margin:.67em 0}
b,strong{font-weight:bold}
abbr{font-size:.9em}
abbr[title]{cursor:help;border-bottom:1px dotted #dddddf;text-decoration:none}
dfn{font-style:italic}
hr{height:0}
mark{background:#ff0;color:#000}
code,kbd,pre,samp{font-family:monospace;font-size:1em}
pre{white-space:pre-wrap}
q{quotes:"\201C" "\201D" "\2018" "\2019"}
small{font-size:80%}
sub,sup{font-size:75%;line-height:0;position:relative;vertical-align:baseline}
sup{top:-.5em}
sub{bottom:-.25em}
img{border:0}
svg:not(:root){overflow:hidden}
figure{margin:0}
audio,video{display:inline-block}
audio:not([controls]){display:none;height:0}
fieldset{border:1px solid silver;margin:0 2px;padding:.35em .625em .75em}
legend{border:0;padding:0}
button,input,select,textarea{font-family:inherit;font-size:100%;margin:0}
button,input{line-height:normal}
button,select{text-transform:none}
button,html input[type=button],input[type=reset],input[type=submit]{-webkit-appearance:button;cursor:pointer}
button[disabled],html input[disabled]{cursor:default}
input[type=checkbox],input[type=radio]{padding:0}
button::-moz-focus-inner,input::-moz-focus-inner{border:0;padding:0}
textarea{overflow:auto;vertical-align:top}
table{border-collapse:collapse;border-spacing:0}
*,::before,::after{box-sizing:border-box}
html,body{font-size:100%}
body{background:#fff;color:rgba(0,0,0,.8);padding:0;margin:0;font-family:"Noto Serif","DejaVu Serif",serif;line-height:1;position:relative;cursor:auto;-moz-tab-size:4;-o-tab-size:4;tab-size:4;word-wrap:anywhere;-moz-osx-font-smoothing:grayscale;-webkit-font-smoothing:antialiased}
a:hover{cursor:pointer}
img,object,embed{max-width:100%;height:auto}
object,embed{height:100%}
img{-ms-interpolation-mode:bicubic}
.left{float:left!important}
.right{float:right!important}
.text-left{text-align:left!important}
.text-right{text-align:right!important}
.text-center{text-align:center!important}
.text-justify{text-align:justify!important}
.hide{display:none}
img,object,svg{display:inline-block;vertical-align:middle}
textarea{height:auto;min-height:50px}
select{width:100%}
.subheader,.admonitionblock td.content>.title,.audioblock>.title,.exampleblock>.title,.imageblock>.title,.listingblock>.title,.literalblock>.title,.stemblock>.title,.openblock>.title,.paragraph>.title,.quoteblock>.title,table.tableblock>.title,.verseblock>.title,.videoblock>.title,.dlist>.title,.olist>.title,.ulist>.title,.qlist>.title,.hdlist>.title{line-height:1.45;color:#7a2518;font-weight:400;margin-top:0;margin-bottom:.25em}
div,dl,dt,dd,ul,ol,li,h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6,pre,form,p,blockquote,th,td{margin:0;padding:0}
a{color:#2156a5;text-decoration:underline;line-height:inherit}
a:hover,a:focus{color:#1d4b8f}
a img{border:0}
p{line-height:1.6;margin-bottom:1.25em;text-rendering:optimizeLegibility}
p aside{font-size:.875em;line-height:1.35;font-style:italic}
h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{font-family:"Open Sans","DejaVu Sans",sans-serif;font-weight:300;font-style:normal;color:#ba3925;text-rendering:optimizeLegibility;margin-top:1em;margin-bottom:.5em;line-height:1.0125em}
h1 small,h2 small,h3 small,#toctitle small,.sidebarblock>.content>.title small,h4 small,h5 small,h6 small{font-size:60%;color:#e99b8f;line-height:0}
h1{font-size:2.125em}
h2{font-size:1.6875em}
h3,#toctitle,.sidebarblock>.content>.title{font-size:1.375em}
h4,h5{font-size:1.125em}
h6{font-size:1em}
hr{border:solid #dddddf;border-width:1px 0 0;clear:both;margin:1.25em 0 1.1875em}
em,i{font-style:italic;line-height:inherit}
strong,b{font-weight:bold;line-height:inherit}
small{font-size:60%;line-height:inherit}
code{font-family:"Droid Sans Mono","DejaVu Sans Mono",monospace;font-weight:400;color:rgba(0,0,0,.9)}
ul,ol,dl{line-height:1.6;margin-bottom:1.25em;list-style-position:outside;font-family:inherit}
ul,ol{margin-left:1.5em}
ul li ul,ul li ol{margin-left:1.25em;margin-bottom:0}
ul.circle{list-style-type:circle}
ul.disc{list-style-type:disc}
ul.square{list-style-type:square}
ul.circle ul:not([class]),ul.disc ul:not([class]),ul.square ul:not([class]){list-style:inherit}
ol li ul,ol li ol{margin-left:1.25em;margin-bottom:0}
dl dt{margin-bottom:.3125em;font-weight:bold}
dl dd{margin-bottom:1.25em}
blockquote{margin:0 0 1.25em;padding:.5625em 1.25em 0 1.1875em;border-left:1px solid #ddd}
blockquote,blockquote p{line-height:1.6;color:rgba(0,0,0,.85)}
@media screen and (min-width:768px){h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{line-height:1.2}
h1{font-size:2.75em}
h2{font-size:2.3125em}
h3,#toctitle,.sidebarblock>.content>.title{font-size:1.6875em}
h4{font-size:1.4375em}}
table{background:#fff;margin-bottom:1.25em;border:1px solid #dedede;word-wrap:normal}
table thead,table tfoot{background:#f7f8f7}
table thead tr th,table thead tr td,table tfoot tr th,table tfoot tr td{padding:.5em .625em .625em;font-size:inherit;color:rgba(0,0,0,.8);text-align:left}
table tr th,table tr td{padding:.5625em .625em;font-size:inherit;color:rgba(0,0,0,.8)}
table tr.even,table tr.alt{background:#f8f8f7}
table thead tr th,table tfoot tr th,table tbody tr td,table tr td,table tfoot tr td{line-height:1.6}
h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{line-height:1.2;word-spacing:-.05em}
h1 strong,h2 strong,h3 strong,#toctitle strong,.sidebarblock>.content>.title strong,h4 strong,h5 strong,h6 strong{font-weight:400}
.center{margin-left:auto;margin-right:auto}
.stretch{width:100%}
.clearfix::before,.clearfix::after,.float-group::before,.float-group::after{content:" ";display:table}
.clearfix::after,.float-group::after{clear:both}
:not(pre).nobreak{word-wrap:normal}
:not(pre).nowrap{white-space:nowrap}
:not(pre).pre-wrap{white-space:pre-wrap}
:not(pre):not([class^=L])>code{font-size:.9375em;font-style:normal!important;letter-spacing:0;padding:.1em .5ex;word-spacing:-.15em;background:#f7f7f8;border-radius:4px;line-height:1.45;text-rendering:optimizeSpeed}
pre{color:rgba(0,0,0,.9);font-family:"Droid Sans Mono","DejaVu Sans Mono",monospace;line-height:1.45;text-rendering:optimizeSpeed}
pre code,pre pre{color:inherit;font-size:inherit;line-height:inherit}
pre>code{display:block}
pre.nowrap,pre.nowrap pre{white-space:pre;word-wrap:normal}
em em{font-style:normal}
strong strong{font-weight:400}
.keyseq{color:rgba(51,51,51,.8)}
kbd{font-family:"Droid Sans Mono","DejaVu Sans Mono",monospace;display:inline-block;color:rgba(0,0,0,.8);font-size:.65em;line-height:1.45;background:#f7f7f7;border:1px solid #ccc;border-radius:3px;box-shadow:0 1px 0 rgba(0,0,0,.2),inset 0 0 0 .1em #fff;margin:0 .15em;padding:.2em .5em;vertical-align:middle;position:relative;top:-.1em;white-space:nowrap}
.keyseq kbd:first-child{margin-left:0}
.keyseq kbd:last-child{margin-right:0}
.menuseq,.menuref{color:#000}
.menuseq b:not(.caret),.menuref{font-weight:inherit}
.menuseq{word-spacing:-.02em}
.menuseq b.caret{font-size:1.25em;line-height:.8}
.menuseq i.caret{font-weight:bold;text-align:center;width:.45em}
b.button::before,b.button::after{position:relative;top:-1px;font-weight:400}
b.button::before{content:"[";padding:0 3px 0 2px}
b.button::after{content:"]";padding:0 2px 0 3px}
p a>code:hover{color:rgba(0,0,0,.9)}
#header,#content,#footnotes,#footer{width:100%;margin:0 auto;max-width:62.5em;*zoom:1;position:relative;padding-left:.9375em;padding-right:.9375em}
#header::before,#header::after,#content::before,#content::after,#footnotes::before,#footnotes::after,#footer::before,#footer::after{content:" ";display:table}
#header::after,#content::after,#footnotes::after,#footer::after{clear:both}
#content{margin-top:1.25em}
#content::before{content:none}
#header>h1:first-child{color:rgba(0,0,0,.85);margin-top:2.25rem;margin-bottom:0}
#header>h1:first-child+#toc{margin-top:8px;border-top:1px solid #dddddf}
#header>h1:only-child{border-bottom:1px solid #dddddf;padding-bottom:8px}
#header .details{border-bottom:1px solid #dddddf;line-height:1.45;padding-top:.25em;padding-bottom:.25em;padding-left:.25em;color:rgba(0,0,0,.6);display:flex;flex-flow:row wrap}
#header .details span:first-child{margin-left:-.125em}
#header .details span.email a{color:rgba(0,0,0,.85)}
#header .details br{display:none}
#header .details br+span::before{content:"\00a0\2013\00a0"}
#header .details br+span.author::before{content:"\00a0\22c5\00a0";color:rgba(0,0,0,.85)}
#header .details br+span#revremark::before{content:"\00a0|\00a0"}
#header #revnumber{text-transform:capitalize}
#header #revnumber::after{content:"\00a0"}
#content>h1:first-child:not([class]){color:rgba(0,0,0,.85);border-bottom:1px solid #dddddf;padding-bottom:8px;margin-top:0;padding-top:1rem;margin-bottom:1.25rem}
#toc{border-bottom:1px solid #e7e7e9;padding-bottom:.5em}
#toc>ul{margin-left:.125em}
#toc ul.sectlevel0>li>a{font-style:italic}
#toc ul.sectlevel0 ul.sectlevel1{margin:.5em 0}
#toc ul{font-family:"Open Sans","DejaVu Sans",sans-serif;list-style-type:none}
#toc li{line-height:1.3334;margin-top:.3334em}
#toc a{text-decoration:none}
#toc a:active{text-decoration:underline}
#toctitle{color:#7a2518;font-size:1.2em}
@media screen and (min-width:768px){#toctitle{font-size:1.375em}
body.toc2{padding-left:15em;padding-right:0}
body.toc2 #header>h1:nth-last-child(2){border-bottom:1px solid #dddddf;padding-bottom:8px}
#toc.toc2{margin-top:0!important;background:#f8f8f7;position:fixed;width:15em;left:0;top:0;border-right:1px solid #e7e7e9;border-top-width:0!important;border-bottom-width:0!important;z-index:1000;padding:1.25em 1em;height:100%;overflow:auto}
#toc.toc2 #toctitle{margin-top:0;margin-bottom:.8rem;font-size:1.2em}
#toc.toc2>ul{font-size:.9em;margin-bottom:0}
#toc.toc2 ul ul{margin-left:0;padding-left:1em}
#toc.toc2 ul.sectlevel0 ul.sectlevel1{padding-left:0;margin-top:.5em;margin-bottom:.5em}
body.toc2.toc-right{padding-left:0;padding-right:15em}
body.toc2.toc-right #toc.toc2{border-right-width:0;border-left:1px solid #e7e7e9;left:auto;right:0}}
@media screen and (min-width:1280px){body.toc2{padding-left:20em;padding-right:0}
#toc.toc2{width:20em}
#toc.toc2 #toctitle{font-size:1.375em}
#toc.toc2>ul{font-size:.95em}
#toc.toc2 ul ul{padding-left:1.25em}
body.toc2.toc-right{padding-left:0;padding-right:20em}}
#content #toc{border:1px solid #e0e0dc;margin-bottom:1.25em;padding:1.25em;background:#f8f8f7;border-radius:4px}
#content #toc>:first-child{margin-top:0}
#content #toc>:last-child{margin-bottom:0}
#footer{max-width:none;background:rgba(0,0,0,.8);padding:1.25em}
#footer-text{color:hsla(0,0%,100%,.8);line-height:1.44}
#content{margin-bottom:.625em}
.sect1{padding-bottom:.625em}
@media screen and (min-width:768px){#content{margin-bottom:1.25em}
.sect1{padding-bottom:1.25em}}
.sect1:last-child{padding-bottom:0}
.sect1+.sect1{border-top:1px solid #e7e7e9}
#content h1>a.anchor,h2>a.anchor,h3>a.anchor,#toctitle>a.anchor,.sidebarblock>.content>.title>a.anchor,h4>a.anchor,h5>a.anchor,h6>a.anchor{position:absolute;z-index:1001;width:1.5ex;margin-left:-1.5ex;display:block;text-decoration:none!important;visibility:hidden;text-align:center;font-weight:400}
#content h1>a.anchor::before,h2>a.anchor::before,h3>a.anchor::before,#toctitle>a.anchor::before,.sidebarblock>.content>.title>a.anchor::before,h4>a.anchor::before,h5>a.anchor::before,h6>a.anchor::before{content:"\00A7";font-size:.85em;display:block;padding-top:.1em}
#content h1:hover>a.anchor,#content h1>a.anchor:hover,h2:hover>a.anchor,h2>a.anchor:hover,h3:hover>a.anchor,#toctitle:hover>a.anchor,.sidebarblock>.content>.title:hover>a.anchor,h3>a.anchor:hover,#toctitle>a.anchor:hover,.sidebarblock>.content>.title>a.anchor:hover,h4:hover>a.anchor,h4>a.anchor:hover,h5:hover>a.anchor,h5>a.anchor:hover,h6:hover>a.anchor,h6>a.anchor:hover{visibility:visible}
#content h1>a.link,h2>a.link,h3>a.link,#toctitle>a.link,.sidebarblock>.content>.title>a.link,h4>a.link,h5>a.link,h6>a.link{color:#ba3925;text-decoration:none}
#content h1>a.link:hover,h2>a.link:hover,h3>a.link:hover,#toctitle>a.link:hover,.sidebarblock>.content>.title>a.link:hover,h4>a.link:hover,h5>a.link:hover,h6>a.link:hover{color:#a53221}
details,.audioblock,.imageblock,.literalblock,.listingblock,.stemblock,.videoblock{margin-bottom:1.25em}
details{margin-left:1.25rem}
details>summary{cursor:pointer;display:block;position:relative;line-height:1.6;margin-bottom:.625rem;outline:none;-webkit-tap-highlight-color:transparent}
details>summary::-webkit-details-marker{display:none}
details>summary::before{content:"";border:solid transparent;border-left:solid;border-width:.3em 0 .3em .5em;position:absolute;top:.5em;left:-1.25rem;transform:translateX(15%)}
details[open]>summary::before{border:solid transparent;border-top:solid;border-width:.5em .3em 0;transform:translateY(15%)}
details>summary::after{content:"";width:1.25rem;height:1em;position:absolute;top:.3em;left:-1.25rem}
.admonitionblock td.content>.title,.audioblock>.title,.exampleblock>.title,.imageblock>.title,.listingblock>.title,.literalblock>.title,.stemblock>.title,.openblock>.title,.paragraph>.title,.quoteblock>.title,table.tableblock>.title,.verseblock>.title,.videoblock>.title,.dlist>.title,.olist>.title,.ulist>.title,.qlist>.title,.hdlist>.title{text-rendering:optimizeLegibility;text-align:left;font-family:"Noto Serif","DejaVu Serif",serif;font-size:1rem;font-style:italic}
table.tableblock.fit-content>caption.title{white-space:nowrap;width:0}
.paragraph.lead>p,#preamble>.sectionbody>[class=paragraph]:first-of-type p{font-size:1.21875em;line-height:1.6;color:rgba(0,0,0,.85)}
.admonitionblock>table{border-collapse:separate;border:0;background:none;width:100%}
.admonitionblock>table td.icon{text-align:center;width:80px}
.admonitionblock>table td.icon img{max-width:none}
.admonitionblock>table td.icon .title{font-weight:bold;font-family:"Open Sans","DejaVu Sans",sans-serif;text-transform:uppercase}
.admonitionblock>table td.content{padding-left:1.125em;padding-right:1.25em;border-left:1px solid #dddddf;color:rgba(0,0,0,.6);word-wrap:anywhere}
.admonitionblock>table td.content>:last-child>:last-child{margin-bottom:0}
.exampleblock>.content{border:1px solid #e6e6e6;margin-bottom:1.25em;padding:1.25em;background:#fff;border-radius:4px}
.sidebarblock{border:1px solid #dbdbd6;margin-bottom:1.25em;padding:1.25em;background:#f3f3f2;border-radius:4px}
.sidebarblock>.content>.title{color:#7a2518;margin-top:0;text-align:center}
.exampleblock>.content>:first-child,.sidebarblock>.content>:first-child{margin-top:0}
.exampleblock>.content>:last-child,.exampleblock>.content>:last-child>:last-child,.exampleblock>.content .olist>ol>li:last-child>:last-child,.exampleblock>.content .ulist>ul>li:last-child>:last-child,.exampleblock>.content .qlist>ol>li:last-child>:last-child,.sidebarblock>.content>:last-child,.sidebarblock>.content>:last-child>:last-child,.sidebarblock>.content .olist>ol>li:last-child>:last-child,.sidebarblock>.content .ulist>ul>li:last-child>:last-child,.sidebarblock>.content .qlist>ol>li:last-child>:last-child{margin-bottom:0}
.literalblock pre,.listingblock>.content>pre{border-radius:4px;overflow-x:auto;padding:1em;font-size:.8125em}
@media screen and (min-width:768px){.literalblock pre,.listingblock>.content>pre{font-size:.90625em}}
@media screen and (min-width:1280px){.literalblock pre,.listingblock>.content>pre{font-size:1em}}
.literalblock pre,.listingblock>.content>pre:not(.highlight),.listingblock>.content>pre[class=highlight],.listingblock>.content>pre[class^="highlight "]{background:#f7f7f8}
.literalblock.output pre{color:#f7f7f8;background:rgba(0,0,0,.9)}
.listingblock>.content{position:relative}
.listingblock code[data-lang]::before{display:none;content:attr(data-lang);position:absolute;font-size:.75em;top:.425rem;right:.5rem;line-height:1;text-transform:uppercase;color:inherit;opacity:.5}
.listingblock:hover code[data-lang]::before{display:block}
.listingblock.terminal pre .command::before{content:attr(data-prompt);padding-right:.5em;color:inherit;opacity:.5}
.listingblock.terminal pre .command:not([data-prompt])::before{content:"$"}
.listingblock pre.highlightjs{padding:0}
.listingblock pre.highlightjs>code{padding:1em;border-radius:4px}
.listingblock pre.prettyprint{border-width:0}
.prettyprint{background:#f7f7f8}
pre.prettyprint .linenums{line-height:1.45;margin-left:2em}
pre.prettyprint li{background:none;list-style-type:inherit;padding-left:0}
pre.prettyprint li code[data-lang]::before{opacity:1}
pre.prettyprint li:not(:first-child) code[data-lang]::before{display:none}
table.linenotable{border-collapse:separate;border:0;margin-bottom:0;background:none}
table.linenotable td[class]{color:inherit;vertical-align:top;padding:0;line-height:inherit;white-space:normal}
table.linenotable td.code{padding-left:.75em}
table.linenotable td.linenos,pre.pygments .linenos{border-right:1px solid;opacity:.35;padding-right:.5em;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none}
pre.pygments span.linenos{display:inline-block;margin-right:.75em}
.quoteblock{margin:0 1em 1.25em 1.5em;display:table}
.quoteblock:not(.excerpt)>.title{margin-left:-1.5em;margin-bottom:.75em}
.quoteblock blockquote,.quoteblock p{color:rgba(0,0,0,.85);font-size:1.15rem;line-height:1.75;word-spacing:.1em;letter-spacing:0;font-style:italic;text-align:justify}
.quoteblock blockquote{margin:0;padding:0;border:0}
.quoteblock blockquote::before{content:"\201c";float:left;font-size:2.75em;font-weight:bold;line-height:.6em;margin-left:-.6em;color:#7a2518;text-shadow:0 1px 2px rgba(0,0,0,.1)}
.quoteblock blockquote>.paragraph:last-child p{margin-bottom:0}
.quoteblock .attribution{margin-top:.75em;margin-right:.5ex;text-align:right}
.verseblock{margin:0 1em 1.25em}
.verseblock pre{font-family:"Open Sans","DejaVu Sans",sans-serif;font-size:1.15rem;color:rgba(0,0,0,.85);font-weight:300;text-rendering:optimizeLegibility}
.verseblock pre strong{font-weight:400}
.verseblock .attribution{margin-top:1.25rem;margin-left:.5ex}
.quoteblock .attribution,.verseblock .attribution{font-size:.9375em;line-height:1.45;font-style:italic}
.quoteblock .attribution br,.verseblock .attribution br{display:none}
.quoteblock .attribution cite,.verseblock .attribution cite{display:block;letter-spacing:-.025em;color:rgba(0,0,0,.6)}
.quoteblock.abstract blockquote::before,.quoteblock.excerpt blockquote::before,.quoteblock .quoteblock blockquote::before{display:none}
.quoteblock.abstract blockquote,.quoteblock.abstract p,.quoteblock.excerpt blockquote,.quoteblock.excerpt p,.quoteblock .quoteblock blockquote,.quoteblock .quoteblock p{line-height:1.6;word-spacing:0}
.quoteblock.abstract{margin:0 1em 1.25em;display:block}
.quoteblock.abstract>.title{margin:0 0 .375em;font-size:1.15em;text-align:center}
.quoteblock.excerpt>blockquote,.quoteblock .quoteblock{padding:0 0 .25em 1em;border-left:.25em solid #dddddf}
.quoteblock.excerpt,.quoteblock .quoteblock{margin-left:0}
.quoteblock.excerpt blockquote,.quoteblock.excerpt p,.quoteblock .quoteblock blockquote,.quoteblock .quoteblock p{color:inherit;font-size:1.0625rem}
.quoteblock.excerpt .attribution,.quoteblock .quoteblock .attribution{color:inherit;font-size:.85rem;text-align:left;margin-right:0}
p.tableblock:last-child{margin-bottom:0}
td.tableblock>.content{margin-bottom:1.25em;word-wrap:anywhere}
td.tableblock>.content>:last-child{margin-bottom:-1.25em}
table.tableblock,th.tableblock,td.tableblock{border:0 solid #dedede}
table.grid-all>*>tr>*{border-width:1px}
table.grid-cols>*>tr>*{border-width:0 1px}
table.grid-rows>*>tr>*{border-width:1px 0}
table.frame-all{border-width:1px}
table.frame-ends{border-width:1px 0}
table.frame-sides{border-width:0 1px}
table.frame-none>colgroup+*>:first-child>*,table.frame-sides>colgroup+*>:first-child>*{border-top-width:0}
table.frame-none>:last-child>:last-child>*,table.frame-sides>:last-child>:last-child>*{border-bottom-width:0}
table.frame-none>*>tr>:first-child,table.frame-ends>*>tr>:first-child{border-left-width:0}
table.frame-none>*>tr>:last-child,table.frame-ends>*>tr>:last-child{border-right-width:0}
table.stripes-all>*>tr,table.stripes-odd>*>tr:nth-of-type(odd),table.stripes-even>*>tr:nth-of-type(even),table.stripes-hover>*>tr:hover{background:#f8f8f7}
th.halign-left,td.halign-left{text-align:left}
th.halign-right,td.halign-right{text-align:right}
th.halign-center,td.halign-center{text-align:center}
th.valign-top,td.valign-top{vertical-align:top}
th.valign-bottom,td.valign-bottom{vertical-align:bottom}
th.valign-middle,td.valign-middle{vertical-align:middle}
table thead th,table tfoot th{font-weight:bold}
tbody tr th{background:#f7f8f7}
tbody tr th,tbody tr th p,tfoot tr th,tfoot tr th p{color:rgba(0,0,0,.8);font-weight:bold}
p.tableblock>code:only-child{background:none;padding:0}
p.tableblock{font-size:1em}
ol{margin-left:1.75em}
ul li ol{margin-left:1.5em}
dl dd{margin-left:1.125em}
dl dd:last-child,dl dd:last-child>:last-child{margin-bottom:0}
li p,ul dd,ol dd,.olist .olist,.ulist .ulist,.ulist .olist,.olist .ulist{margin-bottom:.625em}
ul.checklist,ul.none,ol.none,ul.no-bullet,ol.no-bullet,ol.unnumbered,ul.unstyled,ol.unstyled{list-style-type:none}
ul.no-bullet,ol.no-bullet,ol.unnumbered{margin-left:.625em}
ul.unstyled,ol.unstyled{margin-left:0}
li>p:empty:only-child::before{content:"";display:inline-block}
ul.checklist>li>p:first-child{margin-left:-1em}
ul.checklist>li>p:first-child>.fa-square-o:first-child,ul.checklist>li>p:first-child>.fa-check-square-o:first-child{width:1.25em;font-size:.8em;position:relative;bottom:.125em}
ul.checklist>li>p:first-child>input[type=checkbox]:first-child{margin-right:.25em}
ul.inline{display:flex;flex-flow:row wrap;list-style:none;margin:0 0 .625em -1.25em}
ul.inline>li{margin-left:1.25em}
.unstyled dl dt{font-weight:400;font-style:normal}
ol.arabic{list-style-type:decimal}
ol.decimal{list-style-type:decimal-leading-zero}
ol.loweralpha{list-style-type:lower-alpha}
ol.upperalpha{list-style-type:upper-alpha}
ol.lowerroman{list-style-type:lower-roman}
ol.upperroman{list-style-type:upper-roman}
ol.lowergreek{list-style-type:lower-greek}
.hdlist>table,.colist>table{border:0;background:none}
.hdlist>table>tbody>tr,.colist>table>tbody>tr{background:none}
td.hdlist1,td.hdlist2{vertical-align:top;padding:0 .625em}
td.hdlist1{font-weight:bold;padding-bottom:1.25em}
td.hdlist2{word-wrap:anywhere}
.literalblock+.colist,.listingblock+.colist{margin-top:-.5em}
.colist td:not([class]):first-child{padding:.4em .75em 0;line-height:1;vertical-align:top}
.colist td:not([class]):first-child img{max-width:none}
.colist td:not([class]):last-child{padding:.25em 0}
.thumb,.th{line-height:0;display:inline-block;border:4px solid #fff;box-shadow:0 0 0 1px #ddd}
.imageblock.left{margin:.25em .625em 1.25em 0}
.imageblock.right{margin:.25em 0 1.25em .625em}
.imageblock>.title{margin-bottom:0}
.imageblock.thumb,.imageblock.th{border-width:6px}
.imageblock.thumb>.title,.imageblock.th>.title{padding:0 .125em}
.image.left,.image.right{margin-top:.25em;margin-bottom:.25em;display:inline-block;line-height:0}
.image.left{margin-right:.625em}
.image.right{margin-left:.625em}
a.image{text-decoration:none;display:inline-block}
a.image object{pointer-events:none}
sup.footnote,sup.footnoteref{font-size:.875em;position:static;vertical-align:super}
sup.footnote a,sup.footnoteref a{text-decoration:none}
sup.footnote a:active,sup.footnoteref a:active,#footnotes .footnote a:first-of-type:active{text-decoration:underline}
#footnotes{padding-top:.75em;padding-bottom:.75em;margin-bottom:.625em}
#footnotes hr{width:20%;min-width:6.25em;margin:-.25em 0 .75em;border-width:1px 0 0}
#footnotes .footnote{padding:0 .375em 0 .225em;line-height:1.3334;font-size:.875em;margin-left:1.2em;margin-bottom:.2em}
#footnotes .footnote a:first-of-type{font-weight:bold;text-decoration:none;margin-left:-1.05em}
#footnotes .footnote:last-of-type{margin-bottom:0}
#content #footnotes{margin-top:-.625em;margin-bottom:0;padding:.75em 0}
div.unbreakable{page-break-inside:avoid}
.big{font-size:larger}
.small{font-size:smaller}
.underline{text-decoration:underline}
.overline{text-decoration:overline}
.line-through{text-decoration:line-through}
.aqua{color:#00bfbf}
.aqua-background{background:#00fafa}
.black{color:#000}
.black-background{background:#000}
.blue{color:#0000bf}
.blue-background{background:#0000fa}
.fuchsia{color:#bf00bf}
.fuchsia-background{background:#fa00fa}
.gray{color:#606060}
.gray-background{background:#7d7d7d}
.green{color:#006000}
.green-background{background:#007d00}
.lime{color:#00bf00}
.lime-background{background:#00fa00}
.maroon{color:#600000}
.maroon-background{background:#7d0000}
.navy{color:#000060}
.navy-background{background:#00007d}
.olive{color:#606000}
.olive-background{background:#7d7d00}
.purple{color:#600060}
.purple-background{background:#7d007d}
.red{color:#bf0000}
.red-background{background:#fa0000}
.silver{color:#909090}
.silver-background{background:#bcbcbc}
.teal{color:#006060}
.teal-background{background:#007d7d}
.white{color:#bfbfbf}
.white-background{background:#fafafa}
.yellow{color:#bfbf00}
.yellow-background{background:#fafa00}
span.icon>.fa{cursor:default}
a span.icon>.fa{cursor:inherit}
.admonitionblock td.icon [class^="fa icon-"]{font-size:2.5em;text-shadow:1px 1px 2px rgba(0,0,0,.5);cursor:default}
.admonitionblock td.icon .icon-note::before{content:"\f05a";color:#19407c}
.admonitionblock td.icon .icon-tip::before{content:"\f0eb";text-shadow:1px 1px 2px rgba(155,155,0,.8);color:#111}
.admonitionblock td.icon .icon-warning::before{content:"\f071";color:#bf6900}
.admonitionblock td.icon .icon-caution::before{content:"\f06d";color:#bf3400}
.admonitionblock td.icon .icon-important::before{content:"\f06a";color:#bf0000}
.conum[data-value]{display:inline-block;color:#fff!important;background:rgba(0,0,0,.8);border-radius:50%;text-align:center;font-size:.75em;width:1.67em;height:1.67em;line-height:1.67em;font-family:"Open Sans","DejaVu Sans",sans-serif;font-style:normal;font-weight:bold}
.conum[data-value] *{color:#fff!important}
.conum[data-value]+b{display:none}
.conum[data-value]::after{content:attr(data-value)}
pre .conum[data-value]{position:relative;top:-.125em}
b.conum *{color:inherit!important}
.conum:not([data-value]):empty{display:none}
dt,th.tableblock,td.content,div.footnote{text-rendering:optimizeLegibility}
h1,h2,p,td.content,span.alt,summary{letter-spacing:-.01em}
p strong,td.content strong,div.footnote strong{letter-spacing:-.005em}
p,blockquote,dt,td.content,td.hdlist1,span.alt,summary{font-size:1.0625rem}
p{margin-bottom:1.25rem}
.sidebarblock p,.sidebarblock dt,.sidebarblock td.content,p.tableblock{font-size:1em}
.exampleblock>.content{background:#fffef7;border-color:#e0e0dc;box-shadow:0 1px 4px #e0e0dc}
.print-only{display:none!important}
@page{margin:1.25cm .75cm}
@media print{*{box-shadow:none!important;text-shadow:none!important}
html{font-size:80%}
a{color:inherit!important;text-decoration:underline!important}
a.bare,a[href^="#"],a[href^="mailto:"]{text-decoration:none!important}
a[href^="http:"]:not(.bare)::after,a[href^="https:"]:not(.bare)::after{content:"(" attr(href) ")";display:inline-block;font-size:.875em;padding-left:.25em}
abbr[title]{border-bottom:1px dotted}
abbr[title]::after{content:" (" attr(title) ")"}
pre,blockquote,tr,img,object,svg{page-break-inside:avoid}
thead{display:table-header-group}
svg{max-width:100%}
p,blockquote,dt,td.content{font-size:1em;orphans:3;widows:3}
h2,h3,#toctitle,.sidebarblock>.content>.title{page-break-after:avoid}
#header,#content,#footnotes,#footer{max-width:none}
#toc,.sidebarblock,.exampleblock>.content{background:none!important}
#toc{border-bottom:1px solid #dddddf!important;padding-bottom:0!important}
body.book #header{text-align:center}
body.book #header>h1:first-child{border:0!important;margin:2.5em 0 1em}
body.book #header .details{border:0!important;display:block;padding:0!important}
body.book #header .details span:first-child{margin-left:0!important}
body.book #header .details br{display:block}
body.book #header .details br+span::before{content:none!important}
body.book #toc{border:0!important;text-align:left!important;padding:0!important;margin:0!important}
body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-break-before:always}
.listingblock code[data-lang]::before{display:block}
#footer{padding:0 .9375em}
.hide-on-print{display:none!important}
.print-only{display:block!important}
.hide-for-print{display:none!important}
.show-for-print{display:inherit!important}}
@media amzn-kf8,print{#header>h1:first-child{margin-top:1.25rem}
.sect1{padding:0!important}
.sect1+.sect1{border:0}
#footer{background:none}
#footer-text{color:rgba(0,0,0,.6);font-size:.9em}}
@media amzn-kf8{#header,#content,#footnotes,#footer{padding:0}}
</style>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css">
</head>
<body class="book">
<div id="header">
<h1>SDP Migration and Upgrade Guide</h1>
<div class="details">
<span id="author" class="author">Perforce Professional Services</span><br>
<span id="email" class="email"><a href="mailto:consulting-helix-core@perforce.com">consulting-helix-core@perforce.com</a></span><br>
<span id="revnumber">version v2024.1,</span>
<span id="revdate">2024-06-11</span>
</div>
<div id="toc" class="toc">
<div id="toctitle">Table of Contents</div>
<ul class="sectlevel1">
<li><a href="#_draft_notice">DRAFT NOTICE</a></li>
<li><a href="#_preface">Preface</a></li>
<li><a href="#_introduction">1. Introduction</a>
<ul class="sectlevel2">
<li><a href="#_optimal_helix_core_operating_environment">1.1. Optimal Helix Core Operating Environment</a>
<ul class="sectlevel3">
<li><a href="#_optimal_storage_and_nfs">1.1.1. Optimal Storage and NFS</a></li>
</ul>
</li>
<li><a href="#_motivation">1.2. Motivation</a>
<ul class="sectlevel3">
<li><a href="#_global_topology_upgrades">1.2.1. Global Topology Upgrades</a></li>
<li><a href="#_helix_remote_administration">1.2.2. Helix Remote Administration</a></li>
</ul>
</li>
<li><a href="#_migration_style_upgrades">1.3. Migration Style Upgrades</a></li>
<li><a href="#_big_blue_green_cutover">1.4. Big Blue Green Cutover</a></li>
</ul>
</li>
<li><a href="#_migration_planning">2. Migration Planning</a>
<ul class="sectlevel2">
<li><a href="#_build_a_cross_functional_migration_team">2.1. Build a Cross Functional Migration Team</a></li>
<li><a href="#_define_start_state">2.2. Define Start State</a></li>
<li><a href="#_take_inventory">2.3. Take Inventory</a></li>
<li><a href="#_define_end_state">2.4. Define End State</a></li>
<li><a href="#_define_project_scope">2.5. Define Project Scope</a></li>
<li><a href="#_plan_one_or_more_dry_runs">2.6. Plan One or more Dry Runs</a></li>
<li><a href="#_plan_orchestration_automation">2.7. Plan Orchestration Automation</a></li>
<li><a href="#_define_server_templates">2.8. Define Server Templates</a>
<ul class="sectlevel3">
<li><a href="#_enhanced_studio_pack">2.8.1. Enhanced Studio Pack</a></li>
</ul>
</li>
<li><a href="#_consider_client_upgrades">2.9. Consider Client Upgrades</a></li>
</ul>
</li>
<li><a href="#_migration_preparation">3. Migration Preparation</a>
<ul class="sectlevel2">
<li><a href="#_build_the_green_infrastructure">3.1. Build the Green Infrastructure.</a>
<ul class="sectlevel3">
<li><a href="#_green_infrastructure_setup_timing">3.1.1. Green Infrastructure Setup Timing</a></li>
</ul>
</li>
<li><a href="#_develop_test_plans">3.2. Develop Test Plans</a></li>
<li><a href="#_plan_for_rollback">3.3. Plan for Rollback</a></li>
</ul>
</li>
<li><a href="#_operational_guidance">Appendix A: Operational Guidance</a></li>
<li><a href="#_sample_cutover_procedure">4. Sample Cutover Procedure</a>
<ul class="sectlevel2">
<li><a href="#_one_week_before_cutover">4.1. One Week Before Cutover</a></li>
<li><a href="#_one_day_before_cutover">4.2. One Day Before Cutover</a></li>
<li><a href="#_maintenance_procedure_for_cutover">4.3. Maintenance Procedure for Cutover</a></li>
</ul>
</li>
<li><a href="#_draft_notice_2">Appendix B: DRAFT NOTICE</a></li>
</ul>
</div>
</div>
<div id="content">
<div class="sect1">
<h2 id="_draft_notice">DRAFT NOTICE</h2>
<div class="sectionbody">
<div class="admonitionblock warning">
<table>
<tr>
<td class="icon">
<i class="fa icon-warning" title="Warning"></i>
</td>
<td class="content">
This document is in DRAFT status and should not be relied on yet. It is a preview of a document to be completed in a future release.
</td>
</tr>
</table>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_preface">Preface</h2>
<div class="sectionbody">
<div class="paragraph">
<p>This document is useful in when migrating an existing Perforce Helix Core installation an the optimal operating environment from any starting condition. Whether the starting environment is a large enterprise or "garage band" scale, on-prem or on a public or private cloud, or on any operating system, this guide can help get to the optimal deployment environment for Perforce Helix Core.</p>
</div>
<div class="paragraph">
<p>This document focuses on software rather than hardware aspects of an optimal operating environment. While hardware is not discussed in detail, the migration and upgrade plans described in this document provide an opportunity to test and change out hardware.</p>
</div>
<div class="paragraph">
<p>This document does not discuss case sensitivity or case-conversions. Those are discussed in the <a href="SDP_Win2Linux_Guide.html">SDP Windows to Linux Migration Guide</a>.</p>
</div>
<div class="paragraph">
<p>This guide assumes familiarity with Perforce Helix Core and does not duplicate basic information in the Helix Core documentation.</p>
</div>
<div class="paragraph">
<p><strong>Please Give Us Feedback</strong></p>
</div>
<div class="paragraph">
<p>Perforce welcomes feedback. Please send any suggestions for improving this document to <a href="mailto:consulting-helix-core@perforce.com">consulting-helix-core@perforce.com</a>.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_introduction">1. Introduction</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="_optimal_helix_core_operating_environment">1.1. Optimal Helix Core Operating Environment</h3>
<div class="paragraph">
<p>Getting to an optimal operating environment for Perforce Helix Core first requires defining various aspects of optimal. Any definition of optimal should deliver on expectations of being reliable, robust, performant, and secure.</p>
</div>
<div class="paragraph">
<p>For our purposes in this document, optimal specifically means:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Physical layer (server machines, storage subsystems, etc.) is as desired.</p>
</li>
<li>
<p>Helix Core (P4D) version 2019.1+. The P4D 2013.3 and 2019.1 versions were major architectural overhauls requiring special upgrade procedures. Once the P4D version is 2019.1 or later, future upgrades are standardized.</p>
</li>
<li>
<p>SDP version 2020.1+. This is the SDP version from which SDP upgrades are automated.</p>
</li>
<li>
<p>All Helix server processes (p4d, p4p, p4broker) are operating on Linux servers, on a Linux major version with plenty of support life left in it. As of this writing, that would be RHEL/Rocky Linux 9, or Ubuntu 22.04. RHEL/Rocky Linux 8 and Ubuntu 20.04 are actively supported as well, but with somewhat less runway available. For EOL dates of various Linux distros, see:</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://access.redhat.com/product-life-cycles/">RHEL</a></p>
</li>
<li>
<p><a href="https://wiki.rockylinux.org/rocky/version/">Rocky</a></p>
</li>
<li>
<p><a href="https://ubuntu.com/about/release-cycle">Ubuntu</a></p>
</li>
<li>
<p><a href="https://www.suse.com/products/server/">SuSE</a></p>
</li>
<li>
<p><a href="https://endoflife.date">EOL dates for multiple distros</a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>In public cloud environments, an optimal Helix Core server be deployed instantly in an optimal environment by using the <a href="https://www.perforce.com/blog/vcs/perforce-enhanced-studio-pack">Enhanced Studio Pack (ESP)</a>, an offering by Perforce Software available on Amazon and Azure marketplaces.</p>
</div>
<div class="sect3">
<h4 id="_optimal_storage_and_nfs">1.1.1. Optimal Storage and NFS</h4>
<div class="paragraph">
<p>This document doesn’t focus on hardware or storage components of the definition of optimal for Helix Core. Using NFS isn’t part of the optimal definition. At small scale, the cost and complexity introduced by NFS may not be worth the various benefits. However, as the scale of data increases to and above tens of Terabytes, options involving more scalable filesystem solutions like NFS start making sense and may even be effectively required. NFS can be used in on-prem and public cloud environments.</p>
</div>
<div class="paragraph">
<p>ESP installations do not use NFS, and thus would require adjustment to handle very large Helix Core data sets.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_motivation">1.2. Motivation</h3>
<div class="sect3">
<h4 id="_global_topology_upgrades">1.2.1. Global Topology Upgrades</h4>
<div class="paragraph">
<p>This document is written to support projects commonly referred to as a <strong>global topology upgrade</strong>, sometimes part of an even larger <strong>infrastructure modernization</strong> effort. Such upgrades are commonly driven by desires to maintain performance, security, supportability, access to new product features, and in some cases to escape custom aspects of local infrastructure.</p>
</div>
<div class="paragraph">
<p>If <em>all</em> aspects of the current environment are already optimal as defined above, you don’t need this document. Instead, use the standard upgrade procedure documented in the <a href="https://swarm.workshop.perforce.com/view/guest/perforce_software/sdp/main/doc/SDP_Guide.Unix.html#_upgrades">Upgrades section of the SDP Guide</a>. So for example, if you it’s 2024 and you have SDP 2020.1 and P4D 2019.1, you can upgrade easily in place (unless you also want to change the major version of your operating system, or migrate to a different operating system).</p>
</div>
</div>
<div class="sect3">
<h4 id="_helix_remote_administration">1.2.2. Helix Remote Administration</h4>
<div class="paragraph">
<p>Another potential additional motivation is to get assistance for managing Helix Core servers, or perhaps to get out of that role entirely (perhaps due to departure of key personnel). If there is interest in turning over the keys to your Perforce Helix servers, consider <a href="https://www.perforce.com/support/consulting/helix-remote-admin">the Helix Remote Administration program (HRA)</a>.</p>
</div>
<div class="paragraph">
<p>To be eligible for HRA, customers must be on an optimal environment. Signing up for the program commonly entails a process referred to as "HRA Onboarding," which essentially means doing a Migration Style Upgrade to an optimal environment, as outlined in this document.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_migration_style_upgrades">1.3. Migration Style Upgrades</h3>
<div class="paragraph">
<p>This document focuses on the Migration-Style Upgrade strategy, as opposed to in situ (in place) upgrades. In situ upgrades are preferred when your deployment environment is already optimal, as defined above in <a href="#_optimal_helix_core_operating_environment">Section 1.1, “Optimal Helix Core Operating Environment”</a>. If <em>all</em> of the aspects are currently in the desired state, you don’t need this document. Instead, use the standard upgrade procedure documented in the <a href="https://swarm.workshop.perforce.com/view/guest/perforce_software/sdp/main/doc/SDP_Guide.Unix.html#_upgrades">Upgrades section of the SDP Guide</a>.</p>
</div>
<div class="paragraph">
<p>If the hardware, operating system, or P4D/SDP versions are not as desired, this guide is for you. A Migration-Style Upgrade is great when you need to make a <em>big change</em> in the least disruptive way possible. A key characteristic of a Migration-Style Upgrade is that your original server environment is largely left alone, with little or no change.</p>
</div>
<div class="paragraph">
<p>Typically the original environment remains available for some time after the upgrade, with the general idea that the old environment can eventually be decomissioned, archived, and/or simply deleted.</p>
</div>
</div>
<div class="sect2">
<h3 id="_big_blue_green_cutover">1.4. Big Blue Green Cutover</h3>
<div class="paragraph">
<p>In a Migration-Style Upgrade, a new set of server machines and supporting infrastructure (such as storage) is deployed. This set of servers is referred to as the Green servers or infrastructure. The current, live production equipment is referred to as the Blue servers or infrastructure.</p>
</div>
<div class="paragraph">
<p>In preparation for an Production Cutover, Helix Core data is brought into the Green environment. This is usually non-disruptive to users, who are operating on the Blue (current live production) environment.</p>
</div>
<div class="paragraph">
<p>The Green Environment operates for a time as a test environment, allowing opportunity to test various aspects of the new infrastructure before it becomes the production infrastructure. Depending on risks and needs, testing can be cursory or extensive, lasting days or months.</p>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
If your current method of operating Helix Core does not produce a regular <a href="https://www.perforce.com/manuals/p4sag/Content/P4SAG/backup-recovery-concepts.html">checkpoint</a>, a change to the Blue environment will be required to get at least some basic form of checkpoint process in place. This may involve disruptive operations that might need to be scheduled in a maintenance window before the Green environment can be setup initially.
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>A Big Blue Green Cutover (BBGC) is planned with the eventual goal of cutting over from the entire Blue infrastructure to the Green infrastructure in one single maintenance window. The <em>Big</em> in Big Blue Green Cutover indicates that a phased approach cutover is not an option.</p>
</div>
<div class="paragraph">
<p>In some types of projects, using a phased approach to migrations can mitigate risk. However, in this type of project, a phased cutover actually introduces more risk and complexity, because it requires operating bi-directionally across the Blue and Green infrastructures, which creates its own set of problems. In BBGC, there is a one-way, one-time migration from Blue to Green. After much preparation, ultimately a single Production Cutover completes the project.</p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_migration_planning">2. Migration Planning</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="_build_a_cross_functional_migration_team">2.1. Build a Cross Functional Migration Team</h3>
<div class="paragraph">
<p>As the Start and End States of the migration are defined, it should become clear what human resources and teams, both internal and external to your organization, are to be assembled. This may include Perforce Helix administrators, system administrators, storage architects, build engineers, some representative users (perhaps from different time zones), and Perforce consultants. Also include folks in your organization needed to implement things like DNS changes or load balancer redirection that may be needed as part of the eventual Production Cutover.</p>
</div>
<div class="paragraph">
<p>Migration-Style Upgrade projects involve personnel who are necessary or helpful to successful completion of the project, but whose primary job is something other than the migration. It may be helpful to discuss schedule and availability of various resources for the expected duration of the project. In particular, it is important to ensure key resources are available at critical points in the schedule, including standing up Dry Runs and the eventual cutover.</p>
</div>
<div class="paragraph">
<p>In addition to folks needed to drive the migration, additional resources may be needed for testing, especially for things that need a human eyeball. For example, if your BBGC involves a perfmerge with Helix Swarm, the validation that things "look right" after a Dry Run is best done by someone who uses that software regularly, and has a good sense for what "normal" looks or feels like. Other things, like archive file verification, can be verified mechanically.</p>
</div>
<div class="paragraph">
<p>Migration projects may benefit from having a project manager and/or an executive sponsor to ensure the migration project priorities are aligned with other organizational efforts.</p>
</div>
<div class="paragraph">
<p>Designate communication methods for the project. Start a new discussion thread specific to the migration project. This can be an email thread, a dedicated channel in Slack/Teams/Discord, etc. Include relevant stakeholders. Each Dry Run and eventually the Production Cutover should have its own channel.</p>
</div>
</div>
<div class="sect2">
<h3 id="_define_start_state">2.2. Define Start State</h3>
<div class="paragraph">
<p>The starting environment can pretty much anything:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Any P4D version going back to 1995.</p>
</li>
<li>
<p>P4D server machine(s) operating on Windows, UNIX, Mac OSX, Linux, or other platform.</p>
</li>
<li>
<p>Any legacy method of managing Helix Core:</p>
<div class="ulist">
<ul>
<li>
<p>An older version of SDP (prior to 2020.1).</p>
</li>
<li>
<p>Home-grown custom scripts, which may be simple or extensive.</p>
</li>
<li>
<p>Management scripts provided by a 3rd party such as ICManage.</p>
</li>
<li>
<p>The p4dctl service, possibly installed with the helix-p4d package installation.</p>
</li>
<li>
<p>Manual procedures.</p>
</li>
<li>
<p>No management whatsoever.</p>
</li>
</ul>
</div>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="_take_inventory">2.3. Take Inventory</h3>
<div class="paragraph">
<p>At the outset of a Migration Style Upgrade, take stock of everything that comprises the current Helix Core ecosystem. The inventory should be comprehensive of everything that is part of your infrastructure, regardless of whether you intend for it to be affected by the current upgrade project. In the simplest case, inventory might consist of single server machine with a single p4d commit server.</p>
</div>
<div class="paragraph">
<p>Some items to include in the inventory are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>All server machines involved in the ecosystem, including those running Helix Core software (such as p4d servers) as well as those not operating any Helix Core services but from which automation runs, such as build server farms.</p>
</li>
<li>
<p>All Helix Core software components, such as</p>
<div class="ulist">
<ul>
<li>
<p>Server products p4d/p4broker/p4p</p>
</li>
<li>
<p>p4broker p4p, Helix Swarm, P4DTG, Helix DAM, etc.</p>
</li>
</ul>
</div>
</li>
<li>
<p>Any customization done using Helix Core custom features:</p>
<div class="ulist">
<ul>
<li>
<p>Any custom Helix Core triggers. Triggers in Helix Core are a means of customizing and extending Helix Core behaviors.</p>
</li>
<li>
<p>Any custom Helix Core broker scripts.</p>
</li>
</ul>
</div>
</li>
<li>
<p>Any other custom automation.</p>
</li>
<li>
<p>Any systems integrations with 3rd party or home-grown systems, such as issue/bug trackers, task and agile project management tools, and reporting systems. If in doubt about whether a system is potentially affected by or involved in the upgrade, include it in the inventory for consideration.</p>
</li>
</ul>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
Any custom elements in the ecosystem, such as triggers, require special consideration when planning migrations. Often you want custom elements to survive the migration, though this may require specialized expertise. That may in turn require adding folks to the migration team who are or can get up to speed on the custom elements. In some cases, it may be worth considering whether there is a need for the custom elements to survive the migration.
</td>
</tr>
</table>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
If an ecosystem has production and non-production elements (such as a copy-of-production sandbox environment), those distinctions should be noted in the inventory. Such distinctions may effect whether the elements are in the scope of the effort, and when they are handled in the schedule.
</td>
</tr>
</table>
</div>
</div>
<div class="sect2">
<h3 id="_define_end_state">2.4. Define End State</h3>
<div class="paragraph">
<p>At a minimum, the desired <strong>End State</strong> of a Migration-Style Upgrade is as defined above in <a href="#_optimal_helix_core_operating_environment">Section 1.1, “Optimal Helix Core Operating Environment”</a>. In addition to those aspects, you may choose to add other aspects to your desired <strong>End State</strong>, depending on the goals of your migration. In the mindset of "While we’re at it," some common examples things added to the End State definition are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><strong>Authentication and Single-Sign On (SSO)</strong>: Enable a Single Sign On (SSO) solution using the <a href="https://github.com/perforce/helix-authentication-service/">Helix Authentication Service</a>.</p>
</li>
<li>
<p><strong>Monitoring</strong>: Deploy monitoring with <a href="https://github.com/perforce/p4prometheus">P4Prometheus</a>.</p>
</li>
<li>
<p><strong>Helix Core Data Consolidation (Perfmerge)</strong>: In some cases, one driving goal of the BBGC is to combine formerly silo’d/independent data sets into a single, larger Helix Core data set, with files and history from both server instances. This is often done to enable more collaborative workflows across teams. This complex process can be achieved by doing a <a href="https://ftp.perforce.com/perforce/tools/perfmerge/perfmerge.html">perfmerge</a>, essentially a neurosurgery operation on a set of Helix Core data sets. The complexity and risk of a perfmerge can be mitigated with a Big Blue Green Cutover, which provides the needed infrastructure to do the extensive testing required.</p>
</li>
<li>
<p><strong>Hardware Core Hardware Consolidation</strong>: If the intent is to redeploy Helix Core onto a smaller number of server machines to reduce the hardware footprint of Helix Core, the changes to which data sets operate from which machines can be handled as part of the migration plan. Hardware consolidations are vastly simpler than a data consolidation (ala perfmerge), but still require testing.</p>
</li>
<li>
<p><strong>Hardware Core Topology Expansion</strong>: In some cases a goal of a BBGC is to expand the topology. Merely adding topology components to an existing Helix Core ecosystem, such as adding an edge server or Helix Swarm, does not by itself warrant a BBGC. However, expanding the topology is a common "add on" to a migration plan when the need for doing a BBGC is already established, e.g. due to OS upgrades.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Even if you are near the definition of optimal, say on Rocky Linux 8 but otherwise modern, the Migration-Style Upgrade is the preferred method of getting to the modern topology (on Rocky 9). (If your environment is optimal in all other respects and <em>only</em> the P4D version is older, than you might consider an in situ upgrade). Other options are not discussed in this document, because the Migration-Style Upgrade has significant benefits that often make it preferable even when other options are possible.</p>
</div>
<div class="paragraph">
<p>The specific starting environment will impact migration options and procedures, as will be called out in this document.</p>
</div>
<div class="paragraph">
<p>Some <em>big changes</em> that call for a Migration-Style Upgrade include:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Windows to Linux migration. Those are discussed in the <a href="SDP_Win2Linux_Guide.html">SDP Windows to Linux Migration Guide</a>.</p>
</li>
<li>
<p>Upgrade of a major operating system version, e.g. CentOS 7 → Rocky 8, RHEL 8 → RHEL 9, Ubuntu 20.04 → Ubuntu 22.04, or even a Linux family change, e.g. CentOS 7 → Ubuntu 22.04.</p>
</li>
<li>
<p>Upgrade of SDP from a version prior to 2020.1, where the <a href="https://swarm.workshop.perforce.com/view/guest/perforce_software/sdp/main/doc/SDP_Legacy_Upgrades.Unix.html">SDP Legacy Upgrades</a> applies, a well-documented but manual upgrade procedure. That document is for in situ upgrades of SDP — this document avoids the in situ procedure entirely by using a Migration-Style Upgrade.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The primary scenario this document is focused on is a migration to a cloud provider, though only minor adaptations are needed if going to an on-premises environment. We lean toward a cloud environments for documentation purposes, because we can assume more, and define more, with a cloud as a target. If your target environment is on-prem, there is a greater likelihood of local polices, practices, and perhaps even different teams within your organization that may be involved.</p>
</div>
</div>
<div class="sect2">
<h3 id="_define_project_scope">2.5. Define Project Scope</h3>
<div class="paragraph">
<p>Scoping the migration project starts with the inventory. Decide on a per-item basis whether that item in the inventory is to be included in the migration, left alone, or perhaps decommissioned. Pay attention to whether any software components are obsolete or have been sunset.</p>
</div>
<div class="paragraph">
<p>For example, if P4Web (a legacy server that was sunset in 2012) is part of your inventory in the Blue environment, you’ll want to plan to replace it with its successor, Helix Swarm. If P4Web was used only by humans, this could be as simple as adding Helix Swarm to the Green infrastructure. If instead P4Web was used as part of some external integration, then the migration plan may include a subproject to perhaps replace P4Web URLs or API calls with Helix Swarm equivalents.</p>
</div>
<div class="paragraph">
<p>In addition to scoping based on inventory, the scope must take into account considerations mentioned in the End State, such as adding HA/DR, changing the authentication mechanism, doing a perfmerge, etc.</p>
</div>
</div>
<div class="sect2">
<h3 id="_plan_one_or_more_dry_runs">2.6. Plan One or more Dry Runs</h3>
<div class="paragraph">
<p>The overall project migration plan must account for at least one, and potentially several Dry Runs. A Dry Run exercises parts of the cutover procedure that bring the Green environment to life, but leaves out those parts of the cutover procedure that would bring all users over to the Green environment. Once a Dry Run is completed, the Green environment is online and available for testing.</p>
</div>
<div class="paragraph">
<p>Dry runs are generally tagged with an identifier, usually just a number, e.g. Dry Run 1, Dry Run 2, etc. Dry runs can be formal or informal. Informal dry runs are those available only to a subset of the Migration Team, typically the Perforce admins or someone responsible for crafting and testing the Cutover Procedure. For formal dry runs, the availability of the Green environment is announced to the Migration Team. This is to simplify testing and coordination of testing resources in the environment. For example, representatives from the user community on the Migration Team can focus their test efforts knowing that the Green environment is expected to be ready to use.</p>
</div>
<div class="paragraph">
<p>Typically, only one Dry Run can be operational at a time, as Dry Runs use the Green environment.</p>
</div>
<div class="paragraph">
<p>In the simple case, there will be only one Dry Run, and thus it will be formal. Multiple Dry Runs should be considered if any of the following are true:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>If the Green infrastructure will be delivered in phases, earlier Dry Runs can test those Green components that are available sooner.</p>
</li>
<li>
<p>Orchestration automation needs to be tested. One or more Dry Runs (often informal ones) may be dedicated to the purpose of iterating on the orchestration automation.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>In addition to planned Dry Runs, sometimes learnings from one Dry Run necessitate additional Dry Runs that may not have been planned at the beginning. Expect the unexpected when it comes to Dry Run planning.</p>
</div>
</div>
<div class="sect2">
<h3 id="_plan_orchestration_automation">2.7. Plan Orchestration Automation</h3>
<div class="paragraph">
<p>BBGC migrations often involve automation to orchestrate some steps in the cutover procedure. Done correctly, automation of these "run once" procedures can have significant benefits:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Shortening the duration of the production cutover procedure by removing human lag between any long-running operations in the cutover procedure, such as operations involving checkpoints.</p>
</li>
<li>
<p>Improving the auditability of key aspects of the process by capturing exact commands executed and their outputs, to support diagnostics in event of a failure of some kind. Hopefully issues are discovered and resolved during the dry runs, but having a clear audit trail can be especially valuable in the production cutover.</p>
</li>
<li>
<p>Improving visibility and transparency of the actual commands executed, which can have training value and allow for collaboration, team cross-training, and peer oversight. Seeing actual commands to be executed in code is a more precise way to show the outcome of technical decisions made during migration planning than, say verbal explanation or even documentation. Documentation can quickly become out of date, while code is assured to be current if the procedure relies on execution of the code.</p>
</li>
<li>
<p>Reducing opportunities for human error. Often custover procedures have a sequences of operations that must be executed precisely and in a precise order, the kind of thing best done with automation.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Orchestration automation is not necessary for all BBGC migrations. The more complex the migration, the greater the benefit of using automation. Generally a cutover strategy will start with high-level steps, and then evolve to greater detail. During the planning process, it can be decided where automation adds the most value. The automation may be aware of the Dry Run number, and depending on the steps to be automated, may need to be aware of the distinction between a Dry Run and the Production Cutover.</p>
</div>
<div class="paragraph">
<p>Depending on the scope of the orchestration automation, it may require awareness of the distinction between a dry run and the Production Cutover.</p>
</div>
<div class="paragraph">
<p>When planning the scope of what to automate in the context of a BBGC, be flexible. While there may be multiple Dry Runs, ultimately, the Production Cutover will be done only once. This gives some flexibility in determining the scope of what to automate.</p>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
Don’t be too ambitious with the scope of automation, especially considering the schedule goals of the project and the fact that a BBGC is a "one shot" deal. Automation can reduce the cycle time of Dry Run iterations and ultimately the duration of the Production Cutover procedure, along with other automation benefits as noted above. However, Automation is software development and needs proper testing and iteration. If you’re automating a repetitive process, automate everything that can be automated, but if you’re automating a one-time process, be judicious.
</td>
</tr>
</table>
</div>
</div>
<div class="sect2">
<h3 id="_define_server_templates">2.8. Define Server Templates</h3>
<div class="paragraph">
<p>It is a good idea to define templates for server machines in some form. You may require multiple templates depending on the classes of machine you will operate in the Green environment. You will certainly have a Helix Core (p4d) server. You may have a separate template for Helix Proxy server, and another for a Helix Swarm server, etc. A review the inventory of server machines can guide the list of server templates needed.</p>
</div>
<div class="paragraph">
<p>Some common forms of defining a server template are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>A written language description (perhaps on a wiki page) that defines key characteristics such as server hardware specifics (RAM/CPU), storage system details, operating system, and the like.</p>
</li>
<li>
<p>A "golden image," a labeled virtual image in your preferred virtualization infrastructure.</p>
</li>
<li>
<p>A script of some kind that converts a base operating installation into one suitable for Helix Core.</p>
</li>
<li>
<p><a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html">AWS CloudFormation Templates</a>.</p>
</li>
<li>
<p><a href="https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/overview">Azure Resource Manager Templates</a>.</p>
</li>
<li>
<p><a href="https://developer.hashicorp.com/terraform/intro">Terraform</a>.</p>
</li>
<li>
<p>The Enhanced Studio Pack (ESP) from Perforce. See <a href="#_enhanced_studio_pack">Section 2.8.1, “Enhanced Studio Pack”</a>.</p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="_enhanced_studio_pack">2.8.1. Enhanced Studio Pack</h4>
<div class="paragraph">
<p>The <a href="https://www.perforce.com/blog/vcs/perforce-enhanced-studio-pack">Enhanced Studio Pack (ESP)</a> should be considered if the target environment is Amazon or Azure. ESP is not available for on-prem, and not presently available for other cloud environments. Even in cases where ESP is available, it may not be the best choice. Some cases where ESP is not optimal or would need adjustment afterward include:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>You have a corporate policy that dictates machines must be launched from internally produced baseline virtual machines images. ESP is essentially a set of machine images, but of course not be based on any customer’s unique base images. (In some cases, an exception can be granted because ESP is reasonably well secured, using <a href="https://www.perforce.com/press-releases/support-rocky-linux-and-centos-alternatives">Security Enhanced Rocky Linux provided by Perforce OpenLogic</a>).</p>
</li>
<li>
<p>You plan to use advanced storage solutions like NFS.</p>
</li>
<li>
<p>Your <code>perforce</code> operating system user is defined in LDAP rather than being local. (Note: We recommend using local accounts when you can for optimal reliability, but using an LDAP or NIS account is required when using NFS to ensure numeric user IDs align across a fleet of server machines).</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_consider_client_upgrades">2.9. Consider Client Upgrades</h3>
<div class="paragraph">
<p>Generally speaking, for a global topology upgrade, it is a good idea to plan to upgrade Helix Core client software as well as server components.</p>
</div>
<div class="paragraph">
<p>Helix Core has a powerful feature that allows client and server software to be upgraded independently, so that upgrading the server does force an upgrade of all clients at the same time, and clients can upgrade ahead of the server version as well.</p>
</div>
<div class="paragraph">
<p>However, the greater the disparity between client and server versions, the greater the risk of issues due to version skew between clients and servers. Further, new clients are going to have the latest security features. Lastly, certain product features and security benefits require clients to be upgraded.</p>
</div>
<div class="paragraph">
<p>Upgrading clients should be considered during a Migration Style Upgrade. Typically Migration Style Upgrades are done infrequently compared to in situ upgrades, perhaps once every 3 to 6 years, and thus a good time to address client/server version skew.</p>
</div>
<div class="paragraph">
<p>The following should be considered:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>The <code>p4</code> command line client binary. Updating this may involve notifying users how to download it themselves, or may involve an admin updating the binary on a system used by others.</p>
</li>
<li>
<p>The P4V GUI client, which may be installed by individual users or provided by admins to users.</p>
</li>
<li>
<p>Any software built on any of the Helix Core APIs, such as the C++ API or any of the APIs derived from the C++ API such as P4Perl, P4Python, etc. This may include 3rd party tools. Such software will require recompiling with the new API version, and may possible require code changes and associated testing.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>If your environment has custom automation built with the C++ or derived APIs, updating such things will require specialized expertise. If the automation is from a third party, you may need to explore options for getting updated versions from the vendor to match the new Helix Core version, or information about whether an upgrade is needed.</p>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
If updating clients becomes a complex endeavour, the risk calculus becomes a bit complex. Changing more complex things at once can increase risk, but so can allowing too much version skew. Think of excessive version skew, say 3+ years, as a form of technical debt: Sooner or later you’ll need to make a payment.
</td>
</tr>
</table>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
It you don’t have a good sense for what clients are connecting your Helix Core server, some server log analysis can work wonders. There are various approaches, but the gist is that you can scan a series of p4d server logs covering some span of time (a day, a week, or more), and determine what client programs are connecting to your Helix Core servers over that timeframe.
</td>
</tr>
</table>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_migration_preparation">3. Migration Preparation</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="_build_the_green_infrastructure">3.1. Build the Green Infrastructure.</h3>
<div class="paragraph">
<p>With the End State in mind, build out the Green infrastructure, including all server machines needed, deploying any new storage systems, etc.</p>
</div>
<div class="sect3">
<h4 id="_green_infrastructure_setup_timing">3.1.1. Green Infrastructure Setup Timing</h4>
<div class="paragraph">
<p>You might create the entire green infrastructure at once, or build it out in phases. The longer the infrastructure is available, the more testing can be done prior to it going live — this applies to any individual component of the infrastructure, as well as to the Green infrastructure as a whole. Considerations for phased vs. all at once tend to be schedule and cost. If infrastructure components aren’t ready when needed for dry runs, that can delay the schedule. If they’re available long before they’re needed, that might incur additional costs (especially in cloud environments).</p>
</div>
<div class="paragraph">
<p>If resources can be spun up quickly from templates, it is OK to defer instantiation of those resources until shortly before they’re needed. If spinning up services requires lengthy budget, ordering, acquisition, security review, and/or configuration processes, then spinning up resources sooner can avoid lengthy project delays.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="_develop_test_plans">3.2. Develop Test Plans</h3>
<div class="paragraph">
<p>Two separate test plans are needed for a BBGC. The first test plan, the Dry Run Test Plan, is to be executed during dry run(s), before the Green environment becomes the new production environment. These tests are not tightly constrained by time, and thus allow for anything that needs testing on the new infrastructure. These can be potentially comprehensive, intended to test both the migration process and the new infrastructure.</p>
</div>
<div class="paragraph">
<p>The second test plan, the Cutover Test Plan, is to be executed during the maintenance window for the cutover. Because it occurs during a period of service outage, the Cutover Test Plan is naturally constrained in duration. It is mostly a subset the Dry Run Test Plan, but with some subtle differences.</p>
</div>
<div class="paragraph">
<p>Some things to include in test plans are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><strong>Access Testing</strong>: The server machines in the Green infrastructure need to be accessible from the same end points as the corresponding servers in the Blue infrastructure (unless there are intended changes in access in the End State).</p>
</li>
<li>
<p><strong>Performance Testing</strong>: Often a BBGC involves deployment of new hardware, new server machines, new storage or networking infrastructure, etc. It is a good idea to include performance testing by doing sea trials on the new machines, putting them under some load. Simple things like NFS mount option misconfigurations will not show up with functional testing, and may go unnoticed unless performance testing is done.</p>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="_plan_for_rollback">3.3. Plan for Rollback</h3>
<div class="paragraph">
<p>One of the compelling benefits of a BBGC is that a rollback can be relatively straightforward, since the Blue infrastructure is not affected by the cutover procedure in any way that would interfere with the ability to rollback.</p>
</div>
<div class="paragraph">
<p>That said, planning must account for what the rollback options are, and in particular determine the point after the rollback is no longer an option. Somewhere in the Cutover Procedure, sometime after the Cutover Test Plan is executed, a formal "Go/No Go" decision is made. After that point, a "Fix and roll forward" approach is used.</p>
</div>
<div class="paragraph">
<p>At any point during the execution of the Production Cutover, things may not go according to plan. In the event of a mishap of any kind, decisions must be made about whether to attempt a rollback and how to execute. Planning can and should go into how to prepare for a rollback, and even some "war gaming" thought exercises. However, there is a limit to planning for a rollback, because a rollback is inherently a reactive operation. A rollback is executed in response to some kind of external cause. Actual rollback procedures require a detailed understanding of any potential desire to rollback. After understanding the business and/or technical reasons for a rollback, a plan should be crafted for the specific situation.</p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_operational_guidance">Appendix A: Operational Guidance</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Some basic tips for operating during Dry Runs and the Production Cutover:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Backup Everything Version</p>
</li>
<li>
<p>Capture commands executed and their outputs faithfully.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_sample_cutover_procedure">4. Sample Cutover Procedure</h2>
<div class="sectionbody">
<div class="paragraph">
<p>The following is a SAMPLE cutover procedure. DO NOT be tempted to use this without adaptation to your environment. Cutover procedures need to take into account the specifics of your organization’s planned migration and the Start and End states.</p>
</div>
<div class="paragraph">
<p>The plan must provide high-level steps for the cutover operation. In some cases, these steps are supplemented with great detail, including actual commands to execute, possibily also including diagnostic commands with expected outputs captured during dry runs and later captured for the Production Cutover. Some plans also include expected duration of each step, with projected start and end times, benefitting from experience executing the dry run(s).</p>
</div>
<div class="sect2">
<h3 id="_one_week_before_cutover">4.1. One Week Before Cutover</h3>
<div class="paragraph">
<p>The following steps are to be executed approximately one week before the Production Cutover. These steps are non-disruptive to the current production (Blue) environment:</p>
</div>
<div class="paragraph">
<p>STEP 1. Make sure the commit server and all replica and edge servers have healthy checkpoints and a healthy offline_db.</p>
</div>
<div class="paragraph">
<p>STEP 2. Run the SDP Health Check to verify: <a href="https://swarm.workshop.perforce.com/view/guest/perforce_software/sdp/main/doc/ReleaseNotes.html#_sdp_health_checks" class="bare">https://swarm.workshop.perforce.com/view/guest/perforce_software/sdp/main/doc/ReleaseNotes.html#_sdp_health_checks</a></p>
</div>
<div class="paragraph">
<p>If there are any issues, there is time to address them before the maintenance window for the cutover.</p>
</div>
</div>
<div class="sect2">
<h3 id="_one_day_before_cutover">4.2. One Day Before Cutover</h3>
<div class="paragraph">
<p>The following steps are to be executed approximately one day before the Production cutover. These steps are non-disruptive to the current production (Blue) environment:</p>
</div>
<div class="paragraph">
<p>STEP 1. Make sure the commit server and all replica and edge servers have healthy checkpoints and a healthy offline_db.</p>
</div>
<div class="paragraph">
<p>STEP 2. Run the SDP Health Check to verify: <a href="https://swarm.workshop.perforce.com/view/guest/perforce_software/sdp/main/doc/ReleaseNotes.html#_sdp_health_checks" class="bare">https://swarm.workshop.perforce.com/view/guest/perforce_software/sdp/main/doc/ReleaseNotes.html#_sdp_health_checks</a></p>
</div>
<div class="paragraph">
<p>Because these steps were executed about a week previously, one would hope the status remains the same. If there are any issues, determine if they can be quickly addressed before the maintenance window, or if the maintenance window needs to be rescheduled.</p>
</div>
</div>
<div class="sect2">
<h3 id="_maintenance_procedure_for_cutover">4.3. Maintenance Procedure for Cutover</h3>
<div class="paragraph">
<p>The following describes the overall maintenance window procedure for the Big Blue Green Cutover. Upon successful completion of this procedure, the Green environment will become the new Production Environment. The old Blue environment will remain online for no less than 30 days before being decomissioned.</p>
</div>
<div class="paragraph">
<p>All commands are to be operated as the <code>perforce</code> OS user.</p>
</div>
<div class="paragraph">
<p>STEP 1. Create Upgrade Notes File</p>
</div>
<div class="paragraph">
<p>First, create a directory on the Green commit server for storing, <code>~perforce/CutoverNotes/Prod-2024.1</code>, to contain any text files that may be useful to reference later. (For Dry Runs, use something like <code>~/CutoverNotes/DryRun4-2024.1</code>). If necessary/useful, create the same folder on other server machines in the Green or Blue infrastructure.</p>
</div>
<div class="paragraph">
<p>STEP 2. Disable Alerts.</p>
</div>
<div class="paragraph">
<p>Disable all alerts for the environment to avoid unnecessary alerts during the maintenance window.</p>
</div>
<div class="paragraph">
<p>Consider whatever mechanisms are used Prometheus/Victoria Metrics, P4Prometheus, Grafana, DataDog, NewRelic, etc.</p>
</div>
<div class="paragraph">
<p>STEP 3. Disable Crontabs.</p>
</div>
<div class="paragraph">
<p>Disable crontabs on all server machines in both the Blue and Green infrastructure.</p>
</div>
<div class="paragraph">
<p>Save current crontab on each host with the following:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cd # Go to to the home directory.
crontab -l
crontab -l > crontab.$USER.$(hostname -s)</pre>
</div>
</div>
<div class="paragraph">
<p>Then, clear the crontab on each machine with:</p>
</div>
<div class="paragraph">
<p>crontab -r</p>
</div>
<div class="paragraph">
<p>STEP 4. Lockout Users.</p>
</div>
<div class="paragraph">
<p>Lockout users with a combination of methods described in 3A, 3B, 3C, and 3D. The general gist is to ensure no users can access the Blue environment during the maintenance window, so that there is zero chance of data entered in the Blue environment not being carried over into the Green environment during the Production Cutover. (Upon successful completion, the Blue infrastructure remains offline and awaits decomission, removal, or repurposing of the infrastructure).</p>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
Do Not Rely on communications to users to guarantee that no users are using the Blue systems during this kind of maintenance. Craft operational procedures that provide a guarantee of no data loss even if users don’t receive or read/remember details of the maintenance window schedule.
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>STEP 4A. Lockout Users with Protections</p>
</div>
<div class="paragraph">
<p>Save the current Protections table on the commit server:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cd
p4 protect -o | grep -v ^# | grep -v ^Update: > ~perforce/CutoverNotes/Prod-2024.1/protect.before.p4s
cat ~perforce/CutoverNotes/Prod-2024.1/protect.before.p4s # <-- Make sure it looks right.</pre>
</div>
</div>
<div class="paragraph">
<p>Use a prepared, trimmed Protections "Lockdown" table that allows only the <code>perforce</code> super user.</p>
</div>
<div class="paragraph">
<p>Load it like so, as perforce on the commit server:</p>
</div>
<div class="paragraph">
<p>p4 protect -i < $P4TMP/protect_locked_down.p4s
p4 protect -o</p>
</div>
<div class="paragraph">
<p>STEP 4B. User Lockout with Brokers</p>
</div>
<div class="paragraph">
<p>EDITME</p>
</div>
<div class="paragraph">
<p>STEP 4C. User Lockout with Firewalls</p>
</div>
<div class="paragraph">
<p>EDITME</p>
</div>
<div class="paragraph">
<p>STEP 4D. User Lockout with Private Port</p>
</div>
<div class="paragraph">
<p>THe gist of this approach is to operate services on a port other than the one used for normal operation. For example, if p4d normally runs on port 1666, run it on port 1777 during maintenane.</p>
</div>
<div class="paragraph">
<p>EDITME</p>
</div>
<div class="paragraph">
<p>STEP 5. Prepare for Sanity Checks</p>
</div>
<div class="paragraph">
<p>Capture BEFORE information.</p>
</div>
<div class="paragraph">
<p>Capture the outpout of some basic commands before the migration that will provide a sanity check when compared with the output of the same commands after the migration. Some commands to run:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>p4sanity_check.sh</code></p>
</li>
<li>
<p>On the commit server: <code>p4 clients -a|wc -l</code></p>
</li>
<li>
<p>On each edge server: <code>p4 clients | wc -l</code></p>
</li>
</ul>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
During dry runs, capture precise details of commands to run. Sometimes the output of commands changes across p4d versions. For example, <code>p4 clients|wc -l</code> run on a commit server may yield different results in a commit/edge topology running on p4d 2018.2 then it shows when running on 2023.1. Test the commands you plan to run for both BEFORE and AFTER scenarios in dry runs.
</td>
</tr>
</table>
</div>
<div class="admonitionblock tip">
<table>
<tr>
<td class="icon">
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
Write a single script to capture the bits you’d like to have captured for BEFORE and AFTER comparison.
</td>
</tr>
</table>
</div>
<div class="paragraph">
<p>STEP 6. Stop All Services</p>
</div>
<div class="paragraph">
<p>Stop all p4d, p4broker, and p4p services in both the Blue and Green infrastructure.</p>
</div>
<div class="paragraph">
<p>The Protections change will block most new traffic from coming in, and stopping the services will drop any active connections, as is needed when starting a maintenance window.</p>
</div>
<div class="paragraph">
<p>STEP 7. Get Blue Checkpoints</p>
</div>
<div class="paragraph">
<p>Select the set of Blue servers to get checkpoints from. Start with the commit server, and then add any server that requires a separate checkpoint (ignoring any Blue servers that are to be decomissioned as part of the cutover project). The commit server requires a checkpoint, as does any server that has a data set that differs from the the commit. That includes edge servers and any filtered replicas, but excludes standby (as they are unfiltered by definition). Any edge server whose workspaces are expendable, as may be the case with Build Edges (used only by automation), can be excluded from this list.</p>
</div>
<div class="paragraph">
<p>Start the Blue p4d servers that require checkpoints (and only the p4d servers; no brokers or proxies).</p>
</div>
<div class="paragraph">
<p>From the commit server, use p4 commands to access and request checkpoints from all others servers that need checkpoints with commands like these:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>p4 -p ssl:EdgeServer1:1666 trust -y
p4 -p ssl:EdgeServer1:1666 login -a < $SDP_ADMIN_PASSWORD_FILE
p4 -p ssl:EdgeServer1:1666 pull -lj
p4 -p ssl:EdgeServer1:1666 pull -ls
p4 -p ssl:EdgeServer1:1666 pull -ls</pre>
</div>
</div>
<div class="paragraph">
<p>The get the checkpoints going with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>p4 admin journal</pre>
</div>
</div>
<div class="paragraph">
<p>When those checkpoints are done, stop all services.</p>
</div>
<div class="paragraph">
<p>STEP 8: Stop All Services</p>
</div>
<div class="paragraph">
<p>EDITME</p>
</div>
<div class="paragraph">
<p>STEP 9: Copy checkpoints from all Blue servers to corresponding Green servers.</p>
</div>
<div class="paragraph">
<p>For edge servers, the Green servers will exist near the corresponding Blue servers, so checkpoint copies will be "local-ish."</p>
</div>
<div class="paragraph">
<p>EDITME</p>
</div>
<div class="paragraph">
<p>STEP 10: Load Checkpoitns on Green Servers.</p>
</div>
<div class="paragraph">
<p>The SDP <code>load_checkpoint.sh</code> script will load checkpionts, upgrade data sets, and start services.</p>
</div>
<div class="paragraph">
<p>EDITME</p>
</div>
<div class="paragraph">
<p>STEP 11: Execute Sanity Tests in Green Environment.</p>
</div>
<div class="paragraph">
<p>Exectue the Cutover Test Plan. The cutover plan will start with tests that can be done by administrators, such as archive verification of recent submits, before getting others to spend time doing more detailed testing.</p>
</div>
<div class="paragraph">
<p>Compare output of comamnds captured earlier.</p>
</div>
<div class="paragraph">
<p>Note that the Test Plan may require incremental opening of the Protections table to allow testing in the Green environment.</p>
</div>
<div class="paragraph">
<p>STEP 12: Decide: Go/No Go.</p>
</div>
<div class="paragraph">
<p>Decide to go forward, or else rollback if needed. The specific rollback steps depend on nature and time of failure.</p>
</div>
<div class="paragraph">
<p>STEP 12A: Go!</p>
</div>
<div class="paragraph">
<p>Continue with the plan.</p>
</div>
<div class="paragraph">
<p>STEP 12B: Abort!</p>
</div>
<div class="paragraph">
<p>Enable crontabs and alerting in the Blue Environment. Schedule a future date for another attempt.</p>
</div>
<div class="paragraph">
<p>Plan a retrospective to learn what happened in detail, and how to ensure things go more smoothly next time around.</p>
</div>
<div class="paragraph">
<p>STEP 13. Start all Green services.</p>
</div>
<div class="paragraph">
<p>STEP 14. Direct Traffic to the Green Environment.</p>
</div>
<div class="paragraph">
<p>Redirect user traffic from the the Blue to the Green envirionment. This may involve a mix of DNS changes, load balancer target changes, changes ((hacks?) to <code>/etc/hosts</code> files, NIS changes, or even giving out new IP addresses to end users (to be avoided if at all possible).</p>
</div>
<div class="paragraph">
<p>STEP 15. Restore User Access</p>
</div>
<div class="paragraph">
<p>Restore the normal Protections table:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cd
p4 protect -i < ~perforce/CutoverNotes/Prod-2024.1/protect.before.p4s</pre>
</div>
</div>
<div class="paragraph">
<p>Depending on what actions were taken to lockout users, make whatever changes in the Green environment are needed to let folks in the front door.</p>
</div>
<div class="paragraph">
<p>STEP 16. Restore Crontabs</p>
</div>
<div class="paragraph">
<p>Restore crontabs on Green servers (only).</p>
</div>
<div class="paragraph">
<p>EDITME</p>
</div>
<div class="paragraph">
<p>STEP 17. Celebrate</p>
</div>
<div class="paragraph">
<p>The maintenance is complete. Enjoy a beverge, have a party, or do whatever you do for celebration in your corporate culture.</p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_draft_notice_2">Appendix B: DRAFT NOTICE</h2>
<div class="sectionbody">
<div class="admonitionblock warning">
<table>
<tr>
<td class="icon">
<i class="fa icon-warning" title="Warning"></i>
</td>
<td class="content">
This document is in DRAFT status and should not be relied on yet. It is a preview of a document to be completed in a future release.
</td>
</tr>
</table>
</div>
</div>
</div>
</div>
<div id="footer">
<div id="footer-text">
Version v2024.1<br>
Last updated 2024-06-11 21:56:47 -0400
</div>
</div>
</body>
</html> | # | Change | User | Description | Committed | |
|---|---|---|---|---|---|
| #7 | 30385 | C. Thomas Tyler | Regnerated docs for release. | ||
| #6 | 30294 | C. Thomas Tyler | Updated docs for release. | ||
| #5 | 30177 | C. Thomas Tyler | SDP_Win2Linux_Guide... | ||
| #4 | 30172 | C. Thomas Tyler | DRAFT doc, work in progress. | ||
| #3 | 30152 | C. Thomas Tyler | Documentaion work-in-progress. | ||
| #2 | 30105 | C. Thomas Tyler | WIP. | ||
| #1 | 30102 | C. Thomas Tyler |
Added early DRAFT docs for global topology upgrades. Not yet ready for review. |