# SDP Installation Configuration #============================================================================== # #------------------------------------------------------------------------------ # Storage Mount Points # # See the SDP Guide for optimal storage volume layout information. These # values can reference the absolute paths to storage volume mount points. # Alternately, for demo-grade installations, they may be simple directories. # # If they do not exist, they will be created during installation. # # DB1 = P4ROOT metadata volume during install. # # DB2 = offline_db metadata volume during install. # # Note: Typically DB1 and DB2 are set to the same value (volume), but can also # be set to '/hxmetadata1' and '/hxmetadata2' respectively (if these are validly # mounted, and separate, filesystems). Note that after installation, certain # scripts and/or recovery procedures can swap DB1 and DB2, such that P4ROOT # could be on DB2 and offline_db on DB1. # # DD = Volume for storing versioned files, metadata checkpoints, and # numbered journal files. This contains all critical assets that # must be backed up. # # CD = Optional volume for storing metadata checkpoints separately from DD. # Default is to store on same volume as DD. # # LG = Volume used to store P4LOG (active server log), P4JOURNAL (the active # journal), P4TMP, and various application and script logs. DB1=/hxmetadata DB2=/hxmetadata DD=/hxdepots CD=/hxdepots LG=/hxlogs # If you are sharing the depotdata volume with a replica, change this value to TRUE SHAREDDATA=FALSE # Operating system user/group (owner of all directories and files) - should not be root!! OSUSER=perforce OSGROUP=perforce # Set the PERMS value to change permissions assigned to key files/folers. # OS permissions for SDP for directories and executable files are 700 by # default, and 600 for non-executable files. The values mean that only the # owner (i.e. the OSUSER defined above) can see or execute any files. # This is recommended for optimal security. In some environments, such as # test environments, it makes sense to make the permissions less restrictive, # so that permissions can be extended to include members of the defined # OSGROUP defined above, or to 'other'. Set PERMS to 'Owner' for # 700/600 permissions. Set the value to 'Group' for 750/640 permissions. # Set the value to 'Other' for 755/644 permissions. PERMS=Owner # CASE_SENSITIVE settings: # 0 -- Server will run '-C1' mode, forcing case-insensitive mode on normally case-sensitive platforms # such as Linux # 1 -- Server will run with case sensitivity default of the underlying platform (Unix is case sensitive). CASE_SENSITIVE=1 # Admin user's account name. ADMINUSER=perforce # Admin user's password - this will be written to /p4/common/config/.p4passwd.p4_1.admin (or appropriate # alternative instance file). You can leave this value blank and subsequently edit that file if you wish # but don't forget! P4ADMINPASS=GudL0ngAdminP@sswerd # Default domain - makes it easier to edit this script by avoiding repetition in many cases DEFAULT_DOMAIN="example.com" # Email address from which SDP emails are sent. This must be exactly one email address. Depending on the # configuration of the email server, it may or may not be a distribution list. More strict email # server configurations require it to be an email associated with an individual account. MAILFROM="admin@${DEFAULT_DOMAIN}" # Email to receive emails from SDP scripts. This can be a comma-separated list and/or contain a distribution # list email (e.g. PerforceAdmin@MyCompany.com). If you want something other than $MAILFROM, just # remove $MAILFROM and create your own list in quotes. MAILTO=$MAILFROM # Mail Host Address - this is only needed if you use the p4review.py script MAILHOST="mail.${DEFAULT_DOMAIN}" # SSL_PREFIX should be blank to not use SSL, otherwise ssl: # # If you are planning to use SSL, you need to put certificates in /p4/ssl after # the SDP install or you can generate a self-signed certificate as follows: # Edit /p4/ssl/config.txt to put in the info for your company. # Then run: # /p4/common/bin/p4master_run <instance> /p4/<instance>/p4d_<instance> -Gc # For example using instance 1: # /p4/common/bin/p4master_run 1 /p4/1/bin/p4d_1 -Gc # Note: Editing the config.txt is not strictly necessary, as the certificate # details are not visible to users, only the SSL fingerprint is. SSL_PREFIX=ssl: # Set MASTERINSTANCE to the first instance in your installation. # This is used when installing additional instances on a machine after the # first. It is used to copy the P4ROOT/license file (if available) from the # master instance new instances. MASTERINSTANCE=1 P4_PORT=${SDP_INSTANCE}666 P4BROKER_PORT=${SDP_INSTANCE}667 # Note: The p4ftpd and p4web products are no longer supported. These settings # may be removed in a future release. P4FTP_PORT=202${SDP_INSTANCE} P4WEB_PORT=808${SDP_INSTANCE} # If your SDP_INSTANCE is non numeric, then uncomment the items below to set the PORT setting. # Port for this Perforce server to listen on. # P4_PORT=1666 # If your proxy listening port is different from its master listening port, set the master's target port below P4P_TARGET_PORT=1666 # Ignore the broker port if you are not running a broker. # P4BROKER_PORT=1667 # P4WEB_PORT=80 # P4FTP_PORT=21 # DNS Name (or IP address) of the master/commit server for this instance. # A DNS Name is much easier to manage for failover purposes (although an IP will work). # IMPORTANT NOTE this value must be reachable (valid DNS name or /etc/hosts entry) from this instance!!! # It is also used to set the P4TARGET values for replicas (by mkrep.sh) - so needs # to be valid for that purpose too -- thus a DNS name of the commit is preferred. # It can be a FQDN or a short name depending on how DNS is configured. # Consider network firewall rules (aka Security Groups in some cloud terminology) that may be needed. # Sample Values: # p4.mycompany.com P4MASTERHOST=localhost # Replication service user's password - written to /p4/common/config/.p4passwd.p4_<instance>.service # This value is only required for replicas P4SERVICEPASS=servicepass # The server.id of the master server (always the same value for all servers in a # distributed topology) MASTER_ID=master.${SDP_INSTANCE} # Set SERVER_TYPE to one of: # p4d_master - A master/commit server. # p4d_replica - A replica with all metadata from the master (not filtered in any way). # p4d_filtered_replica - A filtered replica or forwarding replica. # p4d_edge - An edge server # p4d_edge_replica - An replica of an edge server (with no extra filtering). # p4broker - An SDP host running only a broker, with no p4d. # p4proxy - An SDP host running a proxy (maybe with a broker in front), with no p4d. SERVER_TYPE=p4d_master # Replica server.id name. Required if SERVER_TYPE is p4d_replica, p4d_edge, # or p4d_filtered_replica. REPLICA_ID=replica # Email address for p4review complaints for each instance - only required if p4review.py is used. # look something like P4Review_1666@example.com. Set # the COMPLAINFROM_PREFIX (e.g. "P4Review") and # COMPLAINFROM_DOMAIN (e.g. "example.com)" here. Instance # specific values are substituted below. COMPLAINFROM_DOMAIN="${DEFAULT_DOMAIN}" COMPLAINFROM="${SSL_PREFIX}${P4MASTERHOST}:${P4_PORT}_P4Review\@${COMPLAINFROM_DOMAIN}"
# | Change | User | Description | Committed | |
---|---|---|---|---|---|
#21 | 30333 | C. Thomas Tyler |
Added HxCheckpoints, optional setting to store checkpoints on a separate volume from depots. Default is same value as HxDepots. This changes makes the SPD more adaptable to customers, especially using NFS, that prefer to have separate mounts for checkpoints vs. those for archive files. The original SDP design was to have everything that needed to be backed up (i.e. metadata checkpoints and the versioned file tree) on a single volume. This is still the preferred mode, but allowing checkpoints and archives to be separate is useful for some. #review-30334 |
||
#20 | 29851 | C. Thomas Tyler |
Changed default value of P4MASTERHOST from an illustrative sample value, DNS_name_of_master_server_for_this_instance, to a value that will work on Day 1: localhost. Also removed outdated text about the P4MASTERHOST value; it should ALWAYS refer to the commit server. #review-29852 |
||
#19 | 29849 | C. Thomas Tyler |
Updates to configure_new_server.sh: * Added logic to start service so this script can now be run if the service is down at the start. * The 'security' configurable is now set to 4 (was unset). * Enhanced error handling. * Added logic to more fully initialize a new/empty server: - Create perforce P4USER (if there is only a single user on the system). - Create Automation group. - Initialize basic Protections. Updates to configure_new_server.bat: * Added logic to start service so this script can now be run if the service is down at the start. * The 'security' configurable is now set to 4 (was unset). In mkdirs.cfg, change the default password to one good enough to be used with `security=4`. #review-29850 |
||
#18 | 28687 | Robert Cowham | Clarify comments for P4MASTERHOST | ||
#17 | 28192 | C. Thomas Tyler |
Added new optional PERMS setting in mkdirs.cfg, to allow opening of SDP perms when mkdirs.sh is used. This is useful for customer environments (mainly test environments). Documentation for the feature is included in the mkdirs.cfg file. Default behavior is unchanged. |
||
#16 | 27977 | Robert Cowham | Reduce unnecessary repetition in domain names for many instances. | ||
#15 | 27722 | C. Thomas Tyler |
Refinements to @27712: * Resolved one out-of-date file (verify_sdp.sh). * Added missing adoc file for which HTML file had a change (WorkflowEnforcementTriggers.adoc). * Updated revdate/revnumber in *.adoc files. * Additional content updates in Server/Unix/p4/common/etc/cron.d/ReadMe.md. * Bumped version numbers on scripts with Version= def'n. * Generated HTML, PDF, and doc/gen files: - Most HTML and all PDF are generated using Makefiles that call an AsciiDoc utility. - HTML for Perl scripts is generated with pod2html. - doc/gen/*.man.txt files are generated with .../tools/gen_script_man_pages.sh. #review-27712 |
||
#14 | 26986 | C. Thomas Tyler |
Refinements to mkdirs.sh and mkdirs.cfg, per code review: https://swarm.workshop.perforce.com/reviews/26962 In mkdirs.cfg: * Added comments about proper use of DB1/DB2. * Added comment about coming removal of P4FTP/P4WEB settings. In mkdirs.sh v4.1.1: * Replaced -M with set of -M* flags to specify mount points. * Completed implementation of -M* flags. * Enhanced examples in usage function. * Added clarification of difference between '-n' and '-p'. * Fixed some typos. #review @rcowham |
||
#13 | 26982 | C. Thomas Tyler |
mkdirs.sh v4.1.0: * Accounted for directory structure change of Maintenance to Unsupported. * Added standard command line processing with '-h' and '-man' doc flags, and other flags (all documented). * Added in-code docs and updated AsciiDoc. * Enhanced '-test' mode to simulate /hx* mounts. * Enhanced preflight testing, and fixed '-test' mode installs. * Added support for installing to an alternate root directory. * Added '-s <ServerID>' option to override REPLICA_ID. * Added '-S <TargetServerID>' used for replicas of edge servers. * Added '-t <server_type>' option to override SERVER_TYPE. * Added '-M' option to override mount points. * Added '-f' fast option to skip big chown/chmod commands, and moved those commands near the end as well. verify_sdp.sh v5.9.0: * Added check for /p4/Version file, and checked that other legacy SDP methods of checking version * Added sanity check for crontab. * Added 'test skip' mechanism to skip certain tests: - crontab: Skip crontab check. Use this if you do not expect crontab to be configured, perhaps if a different scheduler is used. - license: Skip license related checks. - version: Skip version checks. - excess: Skip checks for excess copies of p4d/p4p/p4broker in PATH. * Added VERIFY_SDP_SKIP_TEST_LIST setting ton instance_vars.template, to define a standard way to have verify_sdp.sh always skip certain tests for a site. * Extended '-online' checks to check for bogus P4MASTERPORT, a common config error. Update test_SDP.py: * Adjusted test suite to account for various changes in mkdirs.sh. * Added 'dir' parameter to run_cmd() and sudo_cmd(), to run a command from a specified directory (as required to test new mkdirs.sh) * Added check_links() similar to existing check_dirs() function. === Upgrade Process Changes === Made /p4/common/bin/p4d/p4/p4broker/p4p shell script rather than binary. This changes the way SDP new binaries are staged for upgrade. For safety, exes are now staged to a director outside the PATH, the /p4/sdp/exes folder. A new 'get_latest_exes.sh' script simplifies the task of pulling executables from the Perforce FTP server. This can be used 'as is' for environments with outbound internet access, and is useful in any case to describe now to acquire binaries. This addresses an issue where a p4d binary staged for a future upgrade might be called before the actual upgrade is performed. upgrade.sh v4.0.0: * All preflight checks are now done first. Added '-p' to abort after preflight. * Added '-n' to show what would be done before anything is executed. * Minimalist logic to start/stop only servers that are upgrade, and apply upgrades only as needed. * Staging of exes for upgrade is now separate from /p4/common/bin * Improved in-code docs, added '-h' and '-man' options. * Retained pre/post P4D 2019.1 upgrade logic. |
||
#12 | 26732 | Robert Cowham |
Change default for DB1/DB2 to /hxmetadata with comment that these can be seperated Add a couple of other clarifying comments. |
||
#11 | 26718 | Robert Cowham |
Rename P4MASTER to P4MASTERHOST for clarity with comments in: - mkdirs.cfg/mkdirs.sh - p4_<instance>.vars - other files which reference Remove unnecessary sed for p4p.template |
||
#10 | 26529 | C. Thomas Tyler |
Corrected comments in mkdirs.cfg for P4MASTER_ID. The P4MASTER_ID should only be set to the master server (aka commit-server in a topology including edge servers). In cases where the P4TARGET of a replica-of-an-edge is needed, the P4TARGET value can be calculated dynamically. We leave P4MASTER_ID to be statically configured, as typically it is only configured once in the life of an instance. Other aspects of the topology are dynamic and require dynamic detection. This is a non-functional, comment-only change. #review-26530 @ashaikh |
||
#9 | 25363 | C. Thomas Tyler |
Fixed bug where /p4/N/bin ownership is not set correctly (owned by root). Fixed issue with missing symlink creation for replicas. Made compliant with shellcheck.sh v0.6.0, driving many changes and fixing potential glob-expansion issues. Fixed an issue with '-test' not setting DB1/DB2. Removed unused/obsolete MD def'n. Added 'p4d_ha' server type to distinguish from existing 'p4d_standby' type, and added comments to mkdirs.cfg to note that p4d_standby is for pre-2018.1 servers, and p4d_ha is for 2018.1+. The distinction is that P4D 2018.1 introduced rpl.journalcopy.location, which changes P4D behaviors and corresponding best practice for placement of the actively pulled journal file for 'journalcopy' replicas. For pre-2018.1, we use /hxlogs/p4/N/journals.rep, and for 2018.1+ with rpl.journalcopy.location=1 set, we us /hxdepots/p4/N/checkpoints.<tag>. Normalized inconsistent indentation. |
||
#8 | 25190 | Robert Cowham |
Change to make instance dir a directory under /p4 (so on root volume). This avoids potential problems where /hxdepots is shared on multiple instances. It also improves performance often. Left the option to put /p4/common on a shared volume if required. |
||
#7 | 25142 | Robert Cowham | Clarify comments - no other change | ||
#6 | 23266 | C. Thomas Tyler |
Fixes and Enhancements: * Enabled daily_checkpoint.sh operate on edge servers, to keep /p4/N/offline_db current on those hosts for site-local recovery w/o requiring a site-local replica (though having a site-local replica can still be useful). * Disabled live_checkpoint.sh for edge servers. * More fully support topologies using edge severs, in both geographically distributed and horizaontal scaling "wokspace server" solutions. * Fix broken EDGESERVER value definition. * Modified name of SDP counter that gets set when a checkpoint is taken to incorporate ServerID, so now the counter name will look like lastSDPCheckpoint.master.1, or lastSDPCheckpoint.p4d_edge_sfo, rather than just lastSDPCheckpoint. There will be multiple such counters in a topology that uses edge servers, and/or which takes checkpoints on replicas. * Added comments for all functions. For the master server, journalPrefix remains: /p4/N/checkpoints/p4_N The /p4/N/checkpoints is reserved for writing by the master/commit server only. For non-standby (possibly filtered) replicas and edge serves, journalPrefix is: /p4/N/checkpoints.<ShortServerID>/p4_N.<ShortServerID> Here, ShortServerID is just the ServerID with the 'p4d_' prefix trimmed, since it is redundant in this context. See mkrep.sh, which enshines a ServerID (server spec) naming standard, with values like 'p4d_fr_bos' (forwarding replica in Boston) and p4d_edge_blr (Edge server in Bangalore). So the journalPrefix for the p4d_edge_bos replica would be: /p4/N/checkpoints.edge_bos/p4_N.edge_bos For "standby" (aka journalcopy) replicas, journalPrefix is set to /p4/N/journals.rep. which is written to the $LOGS volume, due to the nature of standby replicas using journalPrefix to write active server logs to pre-rotated journals. Some take-away to be updated in docs: * The /p4/N/checkpoints folder must be reserved for checkpoints that originate on the master. It should be safe to rsync this folder (with --delete if desired) to any replica or edge server. This is consistent with the current SDP. * I want to change 'journals.rep' to 'checkpoints.<ShortServerID>' for non-standby replicas, to ensure that checkpoints and journals taken on those hosts are written to a volume where they are backed up. * In sites with multiple edge serves, some sharing achive files ('workspace servers'), multiple edge servers will share the same SAN. So we one checkpoints dir per ServerID, and we want that dir to be on the /hxdepots volume. Note that the journalPrefix for replicas was a fixed /p4/N/journals.rep. This was on the /hxlogs volume - a presumably fast-for-writes volume, but typically NOT backed up and not very large. This change puts it under /p4/N/checkpoints.* for edge servers and non-standby replicas, but ensures other replica types and edge servers can generate checkpoints to a location that is backed up and has plenty of storage capacity. For standby replicas only (which cannot be filtered), the journalPrefix remains /p4/N/journals.rep on the /hxlogs volume. |
||
#5 | 22957 | C. Thomas Tyler |
Merged two changes initiated on main down to: Change @22826 by robert_cowham: Ignore some files on Mac. Change @22950 by awkan/ttyler: Provide an override for difference between proxy listening port and target port |
||
#4 | 22571 | C. Thomas Tyler |
Changed default (and defacto standard) value for server.id of a master server to make it unique by incorporating the SDP_INSTANCE name. So for instance 'fgs', the server.id would not be 'master.fgs' instead of just 'master'. This is necessary to enable remote depots, which won't work if two servers involved in remote depot sharing have the same server.id. Affects Unix SDP. |
||
#3 | 22333 | Russell C. Jackson (Rusty) |
The service user always needs to be defined since the proxy base script uses it as well. Added note about making sure to leave replica FALSE when setting up a proxy. If you don't the instance vars script will create an empty database by running the cshow commands. |
||
#2 | 22145 | Robert Cowham | Update mkdirs to help testing. | ||
#1 | 22070 | Russell C. Jackson (Rusty) |
Pull the configuration items out into mkdirs.cfg and source that file in mkdirs.sh. Makes it a little cleaner, and less likely for someone to mess up the mkdirs.sh script. Also preps for potential future configure_sdp.sh script to ask questions and create the mkdirs.cfg file. |