SDP-801 #4

  • //
  • spec/
  • job/
  • SDP-801
  • View
  • Commits
  • Open Download .zip Download (5 KB)
# The form data below was edited by tom_tyler
# Perforce Workshop Jobs
#
#  Job:           The job name. 'new' generates a sequenced job number.
#
#  Status:        Job status; required field.  There is no enforced or
#                 promoted workflow for transition of jobs from one
#                 status to another, just a set of job status values
#                 for users to apply as they see fit.  Possible values:
#
#                 open - Issue is available to be worked on.
#
#                 inprogress - Active development is in progress.
#
#                 blocked - Issue cannot be implemented for some reason.
#
#                 fixed - Fixed, optional status to use before closed.
#                 
#                 closed - Issue has been dealt with definitively.
#
#                 punted - Decision made not to address the issue,
#                    possibly not ever.
#
#                 suspended - Decision made not to address the issue
#                    in the immediate future, but noting that it may
#                    have some merit and may be revisited later.
#
#                 duplicate - Duplicate of another issue that.
#
#                 obsolete - The need behind the request has become
#                    overcome by events.
#
#  Project:       The project this job is for. Required.
#
#  Severity:      [A/B/C] (A is highest)  Required.
#
#  ReportedBy     The user who created the job. Can be changed.
#
#  ReportedDate:  The date the job was created.  Automatic.
#
#  ModifiedBy:    The user who last modified this job. Automatic.
#
#  ModifiedDate:  The date this job was last modified. Automatic.
#
#  OwnedBy:       The owner, responsible for doing the job. Optional.
#
#  Description:   Description of the job.  Required.
#
#  DevNotes:      Developer's comments.  Optional.  Can be used to
#                 explain a status, e.g. for blocked, punted,
#                 obsolete or duplicate jobs.  May also provide
#                 additional information such as the earliest release
#                 in which a bug is known to exist.
#
# Component:      Projects may use this optional field to indicate
#                 which component of the project a givenjob is associated
#                 with.
#
#                 For the SDP, the list of components is defined in:
#                 //guest/perforce_software/sdp/tools/components.txt
#
#  Type:          Type of job [Bug/Feature/Problem].  Required.
#                 Feature and Bug are common terms.
#                 A Problem is suspected bug, or one without a clear
#                 understanding of exactly what is broken.
#
#  Release:       Release in which job is intended to be fixed.

Job:	SDP-801

Status:	open

Project:	perforce-software-sdp

Severity:	C

ReportedBy:	tom_tyler

ReportedDate:	2022/06/08 10:47:12

ModifiedBy:	tom_tyler

ModifiedDate:	2022/06/08 10:51:30

OwnedBy:	tom_tyler

Description:
	If p4d does not stop, upgrade.sh will hang. Doc that.
	
	In one customer environment, the call to 'sudo systemctl
	stop p4d_N' in the stop_p4d() function in backup_functions.sh
	appears not to have stopped p4d, resulting in the upgrade.sh
	hanging for a long while on the stop where it waits for the
	p4d service to stop. The p4d in question was a replica, not
	a master, though that may not be relevant.
	
	The script will, by design, hang indefinitely, because
	we need p4d to be well and truly stopped before
	continuing. This logic was working.
	
	This was on SDP 2022.1.28855 (2022/05/27).
	
	It seems like the call to 'sudo systemctl stop p4d_N'
	that should have been made by the script had no effect,
	but this could not be reproduced manually.
	
	In any case, the eventual fix was to do, as perforce:
	kill $(cat $P4ROOT/server.pid)
	
	That stopped the service, and then the still-running-but-hanging
	upgrade script continued where it left off and finally
	completed the upgrade without further issue.
	
	Background:
	
	Prior to SDP r2020.1, there was an issue where the
	status of p4d according to 'systemctl status p4d_1'
	could be wrong.  In particular, if the servcie was
	started without systemd, then systemd would not know
	the service was up, and would fail to stop it.  If
	systemctl's concept of status of a service being up
	or down is wrong (really a design flaw in systemd),
	then the commands to start/stop the service using
	systemctl can be unreliable.
	
		However, starting with SDP 2020.1, the SDP init scripts
	have logic to require that systemd be used if it is
	configured to be used, which thus ensures that system'd
	concept of up/down for a service is reliable, and thus
	start/stop should be reliable.  If the permissions on
	/etc/systemd/system/p4*.service files are wrong, this
	would introduce a loophole, but the permissions on
	/etc/systemd/system/p4*.service files were checked and
	they were correct.
	
	The logic is functioning as intended, in that the script
	does not proceed until p4d is down. That needs to be
	preserved.
	
	This job is intended to capture the documentation of the
	workaround employed and when to employ it.  Also, this
	job should involve contemplation of whether further
	work is needed on the script.

Component:	doc

Type:	Problem
# Change User Description Committed
#4 default
#3 default
#2 default
#1 default