ston1th/vipman
0
0
Fork 0
Code Issues 2 Releases Activity

analyse deadlock #2

New issue
Open
opened 2021-02-12 08:06:37 +01:00 by ston1th · 0 comments
ston1th commented 2021-02-12 08:06:37 +01:00
Owner
Copy link
Feb 12 07:49:34 dnstest02 vipman[16335]: I0212 07:49:34.740656   16335 vip.go:54] etcd "msg"="following"  "id"="401075b1b18591e3"
Feb 12 07:49:41 dnstest02 vipman[16335]: I0212 07:49:41.047186   16335 vip.go:31] etcd "msg"="leading"  "id"="401075b1b18591e3"
Feb 12 07:49:41 dnstest02 vipman[16335]: I0212 07:49:41.052523   16335 vip.go:59] etcd "msg"="following canceled"  "id"="401075b1b18591e3"
Feb 12 07:49:41 dnstest02 vipman[16335]: I0212 07:49:41.052904   16335 vip.go:37] etcd "msg"="leading canceled"  "id"="401075b1b18591e3"
Feb 12 07:49:41 dnstest02 vipman[16335]: E0212 07:49:41.053719   16335 vip.go:26] etcd "msg"="error running hook" "error"="signal: killed"  "leaderHook"=["/root/start_srv.sh"]

replicate behavior?

  • call panic inside leader controller
  • return early in leader controller
  • report status of leader controller and call stepdown if nessesary
  • Add ping/pong channel

maybe implement some sort of watchdog:

  • check for an increasing counter every 1min
  • if counter does not increase cancel cluster and start over
``` Feb 12 07:49:34 dnstest02 vipman[16335]: I0212 07:49:34.740656 16335 vip.go:54] etcd "msg"="following" "id"="401075b1b18591e3" Feb 12 07:49:41 dnstest02 vipman[16335]: I0212 07:49:41.047186 16335 vip.go:31] etcd "msg"="leading" "id"="401075b1b18591e3" Feb 12 07:49:41 dnstest02 vipman[16335]: I0212 07:49:41.052523 16335 vip.go:59] etcd "msg"="following canceled" "id"="401075b1b18591e3" Feb 12 07:49:41 dnstest02 vipman[16335]: I0212 07:49:41.052904 16335 vip.go:37] etcd "msg"="leading canceled" "id"="401075b1b18591e3" Feb 12 07:49:41 dnstest02 vipman[16335]: E0212 07:49:41.053719 16335 vip.go:26] etcd "msg"="error running hook" "error"="signal: killed" "leaderHook"=["/root/start_srv.sh"] ``` replicate behavior? * call panic inside leader controller * return early in leader controller * report status of leader controller and call stepdown if nessesary * Add ping/pong channel maybe implement some sort of watchdog: * check for an increasing counter every 1min * if counter does not increase cancel cluster and start over
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
ston1th/vipman#2
No description provided.