The central server manager WCCILdata(0) stopped! Restarting the project

[nedzad]

Hello,

Few days ago project on one server in redundant system crashed and it was not able to restart whole day. Whole time this message was written in log:

WCCILpmon (1), 2018.05.27 09:50:07.787, SYS, SEVERE, 10/pmon, The central server manager WCCILdata(0) stopped! Restarting the project
WCCILpmon (1), 2018.05.27 09:50:07.887, SYS, INFO, 14/pmon, Completely stopped the project - restarting it

So this means that server was trying to start, but could not do it. What could be the reason for this? Project on the other server worked normally whole time. I tried to completely stop the project and than to start it, but I kept getting the same error message. The problem disappeared when I restarted the server. Is it possible that some process in background blocked project from starting or what?

This is the whole pmon.log from the moment when crash happened.

WCCILpmon (1), 2018.05.27 09:35:44.456, SYS, INFO, 11/pmon, Detected stopped manager WCCOActrl(110) - restarting it #1
WCCILpmon (1), 2018.05.27 09:35:44.524, SYS, INFO, 1, Manager Start, /opt/WinCC_OA/3.14/bin/WCCOActrl -PROJ TakovoPreljina -pmonIndex 47 asterisk.ctl -num 110
WCCILpmon (1), 2018.05.27 09:49:28.184, SYS, INFO, 22/pmon, Got RESTART_ALL command from WCCILevent, restarting the project
WCCILpmon (1), 2018.05.27 09:49:28.285, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(4) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.285, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(144) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.285, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(142) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.285, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(141) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.285, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(140) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.285, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(138) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.285, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(3) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.285, SYS, INFO, 17/pmon, Stopping Manager WCCOAsnmp(16) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.285, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(34) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.286, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(133) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.286, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(125) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.286, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(119) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.286, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(118) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.286, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(117) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.286, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(116) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.286, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(114) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.286, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(113) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.286, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(112) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.286, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(111) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.286, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(110) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.287, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(108) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.287, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(107) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.287, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(102) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.287, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(101) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.287, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(100) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.287, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(33) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.287, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(31) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.287, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(30) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.287, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(25) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.287, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(24) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.287, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(22) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.287, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(21) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.287, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(20) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.287, SYS, INFO, 17/pmon, Stopping Manager WCCOAmod(14) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.288, SYS, INFO, 17/pmon, Stopping Manager WCCOAsnmp(15) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.288, SYS, INFO, 17/pmon, Stopping Manager WCCOAmod(12) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.288, SYS, INFO, 17/pmon, Stopping Manager WCCOAmod(11) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.288, SYS, INFO, 17/pmon, Stopping Manager WCCOAmod(10) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.288, SYS, INFO, 17/pmon, Stopping Manager WCCOAmod(9) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.288, SYS, INFO, 17/pmon, Stopping Manager WCCOAmod(7) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.288, SYS, INFO, 17/pmon, Stopping Manager WCCOAmod(5) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.288, SYS, INFO, 17/pmon, Stopping Manager WCCOAmod(4) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.288, SYS, INFO, 17/pmon, Stopping Manager WCCOAmod(6) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.288, SYS, INFO, 17/pmon, Stopping Manager WCCOAmod(2) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.288, SYS, INFO, 17/pmon, Stopping Manager WCCOAmod(1) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.288, SYS, INFO, 17/pmon, Stopping Manager WCCOAmod(3) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.288, SYS, INFO, 17/pmon, Stopping Manager WCCOAui(1) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.288, SYS, INFO, 17/pmon, Stopping Manager WCCILdist(1) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.288, SYS, INFO, 17/pmon, Stopping Manager WCCOActrl(2) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.288, SYS, INFO, 17/pmon, Stopping Manager WCCOAvalarch(5) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:49:28.288, SYS, INFO, 118, Last error already repeated 50 times. Until another output occurs, this one will be blocked.
WCCILpmon (1), 2018.05.27 09:50:05.531, SYS, INFO, 17/pmon, Stopping Manager WCCILproxy(1) with signal SIGTERM
WCCILpmon (1), 2018.05.27 09:50:05.619, SYS, INFO, 119, Last error repeated 74 times.
WCCILpmon (1), 2018.05.27 09:50:05.619, SYS, INFO, 14/pmon, Completely stopped the project - restarting it
WCCILpmon (1), 2018.05.27 09:50:05.720, SYS, INFO, 1, Manager Start, /opt/WinCC_OA/3.14/bin/WCCILdata -PROJ TakovoPreljina -pmonIndex 1
WCCILpmon (1), 2018.05.27 09:50:07.787, SYS, SEVERE, 10/pmon, The central server manager WCCILdata(0) stopped! Restarting the project
WCCILpmon (1), 2018.05.27 09:50:07.887, SYS, INFO, 14/pmon, Completely stopped the project - restarting it
WCCILpmon (1), 2018.05.27 09:50:07.988, SYS, INFO, 1, Manager Start, /opt/WinCC_OA/3.14/bin/WCCILdata -PROJ TakovoPreljina -pmonIndex 1
WCCILpmon (1), 2018.05.27 09:50:10.009, SYS, SEVERE, 10/pmon, The central server manager WCCILdata(0) stopped! Restarting the project
WCCILpmon (1), 2018.05.27 09:50:10.109, SYS, INFO, 14/pmon, Completely stopped the project - restarting it
WCCILpmon (1), 2018.05.27 09:50:10.129, SYS, INFO, 1, Manager Start, /opt/WinCC_OA/3.14/bin/WCCILdata -PROJ TakovoPreljina -pmonIndex 1
WCCILpmon (1), 2018.05.27 09:50:12.153, SYS, SEVERE, 10/pmon, The central server manager WCCILdata(0) stopped! Restarting the project
WCCILpmon (1), 2018.05.27 09:50:12.254, SYS, INFO, 14/pmon, Completely stopped the project - restarting it
WCCILpmon (1), 2018.05.27 09:50:12.354, SYS, INFO, 1, Manager Start, /opt/WinCC_OA/3.14/bin/WCCILdata -PROJ TakovoPreljina -pmonIndex 1
WCCILpmon (1), 2018.05.27 09:50:14.380, SYS, SEVERE, 10/pmon, The central server manager WCCILdata(0) stopped! Restarting the project
WCCILpmon (1), 2018.05.27 09:50:14.480, SYS, INFO, 14/pmon, Completely stopped the project - restarting it
.
.
.

Also, next morning I tried to start Arhive Manager 0 on the other server in this system and the project also crashed, writing the same error message in log, but now it was able to restart normally.

Best regards,
Nedžad

[mkerk]

Hello,
from the pmon.log files entries, it´s only possible to see, that the DATA-Manager was not able to (re)start ( "The central server manager WCCILdata(0) stopped! Restarting the project") and that´s why the pmon failed to restart the project. Only from these log-entries it´s not possible to investigate the reason for this.
Please check in the PVSS_II.log file if you can find more clues for this behavior.

Just a remark : there are a relative high number of control-managers in your application. Why? Maybe check if you can use one or more CTRL-Manager as a CTRL script list ( option -f followed by the file name of a CTRL script list) in order to start several scripts with one manager (see chapter CONTROL --> Introduction CONTROL --> Loading Control Libraries --> Several CTRL scripts ).

BR,
Mousser.