The mpi API on beowulf at Livingston

Legend:
green ball Normal status or debugging message
yellow ball Notable condition which may be a non-fatal error
orange ball Error condition not fatal to job
red ball Error condition fatal to job
blue ball Notable condition which is not an error
purple ball Currently undefined
email Condition requires email notification of the responsible administrator of this API
telephone Condition requires phone notification of the responsible administrator of this API

Link: API Status Page for Livingston

11/28/06-13:27:23 CST 11/28/06-19:27:23 GMT 848777257 STARTUP archiveLog file "/ldas_outgoing/logs/LDASmpi.log.html" already closed. (archived as /ldas_outgoing/logs/archive/mpiAPI/LDASmpi.848762600)
11/28/06-13:27:23 CST 11/28/06-19:27:23 GMT 848777257 STARTUP closeListenSock no cid registered for service 'data'
11/28/06-13:27:23 CST 11/28/06-19:27:23 GMT 848777257 STARTUP mpi::init unused data port 10021 closed
11/28/06-13:27:23 CST 11/28/06-19:27:23 GMT 848777257 STARTUP mpi::init port 10021 (jobstate) opened on beowulf as sock7
11/28/06-13:27:23 CST 11/28/06-19:27:23 GMT 848777257 STARTUP bgLoop Looping process watchlogs started
11/28/06-13:27:23 CST 11/28/06-19:27:23 GMT 848777257 STARTUP openListenSock port 10019 (operator) opened on beowulf as sock8
11/28/06-13:27:23 CST 11/28/06-19:27:23 GMT 848777257 STARTUP openListenSock port 10020 (emergency) opened on beowulf as sock9
11/28/06-13:27:23 CST 11/28/06-19:27:23 GMT 848777257 STARTUP leakLogger inital size of mpi API: 21004 kB
11/28/06-13:27:23 CST 11/28/06-19:27:23 GMT 848777257 STARTUP bgLoop Looping process etchosts started
11/28/06-13:27:24 CST 11/28/06-19:27:24 GMT 848777258 IDLE bgLoop Looping process statpagefile started
11/28/06-13:27:24 CST 11/28/06-19:27:24 GMT 848777258 IDLE bgLoop Looping process killedjobreaper started
11/28/06-13:27:24 CST 11/28/06-19:27:24 GMT 848777258 IDLE bgLoop Looping process logrotate started
11/28/06-13:27:24 CST 11/28/06-19:27:24 GMT 848777258 IDLE setFTPandHTTPinfo (::FTPURL 'ftp://130.39.245.245') (::FTPDIR '') (::HTTPURL 'http://130.39.245.245/ldas_outgoing/jobs') (::HTTPDIR '/ldas_outgoing/jobs') (::GRIDFTPURL 'gridftp:/export/grid/ldas') (::GRIDFTPDIR '/export/grid/ldas') (::LDAS_GATEWAY 'ldas 130.39.245.245') (::LDAS_SYSTEM 'ldas-la') (::RUNCODE 'LDAS-LA')
11/28/06-13:27:27 CST 11/28/06-19:27:27 GMT 848777261 IDLE setFTPandHTTPinfo (::FTPURL 'ftp://130.39.245.245') (::FTPDIR '') (::HTTPURL 'http://130.39.245.245/ldas_outgoing/jobs') (::HTTPDIR '/ldas_outgoing/jobs') (::GRIDFTPURL 'gridftp:/export/grid/ldas') (::GRIDFTPDIR '/export/grid/ldas') (::LDAS_GATEWAY 'ldas 130.39.245.245') (::LDAS_SYSTEM 'ldas-la') (::RUNCODE 'LDAS-LA')
11/28/06-13:27:29 CST 11/28/06-19:27:29 GMT 848777263 STARTUP mpi::killAllMpirun cleaning up for user ldas
11/28/06-13:27:30 CST 11/28/06-19:27:30 GMT 848777264 STARTUP mpi::killAllMpirun ran kill 10 times in 1.064 seconds
11/28/06-13:27:30 CST 11/28/06-19:27:30 GMT 848777264 STARTUP mpi::prestartLamds running lamboot for user search01
11/28/06-13:27:31 CST 11/28/06-19:27:31 GMT 848777265 STARTUP mpi::prestartLamds running lamboot for user search02
11/28/06-13:27:32 CST 11/28/06-19:27:32 GMT 848777266 STARTUP mpi::prestartLamds running lamboot for user search03
11/28/06-13:27:33 CST 11/28/06-19:27:33 GMT 848777267 STARTUP mpi::prestartLamds running lamboot for user search04
11/28/06-13:27:34 CST 11/28/06-19:27:34 GMT 848777268 STARTUP mpi::prestartLamds running lamboot for user search05
11/28/06-13:27:35 CST 11/28/06-19:27:35 GMT 848777269 STARTUP mpi::prestartLamds running lamboot for user search06
11/28/06-13:27:36 CST 11/28/06-19:27:36 GMT 848777270 STARTUP mpi::prestartLamds running lamboot for user search07
11/28/06-13:27:38 CST 11/28/06-19:27:38 GMT 848777272 STARTUP mpi::prestartLamds running lamboot for user search08
11/28/06-13:27:39 CST 11/28/06-19:27:39 GMT 848777273 STARTUP mpi::prestartLamds running lamboot for user search09
11/28/06-13:27:40 CST 11/28/06-19:27:40 GMT 848777274 STARTUP mpi::prestartLamds running lamboot for user search10
11/28/06-13:27:41 CST 11/28/06-19:27:41 GMT 848777275 STARTUP mpi::prestartLamds running lamboot for user search11
11/28/06-13:27:42 CST 11/28/06-19:27:42 GMT 848777276 STARTUP mpi::prestartLamds running lamboot for user search12
11/28/06-13:27:43 CST 11/28/06-19:27:43 GMT 848777277 STARTUP mpi::prestartLamds running lamboot for user search13
11/28/06-13:27:44 CST 11/28/06-19:27:44 GMT 848777278 STARTUP mpi::prestartLamds running lamboot for user search14
11/28/06-13:27:45 CST 11/28/06-19:27:45 GMT 848777279 STARTUP mpi::prestartLamds running lamboot for user search15
11/28/06-13:27:46 CST 11/28/06-19:27:46 GMT 848777280 STARTUP mpi::prestartLamds running lamboot for user search16
11/28/06-13:27:48 CST 11/28/06-19:27:48 GMT 848777282 IDLE mpi::updateCmonNodelist updated ::beowulfNodes in cntlmonAPI to 'beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf'
11/28/06-13:27:48 CST 11/28/06-19:27:48 GMT 848777282 STARTUP mpi::prestartLamds STARTUP search01 beowulf ok!
11/28/06-13:27:48 CST 11/28/06-19:27:48 GMT 848777282 STARTUP mpi::prestartLamds STARTUP search02 beowulf ok!
11/28/06-13:27:48 CST 11/28/06-19:27:48 GMT 848777282 STARTUP mpi::prestartLamds STARTUP search03 beowulf ok!
11/28/06-13:27:48 CST 11/28/06-19:27:48 GMT 848777282 STARTUP mpi::prestartLamds STARTUP search04 beowulf ok!
11/28/06-13:27:48 CST 11/28/06-19:27:48 GMT 848777282 STARTUP mpi::prestartLamds STARTUP search05 beowulf ok!
11/28/06-13:27:48 CST 11/28/06-19:27:48 GMT 848777282 STARTUP mpi::prestartLamds STARTUP search06 beowulf ok!
11/28/06-13:27:48 CST 11/28/06-19:27:48 GMT 848777282 STARTUP mpi::prestartLamds STARTUP search07 beowulf ok!
11/28/06-13:27:48 CST 11/28/06-19:27:48 GMT 848777282 STARTUP mpi::prestartLamds STARTUP search08 beowulf ok!
11/28/06-13:27:48 CST 11/28/06-19:27:48 GMT 848777282 STARTUP mpi::prestartLamds STARTUP search09 beowulf ok!
11/28/06-13:27:48 CST 11/28/06-19:27:48 GMT 848777282 STARTUP mpi::prestartLamds STARTUP search10 beowulf ok!
11/28/06-13:27:48 CST 11/28/06-19:27:48 GMT 848777282 STARTUP mpi::prestartLamds STARTUP search11 beowulf ok!
11/28/06-13:27:48 CST 11/28/06-19:27:48 GMT 848777282 STARTUP mpi::prestartLamds STARTUP search12 beowulf ok!
11/28/06-13:27:48 CST 11/28/06-19:27:48 GMT 848777282 STARTUP mpi::prestartLamds STARTUP search13 beowulf ok!
11/28/06-13:27:48 CST 11/28/06-19:27:48 GMT 848777282 IDLE mpi::updateCmonNodelist updated ::beowulfNodes in cntlmonAPI to 'beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf'
11/28/06-13:27:48 CST 11/28/06-19:27:48 GMT 848777282 STARTUP mpi::prestartLamds STARTUP search14 beowulf ok!
11/28/06-13:27:49 CST 11/28/06-19:27:49 GMT 848777283 STARTUP mpi::prestartLamds STARTUP search15 beowulf ok!
11/28/06-13:27:50 CST 11/28/06-19:27:50 GMT 848777284 STARTUP mpi::prestartLamds STARTUP search16 beowulf ok!
11/28/06-13:27:51 CST 11/28/06-19:27:51 GMT 848777285 STARTUP mpi::killAllMpirun {ldas@beowulf:mpirun: child process exited abnormally} {ldas@beowulf:wrapperAPI: child process exited abnormally} {ldas@beowulf:lamd: child process exited abnormally}
03/06/07-09:55:04 CST 03/06/07-15:55:04 GMT 857231718 SHUTDOWN closeListenSock port 10019 (sock8) (operator) closed on beowulf
pehrens@ligo.caltech.edu, igor@ligo-la.caltech.edu 857231718 SHUTDOWN mpi::sHuTdOwN Subject: LDAS Livingston mpi shutdown at 857231718 ( 03/06/07 09:55:04 CST ); Body: mpi shutting down NOW
03/06/07-09:55:05 CST 03/06/07-15:55:05 GMT 857231719 search12 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
03/06/07-09:55:05 CST 03/06/07-15:55:05 GMT 857231719 search04 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
03/06/07-09:55:05 CST 03/06/07-15:55:05 GMT 857231719 search14 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
03/06/07-09:55:05 CST 03/06/07-15:55:05 GMT 857231719 search06 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
03/06/07-09:55:06 CST 03/06/07-15:55:06 GMT 857231720 search16 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
03/06/07-09:55:06 CST 03/06/07-15:55:06 GMT 857231720 search08 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
03/06/07-09:55:06 CST 03/06/07-15:55:06 GMT 857231720 search01 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
03/06/07-09:55:06 CST 03/06/07-15:55:06 GMT 857231720 search11 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
03/06/07-09:55:07 CST 03/06/07-15:55:07 GMT 857231721 search03 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
03/06/07-09:55:07 CST 03/06/07-15:55:07 GMT 857231721 search13 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
03/06/07-09:55:07 CST 03/06/07-15:55:07 GMT 857231721 search05 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
03/06/07-09:55:07 CST 03/06/07-15:55:07 GMT 857231721 search15 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
03/06/07-09:55:08 CST 03/06/07-15:55:08 GMT 857231722 search07 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
03/06/07-09:55:08 CST 03/06/07-15:55:08 GMT 857231722 search10 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
03/06/07-09:55:08 CST 03/06/07-15:55:08 GMT 857231722 search09 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
03/06/07-09:55:08 CST 03/06/07-15:55:08 GMT 857231722 search02 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
03/06/07-09:55:08 CST 03/06/07-15:55:08 GMT 857231722 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search01'
03/06/07-09:55:09 CST 03/06/07-15:55:09 GMT 857231723 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search02'
03/06/07-09:55:09 CST 03/06/07-15:55:09 GMT 857231723 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search03'
03/06/07-09:55:09 CST 03/06/07-15:55:09 GMT 857231723 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search04'
03/06/07-09:55:10 CST 03/06/07-15:55:10 GMT 857231724 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search05'
03/06/07-09:55:10 CST 03/06/07-15:55:10 GMT 857231724 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search06'
03/06/07-09:55:10 CST 03/06/07-15:55:10 GMT 857231724 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search07'
03/06/07-09:55:11 CST 03/06/07-15:55:11 GMT 857231725 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search08'
03/06/07-09:55:11 CST 03/06/07-15:55:11 GMT 857231725 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search09'
03/06/07-09:55:11 CST 03/06/07-15:55:11 GMT 857231725 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search10'
03/06/07-09:55:11 CST 03/06/07-15:55:11 GMT 857231725 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search11'
03/06/07-09:55:12 CST 03/06/07-15:55:12 GMT 857231726 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search12'
03/06/07-09:55:12 CST 03/06/07-15:55:12 GMT 857231726 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search13'
03/06/07-09:55:12 CST 03/06/07-15:55:12 GMT 857231726 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search14'
03/06/07-09:55:12 CST 03/06/07-15:55:12 GMT 857231726 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search15'
03/06/07-09:55:13 CST 03/06/07-15:55:13 GMT 857231727 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search16'
03/06/07-09:55:13 CST 03/06/07-15:55:13 GMT 857231727 SHUTDOWN closeListenSock port 10020 (sock9) (emergency) closed on beowulf
03/06/07-09:55:13 CST 03/06/07-15:55:13 GMT 857231727 SHUTDOWN closeListenSock no cid registered for service 'data'
03/06/07-09:55:13 CST 03/06/07-15:55:13 GMT 857231727 SHUTDOWN closeLog /ldas_outgoing/logs/LDASmpi.log.html (file5) closed