The mpi API on beowulf at Livingston

Legend:
green ball Normal status or debugging message
yellow ball Notable condition which may be a non-fatal error
orange ball Error condition not fatal to job
red ball Error condition fatal to job
blue ball Notable condition which is not an error
purple ball Currently undefined
email Condition requires email notification of the responsible administrator of this API
telephone Condition requires phone notification of the responsible administrator of this API

Link: API Status Page for Livingston

10/27/06-11:44:10 CDT 10/27/06-16:44:10 GMT 846002664 STARTUP archiveLog file "/ldas_outgoing/logs/LDASmpi.log.html" already closed. (archived as /ldas_outgoing/logs/archive/mpiAPI/LDASmpi.845161338)
10/27/06-11:44:10 CDT 10/27/06-16:44:10 GMT 846002664 STARTUP closeListenSock no cid registered for service 'data'
10/27/06-11:44:10 CDT 10/27/06-16:44:10 GMT 846002664 STARTUP mpi::init unused data port 10021 closed
10/27/06-11:44:10 CDT 10/27/06-16:44:10 GMT 846002664 STARTUP mpi::init port 10021 (jobstate) opened on beowulf as sock7
10/27/06-11:44:10 CDT 10/27/06-16:44:10 GMT 846002664 STARTUP bgLoop Looping process watchlogs started
10/27/06-11:44:10 CDT 10/27/06-16:44:10 GMT 846002664 STARTUP openListenSock port 10019 (operator) opened on beowulf as sock8
10/27/06-11:44:10 CDT 10/27/06-16:44:10 GMT 846002664 STARTUP openListenSock port 10020 (emergency) opened on beowulf as sock9
10/27/06-11:44:10 CDT 10/27/06-16:44:10 GMT 846002664 STARTUP leakLogger inital size of mpi API: 21004 kB
10/27/06-11:44:10 CDT 10/27/06-16:44:10 GMT 846002664 STARTUP bgLoop Looping process etchosts started
10/27/06-11:44:11 CDT 10/27/06-16:44:11 GMT 846002665 IDLE bgLoop Looping process statpagefile started
10/27/06-11:44:11 CDT 10/27/06-16:44:11 GMT 846002665 IDLE bgLoop Looping process killedjobreaper started
10/27/06-11:44:11 CDT 10/27/06-16:44:11 GMT 846002665 IDLE bgLoop Looping process logrotate started
10/27/06-11:44:11 CDT 10/27/06-16:44:11 GMT 846002665 IDLE setFTPandHTTPinfo (::FTPURL 'ftp://130.39.245.245') (::FTPDIR '') (::HTTPURL 'http://130.39.245.245/ldas_outgoing/jobs') (::HTTPDIR '/ldas_outgoing/jobs') (::GRIDFTPURL 'gridftp:/export/grid/ldas') (::GRIDFTPDIR '/export/grid/ldas') (::LDAS_GATEWAY 'ldas 130.39.245.245') (::LDAS_SYSTEM 'ldas-la') (::RUNCODE 'LDAS-LA')
10/27/06-11:44:15 CDT 10/27/06-16:44:15 GMT 846002669 IDLE setFTPandHTTPinfo (::FTPURL 'ftp://130.39.245.245') (::FTPDIR '') (::HTTPURL 'http://130.39.245.245/ldas_outgoing/jobs') (::HTTPDIR '/ldas_outgoing/jobs') (::GRIDFTPURL 'gridftp:/export/grid/ldas') (::GRIDFTPDIR '/export/grid/ldas') (::LDAS_GATEWAY 'ldas 130.39.245.245') (::LDAS_SYSTEM 'ldas-la') (::RUNCODE 'LDAS-LA')
10/27/06-11:44:16 CDT 10/27/06-16:44:16 GMT 846002670 STARTUP mpi::killAllMpirun cleaning up for user ldas
10/27/06-11:44:17 CDT 10/27/06-16:44:17 GMT 846002671 STARTUP mpi::killAllMpirun ran kill 10 times in 1.060 seconds
10/27/06-11:44:17 CDT 10/27/06-16:44:17 GMT 846002671 STARTUP mpi::prestartLamds running lamboot for user search01
10/27/06-11:44:18 CDT 10/27/06-16:44:18 GMT 846002672 STARTUP mpi::prestartLamds running lamboot for user search02
10/27/06-11:44:19 CDT 10/27/06-16:44:19 GMT 846002673 STARTUP mpi::prestartLamds running lamboot for user search03
10/27/06-11:44:20 CDT 10/27/06-16:44:20 GMT 846002674 STARTUP mpi::prestartLamds running lamboot for user search04
10/27/06-11:44:21 CDT 10/27/06-16:44:21 GMT 846002675 STARTUP mpi::prestartLamds running lamboot for user search05
10/27/06-11:44:22 CDT 10/27/06-16:44:22 GMT 846002676 STARTUP mpi::prestartLamds running lamboot for user search06
10/27/06-11:44:23 CDT 10/27/06-16:44:23 GMT 846002677 STARTUP mpi::prestartLamds running lamboot for user search07
10/27/06-11:44:24 CDT 10/27/06-16:44:24 GMT 846002678 STARTUP mpi::prestartLamds running lamboot for user search08
10/27/06-11:44:25 CDT 10/27/06-16:44:25 GMT 846002679 STARTUP mpi::prestartLamds running lamboot for user search09
10/27/06-11:44:27 CDT 10/27/06-16:44:27 GMT 846002681 STARTUP mpi::prestartLamds running lamboot for user search10
10/27/06-11:44:29 CDT 10/27/06-16:44:29 GMT 846002683 STARTUP mpi::prestartLamds running lamboot for user search11
10/27/06-11:44:29 CDT 10/27/06-16:44:29 GMT 846002683 STARTUP mpi::prestartLamds running lamboot for user search12
10/27/06-11:44:30 CDT 10/27/06-16:44:30 GMT 846002684 STARTUP mpi::prestartLamds running lamboot for user search13
10/27/06-11:44:31 CDT 10/27/06-16:44:31 GMT 846002685 STARTUP mpi::prestartLamds running lamboot for user search14
10/27/06-11:44:32 CDT 10/27/06-16:44:32 GMT 846002686 STARTUP mpi::prestartLamds running lamboot for user search15
10/27/06-11:44:33 CDT 10/27/06-16:44:33 GMT 846002687 STARTUP mpi::prestartLamds running lamboot for user search16
10/27/06-11:44:35 CDT 10/27/06-16:44:35 GMT 846002689 IDLE mpi::updateCmonNodelist updated ::beowulfNodes in cntlmonAPI to 'beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf'
10/27/06-11:44:35 CDT 10/27/06-16:44:35 GMT 846002689 STARTUP mpi::prestartLamds STARTUP search01 beowulf ok!
10/27/06-11:44:35 CDT 10/27/06-16:44:35 GMT 846002689 STARTUP mpi::prestartLamds STARTUP search02 beowulf ok!
10/27/06-11:44:35 CDT 10/27/06-16:44:35 GMT 846002689 STARTUP mpi::prestartLamds STARTUP search03 beowulf ok!
10/27/06-11:44:35 CDT 10/27/06-16:44:35 GMT 846002689 STARTUP mpi::prestartLamds STARTUP search04 beowulf ok!
10/27/06-11:44:35 CDT 10/27/06-16:44:35 GMT 846002689 STARTUP mpi::prestartLamds STARTUP search05 beowulf ok!
10/27/06-11:44:35 CDT 10/27/06-16:44:35 GMT 846002689 STARTUP mpi::prestartLamds STARTUP search06 beowulf ok!
10/27/06-11:44:35 CDT 10/27/06-16:44:35 GMT 846002689 STARTUP mpi::prestartLamds STARTUP search07 beowulf ok!
10/27/06-11:44:35 CDT 10/27/06-16:44:35 GMT 846002689 STARTUP mpi::prestartLamds STARTUP search08 beowulf ok!
10/27/06-11:44:35 CDT 10/27/06-16:44:35 GMT 846002689 STARTUP mpi::prestartLamds STARTUP search09 beowulf ok!
10/27/06-11:44:35 CDT 10/27/06-16:44:35 GMT 846002689 STARTUP mpi::prestartLamds STARTUP search10 beowulf ok!
10/27/06-11:44:35 CDT 10/27/06-16:44:35 GMT 846002689 STARTUP mpi::prestartLamds STARTUP search11 beowulf ok!
10/27/06-11:44:35 CDT 10/27/06-16:44:35 GMT 846002689 STARTUP mpi::prestartLamds STARTUP search12 beowulf ok!
10/27/06-11:44:35 CDT 10/27/06-16:44:35 GMT 846002689 STARTUP mpi::prestartLamds STARTUP search13 beowulf ok!
10/27/06-11:44:35 CDT 10/27/06-16:44:35 GMT 846002689 STARTUP mpi::prestartLamds STARTUP search14 beowulf ok!
10/27/06-11:44:36 CDT 10/27/06-16:44:36 GMT 846002690 STARTUP mpi::prestartLamds STARTUP search15 beowulf ok!
10/27/06-11:44:37 CDT 10/27/06-16:44:37 GMT 846002691 STARTUP mpi::prestartLamds STARTUP search16 beowulf ok!
10/27/06-11:44:38 CDT 10/27/06-16:44:38 GMT 846002692 STARTUP mpi::killAllMpirun {ldas@beowulf:mpirun: child process exited abnormally} {ldas@beowulf:wrapperAPI: child process exited abnormally} {ldas@beowulf:lamd: child process exited abnormally}
10/27/06-11:44:51 CDT 10/27/06-16:44:51 GMT 846002705 IDLE mpi::updateCmonNodelist updated ::beowulfNodes in cntlmonAPI to 'beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf'
10/27/06-21:00:15 CDT 10/28/06-02:00:15 GMT 846036029 SHUTDOWN closeListenSock port 10019 (sock8) (operator) closed on beowulf
pehrens@ligo.caltech.edu, igor@ligo-la.caltech.edu 846036029 SHUTDOWN mpi::sHuTdOwN Subject: LDAS Livingston mpi shutdown at 846036029 ( 10/27/06 21:00:15 CDT ); Body: mpi shutting down NOW
10/27/06-21:00:15 CDT 10/28/06-02:00:15 GMT 846036029 search12 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
10/27/06-21:00:15 CDT 10/28/06-02:00:15 GMT 846036029 search04 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
10/27/06-21:00:16 CDT 10/28/06-02:00:16 GMT 846036030 search14 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
10/27/06-21:00:16 CDT 10/28/06-02:00:16 GMT 846036030 search06 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
10/27/06-21:00:16 CDT 10/28/06-02:00:16 GMT 846036030 search16 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
10/27/06-21:00:16 CDT 10/28/06-02:00:16 GMT 846036030 search08 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
10/27/06-21:00:17 CDT 10/28/06-02:00:17 GMT 846036031 search01 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
10/27/06-21:00:17 CDT 10/28/06-02:00:17 GMT 846036031 search11 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
10/27/06-21:00:17 CDT 10/28/06-02:00:17 GMT 846036031 search03 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
10/27/06-21:00:17 CDT 10/28/06-02:00:17 GMT 846036031 search13 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
10/27/06-21:00:18 CDT 10/28/06-02:00:18 GMT 846036032 search05 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
10/27/06-21:00:18 CDT 10/28/06-02:00:18 GMT 846036032 search15 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
10/27/06-21:00:18 CDT 10/28/06-02:00:18 GMT 846036032 search07 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
10/27/06-21:00:18 CDT 10/28/06-02:00:18 GMT 846036032 search10 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
10/27/06-21:00:19 CDT 10/28/06-02:00:19 GMT 846036033 search09 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
10/27/06-21:00:19 CDT 10/28/06-02:00:19 GMT 846036033 search02 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
10/27/06-21:00:19 CDT 10/28/06-02:00:19 GMT 846036033 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search01'
10/27/06-21:00:19 CDT 10/28/06-02:00:19 GMT 846036033 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search02'
10/27/06-21:00:19 CDT 10/28/06-02:00:19 GMT 846036033 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search03'
10/27/06-21:00:20 CDT 10/28/06-02:00:20 GMT 846036034 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search04'
10/27/06-21:00:20 CDT 10/28/06-02:00:20 GMT 846036034 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search05'
10/27/06-21:00:20 CDT 10/28/06-02:00:20 GMT 846036034 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search06'
10/27/06-21:00:21 CDT 10/28/06-02:00:21 GMT 846036035 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search07'
10/27/06-21:00:21 CDT 10/28/06-02:00:21 GMT 846036035 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search08'
10/27/06-21:00:21 CDT 10/28/06-02:00:21 GMT 846036035 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search09'
10/27/06-21:00:21 CDT 10/28/06-02:00:21 GMT 846036035 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search10'
10/27/06-21:00:22 CDT 10/28/06-02:00:22 GMT 846036036 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search11'
10/27/06-21:00:22 CDT 10/28/06-02:00:22 GMT 846036036 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search12'
10/27/06-21:00:22 CDT 10/28/06-02:00:22 GMT 846036036 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search13'
10/27/06-21:00:22 CDT 10/28/06-02:00:22 GMT 846036036 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search14'
10/27/06-21:00:23 CDT 10/28/06-02:00:23 GMT 846036037 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search15'
10/27/06-21:00:23 CDT 10/28/06-02:00:23 GMT 846036037 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search16'
10/27/06-21:00:23 CDT 10/28/06-02:00:23 GMT 846036037 SHUTDOWN closeListenSock port 10020 (sock9) (emergency) closed on beowulf
10/27/06-21:00:23 CDT 10/28/06-02:00:23 GMT 846036037 SHUTDOWN closeListenSock no cid registered for service 'data'
10/27/06-21:00:23 CDT 10/28/06-02:00:23 GMT 846036037 SHUTDOWN closeLog /ldas_outgoing/logs/LDASmpi.log.html (file5) closed