The mpi API on beowulf at Livingston

Legend:
green ball Normal status or debugging message
yellow ball Notable condition which may be a non-fatal error
orange ball Error condition not fatal to job
red ball Error condition fatal to job
blue ball Notable condition which is not an error
purple ball Currently undefined
email Condition requires email notification of the responsible administrator of this API
telephone Condition requires phone notification of the responsible administrator of this API

Link: API Status Page for Livingston

10/28/06-10:42:40 CDT 10/28/06-15:42:40 GMT 846085374 STARTUP archiveLog file "/ldas_outgoing/logs/LDASmpi.log.html" already closed. (archived as /ldas_outgoing/logs/archive/mpiAPI/LDASmpi.846036037)
10/28/06-10:42:40 CDT 10/28/06-15:42:40 GMT 846085374 STARTUP closeListenSock no cid registered for service 'data'
10/28/06-10:42:40 CDT 10/28/06-15:42:40 GMT 846085374 STARTUP mpi::init unused data port 10021 closed
10/28/06-10:42:40 CDT 10/28/06-15:42:40 GMT 846085374 STARTUP mpi::init port 10021 (jobstate) opened on beowulf as sock7
10/28/06-10:42:40 CDT 10/28/06-15:42:40 GMT 846085374 STARTUP bgLoop Looping process watchlogs started
10/28/06-10:42:40 CDT 10/28/06-15:42:40 GMT 846085374 STARTUP openListenSock port 10019 (operator) opened on beowulf as sock8
10/28/06-10:42:40 CDT 10/28/06-15:42:40 GMT 846085374 STARTUP openListenSock port 10020 (emergency) opened on beowulf as sock9
10/28/06-10:42:40 CDT 10/28/06-15:42:40 GMT 846085374 STARTUP leakLogger inital size of mpi API: 21008 kB
10/28/06-10:42:40 CDT 10/28/06-15:42:40 GMT 846085374 STARTUP bgLoop Looping process etchosts started
10/28/06-10:42:40 CDT 10/28/06-15:42:40 GMT 846085374 IDLE bgLoop Looping process statpagefile started
10/28/06-10:42:40 CDT 10/28/06-15:42:40 GMT 846085374 IDLE bgLoop Looping process killedjobreaper started
10/28/06-10:42:40 CDT 10/28/06-15:42:40 GMT 846085374 IDLE bgLoop Looping process logrotate started
10/28/06-10:42:41 CDT 10/28/06-15:42:41 GMT 846085375 IDLE setFTPandHTTPinfo (::FTPURL 'ftp://130.39.245.245') (::FTPDIR '') (::HTTPURL 'http://130.39.245.245/ldas_outgoing/jobs') (::HTTPDIR '/ldas_outgoing/jobs') (::GRIDFTPURL 'gridftp:/export/grid/ldas') (::GRIDFTPDIR '/export/grid/ldas') (::LDAS_GATEWAY 'ldas 130.39.245.245') (::LDAS_SYSTEM 'ldas-la') (::RUNCODE 'LDAS-LA')
10/28/06-10:42:44 CDT 10/28/06-15:42:44 GMT 846085378 IDLE setFTPandHTTPinfo (::FTPURL 'ftp://130.39.245.245') (::FTPDIR '') (::HTTPURL 'http://130.39.245.245/ldas_outgoing/jobs') (::HTTPDIR '/ldas_outgoing/jobs') (::GRIDFTPURL 'gridftp:/export/grid/ldas') (::GRIDFTPDIR '/export/grid/ldas') (::LDAS_GATEWAY 'ldas 130.39.245.245') (::LDAS_SYSTEM 'ldas-la') (::RUNCODE 'LDAS-LA')
10/28/06-10:42:45 CDT 10/28/06-15:42:45 GMT 846085379 STARTUP mpi::killAllMpirun cleaning up for user ldas
10/28/06-10:42:46 CDT 10/28/06-15:42:46 GMT 846085380 STARTUP mpi::killAllMpirun ran kill 10 times in 1.146 seconds
10/28/06-10:42:46 CDT 10/28/06-15:42:46 GMT 846085380 STARTUP mpi::prestartLamds running lamboot for user search01
10/28/06-10:42:47 CDT 10/28/06-15:42:47 GMT 846085381 STARTUP mpi::prestartLamds running lamboot for user search02
10/28/06-10:42:48 CDT 10/28/06-15:42:48 GMT 846085382 STARTUP mpi::prestartLamds running lamboot for user search03
10/28/06-10:42:49 CDT 10/28/06-15:42:49 GMT 846085383 STARTUP mpi::prestartLamds running lamboot for user search04
10/28/06-10:42:49 CDT 10/28/06-15:42:49 GMT 846085383 STARTUP mpi::prestartLamds running lamboot for user search05
10/28/06-10:42:50 CDT 10/28/06-15:42:50 GMT 846085384 STARTUP mpi::prestartLamds running lamboot for user search06
10/28/06-10:42:51 CDT 10/28/06-15:42:51 GMT 846085385 STARTUP mpi::prestartLamds running lamboot for user search07
10/28/06-10:42:52 CDT 10/28/06-15:42:52 GMT 846085386 STARTUP mpi::prestartLamds running lamboot for user search08
10/28/06-10:42:53 CDT 10/28/06-15:42:53 GMT 846085387 STARTUP mpi::prestartLamds running lamboot for user search09
10/28/06-10:42:54 CDT 10/28/06-15:42:54 GMT 846085388 STARTUP mpi::prestartLamds running lamboot for user search10
10/28/06-10:42:55 CDT 10/28/06-15:42:55 GMT 846085389 STARTUP mpi::prestartLamds running lamboot for user search11
10/28/06-10:42:56 CDT 10/28/06-15:42:56 GMT 846085390 STARTUP mpi::prestartLamds running lamboot for user search12
10/28/06-10:42:57 CDT 10/28/06-15:42:57 GMT 846085391 STARTUP mpi::prestartLamds running lamboot for user search13
10/28/06-10:42:58 CDT 10/28/06-15:42:58 GMT 846085392 STARTUP mpi::prestartLamds running lamboot for user search14
10/28/06-10:42:59 CDT 10/28/06-15:42:59 GMT 846085393 STARTUP mpi::prestartLamds running lamboot for user search15
10/28/06-10:42:59 CDT 10/28/06-15:42:59 GMT 846085393 STARTUP mpi::prestartLamds running lamboot for user search16
10/28/06-10:43:01 CDT 10/28/06-15:43:01 GMT 846085395 IDLE mpi::updateCmonNodelist updated ::beowulfNodes in cntlmonAPI to 'beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf'
10/28/06-10:43:01 CDT 10/28/06-15:43:01 GMT 846085395 STARTUP mpi::prestartLamds STARTUP search01 beowulf ok!
10/28/06-10:43:01 CDT 10/28/06-15:43:01 GMT 846085395 STARTUP mpi::prestartLamds STARTUP search02 beowulf ok!
10/28/06-10:43:01 CDT 10/28/06-15:43:01 GMT 846085395 STARTUP mpi::prestartLamds STARTUP search03 beowulf ok!
10/28/06-10:43:01 CDT 10/28/06-15:43:01 GMT 846085395 STARTUP mpi::prestartLamds STARTUP search04 beowulf ok!
10/28/06-10:43:01 CDT 10/28/06-15:43:01 GMT 846085395 STARTUP mpi::prestartLamds STARTUP search05 beowulf ok!
10/28/06-10:43:01 CDT 10/28/06-15:43:01 GMT 846085395 STARTUP mpi::prestartLamds STARTUP search06 beowulf ok!
10/28/06-10:43:01 CDT 10/28/06-15:43:01 GMT 846085395 STARTUP mpi::prestartLamds STARTUP search07 beowulf ok!
10/28/06-10:43:01 CDT 10/28/06-15:43:01 GMT 846085395 STARTUP mpi::prestartLamds STARTUP search08 beowulf ok!
10/28/06-10:43:01 CDT 10/28/06-15:43:01 GMT 846085395 STARTUP mpi::prestartLamds STARTUP search09 beowulf ok!
10/28/06-10:43:01 CDT 10/28/06-15:43:01 GMT 846085395 STARTUP mpi::prestartLamds STARTUP search10 beowulf ok!
10/28/06-10:43:01 CDT 10/28/06-15:43:01 GMT 846085395 STARTUP mpi::prestartLamds STARTUP search11 beowulf ok!
10/28/06-10:43:01 CDT 10/28/06-15:43:01 GMT 846085395 STARTUP mpi::prestartLamds STARTUP search12 beowulf ok!
10/28/06-10:43:01 CDT 10/28/06-15:43:01 GMT 846085395 STARTUP mpi::prestartLamds STARTUP search13 beowulf ok!
10/28/06-10:43:02 CDT 10/28/06-15:43:02 GMT 846085396 STARTUP mpi::prestartLamds STARTUP search14 beowulf ok!
10/28/06-10:43:02 CDT 10/28/06-15:43:02 GMT 846085396 STARTUP mpi::prestartLamds STARTUP search15 beowulf ok!
10/28/06-10:43:04 CDT 10/28/06-15:43:04 GMT 846085398 STARTUP mpi::prestartLamds STARTUP search16 beowulf ok!
10/28/06-10:43:04 CDT 10/28/06-15:43:04 GMT 846085398 STARTUP mpi::killAllMpirun {ldas@beowulf:mpirun: child process exited abnormally} {ldas@beowulf:wrapperAPI: child process exited abnormally} {ldas@beowulf:lamd: child process exited abnormally}
10/28/06-10:43:08 CDT 10/28/06-15:43:08 GMT 846085402 IDLE mpi::updateCmonNodelist updated ::beowulfNodes in cntlmonAPI to 'beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf'
11/28/06-09:22:57 CST 11/28/06-15:22:57 GMT 848762591 SHUTDOWN closeListenSock port 10019 (sock8) (operator) closed on beowulf
pehrens@ligo.caltech.edu, igor@ligo-la.caltech.edu 848762591 SHUTDOWN mpi::sHuTdOwN Subject: LDAS Livingston mpi shutdown at 848762591 ( 11/28/06 09:22:57 CST ); Body: mpi shutting down NOW
11/28/06-09:22:57 CST 11/28/06-15:22:57 GMT 848762591 search12 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
11/28/06-09:22:57 CST 11/28/06-15:22:57 GMT 848762591 search04 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
11/28/06-09:22:58 CST 11/28/06-15:22:58 GMT 848762592 search14 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
11/28/06-09:22:58 CST 11/28/06-15:22:58 GMT 848762592 search06 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
11/28/06-09:22:58 CST 11/28/06-15:22:58 GMT 848762592 search16 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
11/28/06-09:22:59 CST 11/28/06-15:22:59 GMT 848762593 search08 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
11/28/06-09:22:59 CST 11/28/06-15:22:59 GMT 848762593 search01 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
11/28/06-09:22:59 CST 11/28/06-15:22:59 GMT 848762593 search11 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
11/28/06-09:22:59 CST 11/28/06-15:22:59 GMT 848762593 search03 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
11/28/06-09:23:00 CST 11/28/06-15:23:00 GMT 848762594 search13 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
11/28/06-09:23:00 CST 11/28/06-15:23:00 GMT 848762594 search05 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
11/28/06-09:23:00 CST 11/28/06-15:23:00 GMT 848762594 search15 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
11/28/06-09:23:00 CST 11/28/06-15:23:00 GMT 848762594 search07 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
11/28/06-09:23:01 CST 11/28/06-15:23:01 GMT 848762595 search10 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
11/28/06-09:23:01 CST 11/28/06-15:23:01 GMT 848762595 search09 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
11/28/06-09:23:01 CST 11/28/06-15:23:01 GMT 848762595 search02 mpi::abortJobInDcApi datacond API unreachable!!: sock::open: could not connect to datacond emergency on port 10017 on datacon. {connection refused}
11/28/06-09:23:01 CST 11/28/06-15:23:01 GMT 848762595 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search01'
11/28/06-09:23:02 CST 11/28/06-15:23:02 GMT 848762596 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search02'
11/28/06-09:23:02 CST 11/28/06-15:23:02 GMT 848762596 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search03'
11/28/06-09:23:02 CST 11/28/06-15:23:02 GMT 848762596 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search04'
11/28/06-09:23:03 CST 11/28/06-15:23:03 GMT 848762597 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search05'
11/28/06-09:23:03 CST 11/28/06-15:23:03 GMT 848762597 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search06'
11/28/06-09:23:03 CST 11/28/06-15:23:03 GMT 848762597 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search07'
11/28/06-09:23:04 CST 11/28/06-15:23:04 GMT 848762598 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search08'
11/28/06-09:23:04 CST 11/28/06-15:23:04 GMT 848762598 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search09'
11/28/06-09:23:04 CST 11/28/06-15:23:04 GMT 848762598 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search10'
11/28/06-09:23:04 CST 11/28/06-15:23:04 GMT 848762598 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search11'
11/28/06-09:23:05 CST 11/28/06-15:23:05 GMT 848762599 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search12'
11/28/06-09:23:05 CST 11/28/06-15:23:05 GMT 848762599 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search13'
11/28/06-09:23:05 CST 11/28/06-15:23:05 GMT 848762599 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search14'
11/28/06-09:23:05 CST 11/28/06-15:23:05 GMT 848762599 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search15'
11/28/06-09:23:06 CST 11/28/06-15:23:06 GMT 848762600 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search16'
11/28/06-09:23:06 CST 11/28/06-15:23:06 GMT 848762600 SHUTDOWN closeListenSock port 10020 (sock9) (emergency) closed on beowulf
11/28/06-09:23:06 CST 11/28/06-15:23:06 GMT 848762600 SHUTDOWN closeListenSock no cid registered for service 'data'
11/28/06-09:23:06 CST 11/28/06-15:23:06 GMT 848762600 SHUTDOWN closeLog /ldas_outgoing/logs/LDASmpi.log.html (file5) closed