Windows HPC Server Message Passing Interface (MPI) Forum

failed to summit MPI application builded with 64-bit binary code

Thu, 05 Nov 2009 21:24:33 Z

I have an application running on Windows HPC 2008. There is no problem with 32-bit binary code. However, I can not submit the job for 64-bit binary code. It was release build with amd64 msmpi.lib. I can run the code on one node through mpiexec command. However, if I submit the job on multiple node it failed with following error message.

Aborting: failed to launch '\\TDCMFCPWOPHpF01\HPC_APPS\rluo\mpi_demo_64.exe' on TDCMFCPWOPCNA01
Error (14001) The application has failed to start because its side-by-side configuration is incorrect. Please see the application event log for more detail.

program not working when using more than 2 processes...

Tue, 24 Nov 2009 19:18:05 Z

Hello!

I'm a very beginner in MPI and the answer to my question might be obvious, but I just can't see it now.

Could anyone say what causes the program containing the following lines not to work when using more than 2 processes? (it works fine in a serial manner and also when using 2 processes)

	if (myrank == 0)
	{

		// read from file
			...

		// send target value to all other processes
		for (i = 1; i < nrprocs; i++)
		{
			MPI_Send (&target, 1, MPI_INT, i, TAG1, MPI_COMM_WORLD);
		}


		// calculate len
		len = n / nrprocs;

		// send len to all other processes
		for (i = 1; i < nrprocs; i++)
		{
			MPI_Send (&len, 1, MPI_INT, i, TAG2, MPI_COMM_WORLD);
		}
		

		// send to each process a part of the array to be searched
		for (i = 1; i < nrprocs; i++)
		{
			MPI_Send (&b[(i-1)*len], len, MPI_INT, i, TAG3, MPI_COMM_WORLD);
		}



		// the root process searches the last portion
		for (i = ((nrprocs - 1)*len); i < n; i++)
		{
			if (b[i] == target)
				printf ("%d ", i);
		}

		no = 0;
		while ( no != (nrprocs - 1) )
		{
			MPI_Recv (&index, 1, MPI_INT, 1, MPI_ANY_TAG, MPI_COMM_WORLD, &status);

			if (status.MPI_TAG == END_TAG)
			{
				no++;
			}
			else
			{
				printf ("%d ", index);
			}
		}	
	}
	else
	{
		MPI_Recv (&target, 1, MPI_INT, 0, TAG1, MPI_COMM_WORLD, &status);
		MPI_Recv (&len, 1, MPI_INT, 0, TAG2, MPI_COMM_WORLD, &status);
		MPI_Recv (&b[0], len, MPI_INT, 0, TAG3, MPI_COMM_WORLD, &status);

		index = -1;
		for (i = 0; i < len; i++)
		{
			if (b[i] == target)
			{
				index = (myrank - 1)*len + i;
				MPI_Send (&index, 1, MPI_INT, 0, TAG, MPI_COMM_WORLD);
			}
		}

		// message saying that the current process has finished searching
		MPI_Send (&index, 1, MPI_INT, 0, END_TAG, MPI_COMM_WORLD);
	}

Thanks in advance!

mpi.dll

Tue, 24 Nov 2009 09:19:34 Z

My PC OS is Windows HPC Server 2008

When I use HPC Cluster Manager add Task to MPI.exe (Hello world) , the task error message:

ERROR: unable to read the cmd header on the pmi context, Error= -1

How do I solve this problem !?

Did OS lack "mpi.dll" !? If it did,how can I found the "mpi.dll" !?

Or problem have else !?

by the way....

Do you install MPICH2 (x86 or x64) ?

The MPICH2 have to install !?

MPI environment

Mon, 23 Nov 2009 16:52:25 Z

Hi,

We have been using pallas mpi for long time and usually has MPICH_DISABLE_SOCK 1 as environment variable only.

It may be wrong to not use any other parameters while validation of network direct on windows hpc platform.

Is there any open list available which we should consider in command line of 'mpiexec' to validate Network Direct to some more depth level?

Can anyone share thier experience using various pallas mpi environement variables in thier testing of Network Direct?

Thanks in advance,

HPC Cluster Manager And MPI Task

Tue, 17 Nov 2009 13:45:12 Z

When I use HPC Cluster Manager run a sample MPI program (EX:Hello world).
I find that always failed. I guess the MPI environment set be problem.
I have 5 PC, 1 HeadNode 4 ComputerNode.
Each PC installation MPICH2,the MPICH2 need for the installation on Windows HPC Server 2008 !?

When use HPC Cluster Manager run Diagnostics MPI test is Succeeded
http://img163.imageshack.us/img163/7493/23749122.jpg (HeadNode)
http://img29.imageshack.us/img29/9378/44036017.jpg (ComputerNode)
And than add new job to add task(MPI) always failed.
How can I solve this problem !?

By the way....
"Required Resources" under add task, what does it mean mean !?

MPI programs to provide test ....
download :http://www.xun6.com/file/899af4f11/MPI.exe.html

MPI Job Question

Thu, 12 Nov 2009 07:40:14 Z

On the Job Management ,when I new job and add task,the "required nodes" under Required Resources.
What 's the "required nodes" that it mean !?
http://img91.imageshack.us/img91/869/132456.jpg

by the way .... when I create a new job,the task is failed .
The task failed data (View Failed Tasks) download the file
→ http://www.xun6.com/file/b92173a11/mpierror.txt.html
How can I solve this problem !?

MPI problem

Fri, 06 Nov 2009 06:10:56 Z

Hi. I have got an MPI job which makes the following error. I have no problem if I use one node with multiple processes but if I use multiple nodes, I got always the following error. My HPC clusters are using Infiniband. Is there any option I should use for this type of network? I don't know where I should look into.

job aborted:

[ranks] message

[0] fatal error

Fatal error in MPI_Scatterv: Other MPI error, error stack:

MPI_Scatterv(358).......................: MPI_Scatterv(sbuf=0x06F47100, scnts=0x02138718, displs=0x0

1E21180, MPI_DOUBLE, rbuf=0x01C19660, rcount=1, MPI_DOUBLE, root=0, comm=0x84000001) failed

MPIR_Scatterv(119)......................:

MPIC_Send(39)...........................:

MPIC_Wait(277)..........................:

CH3_ND::CCq::Poll(136)..................:

CH3_ND::CEndpoint::RecvSucceeded(1476)..:

CH3_ND::CEndpoint::ProcessReceives(1120):

CH3_ND::CEndpoint::ProcessDataMsg(1281).:

MPIDI_CH3_RndvSend(271).................: failure occurred while attempting to send message data

CH3_ND::CEndpoint::ProcessSends(869)....:

CH3_ND::CEnvironment::CreateMr(490).....:

CH3_ND::CMr::Create(91).................:

CH3_ND::CMr::Init(66)...................:

CH3_ND::CAdapter::RegisterMemory(293)...: [ch3:nd] INDAdapter::RegisterMemory failed with 0xc0000001

[1-4] terminated

Any advice or help will be greatly appreciated.

Thanks,

Jong

MPI command line

Thu, 05 Nov 2009 08:55:10 Z

My PC system is Windows HPC Server 2008
I have problem on command line
follow this picture http://img18.imageshack.us/img18/9416/123ik.jpg
How can I solve this problem ?

The "sendmpi.exe" that I use Microsoft Visual C++ compile
And I haven install MPICH2 on PC.

If I debug the code , it is no problem.

Configuration Test MPI Ping Pong Lightweight Throughput failed

Wed, 04 Nov 2009 12:58:21 Z

Hi, all!

I tried to run the test "MPI Ping Pong Lightweight Throughput" from "HPC Cluster Manager" and obtained the following output:

Node Message HPC-HEAD There is an error in XML document (1, 1). --> Data at the root level is invalid. Line 1, position 1.

I am not sure, that I understand this result, hence any help will be appreciated.

My cluster is based on HPC Server 2008 and has two nodes in topology 5 (all nodes on eneterprise network)

cannot connet to head node from local host

Fri, 30 Oct 2009 02:45:26 Z

Hi, I am using the ccs 2003. I want to use computing node 1 and 2 (16 processor in total), I can send the job using the user interface on CN01 and there is no problem. But when I tried the command line: c:\documents and settings\hpcuser> job submit /numprocessors:16 /askednodes:cn01,cn02 mpiexec -hosts 2 cn01 cn02 mpitest2.exe
the system tells me that cannot connect to head node from 'localhost'.

Does someone experience the same problem? Furthermore, the reason I use the command line is that I hope to place rank 0 on CN01, rank 1 on CN02. Then rank 0 and rank 1 can pass message between each other, then they can initialize OpenMP on their own node for future use. Is it possible to do this on ccs 2003?

Thanks and regards
Skiff

Using one processor from each of multiple nodes on ccp 2003

Thu, 29 Oct 2009 02:27:51 Z

Dear all, I am using the windows ccp 2003. I want to run my program on 2 nodes. Inside each node, I want to use OpenMP since all the processors on one node have access to the shared memory.
Can I use only one processor from each of the two nodes? Then the two processors will be the master thread in their own node and pass message between each other. But in ccp2003, I can't find the right way to specify two processors from 2 nodes with each node having 8 processors.

Thanks and regards
skiff

New MPI user question.

Wed, 14 Oct 2009 17:51:42 Z

I have an HP laptop computer with dual core CPU's and 6 GB of RAM, running the MS Vista 64-bit operating system and
the Intel 64-bit FORTRAN compiler, version 10.0.027. I am trying to follow the MPI FORTRAN example codes in the book,
"Using MPI: Portable Parallel Programming with the Message-Passing Interface," 2nd ed., by W. Gropp, et. al. Specifically,
I am trying to run the simple FORTRAN program shown on page 25 which calculates pi, CalcPi. I have successfully, I think, loaded,
compiled, and linked this short FORTRAN program with the MPI library--no errors. When I run it, just by initiating the
CalcPi.exe, as in C:> CalcPi the program runs and produces the correct pi result; however, the variable "numprocs," which is
the number of processes used to do the calculation, is always 1, no matter how many integration steps I request. Doing more
research, I found that I was not properly starting this MPI code: I should be initiating the command:
C:> mpiexec -n 4 CalcPi.exe if I wanted more than 1 process (4 in this case). However, when I run this, I get this error message:

Error while connecting to host. No connection could be made because the target machine actively refused it. (10061)
Connect on sock (host=Owner-PC, port=8676) failed, exhaused [sic] all end points
Unable to connect to 'Owner-PC:8676', sock error: Error = -1

I am the administrator on this laptop, and I am not trying to connect to another machine, to my knowledge. So, what am I doing wrong?
As I say, when I run CalcPi.exe as C:> CalcPi all appears to run well, except that "numprocs" always equals 1 no matter how many
integrations steps I request, and I have gone as high as 1 billion.

Mpi Error Question

Wed, 28 Oct 2009 13:02:16 Z

My PC system is Windows HPC Server 2008
on Command line :
mpiexec sendmpi.exe
I have two error message
Link the picture http://img163.imageshack.us/img163/2276/789r.jpg

how solve it !?

Building a function out of a mpi program and inserting it into the new main function....?

Tue, 06 Oct 2009 16:59:09 Z

Hi:
I have a mpi program with its bash file which works completely fine. It is as follows:

    int main(int argc, char *argv[])
{

    MPI_Status status;
    MPI_Init(&argc,&argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &MYRANK);
    MPI_Comm_size(MPI_COMM_WORLD, &NP);

.......CODES.......

     MPI_Finalize();
    return 0;
}

Its final results is a double vector that I need. Now what I want to do is to write a function out of it (for example changing the int main(int argc, char *argv[]) to double foo("some arguments")) so I use it couple of times in another main program. I don't know how to do it. I inserted the function foo as it was (with all mpi stuff) in the new main program that has not mpi stuff but it doesn't work (it takes the error on arguments of MPI_Init(&argc,&argv) since they are not defined in foo ofcourse. I have even tried to pass these arguments from the new main to foo but it doesn't work.).
Any help is really appreciated.
Thanks,
Amin

Forcing Rank 0 to a particular cluster node

Mon, 28 Sep 2009 11:57:19 Z

Our cluster has a mix of machines, and I need rank 0 in my application to run on a particular node that has the capabilities the controlling rank needs. Is there some way to do this I'm missing?

Cannot link to msmpi.lib with both g77 and gfortran

Wed, 16 Sep 2009 16:02:27 Z

Parpack is the parallel version of arpack, a well-known package for sparse eigenproblems written in Fortran. I am trying to port this package and let it run on Microsoft HPC clusters. However, I encontered problems when linking with Micorosft MPI SDK. I already successfully built the package on both MPICH and MPICH2. Now I am quite sure the problem should not be related to parapck itself.

I am using the 32bit SDK, and two libraries msmpifc.lib and msmpi.lib. The order of the libraries are important. If I put msmpi.lib before msmpifc.lib, there will be lots of link errors (>100). I put msmpifc.lib before msmpi.lib and obtained 15 link errors (two root errors):

undefined reference to _imp_MPI_F_STATUSES_IGNOR in mpif.obj
undefined reference to _imp_MPI_F_STATUS_IGNOR in mpif.obj

Did anybody encounter the similar problem? And solution?

BTW: I also tried the combination of msmpifmc.lib and msmpi.lib. The errors are the same.

Thanks,
Gu

MS MPI and GUI, COM

Fri, 17 Jul 2009 10:47:10 Z

I have an application which has its GUI and uses COM technology. I want to use MPI in one of the modules of the application. I wonder how will the system behave concerning COM technology in general.

And how will the existance of GUI affect the MPI parallesation.

I would be thankfull if someone could give a link to a reference about internal work principals of the MS MPI, I mean what REALLY happens when I call MPI_Init() - is it something like UNIX fork() with copying the whole process envirorment via network (including all dlls that proccess loads) or just some binary code delegation?

MPI_FINALIZE fails with error: rank 0 unable to connect to rank 1

Sat, 12 Sep 2009 21:44:13 Z

Hi all,

I have MPI fortran code that I would like to run on two computers, one my home computer , the other work, connected through VPN. I run mpiexec with two hosts and I get the output from the program but MPI Finalize crashes with the above error message. job aborted:

rank: node: exit code[: error message]
0: <IP rank0>: 1: Fatal error in MPI_Finalize: Other MPI error, error stack:
MPI_Finalize(307)............: MPI_Finalize failed
MPI_Finalize(198)............:
MPID_Finalize(92)............:
PMPI_Barrier(476)............: MPI_Barrier(comm=0x44000002) failed
MPIR_Barrier(82).............:
MPIC_Sendrecv(158)...........:
MPID_Isend(116)..............: failure occurred while attempting to send an eage
r message
MPIDI_CH3_iSend(175).........:
MPIDI_CH3I_Sock_connect(1215): [ch3:sock] rank 0 unable to connect to rank 1 usi
ng business card <port=2652 description=<rank 1's network here> ifname=<rank 1s IP>>
MPIDU_Sock_post_connect(1231): unable to connect to ... on p
ort 2652, exhausted all endpoints (errno -1)
MPIDU_Sock_post_connect(1247): gethostbyname failed, The requested name is valid
and was found in the database, but it does not have the correct associated data
being resolved for. (errno 11004)
1: ...: 1

I set MPICH_NETMASK to the correct IP but it doesnt seem to make a difference.

Does anyone have a hint for me? Let me know if you need more information...

Benjamin

How to evenly create the processors on the nodes.

Mon, 03 Aug 2009 05:48:56 Z

I have a MPI program running on the Windows HPC cluster. The cluster have 50 nodes, each nodes have 8 processors. I tried to run my program using 50 processors. The windows HPC always creates 8 processors on 1 nodes. How can I use 25 nodes and each nodes create 2 processors. I tried to use machine file, however, The windoes HPC didn't create processors as my machine file specified.

Running Intel MPI Benchmark for Network Direct

Mon, 16 Mar 2009 18:54:55 Z

Hi,
   I am new to HPC clusters and running MPI job, I am trying to understand if I am missing something while running Intel MPI Benchmark(IMB) over Network Direct on a Windows HPC cluster.

   I have a IMB(source downloded from Intel site) compiled with MSMPI.lib(from MS Compute Cluster Pack) and tried running through the mpiexec and job submit which failed to run. Below is the error I got,

"Error (14001) The application has failed to start because its side-by-side configuration is incorrect. Please see the application event log for more detail."

   Later I found the IMB source for compiling them for Windows 2008 server and compiled them with impi.lib which came with that source. I am able to start that job only with the mpiexec that cmae with the same source. I got some performance numbers too. I am confused now if the results I got is for Network Direct, since the IMB-MPI1 was not compiled with MSMPI.lib & the job was not run with mpiexec command from MS HPC Pack. How to confirm the same?

Thanks,
N

Other MPI error, error stacK

Thu, 30 Jul 2009 03:06:50 Z

Hi, I am working with MPI . try to connect 2 process with mpi_open_port and/or mpi_publish_name.

When i run the program in VC+2008 i dont have problem, but when i run the programm with mpiexec appear the problem.

the message of error is the next:

"Fatal error in MPI_Open_port: Other MPI error, error stack:

MPI_Open_port(119): MPI_Open_port(MPI_INFO_NULL, port=0012FD1C) failed

MPID_Open_port(69): Function not implemented

thanks

Can different MPI processes write data in one file (with different offset)?

Wed, 05 Aug 2009 09:38:33 Z

I'm developing serious mathematics system, which using MPI technology. And I've faced a problem: "Can different MPI processes write data in one file (with different offset)?" If it is possible, I ask you to tell me how to do it. It will be very healthy, if you'll show the source code solving my problem. P.S. English language is not my native language.

No MPI - clusrun doesn't work

Mon, 03 Aug 2009 14:30:32 Z

Hi,

i am an absolute newbie in using the HPC and i hope anyone can help me: I have a small Cluster with 1 Headnode and 2 compute Nodes. If i run a job on one compute node only, it works fine. But if i start a job wich should run on booth compute nodes, the job runs only on the first node. Now i tried "clusrun smpd -status" and i got the following error:

****************
Command has failed on node CLUSTERHEAD. Message:Task failed during execution wit
h exit code 1. Please check task's output for error details.
-------------------------- CLUSTERHEAD returns 1 --------------------------
'smdp' is not recognized as an internal or external command,
operable program or batch file.
Command has failed on node CLUSTERPC02. Message:Error from node: CLUSTERPC02:Log
on failure: unknown user name or bad passwordException of type 'Microsoft.Hpc.Ac
tivation.NodeManagerException' was thrown.
-------------------------- CLUSTERPC02 returns 1 --------------------------
Command has failed on node CLUSTERPC01. Message:Error from node: CLUSTERPC01:Log
on failure: unknown user name or bad passwordException of type 'Microsoft.Hpc.Ac
tivation.NodeManagerException' was thrown.
-------------------------- CLUSTERPC01 returns 1 --------------------------

-------------------------- Summary --------------------------
0 Nodes succeeded
3 Nodes failed:CLUSTERHEAD,CLUSTERPC01,CLUSTERPC02
***************

By using the command: clusrun /all hostname.exe i get this following error-message:

***************
Command proxy has failed on node CLUSTERPC01. Message:Error from node: CLUSTERPC
01:Logon failure: unknown user name or bad passwordException of type 'Microsoft.
Hpc.Activation.NodeManagerException' was thrown.
Command has been canceled on node CLUSTERHEAD. Message:Command output proxy has
failed.
Command has been canceled on node CLUSTERPC01. Message:Command output proxy has
failed.
Command has been canceled on node CLUSTERPC02. Message:Command output proxy has
failed.

-------------------------- Summary --------------------------
0 Nodes succeeded
3 Nodes failed:CLUSTERHEAD,CLUSTERPC01,CLUSTERPC02
****************

Sorry, but i dont know what it means. Can anyone help me???

Thank you, Albert

MS MPI on non-HPC Servers

Wed, 01 Jul 2009 23:29:45 Z

I am migrating an HPC application from Linux to Windows platform. Got a few questions -

1) My platform has a few nodes, and they currently have Windows 2008 Embedded Server on it...will the MS-MPI in the HPC 2008 Pack work with this OS. In otherwords, can I run MPI application without the HPC Server's 'Job Scheduler' ?

2) What's link to the MS-MPI Reference documents - i.e document which details all the MPI calls, and more.
I am especially interested in the 'One Sided Communication' calls, with 'Passive' sychronization, that uses RDMA feature of Infiniband.

Appreciate a prompt response.

Thanks,
Sashi

API to determine MSMPI version

Thu, 23 Jul 2009 19:49:39 Z

Is there a way to determine from within my app what version of the msmpi dll was loaded (CCS/2003 or HPC server/2008) ?
MPI_Get_Version appears to be for querying the version of the MPI standard that is supported (2.0 for both ?).

I can't find the MS MPI API on MSDN, and browsing mpi.h didn't reveal anything to me.

Thanks,
Chris

unresolved external symbol MPI_INIT referenced in function MAIN___fpi.obj_

Fri, 10 Jul 2009 05:54:05 Z

aHi every one:
I am tried to compile a simple MPI (fpi) program using MSMPI in VS2008 and intel fortran.
The code is the example code named "fpi.f" in MPICH2 installation folder.
The compiler is intel fortran 11 which integrated in VS 2008.
Before I start compile the program, several steps of setting are made:
1. set "C:\Program Files\Microsoft HPC Pack 2008 SDK\Include" as additional include directories in Fortran tab.
2. set "C:\Program Files\Microsoft HPC Pack 2008 SDK\Lib\amd64" as additional library directories in Link tab.
3. set "msmpi.lib msmpifec.lib" as additional dependencies in link tab.
Then I compiled the program to X64 program. but the compiler reported following errors:
Error    1    error LNK2019: unresolved external symbol MPI_INIT referenced in function MAIN__    fpi.obj
Error    2    error LNK2019: unresolved external symbol MPI_COMM_RANK referenced in function MAIN__    fpi.obj
Error    3    error LNK2019: unresolved external symbol MPI_COMM_SIZE referenced in function MAIN__    fpi.obj
Error    4    error LNK2019: unresolved external symbol MPI_BCAST referenced in function MAIN__    fpi.obj
Error    5    error LNK2019: unresolved external symbol MPI_FINALIZE referenced in function MAIN__    fpi.obj
Error    6    error LNK2019: unresolved external symbol MPI_REDUCE referenced in function MAIN__    fpi.obj
It looks like the program can not link with the "msmpi.lib" . I also tried to link the program to "C:\Program Files\Microsoft HPC Pack 2008 SDK\Lib\i386". the compiler also report the above errors. By the way, if I link the program with MPCIH2 then everything is fine.
Can any can tell me how to set the link library for Fortran?
Many thanks.

How to link msMPI with intel fortran?

Fri, 10 Jul 2009 05:57:08 Z

WHPCS2008 MPI diagnostics failing randomly

Fri, 19 Jun 2009 06:10:44 Z

Hi,

I'm trying to help a customer who is running WHPCS 2008 on an SGI cluster (of Altix XE310 servers). WHPCS 2008 passes all the built-in diagnostic tests except for the MPI ping-pong and lightweight throughput tests. These fail on random nodes. The cluster is setup with a private GigE network for management and an Infiniband (IB) network for MPI traffic. The IB network is connected to a Cisco SFS7000d switch which us running the subnet manager.

When running the MPI ping-pong and lightweight throughput tests, multiple nodes fail randomly. For example, the tests are run on a subset of the cluster works OK, but after they get up to around 14 nodes, the test will fail on a random node. (The cluster has about 20 nodes in total.) However, a "clusrun" of simple commands works OK across all nodes. The log output from the MPI tests is as follows:

Time Message

18/06/2009 12:32:06 PM Reverted

18/06/2009 12:32:05 PM The operation failed due to errors during execution.

18/06/2009 12:32:05 PM The operation failed and will not be retried.

18/06/2009 12:32:05 PM ---- error analysis -----

18/06/2009 12:32:05 PM

18/06/2009 12:32:05 PM mpi has detected a fatal error and aborted mpipingpong.exe

18/06/2009 12:32:05 PM [5] on MICRO15

18/06/2009 12:32:05 PM

18/06/2009 12:32:05 PM ---- error analysis -----

18/06/2009 12:32:05 PM

18/06/2009 12:32:05 PM [6-19] terminated

18/06/2009 12:32:05 PM

18/06/2009 12:32:05 PM CH3_ND::CEndpoint::ConnReqFailed(407): [ch3:nd] INDConnector::Connect to 192.168.2.111:1 failed with 0x80070043

18/06/2009 12:32:05 PM CH3_ND::CEndpoint::Connect(236)......:

18/06/2009 12:32:05 PM CH3_ND::CEnvironment::Connect(400)...:

18/06/2009 12:32:05 PM MPIDI_CH3I_VC_post_connect(426)......: MPIDI_CH3I_Nd_connect failed in VC_post_connect

18/06/2009 12:32:05 PM MPIDI_CH3_iSendv(239)................:

18/06/2009 12:32:05 PM MPIDI_EagerContigIsend(519)..........: failure occurred while attempting to send an eager message

18/06/2009 12:32:05 PM MPIC_Sendrecv(120)...................:

18/06/2009 12:32:05 PM MPIR_Allgather(487)..................:

18/06/2009 12:32:05 PM MPI_Allgather(864)...................: MPI_Allgather(sbuf=0x000000000022F750, scount=128, MPI_CHAR, rbuf=0x0000000000B71100, rcount=128, MPI_CHAR, MPI_COMM_WORLD) failed

18/06/2009 12:32:05 PM Fatal error in MPI_Allgather: Other MPI error, error stack:

18/06/2009 12:32:05 PM [5] fatal error

18/06/2009 12:32:05 PM

18/06/2009 12:32:05 PM [0-4] terminated

18/06/2009 12:32:05 PM

18/06/2009 12:32:05 PM [ranks] message

18/06/2009 12:32:05 PM job aborted:

18/06/2009 12:32:05 PM

18/06/2009 12:31:50 PM Connecting to scheduler service on node micro.

Has anyone seen this type f problem? Any suggestions for resolving it?

Thanks very much.

Regards,

David

Problems with while running an MPICH job

Sat, 16 May 2009 07:59:37 Z

when I run a code Like this on cluster with two nodes using one core on each:

use mpi

implicit real*8(a-h,o-z)

call MPI_Init ( ierr )

call MPI_Comm_rank ( MPI_COMM_WORLD, my_id, ierr )

CALL MPI_COMM_SIZE ( MPI_COMM_WORLD, NUM_PROCS, IERR )

print*,my_id

call mpi_barrier(mpi_comm_world,ierr)

call MPI_Finalize ( ierr )

end

It does the first printing , but on mpi_barrier it return error including message: rank 0 unable to connect to rank 1 using business card (port=30776,....)

I can not use any of the command mpi_bcast, mpi_reduce ,... they all don't work, but it can read from each node and print, but

It comes to communication, It returns this error

I use windows HPC cluster

I have run this code with this version of mpi on win XP and vista PC and laptop computers. please help me pass the conundrum.

Thanks

Parsa

problems linking fortran 90 code with ms mpi

Tue, 19 May 2009 11:05:30 Z

hi,
I installed microsoft compute cluster pack and there is no msmpif.lib or any other fortran library.
I used msmpi.lib and It returns that "check your include directories: module [mpi]"
It is said that msmpi works for fortran90 too.
please help me out

Regards
Parsa

cluster job manager fails all the tasks with the program I try to launch "Could not load file or assembly 'tmpWin32withCLR"

Thu, 27 Nov 2008 16:39:34 Z

I've created the parallel program for computer cluster managed by WCCS2003. Debugging using SDK on separate computer is absolutly succesfull. The only problem is that cluster job manager fails all the tasks with the program I try to launch. Using the only node that was used for developing returns the same result.

The program has been developed using managed C++ (CLR Console Application) and MPI in Microsoft Visual Studio 2005.

This exception occurred when I try to execute it on cluster:

Unhandled Exception: System.IO.FileLoadException: Could not load file or assembly 'tmpWin32withCLR, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. Failed to grant minimum permission requests. (Exception from HRESULT: 0x80131417) File name: 'tmpWin32withCLR, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' ---> System.Security.Policy.PolicyException: Required permissions cannot be acquired.

at System.Security.SecurityManager.ResolvePolicy(Evidence evidence, PermissionSet reqdPset, PermissionSet optPset, PermissionSet denyPset, PermissionSet& denied, Boolean checkExecutionPermission)

at System.Security.SecurityManager.ResolvePolicy(Evidence evidence, PermissionSet reqdPset, PermissionSet optPset, PermissionSet denyPset, PermissionSet& denied, Int32& securitySpecialFlags, Boolean checkExecutionPermission)

Please help or give instructions how I can go in to get around this error.

-trace permission trouble

Mon, 22 Jun 2009 18:42:38 Z

I'm currently trying to do a mpiexec -trace MPIApp.exe but I am getting the error.

Aborting: failed to start tracing on ...
Error (5) Access is denied.

I know the problem is that I don't have access to the cluster, and the cluster administrator does not want to give me access to use cluster manager, but he asked me to post here if there was a way around it. Can he give me the capabilities of using -trace without giving me access to the cluster manager? Can we setup a certain job template that I can use the -trace function?

Any help is appreciated,

AJ

32-bit SDK for HPC Server 2008 fails --- "The procedure entry point GetProcessIdOfThread could not be located..."

Wed, 08 Oct 2008 19:18:26 Z

I'm wondering if anyone has run into the following with the RTM of the SDK for HPC Server 2008. I have various 64-bit MPI apps that run fine with the 64-bit version of the new SDK. But I wanted to do some demos under 32-bit Windows XP Pro, so I rebuilt the apps for 32-bit, installed the 32-bit SDK on the Windows XP machine, and then tried to run my apps via mpiexec:

mpiexec -n 4 MyMPIApp.exe

Immediately, a dialog pops up that says "The procedure entry point GetProcessIdOfThread could not be located in the dynamic link library KERNEL32.dll". I close the dialog, and the console window then displays:

ReadFile() failed, error 109
Error: unable to start the local smpd manager

I disabled the firewall, same result. Rebooted, same result. Installed MPI.NET and ran one of my MPI.NET apps, exactly the same error. If I uninstall, and install the SDK for Compute Cluster Pack, the error goes away. So something is up with the new SDK.

Any ideas? Thanks!

how to select cores from nodes to run job on multiple nodes.

Wed, 17 Jun 2009 07:29:54 Z

Hi all,

    I have 2 nodes in my cluster with 4 core on each node.

    I have one exe file called sleep.exe. I submitted job with Job submit /numnodes:2 mpiexec -cores 2 sleep.exe then it was open 2 sleep.exe processes on each node.

    And I have a 4 core Ansys CFX job, and I want to run this job on 2 core from first node and other 2 core from second node.

    I have tried with job submit /numnodes:2 /workdir:<working directory path> /stdout:out.log /stderr:error.log mpiexec -cores 2 cfx5solve.exe -v -def <.def file> -start-method MSMPI -part 4. Then the job got failed and generated below error information in error.log file

"An error has occurred in cfx5solve:

Error reported by IO module: readIntFmtData: (fgets failed) syserr:: No
error

An error has occurred in cfx5solve:

Error reported by IO module: iif_set_lock: error reading lock file
//litocmaster/work/benchmark.def.lck: No error

An error has occurred in cfx5solve:

Neither Start Command nor Option is defined for start method MSMPI; check
that you have given the method name correctly.

An error has occurred in cfx5solve:

Neither Start Command nor Option is defined for start method MSMPI; check
that you have given the method name correctly.

Can't call method "name" on an undefined value at C:\Program Files\ANSYS Inc\v110\CFX\bin\/perllib/CFX5/Job/Settings.pm line 2464.
An error has occurred in cfx5solve:

Neither Start Command nor Option is defined for start method MSMPI; check
that you have given the method name correctly.

An error has occurred in cfx5solve:

Neither Start Command nor Option is defined for start method MSMPI; check
that you have given the method name correctly.

Can't call method "name" on an undefined value at c:\Program Files\ANSYS Inc\v110\CFX\bin\/perllib/CFX5/Job/Settings.pm line 2464.
An error has occurred in cfx5solve:

Neither Start Command nor Option is defined for start method MSMPI; check
that you have given the method name correctly.

An error has occurred in cfx5solve:

Neither Start Command nor Option is defined for start method MSMPI; check
that you have given the method name correctly.

Can't call method "name" on an undefined value at c:\Program Files\ANSYS Inc\v110\CFX\bin\/perllib/CFX5/Job/Settings.pm line 2464.
An error has occurred in cfx5solve:

Neither Start Command nor Option is defined for start method MSMPI; check
that you have given the method name correctly.

An error has occurred in cfx5solve:

Neither Start Command nor Option is defined for start method MSMPI; check
that you have given the method name correctly.

Can't call method "name" on an undefined value at C:\Program Files\ANSYS Inc\v110\CFX\bin\/perllib/CFX5/Job/Settings.pm line 2464. "

But when I submit the job with out mpiexec option, The job is running fine on available resources.

Will mpiexec works with all applications or not. Please give me suggessions on this. And any body tested this kind of scenario with Starccm application.

Regards,
P. Kalyan Rao

Getting mpiexec.exe to run simple hello world

Fri, 27 Jun 2008 15:42:21 Z

I have just installed the HPC Pack 2008 Beta 2, and I am trying to execute a simple hello world program with the command:

mpiexec.exe -d 3 C:\Hello\Hello.exe

The command just hangs, and the last couple lines of the debug output say:

ERROR: AcceptSecurityContext failed with error 0x80090308
ERROR: unable to process the sspi server iteration buffer

If anyone has any ideas what my problem might be, I would greatly appreciate some help getting going with this.

a sudden interuption of the MPI job

Thu, 11 Jun 2009 10:46:39 Z

I am runing a MPI job on MSHPC cluster. it is runing ok at the beginging of the calculation, the output was written correctly, but the job was just suddenly stoped in the middle. The error message was as follows. I checked the code line where the "tag=273" pointed at, it is correct. could anybody know the reason? Cheers

Cindy

****************************************
job aborted:
rank: node: exit code: message
0: SB-NODE011: fatal error: Fatal error in MPI_Recv: Other MPI error, error stack:
MPI_Recv(179)...........: MPI_Recv(buf=0x0000000001A4F408, count=1, MPI_INTEGER, src=4, tag=273, MPI_COMM_WORLD, status=0x0000000000712400) failed
MPIDI_CH3I_Progress(165): handle_sock_op failed
handle_sock_read(530)...:
ReadFailed(1518)........: An existing connection was forcibly closed by the remote host. (errno 10054)
1: SB-NODE011: fatal error: Fatal error in MPI_Bcast: Other MPI error, error stack:
MPI_Bcast(791)..........: MPI_Bcast(buf=0x0000000001A4F408, count=1, MPI_INTEGER, root=0, MPI_COMM_WORLD) failed
MPIR_Bcast(192).........:
MPIC_Recv(98)...........:
MPIC_Wait(321)..........:
MPIDI_CH3I_Progress(165): handle_sock_op failed
handle_sock_read(530)...:
ReadFailed(1518)........: An existing connection was forcibly closed by the remote host. (errno 10054)
2: SB-NODE011: fatal error: Fatal error in MPI_Bcast: Other MPI error, error stack:
MPI_Bcast(791)..........: MPI_Bcast(buf=0x0000000001A4F408, count=1, MPI_INTEGER, root=0, MPI_COMM_WORLD) failed
MPIR_Bcast(192).........:
MPIC_Recv(98)...........:
MPIC_Wait(321)..........:
MPIDI_CH3I_Progress(165): handle_sock_op failed
handle_sock_read(530)...:
ReadFailed(1518)........: An existing connection was forcibly closed by the remote host. (errno 10054)
3: SB-NODE011: fatal error: Fatal error in MPI_Bcast: Other MPI error, error stack:
MPI_Bcast(791)..........: MPI_Bcast(buf=0x0000000001A4F408, count=1, MPI_INTEGER, root=0, MPI_COMM_WORLD) failed
MPIR_Bcast(192).........:
MPIC_Recv(98)...........:
MPIC_Wait(321)..........:
MPIDI_CH3I_Progress(165): handle_sock_op failed
handle_sock_read(530)...:
ReadFailed(1518)........: An existing connection was forcibly closed by the remote host. (errno 10054)
4: SB-NODE013: 157: process exited without calling finalize
5: SB-NODE013: terminated
6: SB-NODE013: terminated
7: SB-NODE013: terminated

---- error analysis -----

4: sotoncaa.exe ended prematurely and may have crashed on SB-NODE013
**********************************************************

C#, MPI, and VS2008 trouble

Wed, 10 Jun 2009 20:04:41 Z

I found a simple program that is basically a hello world! program for C# in VS2008.

I am trying to submit this job to my cluster using mpiexec -n 10 MPIHello.exe. on the HPC Job Manager and I am currently getting this error.

Unhandled Exception: System.IO.FileNotFoundException: Could not load file or assembly 'MPI, Version=1.0.0.0, Culture=neutral, PublicKeyToken=29b4a045737654fe' or one of its dependencies. The system cannot find the file specified.
File name: 'MPI, Version=1.0.0.0, Culture=neutral, PublicKeyToken=29b4a045737654fe'
at MPIHello.Program.Main(String[] args)

I am executing the program as a release version and I added the MPI.NET reference into VS2008 by the tutorial I found here.

http://www.osl.iu.edu/research/mpi.net/documentation/tutorial/hello.php

I submitted a simple hello program using just C# with no problems which leads me to believe that I am missing a couple configuration steps.

Any help with my problem would be appreciated.

AJ Fret

Performance problem on multi-core system with multithreaded applications

Thu, 28 May 2009 06:59:40 Z

Hello all,
I have two different multi-threaded applications that when run _alone_ on 16-core Intel Xeon box with Linux, take 4 hours(app1) and 10 hours(app2) respectively. With MPI, I specify 8 threads for each of the application and fire them simultaneously.
Now, when app1 gets completed, app2 runs with its 8 threads. I am wondering if it is possible to create more threads dynamically when you have resources lying idle? Here app2 just doesn't use remaining 8 cores, right?

Please let me know if I am assuming something wrong.

Thanks,
Blue

Can MPI take raw data stream?

Thu, 09 Apr 2009 03:47:10 Z

This is also posted in the developer section as I wasn't sure which forum would be the best to find the solution to my issue....any help VERY appreciated, TIA!

We have CCS 2003 SP1 implemented currently, but may need to move to HPC 2008 for this, which is part of the question. The goal is to take a streams of raw data, currently transported by tcpip/winsock, convert this into something that can be passed through the/a scheduler so that it can manage near real-time analysis, very minimal latency through the cluster which may alter the stream slightly, and then be able to convert and send traffic back through to the raw data interface.

There have been limitations with processing through one serial channel, so we'd like to run multiple channels and we think we can produce what we need, as long as we can get the data in and out fast enough.

Setup:
Right now we have a small cluster, but would only be used for this one purpse, 2 nodes, quad-cores, with 8Gb of RAM each, Cisco 2960G, teamed NIC interfaces, and TOE cards as well. All completely isolated to this task and the source/destination interface plugged into the switch. Although I'm not sure if this is enough firepower to do it?

We intend to run a job that opens a socket and streams the data into memory, which would be very small, but lots of data passing, still only 2Gb for the entire duration, each stream would represent one channel, which we hope can be spread across multiple nodes. So this brings us to the challenges....

How do we write the interface that recieves the data? Does this need to run through MPI? Can the interface be real-time, or does it have to be batched in? i.e. do we need to try and break the data up in smaller transactions and batch it through to get the result? How do we get the cluster to accept the data stream and allow us to view it and manipulate it?

Thanks so much for your help on this!

MPI job aborts for unexplained reasons

Tue, 12 May 2009 22:12:14 Z

Hi there--

We're upgrading to HPC Pack 2008 from CCS Server 2003, moving our existing C++ and C# MPI programs to HPC Pack 2008. Generally things seem to progress, but one C++ MPI program is failing.

We fire off a job, and things work for a while, then we get a crash.

In the task properties, the Error message is: "Task failed during execution with exit code -4. Please check task's output for error details."

The "Output:" box in the HPC Job manager is empty. However if we look in the log file that captured stdout/stderr from the job, we see:

   Aborting: smpd on MT-CCS-01 failed to communicate with smpd on MT-CCS-11
   Other MPI error, error stack:
   ReadFailed(1317): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.  (errno 10060)

No indiciation of failures on the logs at mt-ccs-11. Under 2003, I would look for %windir%\pchealth\ErrorRep\UserDumps, but that directory isn't here. Should I enable error reporing? Any suggestions on how to diagnose?

Thanks for your help!