Error running DryadLINQ samples in "Local" mode RRS feed

  • Question

  • In another thread I posted this as a follow up to issues that surrounded getting the samples to run on the HPC Cluster. That worked, but I still had the issues below when trying to run the same code configured to only execute locally, not on the cluster.
    I have been able to get this sample to run just fine on the cluster, but local mode fails with the following exception:
    Microsoft.Distributed.Linq.DscException was unhandled
     Message=Error communicating with DSC. Error code=1 Message=Failed
     at Microsoft.Distributed.Internal.DscUtils.ThrowIfFailed(DscResult dscResult)
     at Microsoft.Distributed.Internal.DscUtils.CreateStreamFromLocalFiles(Uri fullyQualifiedUri, DataInfo dataInfo, Type recordType, String[] partitionDataFullPaths)
     at Microsoft.Distributed.Internal.DscUtils.CreateStreamFromLocalFiles[TRecord](Uri fullyQualifiedUri, DataInfo`1 dataInfo, String[] partitionDataFullPaths)
     at Microsoft.Distributed.Internal.DistributedDataHelper.CreatePartitionsAsDsc[TRecord](String tmpDir, Uri fullyQualifiedUri, DataInfo`1 dataInfo, IEnumerable`1 dataSets, IReaderWriterFactory`1 readerWriterFactory)
     at Microsoft.Distributed.Linq.AsDistributedInputQueryNode`1.BuildQueryPlanAndCodeDom(BuildQueryState queryExec)
     at Microsoft.Distributed.Linq.UnaryQueryNode`2.BuildQueryPlanAndCodeDom(BuildQueryState queryExec)
     at Microsoft.Distributed.Linq.HashPartitionQueryNode`2.BuildQueryPlanAndCodeDom(BuildQueryState queryExec)
     at Microsoft.Distributed.Linq.UnaryQueryNode`2.BuildQueryPlanAndCodeDom(BuildQueryState queryExec)
     at Microsoft.Distributed.Linq.UnaryQueryNode`2.BuildQueryPlanAndCodeDom(BuildQueryState queryExec)
     at Microsoft.Distributed.Linq.UnaryQueryNode`2.BuildQueryPlanAndCodeDom(BuildQueryState queryExec)
     at Microsoft.Distributed.Linq.DistributedOutputQueryNode`1.BuildQueryPlanAndCodeDom(BuildQueryState queryExec)
     at Microsoft.Distributed.Linq.DistributedQuery`1.ExecuteInternal(DistributedQueryConfiguration config, Uri outputUri, IReaderWriterFactory`1 readerWriterFactory)
     at Microsoft.Distributed.Linq.DistributedQuery`1.ExecuteInternal(Uri outputUri, IReaderWriterFactory`1 readerWriterFactory)
     at Microsoft.Distributed.Linq.DistributedQuery.Execute[TSource](DistributedQuery`1 source)
     at Microsoft.Distributed.Linq.DistributedQuery`1.<GetEnumerator>d__1.MoveNext()
     at DupPic2.Program.Main(String[] args) in D:\Dryad-SampleCode-CTP1\ProgrammingGuideSamples\DupPic2\DupPic2\Program.cs:line 28
     at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
     at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
     at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
     at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
     at System.Threading.ThreadHelper.ThreadStart()
    using System.Security.Cryptography;
    using Microsoft.Distributed.Linq;
    namespace DupPic2
     public class Program
     public static void Main(string[] args)
     string directoryName = @"\\MyServerName\Temp\pics";
     var duplicatedFiles =
     Directory.GetFiles(directoryName, "*.jpg",
     .AsDistributed() //<== comment out this line and everything works fine in local mode, or when running against the cluster
     .Select(filename => new
      hash = GetChecksum(filename),
      name = filename
     .GroupBy(record => record.hash)
     .Where(group => group.Count() > 1)
     .SelectMany(group =>
      group.Select(record =>;
     foreach (var file in duplicatedFiles)
     public static string GetChecksum(string file)
     using (FileStream stream = File.OpenRead(file))
     SHA256Managed sha = new SHA256Managed();
     byte[] checksum = sha.ComputeHash(stream);
     return BitConverter
      .Replace("-", String.Empty);
    I have followed all of the instructions for installation on the client side.  The ONLY thing that I have noticed is that I am running Win7 Ultimate 64 instead of Enterprise 64.  Could this possibly make the difference? (All the docs out there comparing the 2 editions indicate it should work, except for the DryadLINQ install doc itself - which says it MAY)
    I have verified the local XC share and DryadData share exist but that are never being populated.  I created the picture directory, shared it and ran it on the cluster and it works fine.  Commenting out the cluster configuration stuff in the app.config so that it runs locally causes the above exception.  Commenting out the .AsDistributed() line makes the sample work fine locally when the config says to run locally.  VS was launched in Admin mode.
    I have uninstalled and reinstalled the HPC DISC Server, Vertex, and Client in that order (all 64 bits).



    Thanks- Steve
    • Edited by SAFti618 Tuesday, February 1, 2011 8:59 PM
    Tuesday, February 1, 2011 6:51 PM


All replies

  • More information -- this appears to be a SETUP error.

    I believe that this is a SETUP error during creation of database HBCDsc.

    I ran the SQL Profiler during the run and found the following error messages:

    Error: 515, Severity: 16, State: 2

    Cannot insert the value NULL into column 'ReplicationFactor', table 'HPCDsc.dbo.Streams'; column does not allow nulls. INSERT fails.

    The statement has been terminated.

    declare @p1 int

    set @p1=7

    exec sp_prepexec @p1 output,N'@sddl nvarchar(max) ,@name nvarchar(375)',N'DECLARE @tablevar table(id bigint, sddl nvarchar(max), name nvarchar(375), rf tinyint, len bigint, cre datetime2, lu datetime2); INSERT INTO [dbo].[Streams] (SDDL, Name, ReplicationFactor, Length, CreationTime, LastUseTime) OUTPUT inserted.s_id, inserted.SDDL, inserted.Name, inserted.ReplicationFactor, inserted.Length, inserted.CreationTime, inserted.LastUseTime INTO @tablevar VALUES (@sddl, @name, (SELECT TOP(1) replicationFactor FROM [dbo].[ClusterConfig]), 0, GETDATE(), GETDATE()) SELECT id, sddl, name, rf, len, cre, lu FROM @tablevar',@sddl=N'O:S-1-5-21-BLAH:BAD:(A;;FA;;;BA)(A;;FR;;;PU)(A;;FA;;;S-1-5-21-BLAH)',@name=N'hpcdsc://__HPC_Temp/d50454ec-4814-426f-a7c1-3e74fdded9aa' select @p1

    The ClusterConfig table is empty, so the TOP(1) query returns null.

    Inserting a 1 made the sample work, as did inserting a 2.

    QUESTION:  What is the proper replicationFactor value to insert into the ClusterConfig table?  Note that I expected that it should have been specified during SETUP.



    Thanks- Steve
    Tuesday, February 1, 2011 7:18 PM
  • Anyone?
    Thanks- Steve
    Friday, February 4, 2011 11:22 PM
  • For issues concerning the pre-release Dryad software please either log in to the Connect beta website and submit a 'feedback' item or use the beta forum at 
    Saturday, February 5, 2011 12:21 AM