locked
Import data from XML into programtically created data model RRS feed

  • Question

  • Hi,

    I am trying to write a single C# executable that creates a Zentity data model programatically via a XML and XSD file and then import data.

    I need assistance on the following:

    1) I have programatically created a zentity data model by following the CreateFromRdfs example code (i.e. Documentation>en>Zentity.Core>DataModelModule>CreateFromRdfs)

    2) To my knowledge, the next step will be to import data via XML files. The problem I am faced is where can i find sample codes on importing data into a programtically created data model? I have gone thru the PresidentDataImport project (i.e. importing data via XML) and was wondering how to make use of the ZentityContext

    zenContext.USPresident

    given that the model I have is programatically created?

    Some sample codes will really help.

    Thank you.

    Wednesday, April 25, 2012 3:21 PM

Answers

  • "However, I notice the need to use the generated assembly via .NET reflection for retrieving and creating.
    Is there any alternatives than using .NET reflection?"

    Ron, There is no way this can be removed. One has to use refelction to add resources in the Zentity. Using reflection is not recommended but there is no other way.

    There are two times when you want to add resources to the Zentity.

    1) At the time of creating the model as you have done in the above post, and

    2) Any time later after creation of the model.

    first point
    If you are adding the resource at the time of creating the model (in the running app) then reflection is the only way to get it. So its better to defer the addition of resources for later time and use Zentity Resource Manager to accomplish such task of updating/adding/deleting resources.

    Second point
    When you want to add the resource at some later point then there are two options.

    1) Create a sample client application for each type of data model and add the resources through this application without using reflection as you can refer the assembly directly. So I will have to create as many sample applications as number of models, one for each domain.

    2) Using Zentity Applications to add resources. Here again Reflection is the ONLY option. Any automated application of whatever type cannot avoid reflection to work with any resource and resource type. I even thought of creating some clever application (using Unity and/or code generation and/or MVVM pattern) but no way to overcome reflection. It is inherent to the Zenity design or better "concept of RDF resources" where a resource is eternal and we only discover and extend resources.

    I hope i made this point clear and I was also confused about this point.

    Thanks!

    Please let me know if this helps you.

    • Proposed as answer by Kalnemi Tuesday, May 22, 2012 6:21 AM
    • Marked as answer by ronbutan Tuesday, May 22, 2012 12:31 PM
    Tuesday, May 22, 2012 6:20 AM

All replies

  • Thanks for showing interest in Zentity.

    USPresident is a nice example for the sample code you are looking for. Please update the zenDataService reference by giving the updated url of your machine and and the sql server name.

    Also, please see the following threads which have helpful details for similar query.

    (PowerShell Script to "load" a new Data Model Module?)

    (webui error)

    Let me know if you have any further query.

    -Regards

    • Proposed as answer by Kalnemi Thursday, April 26, 2012 6:05 AM
    Thursday, April 26, 2012 6:01 AM
  • Hi Kalnemi,

    Thank you for the reply. I have gone thru the sample code for the USPresident example. Basically, I would like my C# program to do these steps in a single run:

    static void Main (string[] args) { Assembly extensions = SynchronizeAndGenerateAssembly(); //Code to update ZenDataService service reference //Code to import data from respective XML files } private static Assembly SynchronizeAndGenerateAssembly() { using (ZentityContext context = new ZentityContext(string.Format(connString, "Zentity.Core"))) { ResourceType defaultBaseResourceType = context.DataModel.Modules["Zentity.Core"].ResourceType["Resource"]; DataModelModule module = DataModelModule.CreateFromRdfs(defaultBaseResourceType, RDF_PATH, XSD_PATH, false); context.DataModel.Modules.Add(module); context.CommandTimeout = 300; context.DataModel.Synchronize(); byte [] rawAssembly = context.DataModel.GenerateExtensionsAssembly(extensionsAssemblyName, true, new string [] {namespace}, new string[] {namespace}, null); return Assembly.Load(rawAssembly); } }

    The vision for this is to allow the creation of ZDM, import data & create collection all in one single run of the C# executable.

    On another note, is it a better solution to execute Powershell and run the scripts by invoking Powershell via the C# program than doing it thru C# code as I posted above?

    Appreciate your inputs.

    Thursday, April 26, 2012 6:59 AM
  • 1) On another note, is it a better solution to execute Powershell and run the scripts by invoking Powershell via the C# program than doing it thru C# code as I posted above?

    It depends on your convenience. Creating and updating through C# program is more flexible than via xml but requires more work. Also, once you created the data model, either by using the xml or through code, you can use PowerShell script for the rest of the work. For user's  convenience a new document describing all the powershell functions with examples and parameyer usage will be posted soon, latest by Friday.

    2) creation of ZDM, import data & create collection all in one single run of the C# executable

    Yes, you can do this. To get more insight on its implementation you may go through the powershell scripts files.

    -Regards

    Thursday, April 26, 2012 9:45 AM
  • Hi,

    Appreciate the prompt and enlightening reply, I look forward to the new documentation. I have one more doubt. Is running this piece of code

    private static Assembly SynchronizeAndGenerateAssembly()
    {
    using (ZentityContext context = new ZentityContext(string.Format(connString, "Zentity.Core")))
    {
    ResourceType defaultBaseResourceType = context.DataModel.Modules["Zentity.Core"].ResourceType["Resource"];
    DataModelModule module = DataModelModule.CreateFromRdfs(defaultBaseResourceType, RDF_PATH, XSD_PATH, false);
    context.DataModel.Modules.Add(module);
    context.CommandTimeout = 300;
    context.DataModel.Synchronize();
    byte [] rawAssembly = context.DataModel.GenerateExtensionsAssembly(extensionsAssemblyName, true, new string [] {namespace}, new string[] {namespace}, null);
    return Assembly.Load(rawAssembly);
    }

    equivalent to running this cmdlet in Powershell?

    # I understand that this cmdlet uses functions from 2 other .ps1 files
    
    Create-ZentityDataModelClientSetup “C:\Program Files (x86)\Zentity\Samples\US Presidents Sample\RDFs\PivotDemo.xml” “C:\Program Files (x86)\Zentity\Samples\US Presidents Sample\RDFS\PivotDemo.xsd” PivotDemo.Model

    If they are different, what changes are needed for me to achieve the equivalent in C# code?

    Thank you.

    Thursday, April 26, 2012 3:53 PM
  • Hi Ron,

    This code is complete for the creation of custom model but it is not yet complete and it is not updating the "Clientsetup". Please see the thread

    (PowerShell Script to "load" a new Data Model Module?)

    which contains the answer to this query. The above thread also explains exactly which "part" is lacking and where you can find that in powershell scripts along with the function name. Also, you may follow that script code to create your program.

    Please let me know if you have any further query.

    -Regards

    Friday, April 27, 2012 3:25 AM
  • The Zentity Powershell cmdlet is now available for download. Please find it from the this link. Zentity 2.1 Poweshell cmdlet Guide


    • Edited by Kalnemi Friday, May 4, 2012 1:29 AM link check
    Friday, May 4, 2012 1:27 AM
  • Thank you for the reply.

    From my understanding after going thru the samples (i.e. USPresident), the import of data into Zentity database is via XML or database is via C# program.

    I will like to know if there are any other means of performing data import into Zentity? 

    Tuesday, May 8, 2012 12:53 AM

  • Import via xml or import via database (although wrongly) points to the initial storage of the data. You may also import data from other locations. The only care you have to take is the conversion of the data as they are shown in the samples. The other place to import Zentity data is from PubMed online. The Zentity data import sample uses the PubMed online service to retrieve data and then save it according to the way defined in scholarly work.

    Please let me know if you have any further query.

    Tuesday, May 8, 2012 8:32 AM
  • Apologies. My question was vague.

    What I meant to ask was if there are other methods of importing data into Zentity database besides using C# code? Is there a Powershell cmdlet that does this import of data?

    Tuesday, May 8, 2012 2:01 PM

  • No, there is no other way to import data into Zentity. A general command to import data to zentity is not practical if the incoming data are from different sources or having diferent formats.
    Tuesday, May 8, 2012 6:17 PM
  • I am attempting the simplify the "President data import via XML" example comprising of a 4 step process:

    1) running cmdlet in Powershell to create data model (which creates a .dll file in /DataService/bin)

    2) changing visual type to image via Powershell

    3) Import data via running PresidentDataImport.exe with ZenDataService updated with PivotDemo.Modelcontext

    4) Create collection via Powershell

    I would like to condense the above steps into a single step via running a C# program i.e. all one has to do is to run the C# executable and one should be able to view a custom model in visual explorer like in the USPresident example instead of having to perform the 4 steps above manually. I have examined the code below (extracted from documentation) and it seems to fit my requirement with further tweaks to create collection and publishing the resource type.

    However, I notice the need to use the generated assembly via .NET reflection for retrieving and creating.

    Is there any alternatives than using .NET reflection?

    using System;
                using System.Linq;
                using Zentity.Core;
                using System.Reflection;
                using System.Collections;
                
                namespace ZentitySamples
                {
                    public class Program
                    {
                        static string nameSpace = "Namespace" + Guid.NewGuid().ToString("N");
                        const string connectionStringFormat = @"provider=System.Data.SqlClient;
                                metadata=res://{0}; provider connection string='Data Source=.;
                                Initial Catalog=Zentity;Integrated Security=True;MultipleActiveResultSets=True'";
                        const string rdfsPath = @"C:\Sample.rdf";
                        const string dataTypesXsdPath = @"C:\Sample.xsd";
                        static string extensionsAssemblyName = nameSpace;
                
                        public static void Main(string[] args)
                        {
                            Assembly extensions = SynchronizeAndGenerateAssembly();
                            CreateRepositoryItems(extensions);
                            FetchRepositoryItems(extensions);
                        }
                
                        private static void FetchRepositoryItems(Assembly extensionsAssembly)
                        {
                            using (ZentityContext context = new ZentityContext(
                                string.Format(connectionStringFormat, extensionsAssemblyName)))
                            {
                                Console.WriteLine("Getting books...");
                                Type resourceTypeBook = extensionsAssembly.GetType(nameSpace + ".Book");
                                MethodInfo ofTypeMethod = context.Resources.GetType().GetMethod("OfType").
                                    MakeGenericMethod(resourceTypeBook);
                                var customTypeInstances = ofTypeMethod.Invoke(context.Resources, null);
                                foreach (Resource book in (IEnumerable)customTypeInstances)
                                    Console.WriteLine(book.Id);
                
                                Console.WriteLine("\nGetting persons...");
                                Type resourceTypePerson = extensionsAssembly.GetType(nameSpace + ".Person");
                                ofTypeMethod = context.Resources.GetType().GetMethod("OfType").
                                    MakeGenericMethod(resourceTypePerson);
                                customTypeInstances = ofTypeMethod.Invoke(context.Resources, null);
                                foreach (Resource person in (IEnumerable)customTypeInstances)
                                    Console.WriteLine(person.Id);
                
                                Console.WriteLine("\nGetting AuthoredBy relationships...");
                                NavigationProperty authoredByProperty = context.DataModel.Modules[nameSpace].
                                    ResourceTypes["Book"].NavigationProperties["AuthoredBy"];
                                Predicate predicate = context.Predicates.
                                    Where(tuple => tuple.Id == authoredByProperty.Association.PredicateId).First();
                                predicate.Relationships.Load();
                                foreach (Relationship rel in predicate.Relationships)
                                    Console.WriteLine("[{0}] <--{1}--> [{2}]", rel.Subject.Id, predicate.Name,
                                        rel.Object.Id);
                            }
                        }
                
                        private static void CreateRepositoryItems(Assembly extensionsAssembly)
                        {
                            using (ZentityContext context = new ZentityContext(
                                string.Format(connectionStringFormat, extensionsAssemblyName)))
                            {
                                Type resourceTypeBook = extensionsAssembly.GetType(nameSpace + ".Book");
                                PropertyInfo propertyAuthoredBy = resourceTypeBook.GetProperty("AuthoredBy");
                                Type resourceTypePerson = extensionsAssembly.GetType(nameSpace + ".Person");
                
                                var aPerson = Activator.CreateInstance(resourceTypePerson);
                                var aBook = Activator.CreateInstance(resourceTypeBook);
                                var anotherBook = Activator.CreateInstance(resourceTypeBook);
                
                                // AuthoredBy is actually a an EntityCollection<Person>.
                                var instanceAuthoredBy = propertyAuthoredBy.GetValue(aBook, null);
                                Type entityCollectionOfPerson = instanceAuthoredBy.GetType();
                                // Get the "Add" method.
                                MethodInfo methodAdd = entityCollectionOfPerson.GetMethod("Add");
                                // Add author to book.
                                methodAdd.Invoke(instanceAuthoredBy, new object[] { aPerson });
                
                                instanceAuthoredBy = propertyAuthoredBy.GetValue(anotherBook, null);
                                methodAdd.Invoke(instanceAuthoredBy, new object[] { aPerson });
                
                                // Save the entities to repository. 
                                context.AddToResources((Resource)aPerson);
                                context.SaveChanges();
                            }
                        }
                
                        private static Assembly SynchronizeAndGenerateAssembly()
                        {
                            using (ZentityContext context = new ZentityContext(
                                string.Format(connectionStringFormat, "Zentity.Core")))
                            {
                                ResourceType defaultBaseResourceType = context.DataModel.
                                    Modules["Zentity.Core"].ResourceTypes["Resource"];
                
                                DataModelModule module = DataModelModule.CreateFromRdfs(defaultBaseResourceType,
                                    rdfsPath, dataTypesXsdPath, nameSpace, false);
                
                                context.DataModel.Modules.Add(module);
                
                                // Synchronize to alter the database schema.
                                // This method takes a few minutes to complete depending on the actions taken by 
                                // other modules (such as change history logging) in response to schema changes.
                                // Provide a sufficient timeout.
                                context.CommandTimeout = 300;
                                context.DataModel.Synchronize();
                
                                // Generate the module assembly to use.
                                byte[] rawAssembly = context.DataModel.GenerateExtensionsAssembly(
                                    extensionsAssemblyName, true, new string[] { nameSpace },
                                    new string[] { nameSpace }, null);
                
                                return Assembly.Load(rawAssembly);
                            }
                        }
                    }
                }




    • Edited by ronbutan Wednesday, May 9, 2012 2:01 PM
    • Proposed as answer by Kalnemi Tuesday, May 22, 2012 6:21 AM
    • Unproposed as answer by Kalnemi Tuesday, May 22, 2012 6:21 AM
    Wednesday, May 9, 2012 1:59 PM
  • Ron,

    I have not tested the code yet. Let me once test it out at my end. I will update you by tomorrow.

    -Thanks!

    Thursday, May 10, 2012 4:11 PM
  • Hi Ron,

    I tried testing the above at my end with a own created sample. I am getting errors. Can you please provide me the sample.xml and sample.xsd?

    -Regards

    Monday, May 14, 2012 9:28 AM
  • Hi Kalnemi,

    Thank you for taking effort to try the codes. Sample.rdf & Sample.xsd were taken from Zentity.Core.chm>Zentity.Core>DataModelModule>CreateFromRdfs

    Sample.rdf

    <?xml version="1.0"?>
                <!DOCTYPE rdf:RDF [
                  <!ENTITY dt "http://contoso.com/2008/" >
                ]>
                <rdf:RDF
                   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
                   xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
                   xml:base="http://www.sample.org/">
                
                  <rdfs:Class rdf:about="Person">
                  </rdfs:Class>
                
                  <rdfs:Class rdf:about="Employee">
                    <rdfs:subClassOf rdf:resource="Person"/>
                  </rdfs:Class>
                
                  <rdfs:Class rdf:about="Book">
                  </rdfs:Class>
                
                  <rdf:Property rdf:about="AuthoredBy">
                    <rdfs:range rdf:resource="Person"/>
                    <rdfs:domain rdf:resource="Book"/>
                  </rdf:Property>
                
                  <rdf:Property rdf:about="EditedBy">
                    <rdfs:range rdf:resource="Person"/>
                    <rdfs:domain rdf:resource="Book"/>
                  </rdf:Property>
                
                  <rdf:Property rdf:about="Name">
                    <rdfs:range rdf:resource="&dt;string128"/>
                    <rdfs:domain rdf:resource="Person"/>
                  </rdf:Property>
                
                  <rdf:Property rdf:about="Designation">
                    <rdfs:range rdf:resource="&dt;string256"/>
                    <rdfs:domain rdf:resource="Employee"/>
                  </rdf:Property>
                
                  <rdf:Property rdf:about="//Book/Name">
                    <rdfs:range rdf:resource="&dt;string128"/>
                    <rdfs:domain rdf:resource="Book"/>
                  </rdf:Property>
                
                </rdf:RDF>
    

    Sample.xsd

    <xs:schema
                  targetNamespace="http://contoso.com/2008/"
                  xmlns:xs="http://www.w3.org/2001/XMLSchema">
                
                  <xs:simpleType name="string128" id="string128">
                    <xs:restriction base="xs:string">
                      <xs:maxLength value="128"/>
                    </xs:restriction>
                  </xs:simpleType>
                
                  <xs:simpleType name="string256" id="string256">
                    <xs:restriction base="xs:string">
                      <xs:maxLength value="256"/>
                    </xs:restriction>
                  </xs:simpleType>
                
                </xs:schema>
    

    The reason for doing this is to automate the creation of model and importing of data (instead of having to run Powershell scripts) from RDFs that could be generated from other data sources.

    Can PivotCollection be created via code? i.e. Step 4 of President Data Import via XML in Getting Started Guide.docx

    Monday, May 14, 2012 1:16 PM
  • Hi Ron,

    Thanks for this info. I successfully created new data model using your code. But i got error when i tried to add resource using Zentity Rwesource Manager. Can you help me out in this if you have already able to add resource using ZRM. Below i am pasting the details of the errors. The screen shot for the stack trace is above.

    <?xml version="1.0" encoding="utf-8"?>
    <Root>
      <Items>
        <Item>
          <DateTime>05/16/2012 11:13:35</DateTime>
          <Severity>Error</Severity>
          <Content>An error occurred while retrieving Resource Types</Content>
        </Item>
        <Item>
          <DateTime>05/16/2012 11:12:31</DateTime>
          <Severity>Error</Severity>
          <Content>An error occurred while retrieving Resource Types</Content>
        </Item>
        <Item>
          <DateTime>02/13/2012 01:34:57</DateTime>
          <Severity>Error</Severity>
          <Content>Service error log -&gt;
    Message: An error occured while trying to search the resources.
    Context: 
    </Content>
        </Item>
      </Items>
    </Root>


    • Edited by Kalnemi Thursday, May 17, 2012 6:54 AM
    Thursday, May 17, 2012 6:53 AM
  • WebUI Error for book2
    Thursday, May 17, 2012 6:54 AM
  • Hi Kalnemi,

    I added the resources via C# code. I did not use the Zentity Resource Manager (ZRM) UI to add resources. I tried using ZRM on my end (for the first time) but it seems that I cannot add any resources via the UI except view the data models that had been added.

    The codes above for adding data models by reading RDFs files and adding resources to the created resource were taken from the Zentity documentation, Zentity.Core.chm>Zentity.Core>DataModelModule>CreateFromRdfs.

    I am afraid I cannot provide further insights to adding resource via ZRM.

    Monday, May 21, 2012 1:59 AM
  • Hi Ron,

    Thanks for you reply.

    I am working on a complete sample application to show you as Demo and that will also be "modifiable" using ZRM. It will fit the requirements posted in the response from you on 9th May. I need some time, around 2 days, but I will provide you a complete sample by wednesday 23rd May.

    -Regards

    Monday, May 21, 2012 5:04 AM
  • "However, I notice the need to use the generated assembly via .NET reflection for retrieving and creating.
    Is there any alternatives than using .NET reflection?"

    Ron, There is no way this can be removed. One has to use refelction to add resources in the Zentity. Using reflection is not recommended but there is no other way.

    There are two times when you want to add resources to the Zentity.

    1) At the time of creating the model as you have done in the above post, and

    2) Any time later after creation of the model.

    first point
    If you are adding the resource at the time of creating the model (in the running app) then reflection is the only way to get it. So its better to defer the addition of resources for later time and use Zentity Resource Manager to accomplish such task of updating/adding/deleting resources.

    Second point
    When you want to add the resource at some later point then there are two options.

    1) Create a sample client application for each type of data model and add the resources through this application without using reflection as you can refer the assembly directly. So I will have to create as many sample applications as number of models, one for each domain.

    2) Using Zentity Applications to add resources. Here again Reflection is the ONLY option. Any automated application of whatever type cannot avoid reflection to work with any resource and resource type. I even thought of creating some clever application (using Unity and/or code generation and/or MVVM pattern) but no way to overcome reflection. It is inherent to the Zenity design or better "concept of RDF resources" where a resource is eternal and we only discover and extend resources.

    I hope i made this point clear and I was also confused about this point.

    Thanks!

    Please let me know if this helps you.

    • Proposed as answer by Kalnemi Tuesday, May 22, 2012 6:21 AM
    • Marked as answer by ronbutan Tuesday, May 22, 2012 12:31 PM
    Tuesday, May 22, 2012 6:20 AM
  • Hi Kalnemi,

    How is the Demo as mentioned on 21st May 2012 going?

    Friday, June 1, 2012 12:38 AM
  • I am working on it. Its still not completed (nothing great achieved). Needed some more time. I will update on this thread with the source code once finished.

    -Regards

    Friday, June 1, 2012 8:30 AM