CRM Import - Data file too large

All Replies

• Monday, December 29, 2008 4:57 PM

  Owner

   

   
  

  0
  I am not aware of any other options.  You will want to include only a few rows of data in your sample file.  You may need to sort your data file on the picklist values and complete several interations of the import process.  That will allow you to modify your data map to accommodate the values needed.

   

  • Reply
  • Quote

   
• Monday, December 29, 2008 5:04 PM

   

   
  

  0

  Do you think that it would be a better option for me to use the Data Migration Manager? We were having a little bit of trouble with the .cvs files being compatable with the Migration tool, but the import process will be extremely lengthy if I have to go through and edit the information, and break apart the different 'picklist' sections.

   

  • Reply
  • Quote

   
• Monday, December 29, 2008 5:49 PM

  Owner

   

   
  

  0

  I would consider trying the DMM. 

   

  One other option that just came to mind is that you can go to Settings, Data Management, Data Maps, and export your data map.  It exports as a xml file.  You can try to manually add the additional picklist values needed using the same format as what currently exists in the data map.  You can then save your file and import it for use.  Keep a backup copy of the original in the event something does not go as expected.  It can get a little tricky to get the xml in the right format.  I suggest adding just one value manually, import the map and try it with one or two records as a test.  If that works then move forward with adding additional values until you are comfortable that you can add all the values needed.

   

  Here is a link to more information about modifying the xml file.  I hope this helps.

   

  http://blogs.msdn.com/crm/archive/2008/02/08/pick-list-and-bit-data-migration-using-dmm.aspx

   

  • Reply
  • Quote

   
• Tuesday, January 06, 2009 11:19 AM

   

   
  

  0
  The file upload size is limited to the IIS file size . You can try changing the max request limit in Web.Config.

   

  • Reply
  • Quote

   
• Wednesday, January 07, 2009 10:29 AM

   

   
  

  0

  Hi Guys

   

  I am under the impression that when completing the data map, you do not need to include any data, just the header row in order to carry out the mappings.

   

  With regards to the picklist values, again I do not believe that these are required in the data mapping as the list attributes are actually drawn from the DB rather than your sample file.

   

  If you try creating your data map with just the header row, you should find it work, then run the import as normal with your complete data CSV

   

  hope that helps

   

  http://allanvirtualitbiz.blogspot.com

   

  • Reply
  • Quote

   
• Wednesday, January 07, 2009 1:09 PM

  Owner

   

   
  

  0

  With the Import Tool, there is a place to map attribute values.   The tool provides the ability to map attribute values from the file to attribute values that exist in the db.

   

  In the DMM if the values are a complete match then the DMM will auto generate them or add them if the option to adjust values was selected when initiating the DMM.

  • Reply
  • Quote

   
• Friday, January 09, 2009 8:37 PM

   

   
  

  0

  Thank you for all of the recommendations.  After reviewing everything, I believe that working with the DMM is the best bet in our case.  However, I'm having trouble with the migrations.  Every time I run the DMM, I get an error that states "An error has occurred.  Restart the Data Migration Manager."  I have yet to get a successful migration, and I have yet to find a solution.

   

  Let me know if you have any suggestions! Thank you!!

  • Reply
  • Quote

   
• Friday, January 09, 2009 9:27 PM

  Owner

   

   
  

  0
  Try installing the DMM on a different PC and select the installation type that does not connect directly to the database.  Once it is installed on a new PC, create a simple test file with 1 lead, contact or account record and see if the DMM works as expected.

   

  • Proposed As Answer by Donna EdwardsMVP, Owner Tuesday, June 30, 2009 3:40 PM
  •  
  • Reply
  • Quote

   
• Monday, June 01, 2009 8:30 PM

   

   
  

  0
  In my experience that generally indicates flaws in the source csv file (or files) - embedded carriage returns or non-printing characters are the most common culprits, usually found in Comment/Description/Note fields. 
  
  I often find it useful to break large files down into smaller batches for processing through the DMM, easier to isolate the data issues that way.  Still, it can be a pain to find those things, the file will often open up OK in Excel; in Notepad you can sometimes see what's wrong, but scrolling through thousands of lines of text is not a very appealing or elegant way to manage data. 
  
  If you can import the csv to a SQL Server database you can scrub out offending characters by running the text blobs through a udf like this:
  
  create function [dbo].[ConvertNText]( @input ntext )
  returns varchar(max)
  as
  begin
   declare @output varchar(max)
   set @output = replace(convert(varchar(max), @input), '"', '')
   while PATINDEX('%[^a-zA-Z0-9 '''''']%', @output) <> 0
   begin
    set @output = REPLACE(@output, SUBSTRING(@output, PATINDEX('%[^a-zA-Z0-9 '''''']%', @output), 1), '')
   end
   return @output
  end
  
  Then export back out to csv and fire up the DMM again.
  • Reply
  • Quote