locked
Using SQL triggers + Job Agent + SSIS to sent XML data to ftp servers or OOTB CRM Solution? RRS feed

  • Question

  • Hello Community,

    we are looking for a solution to automatically transfer data (in XML format) from our CRM System to the FTP-Server of our webhoster. To this point we intent to go this way:

    1. Trigger on the FilteredView (on update/insert/delete). Within the trigger, a SQL job agent is started.

    2. SQL Job Agent starts a password secured SSIS package.

    3. The SSIS package uses a select clause only and stores the data as an XML file on disk. This XML file is sent within a  FTP task to the webhoster's server.

    I know that this is an unsupported way, but unfortunately we only have the community edition of VS + BIDS. Plug-In or custom workflow development is therefore not possible.

    Is there any Dynamics solution OOTB to automatically sent XML data to servers?

    Which kind of problems (security, cached data...) may occur with the above described "solution"?

    Thanks in advance

    oppemaniac

    Monday, May 30, 2016 11:59 AM

Answers

  • You've already identified that this is unsupported. I would also worry about the performance implications of running an SSIS package via a trigger.

    Instead, I'd use a different mechanism to run the process, which is fully supported:

    • Still use an SSIS package
    • Create a custom entity in CRM - let's call it triggerPackage
    • Create a workflow on the data operation(s) that should send data. In that workflow create a triggerPackage record
    • Add a step in the SSIS package to check for any active triggerPackage records. If there are, then create the XML, and as a final step, deactivate the active triggerPackage record(s)
    • Run the SSIS package via a SqlAgent job that runs on a schedule - maybe every 5 minutes

    You won't find any OOTB solution to send xml data, as you'd need to specify what data to send, and how to structure it. As you're already using SQL, I'd use a FOR XML clause on the SELECT statement to generate the XML, which you can then write to a file and send via FTP within SSIS.


    Microsoft CRM MVP - http://mscrmuk.blogspot.com/ http://www.excitation.co.uk

    • Marked as answer by oppemaniac Thursday, June 2, 2016 1:40 PM
    Tuesday, May 31, 2016 5:02 PM
    Moderator
  • The main reason for suggesting a scheduled job based on checking data is that this would be supported, whereas adding triggers on the CRM tables is not supported. Performance was a secondary consideration. As it seems like the total processing requirements are low, then the overall performance is not an issue, but you would want to minimise any delay in saving a record. By default, CRM will timeout an update/create after 30 seconds.

    I've never tried starting an SSIS package in a trigger, but if it runs synchronously then that will delay the overall operation, whereas I think running the package as a SqlAgent job from a trigger will probably be asynchronous


    Microsoft CRM MVP - http://mscrmuk.blogspot.com/ http://www.excitation.co.uk

    • Marked as answer by oppemaniac Thursday, June 2, 2016 1:40 PM
    Wednesday, June 1, 2016 3:48 PM
    Moderator

All replies

  • You've already identified that this is unsupported. I would also worry about the performance implications of running an SSIS package via a trigger.

    Instead, I'd use a different mechanism to run the process, which is fully supported:

    • Still use an SSIS package
    • Create a custom entity in CRM - let's call it triggerPackage
    • Create a workflow on the data operation(s) that should send data. In that workflow create a triggerPackage record
    • Add a step in the SSIS package to check for any active triggerPackage records. If there are, then create the XML, and as a final step, deactivate the active triggerPackage record(s)
    • Run the SSIS package via a SqlAgent job that runs on a schedule - maybe every 5 minutes

    You won't find any OOTB solution to send xml data, as you'd need to specify what data to send, and how to structure it. As you're already using SQL, I'd use a FOR XML clause on the SELECT statement to generate the XML, which you can then write to a file and send via FTP within SSIS.


    Microsoft CRM MVP - http://mscrmuk.blogspot.com/ http://www.excitation.co.uk

    • Marked as answer by oppemaniac Thursday, June 2, 2016 1:40 PM
    Tuesday, May 31, 2016 5:02 PM
    Moderator
  • Hey David,

    thank you for your reply. In the SELECT statement I already use a FOR XML clause. Since we only update / insert / delete entries in the respective tables on a very irregular basis (maybe two or three days month), I thought that running a "regular" SQL Agent Job is worse compared to a trigger invoked Agent in perfomance terms?

    The SSIS package is started by the SQL JobAgent which itself is started by the trigger. I read in some comment that this way is prefered over starting the SSIS package directly in the trigger. Is that true?

    Thanks

    oppemaniac

    Wednesday, June 1, 2016 12:43 PM
  • The main reason for suggesting a scheduled job based on checking data is that this would be supported, whereas adding triggers on the CRM tables is not supported. Performance was a secondary consideration. As it seems like the total processing requirements are low, then the overall performance is not an issue, but you would want to minimise any delay in saving a record. By default, CRM will timeout an update/create after 30 seconds.

    I've never tried starting an SSIS package in a trigger, but if it runs synchronously then that will delay the overall operation, whereas I think running the package as a SqlAgent job from a trigger will probably be asynchronous


    Microsoft CRM MVP - http://mscrmuk.blogspot.com/ http://www.excitation.co.uk

    • Marked as answer by oppemaniac Thursday, June 2, 2016 1:40 PM
    Wednesday, June 1, 2016 3:48 PM
    Moderator