Insert & Update SQL Table from XML file data using VB.NET 2010

I have been researching and writing/re-writing a program to do this task for a week now. I need some collaboration on this to maybe bring up something I haven't though of before. Specifically, we have an auto-generated XML file sent to us daily with ~70k records (~75MB in size.) I was asked to make a table on one of the servers (SQL) which contains this information so that it can queried. Also, this program must Update existing records (if data has changed) and Insert new records DAILY. Records must not be deleted from the db


Here is the list of methods I have attempted (so far) and reasons they did not work.


  1. SQLXMLBulkLoad - This worked excellent for importing the data. However, the limitation of the Bulk Load class is that it can not Update and/or Insert. Time for a re-write.

    SQLXMLBulkLoad - 這非常適合導入數據。但是,Bulk Load類的限制是它不能更新和/或插入。是時候重寫了。

  2. SQL OpenRowSet (using SQLCommand, etc.) - This does not work because the server, program, and XML file will all 3 be on different computers. These devices CAN be configured to allow each other access to the file (specifically the server), however this method was deemed "Not realistic, too much overhead" Time for a re-write.

    SQL OpenRowSet(使用SQLCommand等) - 這不起作用,因為服務器,程序和XML文件都將在不同的計算機上。這些設備可以配置為允許彼此訪問文件(特別是服務器),但是這種方法被認為是“不現實,開銷太大”重寫時間。

  3. DataSet Merge, then TableAdapter.Update - This method intitially seemed like it would definitely work. The idea is simple, use DataSet.XMLRead() method to put the XML data into a table in the dataset, then just add the SQL table to the dataset (Using SQLCommand, etc.), merge the two tables, and then use Table Adapter to Update/Insert the table into the existing SQL table. This method seems not to work because the XML file has two nodes (columns) which contains dates. Unfortunately, there is not a uniform Date datatype between SQL and XML. I even attempted changing all of the date formats from the XML file to the DateTime SQL format, which worked, but still deemed a datatype mismatch exception upon running.

    DataSet Merge,然后是TableAdapter.Update - 這個方法在內部看起來肯定會起作用。這個想法很簡單,使用DataSet.XMLRead()方法將XML數據放入數據集中的表中,然后只需將SQL表添加到數據集(使用SQLCommand等),合並兩個表,然后使用Table適配器更新/將表插入現有SQL表。此方法似乎不起作用,因為XML文件有兩個包含日期的節點(列)。不幸的是,SQL和XML之間沒有統一的Date數據類型。我甚至嘗試將所有日期格式從XML文件更改為DateTime SQL格式,這種格式有效,但在運行時仍被視為數據類型不匹配異常。

At this point, I am out of ideas. This seems to be a task that has surely been done before. I am not necessarily looking for someone to write this code for me (I am fully capable of this), I just need some collaboration on the topic.


Thank You


1 个解决方案



We do something similar with database imports received in XML format, and all I do is pass the XML directly to a stored procedure and then shred the XML using XQuery and OPENXML.


Both of those technologies allow you to query XML in SQL as if it was a table in your database. Taking that approach, you can just pass your XML to a script or stored procedure, query it in SQL, and insert the results wherever you need them. Anecdotally, OPENXML is better for processing large XML files, but you could try both and see how they work for you. Below is an example using OPENXML and a simple merge statement.


create procedure ImportXml
    @importXml xml
--with OPENXML you have to prepare the document before you use it
--this is unecessary with XQuery
DECLARE @idoc int
EXEC sp_xml_preparedocument @idoc OUTPUT, @importXml;

--this is just a typical Merge statement that will update data if it exists
--and insert it if it does not
merge NormalDataTable
     --here is where you are querying the XML document directly.  You can 
     --see, it works just like a SQL statement, with a special syntax for
     --specifying where to get data out of the XML document and how to map
     --it to a table structure
     select *
     from openxml(@idoc, '/Root/Element')
         ElementID int '@ElementID',
         ElementValueName varchar(50) '@ElementValueName'
) source
on NormalDataTable.ElementID = source.ElementID
when not matched then
    insert ...
when matched then
    update ...

exec sp_xml_removedocument @idoc



粤ICP备14056181号  © 2014-2021