Insert & Update SQL Table from XML file data using VB.NET 2010


I have been researching and writing/re-writing a program to do this task for a week now. I need some collaboration on this to maybe bring up something I haven't though of before. Specifically, we have an auto-generated XML file sent to us daily with ~70k records (~75MB in size.) I was asked to make a table on one of the servers (SQL) which contains this information so that it can queried. Also, this program must Update existing records (if data has changed) and Insert new records DAILY. Records must not be deleted from the db

我一直在研究和編寫/重寫一個程序來執行這項任務一周。我需要在這方面進行一些合作,以便提出我以前沒有的東西。具體來說,我們每天都有一個自動生成的XML文件,大約有70,000條記錄(大小約為75MB)。我被要求在其中一個包含此信息的服務器(SQL)上創建一個表,以便查詢它。此外,此程序必須更新現有記錄(如果數據已更改)和插入新記錄每日。不得從db中刪除記錄

Here is the list of methods I have attempted (so far) and reasons they did not work.

這是我嘗試過的方法列表(到目前為止)以及它們不起作用的原因。

  1. SQLXMLBulkLoad - This worked excellent for importing the data. However, the limitation of the Bulk Load class is that it can not Update and/or Insert. Time for a re-write.

    SQLXMLBulkLoad - 這非常適合導入數據。但是,Bulk Load類的限制是它不能更新和/或插入。是時候重寫了。

  2. SQL OpenRowSet (using SQLCommand, etc.) - This does not work because the server, program, and XML file will all 3 be on different computers. These devices CAN be configured to allow each other access to the file (specifically the server), however this method was deemed "Not realistic, too much overhead" Time for a re-write.

    SQL OpenRowSet(使用SQLCommand等) - 這不起作用,因為服務器,程序和XML文件都將在不同的計算機上。這些設備可以配置為允許彼此訪問文件(特別是服務器),但是這種方法被認為是“不現實,開銷太大”重寫時間。

  3. DataSet Merge, then TableAdapter.Update - This method intitially seemed like it would definitely work. The idea is simple, use DataSet.XMLRead() method to put the XML data into a table in the dataset, then just add the SQL table to the dataset (Using SQLCommand, etc.), merge the two tables, and then use Table Adapter to Update/Insert the table into the existing SQL table. This method seems not to work because the XML file has two nodes (columns) which contains dates. Unfortunately, there is not a uniform Date datatype between SQL and XML. I even attempted changing all of the date formats from the XML file to the DateTime SQL format, which worked, but still deemed a datatype mismatch exception upon running.

    DataSet Merge,然后是TableAdapter.Update - 這個方法在內部看起來肯定會起作用。這個想法很簡單,使用DataSet.XMLRead()方法將XML數據放入數據集中的表中,然后只需將SQL表添加到數據集(使用SQLCommand等),合並兩個表,然后使用Table適配器更新/將表插入現有SQL表。此方法似乎不起作用,因為XML文件有兩個包含日期的節點(列)。不幸的是,SQL和XML之間沒有統一的Date數據類型。我甚至嘗試將所有日期格式從XML文件更改為DateTime SQL格式,這種格式有效,但在運行時仍被視為數據類型不匹配異常。

At this point, I am out of ideas. This seems to be a task that has surely been done before. I am not necessarily looking for someone to write this code for me (I am fully capable of this), I just need some collaboration on the topic.

在這一點上,我沒有想法。這似乎是以前肯定已經完成的任務。我不一定要找人為我編寫這段代碼(我完全有能力),我只需要就這個主題進行一些合作。

Thank You

謝謝

1 个解决方案

#1


1  

We do something similar with database imports received in XML format, and all I do is pass the XML directly to a stored procedure and then shred the XML using XQuery and OPENXML.

我們做了類似於以XML格式接收的數據庫導入,我所做的就是將XML直接傳遞給存儲過程,然后使用XQuery和OPENXML碎化XML。

Both of those technologies allow you to query XML in SQL as if it was a table in your database. Taking that approach, you can just pass your XML to a script or stored procedure, query it in SQL, and insert the results wherever you need them. Anecdotally, OPENXML is better for processing large XML files, but you could try both and see how they work for you. Below is an example using OPENXML and a simple merge statement.

這兩種技術都允許您在SQL中查詢XML,就好像它是數據庫中的表一樣。采用這種方法,您只需將XML傳遞給腳本或存儲過程,在SQL中查詢,然后將結果插入到您需要的任何位置。有趣的是,OPENXML更適合處理大型XML文件,但您可以嘗試兩種方式,看看它們如何為您工作。下面是使用OPENXML和簡單合並語句的示例。

create procedure ImportXml
(
    @importXml xml
)
as
--with OPENXML you have to prepare the document before you use it
--this is unecessary with XQuery
DECLARE @idoc int
EXEC sp_xml_preparedocument @idoc OUTPUT, @importXml;

--this is just a typical Merge statement that will update data if it exists
--and insert it if it does not
merge NormalDataTable
using
(
     --here is where you are querying the XML document directly.  You can 
     --see, it works just like a SQL statement, with a special syntax for
     --specifying where to get data out of the XML document and how to map
     --it to a table structure
     select *
     from openxml(@idoc, '/Root/Element')
     with
     (
         ElementID int '@ElementID',
         ElementValueName varchar(50) '@ElementValueName'
     ) 
) source
on NormalDataTable.ElementID = source.ElementID
when not matched then
    insert ...
when matched then
    update ...

exec sp_xml_removedocument @idoc

注意!

本站转载的文章为个人学习借鉴使用,本站对版权不负任何法律责任。如果侵犯了您的隐私权益,请联系我们删除。



 
粤ICP备14056181号  © 2014-2021 ITdaan.com