解決mysqldb查詢大量數據導致內存使用過高的問題


1.源碼
connection=MySQLdb.connect( host="thehost",user="theuser", passwd="thepassword",db="thedb") cursor=connection.cursor() cursor.execute(query) for row in cursor.fetchall(): print(row)
2.問題
普通的操作不管是fetchall()還是fetchone()都是先將數據加載到本地再進行計算,大量的數據會導致內存資源消耗光。解決的方法是使用SSCurosr光標來處理。


3.優化后的代碼  
import MySQLdb.cursors connection=MySQLdb.connect( host="thehost",user="theuser", passwd="thepassword",db="thedb", cursorclass = MySQLdb.cursors.SSCursor) cursor=connection.cursor() cursor.execute(query) for row in cursor: print(row)

參考文檔:http://mysql-python.sourceforge.net/MySQLdb.html#

關鍵段落截取:
BaseCursor
The base class for Cursor objects. This does not raise Warnings.
CursorStoreResultMixIn
Causes the Cursor to use the  mysql_store_result() function to get the query result. The entire result set is stored on the client side.
CursorUseResultMixIn
Causes the cursor to use the  mysql_use_result() function to get the query result. The result set is stored on the server side and is transferred row by row using fetch operations.
CursorTupleRowsMixIn
Causes the cursor to return rows as a tuple of the column values.

CursorDictRowsMixIn

Causes the cursor to return rows as a dictionary, where the keys are column names and the values are column values. Note that if the column names are not unique, i.e., you are selecting from two tables that share column names, some of them will be rewritten as  table.column. This can be avoided by using the SQL  ASkeyword. (This is yet-another reason not to use  * in SQL queries, particularly where  JOIN is involved.)
Cursor
The default cursor class. This class is composed of  CursorWarningMixInCursorStoreResultMixInCursorTupleRowsMixIn, and  BaseCursor, i.e. it raises  Warning, uses mysql_store_result(), and returns rows as tuples.
DictCursor
Like  Cursor except it returns rows as dictionaries.
SSCursor
A "server-side" cursor. Like  Cursor but uses  CursorUseResultMixIn. Use only if you are dealing with potentially large result sets.
SSDictCursor
Like  SSCursor except it returns rows as dictionaries.




注意!

本站转载的文章为个人学习借鉴使用,本站对版权不负任何法律责任。如果侵犯了您的隐私权益,请联系我们删除。



 
粤ICP备14056181号  © 2014-2021 ITdaan.com