使用numpy.genfromtxt在Python 3中加載UTF-8文件

[英]Loading UTF-8 file in Python 3 using numpy.genfromtxt


I have a CSV file that I downloaded from WHO site (http://apps.who.int/gho/data/view.main.52160 , Downloads, "multipurpose table in CSV format"). I try to load the file into a numpy array. Here's my code:

我有一個CSV文件,我從世衛組織網站下載(http://apps.who.int/gho/data/view.main.52160,下載,“CSV格式的多用途表格”)。我嘗試將文件加載到numpy數組中。這是我的代碼:

import numpy
#U75 - unicode string of max. length 75
world_alcohol = numpy.genfromtxt("xmart.csv", dtype="U75", skip_header=2, delimiter=",")
print(world_alcohol)

And I get

我明白了

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2: ordinal not in range(128).

UnicodeDecodeError:'ascii'編解碼器無法解碼位置2中的字節0xc3:序數不在范圍內(128)。

I guess that numpy has a problem reading the string "Côte d'Ivoire". The file is properly encoded UTF-8 (according to my text editor). I am using Python 3.4.3 and numpy 1.9.2.

我猜numpy在閱讀字符串“Côted'Ivoire”時遇到了問題。該文件正確編碼為UTF-8(根據我的文本編輯器)。我正在使用Python 3.4.3和numpy 1.9.2。

What am I doing wrong? How can I read the file into numpy?

我究竟做錯了什么?如何將文件讀入numpy?

1 个解决方案

#1


8  

In Python3 I can do:

在Python3中我可以做到:

In [224]: txt = "Côte d'Ivoire"
In [225]: x = np.zeros((2,),dtype='U20')
In [226]: x[0] = txt
In [227]: x
Out[227]: 
array(["Côte d'Ivoire", ''],   dtype='<U20')

Which means I probably could open a 'UTF-8' file (regular, not byte mode), and readlines, and assign them to elements of an array like x.

這意味着我可能打開一個'UTF-8'文件(常規,非字節模式)和readlines,並將它們分配給像x這樣的數組元素。

But genfromtxt insists on operating with byte strings (ascii) which can't handle the larger UTF-8 set (7 bytes v 8). So I need to apply decode at some point to get an U array.

但genfromtxt堅持使用字節字符串(ascii)進行操作,這些字符串無法處理較大的UTF-8集(7字節v 8)。所以我需要在某些時候應用解碼來獲得一個U數組。

I can load it into a 'S' array with genfromtxt:

我可以使用genfromtxt將其加載到'S'數組中:

In [258]: txt="Côte d'Ivoire"
In [259]: a=np.genfromtxt([txt.encode()],delimiter=',',dtype='S20')
In [260]: a
Out[260]: 
array(b"C\xc3\xb4te d'Ivoire",  dtype='|S20')

and apply decode to individual elements:

並將解碼應用於單個元素:

In [261]: print(a.item().decode())
Côte d'Ivoire

In [325]: print _
Côte d'Ivoire

Or use np.char.decode to apply it to each element of an array:

或者使用np.char.decode將其應用於數組的每個元素:

In [263]: np.char.decode(a)
Out[263]: 
array("Côte d'Ivoire", dtype='<U13')
In [264]: print(_)
Côte d'Ivoire

genfromtxt lets me specify converters:

genfromtxt讓我指定轉換器:

In [297]: np.genfromtxt([txt.encode()],delimiter=',',dtype='U20',
    converters={0:lambda x: x.decode()})
Out[297]: 
array("Côte d'Ivoire", dtype='<U20')

If the csv has a mix of strings and numbers, this converters approach will be easier to use than the np.char.decode. Just specify the converter for each string column.

如果csv混合了字符串和數字,那么這種轉換器方法將比np.char.decode更容易使用。只需為每個字符串列指定轉換器。

(See my earlier edits for Python2 tries).

(請參閱我之前對Python2嘗試的編輯)。


注意!

本站翻译的文章,版权归属于本站,未经许可禁止转摘,转摘请注明本文地址:https://www.itdaan.com/blog/2015/10/07/729cacd754ae0c3e701623ddbd67ce0.html



 
粤ICP备14056181号  © 2014-2020 ITdaan.com