最有效的方法来映射函数超过numpy数组。

[英]Most efficient way to map function over numpy array


What is the most efficient way to map a function over a numpy array? The way I've been doing it in my current project is as follows:

在numpy数组上映射函数最有效的方法是什么?在我目前的项目中,我一直是这样做的:

import numpy as np 

x = np.array([1, 2, 3, 4, 5])

# Obtain array of square of each element in x
squarer = lambda t: t ** 2
squares = np.array([squarer(xi) for xi in x])

However, this seems like it is probably very inefficient, since I am using a list comprehension to construct the new array as a Python list before converting it back to a numpy array.

然而,这似乎是非常低效的,因为在将新数组转换回numpy数组之前,我使用了一个列表理解来构造新的数组作为Python列表。

Can we do better?

我们可以做得更好吗?

8 个解决方案

#1


63  

How about using numpy.vectorize.

如何使用numpy.vectorize。

>>> import numpy as np
>>> x = np.array([1, 2, 3, 4, 5])
>>> squarer = lambda t: t ** 2
>>> vfunc = np.vectorize(squarer)
>>> vfunc(x)
array([ 1,  4,  9, 16, 25])

http://docs.scipy.org/doc/numpy-1.10.1/reference/generated/numpy.vectorize.html

http://docs.scipy.org/doc/numpy-1.10.1/reference/generated/numpy.vectorize.html

#2


59  

I've tested all suggested methods plus np.array(map(f, x)) with perfplot (a small project of mine).

我已经测试了所有建议的方法加上np。数组(map(f, x))和perfplot(我的一个小项目)。

Message #1: If you can use numpy's native functions, do that.

消息#1:如果您可以使用numpy的本机函数,请这样做。

If the function you're trying to vectorize already is vectorized (like the x**2 example in the original post), using that is much faster than anything else (note the log scale):

如果你想要向量化的函数已经向量化了(如前一篇文章中的x**2示例),那么使用它要比使用其他函数快得多(请注意日志比例):

enter image description here

If you actually need vectorization, it doesn't really matter much which variant you use.

如果您确实需要矢量化,那么使用哪种变体并不重要。

enter image description here


Code to reproduce the plots:

复制图的代码:

import numpy as np
import perfplot
import math


def f(x):
    # return math.sqrt(x)
    return np.sqrt(x)


vf = np.vectorize(f)


def array_for(x):
    return np.array([f(xi) for xi in x])


def array_map(x):
    return np.array(list(map(f, x)))


def fromiter(x):
    return np.fromiter((f(xi) for xi in x), x.dtype)


def vectorize(x):
    return np.vectorize(f)(x)


def vectorize_without_init(x):
    return vf(x)


perfplot.show(
    setup=lambda n: np.random.rand(n),
    n_range=[2**k for k in range(20)],
    kernels=[
        f,
        array_for, array_map, fromiter, vectorize, vectorize_without_init
        ],
    logx=True,
    logy=True,
    xlabel='len(x)',
    )

#3


27  

TL;DR

In my experience, np.fromiter is generally the fastest for different sizes of arrays, and different versions of Python and NumPy from different compilers. Here is a short example:

在我的经验中,对于不同大小的数组、不同版本的Python和不同编译器的NumPy, np.fromiter通常是最快的。这里有一个简短的例子:

import numpy as np
x = np.array([1, 2, 3, 4, 5])
f = lambda x: x ** 2
squares = np.fromiter((f(xi) for xi in x), x.dtype, count=len(x))

Note that count is optional. It may be worthwhile to use np.vectorize for arrays with at least hundreds of elements (see the plot in Nico's answer).

注意计数是可选的。使用np或许是值得的。使用至少有数百个元素的数组进行矢量化(参见Nico的答案)。

Comparison of methods

Here are some simple tests to compare three methods to map a function, this example using with Python 2.7 and NumPy 1.9. First, the set-up functions for testing:

这里有一些简单的测试,可以比较映射函数的三种方法,本例使用的是Python 2.7和NumPy 1.9。首先,测试的设置功能:

import timeit
import numpy as np

f = lambda x: x ** 2
vf = np.vectorize(f)

def test_array(x, n):
    t = timeit.timeit(
        'np.array([f(xi) for xi in x])',
        'from __main__ import np, x, f', number=n)
    print('array: ' + str(t))

def test_fromiter(x, n):
    t = timeit.timeit(
        'np.fromiter((f(xi) for xi in x), x.dtype, count=len(x))',
        'from __main__ import np, x, f', number=n)
    print('fromiter: ' + str(t))

def test_vectorized(x, n):
    t = timeit.timeit(
        'vf(x)',
        'from __main__ import x, vf', number=n)
    print('vectorized: ' + str(t))

With five elements, np.fromiter is fastest, and np.vectorize is much slower (due to set-up costs):

np.fromiter有五个元素,是最快的,np也是。矢量化速度要慢得多(由于设置成本):

x = np.array([1, 2, 3, 4, 5])
n = 100000
test_array(x, n)       # 0.616514921188
test_fromiter(x, n)    # 0.585698843002
test_vectorized(x, n)  # 2.6228120327

With 100s of elements all methods are about the same:

有了100多个元素,所有的方法都差不多:

x = np.arange(100)
n = 10000
test_array(x, n)       # 0.519502162933
test_fromiter(x, n)    # 0.500586986542
test_vectorized(x, n)  # 0.525988101959

But with 1000s of array elements or more, the vectorized approach is most efficient:

但如果数组元素超过1000个,矢量化方法是最有效的:

x = np.arange(1000)
n = 1000
test_array(x, n)       # 0.472352981567
test_fromiter(x, n)    # 0.453316926956
test_vectorized(x, n)  # 0.291934967041

However, different versions of Python/NumPy and compiler optimization will have different results, so do a similar test for your environment.

但是,不同版本的Python/NumPy和编译器优化会有不同的结果,所以对您的环境进行类似的测试。

#4


8  

squares = squarer(x)

Arithmetic operations on arrays are automatically applied elementwise, with efficient C-level loops that avoid all the interpreter overhead that would apply to a Python-level loop or comprehension.

数组上的算术操作是自动应用的元素,使用高效的c级循环,避免了所有应用于python级循环或理解的解释器开销。

Most of the functions you'd want to apply to a NumPy array elementwise will just work, though some may need changes. For example, if doesn't work elementwise. You'd want to convert those to use constructs like numpy.where:

您希望应用到NumPy数组elementwise的大多数函数都将正常工作,尽管有些函数可能需要更改。例如,if在元素上不起作用。你想要转换它们来使用像numpy.where这样的结构:

def using_if(x):
    if x < 5:
        return x
    else:
        return x**2

becomes

就变成了

def using_where(x):
    return numpy.where(x < 5, x, x**2)

#5


8  

I believe in newer version( I use 1.13) of numpy you can simply call the function by passing the numpy array to the fuction that you wrote for scalar type, it will automatically apply the function call to each element over the numpy array and return you another numpy array

我相信更新的numpy版本(我用的是1.13)你可以通过将numpy数组传递给你为标量类型编写的函数来调用这个函数,它会自动地对numpy数组中的每个元素应用函数调用并返回另一个numpy数组

>>> import numpy as np
>>> squarer = lambda t: t ** 2
>>> x = np.array([1, 2, 3, 4, 5])
>>> squarer(x)
array([ 1,  4,  9, 16, 25])

#6


1  

As mentioned in this post, just use generator expressions like so:

如本文所述,只需使用生成器表达式如下:

numpy.fromiter((<some_func>(x) for x in <something>),<dtype>,<size of something>)

#7


0  

Maybe this is not directly answering the question, but I've heard that numba can compile existing python code into parallel machine instructions. I'll revisit and revise this post when I actually have a chance to use that.

也许这并不能直接回答这个问题,但是我听说numba可以将现有的python代码编译成并行机器指令。当我有机会使用这篇文章的时候,我会重新审视和修改它。

#8


-2  

Maybe using vectorize is better

也许使用矢量化更好

def square(x):
   return x**2

vfunc=vectorize(square)

vfunc([1,2,3,4,5])

output:array([ 1,  4,  9, 16, 25])

注意!

本站翻译的文章,版权归属于本站,未经许可禁止转摘,转摘请注明本文地址:http://www.itdaan.com/blog/2016/02/05/58acff648932a41dc9631eadd2e4fe75.html



 
© 2014-2018 ITdaan.com 粤ICP备14056181号