Author Message
acorrigan

Joined: 30 May 2007
Posts: 2
Location: GMU

 Posted: Wed May 30, 2007 7:43 pm    Post subject: numpy array support? I really like PyD and I'm considering using it write numerical code. Is it possible to write functions in D involving D-arrays and have PyD generate a module which maps automatically between D-arrays and NumPy arrays? This is basically what f2py does with Fortran arrays. I was hoping that something similar might be possible with PyD and D.
KirkMcDonald

Joined: 22 Jun 2006
Posts: 23

Posted: Fri Jun 01, 2007 2:46 am    Post subject:

In order to map between D arrays and NumPy arrays directly, I would have to find some way to reach into NumPy's innards and grab a slice of the NumPy array's internal buffer. I don't know nearly enough about how NumPy works to do this, and have serious doubts that it's even possible to do it in a useful way.

Therefore, the next best thing is writing a D type that wraps NumPy array objects and overloads appropriate operators so that it acts like an array. As it happens, Pyd already has PydObject, a type which wraps any arbitrary Python object. Your best bet is to have your function parameters that accept these arrays to be of type PydObject. You should then be able to index the PydObject normally.

This would get trickier with multi-dimensional arrays. To index a multi-dimensional NumPy array, you pass a tuple as the index. Therefore, you would have to pass a PydObject holding a tuple to PydObject.opIndex. Building PydObjects like this is one of Pyd's rougher edges. Making a PydObject-tuple from D items the easy way looks like this:

 Code: new PydObject(PyTuple_FromItems(1, 2, 3));

PyTuple_FromItems is an undocumented function inside of Pyd, which looks like this:

 Code: PyObject* PyTuple_FromItems(T ...) (T t);

It wouldn't be hard to shorten the above with a simple wrapper function, and I fully plan on adding such a thing to Pyd in the future. (There are a number of PydObject features that need to be fleshed out, but the bits that are there should work.)

However, the use of PydObject implies a certain overhead. (It is a class, after all, and goes on the heap.) If you want to minimize overhead for your numerics code, then you are effectively on your own, and must deal with the C API directly. This is made available by saying "import python;". After that, you may have your functions accept PyObject*s, and Python objects will be passed in directly as borrowed references. You are responsible for reference counts and exception handling whenever you call the Python/C API directly. Pyd's handle_exception() function is the recommended way to deal with Python/C API exceptions; simply call it whenever an exception may have occurred (it will do nothing if no exception has occurred, and throw the exception as a PythonException if one has).

In short: My personal recommendation is to use PydObject, though I am very interested in whatever shortcomings you may come across with it.
acorrigan

Joined: 30 May 2007
Posts: 2
Location: GMU

 Posted: Sun Jun 03, 2007 10:01 am    Post subject: Thanks.
baxissimo

Joined: 23 Oct 2006
Posts: 241
Location: Tokyo, Japan

 Posted: Fri Aug 10, 2007 2:13 am    Post subject: Numpy's C Array You can most definitely operate directly on the memory used by a Numpy array. It has a very carefully thought out C API precisely for that purpose. So probably what's really needed is a D translation of that C API. If you have numpy installed, the main API can be found in a path like: C:/Python25/Lib/site-packages/numpy/core/include/numpy/ndarrayobject.h Unfortunately it contains LOTS of #defines.
Amayng

Joined: 23 Oct 2007
Posts: 1

Posted: Tue Oct 23, 2007 9:15 am    Post subject:

Boost.Python also lacks a seemless interface to numpy.
There is a nice utility called num_util which which takes care of all the ugly Python C-api. It can be downloaded from http://www.eos.ubc.ca/research/clouds/software/pythonlibs/num_util/num_util_release2/

Would be awesome to have something like that in D!

This is what I want to do in Python: Using only standard Python/Numpy data types.

test.py
 Code: #!/usr/bin/env python import numpy as npy import pylab as pyl import AffinityPropagation as ap x = pyl.load('data/ToyProblemData.txt') N = npy.shape(x)[0] print 'preparing similarity matrix' S = ap.outer_dot(x) print 'putting preferences in the similarity matrix' median = npy.median(npy.median(S)) P = npy.repeat(median,N) S[range(N), range(N)] = P S = -S print 'running affinity propagation' dic = ap.ap(S, 100,50,0.5) print dic #OUTPUT: #{'lam': 0.5, 'K': 3, 'it': 58, #'cl': array([0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 2, 1, 0, 2, 1, 2, 2, 2, 2, 2, 2, 0, 2,, 1]), #'dpex': array([ 2,  2,  2,  2,  2,  2,  6,  6,  6,  6, 22,  6,  2, 22,  6, 22, 22, 22, 22, 22, 22,  2, 22, 22,  6]), }

This is what it looks like in C++ using Boost.Python and num_util.h and num_util.cpp. Since a Numpy stores data as contiguous C array (unless you do stuff like transposing multidimensional arrays)

ap.hpp
 Code: #ifndef AP_HPP #define AP_HPP #define PY_ARRAY_UNIQUE_SYMBOL PyArrayHandle #include "num_util.h" using namespace std; namespace b = boost; namespace bp = boost::python; namespace bpn = boost::python::numeric; namespace nu = num_util; bp::dict ap(bpn::array &inSimilaritiesMatrix, uint maxit, uint convit, double lam); BOOST_PYTHON_MODULE(AffinityPropagation) {    import_array();    bpn::array::set_module_and_type("numpy", "ndarray");    def("ap",ap);    def("outer_dot",outer_dot); } #endif

app.cpp
 Code: #include #include "ap.hpp" bp::dict ap(bpn::array &inSimilaritiesMatrix, uint maxit, uint convit, double lam){        /* CHECKING INPUT VALUES */    nu::check_rank(inSimilaritiesMatrix,2);    vector shp(nu::shape(inSimilaritiesMatrix));    int N = shp[0];    if(N != shp[1]){       PyErr_SetString(PyExc_ValueError, "Expected a similarity matrix in (N,N) array format!");       bp::throw_error_already_set();    }    /* SETUP VARIABLES */    double* dataPtr = (double*) nu::data(inSimilaritiesMatrix);    double_matrix_ref    s(dataPtr,b::extents[shp[0]][shp[1]]);   //similarities    ...    CODE    ...    /* PREPARING OUTPUT TO PYTHON */    bpn::array ret_dpex    =  nu::makeNum( &dpex[0], N);    bpn::array ret_cl   =  nu::makeNum( &cl[0], N);           bp::dict retvals;    retvals["K"] = K;    retvals["lam"] = lam;    retvals["maxit"] = maxit;    retvals["convit"] = convit;    retvals["it"] = it;    retvals["dpex"] = ret_dpex;    retvals["cl"] = ret_cl;    retvals["net_similarity"] = net_similarity;    retvals["average_preference"] = average_preference;    retvals["net_self_responsibility"] = net_self_responsibility;    retvals["net_responsibility"] = net_responsibility;    retvals["net_availability"] = net_availability;        return retvals; }
 Display posts from previous: All Posts1 Day7 Days2 Weeks1 Month3 Months6 Months1 Year Oldest FirstNewest First
 All times are GMT - 6 Hours Page 1 of 1