[Tutor] improving speed using and recalling C functions
Gabriele Brambilla
gb.gabrielebrambilla at gmail.com
Fri Apr 11 16:23:47 CEST 2014
this is the profile for a sample of 1000 elements
Fri Apr 11 10:21:21 2014 restats
31594963 function calls in 103.708 seconds
Ordered by: internal time
List reduced from 47 to 20 due to restriction <20>
ncalls tottime percall cumtime percall filename:lineno(function)
1 57.133 57.133 103.692 103.692 skymapsI.py:44(mymain)
176832 9.898 0.000 11.710 0.000
interpolate.py:394(_call_linear)
18101000 7.808 0.000 7.808 0.000 {method 'write' of 'file'
objects}
1237824 3.794 0.000 3.794 0.000 {numpy.core.multiarray.array}
1001000 3.610 0.000 38.383 0.000 instruments.py:10(kappa)
353664 3.314 0.000 3.314 0.000 {method 'reduce' of
'numpy.ufunc'
objects}
176832 3.157 0.000 7.428 0.000
interpolate.py:454(_check_bounds)
176832 2.074 0.000 10.213 0.000 interpolate.py:330(__init__)
176832 2.053 0.000 21.522 0.000 interpolate.py:443(_evaluate)
176832 1.253 0.000 4.404 0.000 polyint.py:82(_set_yi)
176832 0.769 0.000 0.769 0.000 {method 'clip' of
'numpy.ndarray'
objects}
353664 0.706 0.000 0.706 0.000 {method 'reshape' of
'numpy.ndarra
y' objects}
353664 0.667 0.000 1.205 0.000
numerictypes.py:735(issubdtype)
707328 0.637 0.000 2.451 0.000 numeric.py:392(asarray)
176832 0.601 0.000 23.555 0.000 polyint.py:37(__call__)
353664 0.569 0.000 3.883 0.000 _methods.py:31(_any)
176832 0.504 0.000 1.429 0.000 polyint.py:74(_reshape_yi)
176832 0.473 0.000 0.473 0.000 {method 'searchsorted' of
'numpy.n
darray' objects}
176832 0.440 0.000 1.645 0.000 polyint.py:102(_set_dtype)
176832 0.426 0.000 4.830 0.000 polyint.py:30(__init__)
thanks
Gabriele
2014-04-11 10:18 GMT-04:00 Gabriele Brambilla <
gb.gabrielebrambilla at gmail.com>:
> ok
> modifying the for in this way (zipping an array of matrix drive it crazy)
> it works
>
> dko=0
>
> for gammar in gmils:
>
>
> omC = (1.5)*(gammar**3)*c/(rho*rlc)
>
> gig = omC*hcut/eVtoErg
>
>
> #check the single emission
>
>
>
> for w in eel:
>
> omega =
> (10**(w*stepENE+Lemin))*eVtoErg/hcut
>
> x = omega/omC
>
> kap = instruments.kappa(x)
>
> Iom = (1.732050808/c)*(e**2)*gammar*kap
> #jackson dI/domega
>
> P = Iom*(c/(rho*rlc))/(2*pi) #jackson P
>
> phps = P/(hcut*omega) #photons per second
>
> www = phps/(stepPHA*sin(zobs)*stepOB)
>
> MYMAPS[dko][i,j,w] += www
>
> dko += 1
>
>
>
> count = count + 1
>
> Now I will tell you how much it takes.
>
> Thanks
>
> Gabriele
>
>
>
> 2014-04-11 10:05 GMT-04:00 Gabriele Brambilla <
> gb.gabrielebrambilla at gmail.com>:
>
> ok, it seems that the code don't enter in this for loop
>>
>> for gammar, MYMAP in zip(gmlis, MYMAPS):
>>
>> I don't understand why.
>>
>> Thanks
>>
>> Gabriele
>>
>>
>> 2014-04-11 9:56 GMT-04:00 Gabriele Brambilla <
>> gb.gabrielebrambilla at gmail.com>:
>>
>> Hi, I'm sorry but there is a big problem.
>>> the code is producing empty file.dat.
>>>
>>> I think it's because of this that previously I have done that strange
>>> trick of myinternet...
>>>
>>> So:
>>>
>>> for my_line in open('data.dat'):
>>>
>>> myinternet = []
>>>
>>> gmlis = []
>>>
>>> print('reading the line', count, '/599378')
>>>
>>> my_parts = [float(i) for i in my_line.split()]
>>>
>>> phase = my_parts[4]
>>>
>>> zobs = my_parts[5]
>>>
>>> rho = my_parts[6]
>>>
>>>
>>>
>>> gmils=[my_parts[7], my_parts[8], my_parts[9],
>>> my_parts[10], my_parts[11]]
>>>
>>>
>>>
>>> i = int((phase-phamin)/stepPHA)
>>>
>>> j = int((zobs-obamin)/stepOB)
>>>
>>>
>>>
>>> for gammar, MYMAP in zip(gmlis, MYMAPS):
>>>
>>>
>>>
>>> omC = (1.5)*(gammar**3)*c/(rho*rlc)
>>>
>>> gig = omC*hcut/eVtoErg
>>>
>>> #check the single emission
>>>
>>>
>>>
>>> for w in eel:
>>>
>>> omega =
>>> (10**(w*stepENE+Lemin))*eVtoErg/hcut
>>>
>>> x = omega/omC
>>>
>>> kap = instruments.kappa(x)
>>>
>>>
>>> Iom = (1.732050808/c)*(e**2)*gammar*kap
>>> #jackson dI/domega
>>>
>>> P = Iom*(c/(rho*rlc))/(2*pi) #jackson P
>>>
>>> phps = P/(hcut*omega) #photons per second
>>>
>>> www = phps/(stepPHA*sin(zobs)*stepOB)
>>>
>>> MYMAP[i,j,w] += www
>>>
>>>
>>>
>>> count = count + 1
>>>
>>> when I exit here the MYMAP matrix has all the cells = 0.
>>>
>>> Now I will try to fiugre it out why.
>>>
>>> Thanks
>>>
>>> Gabriele
>>>
>>>
>>>
>>> 2014-04-11 9:20 GMT-04:00 Gabriele Brambilla <
>>> gb.gabrielebrambilla at gmail.com>:
>>>
>>> Hi Danny,
>>>> I'm quiet impressed.
>>>> the program takes near 30 minutes instead of more than 8 hours!
>>>>
>>>> this is the profile:
>>>> Fri Apr 11 09:14:04 2014 restats
>>>>
>>>> 19532732 function calls in 2105.024 seconds
>>>>
>>>> Ordered by: internal time
>>>>
>>>> ncalls tottime percall cumtime percall filename:lineno(function)
>>>> 1 2087.606 2087.606 2105.006 2105.006 skymapsI.py:44(mymain)
>>>> 18101000 12.757 0.000 12.757 0.000 {method 'write' of 'file'
>>>> objects}
>>>>
>>>> 715853 3.473 0.000 3.473 0.000 {method 'split' of 'str'
>>>> objects}
>>>> 715854 1.162 0.000 1.162 0.000 {zip}
>>>> 1 0.018 0.018 2105.024 2105.024 <string>:1(<module>)
>>>> 6 0.006 0.001 0.006 0.001 {open}
>>>> 5 0.002 0.000 0.002 0.000 {method 'close' of 'file'
>>>> objects}
>>>>
>>>> 1 0.000 0.000 0.000 0.000
>>>> function_base.py:8(linspace)
>>>> 5 0.000 0.000 0.000 0.000
>>>> {numpy.core.multiarray.zeros}
>>>> 1 0.000 0.000 0.000 0.000
>>>> function_base.py:93(logspace)
>>>> 1 0.000 0.000 0.000 0.000
>>>> {numpy.core.multiarray.arange}
>>>> 3 0.000 0.000 0.000 0.000 {range}
>>>> 1 0.000 0.000 0.000 0.000 {method 'disable' of
>>>> '_lsprof.Prof
>>>> iler' objects}
>>>>
>>>> I hope to have similar problems in the future to learn better how to do
>>>> with them!
>>>> but in the profile I don't see any operation regarding reading the file
>>>> or the mathematical operations...are them hidden in mymain()?
>>>>
>>>> thanks
>>>>
>>>> Gabriele
>>>>
>>>>
>>>>
>>>>
>>>> 2014-04-10 21:38 GMT-04:00 Danny Yoo <dyoo at hashcollision.org>:
>>>>
>>>> > Comment: You are looping over your sliced eel five times. Do you
>>>>> > need to? I like eel salad a great deal, as well, but, how about:
>>>>> >
>>>>> >
>>>>> > for k in eel:
>>>>> > MYMAP1[i, j, k] = MYMAP1[i, j, k] + myinternet[oo]
>>>>> > MYMAP2[i, j, k] = MYMAP2[i, j, k] + myinternet[oo]
>>>>> > MYMAP3[i, j, k] = MYMAP3[i, j, k] + myinternet[oo]
>>>>> > MYMAP4[i, j, k] = MYMAP4[i, j, k] + myinternet[oo]
>>>>> > MYMAP5[i, j, k] = MYMAP5[i, j, k] + myinternet[oo]
>>>>> > oo = oo + 1
>>>>>
>>>>>
>>>>> Hi Gabriele,
>>>>>
>>>>> Also note that, when Martin looked at this part of the code, he
>>>>> unfortunately misinterpreted its effect; Martin's proposed rewrite
>>>>> here does not preserve the meaning of the original code. But rather
>>>>> than wag my finger at how Martin interpreted the code, I'd rather make
>>>>> the observation that this is a warning sign that the original code
>>>>> here was not easy to understand.
>>>>>
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/tutor/attachments/20140411/a1f64b76/attachment-0001.html>
More information about the Tutor
mailing list