Question

I spend the whole day waiting for a loop to exit without hope!. I know that python is not so efficient when it comes to performance as such I would really appreciate any speed-up suggestions to my problem.

I have captured a large number of packets (around 500,000) using wireshark and saved them to .pcap file. After that I read the packets from the saved file by using Scapy rdpcap() function and then I accessed each packet in a loop to extract the source IP Address. My code is as follows:

from scaly.all import *

srcList =[]
Packets = rdpcap("pcapfile")

for pkt in Packets:
    src = Packets[Packets.index(pkt)][1].src
    srcList.append(src)

Note: I have done some digging and I found that Cython is used to speed-up nested loop, but honestly I have no idea how to use that in my case. any insight would be great

Answer 1

If I am not misunderstanding your intention, you can simplify your code, that should also speed it up:

from scaly.all import *

Packets = rdpcap("pcapfile")
srcList = [pkt[1].src for pkt in Packets]

The difference between this solution and yours can be illustrated with a simple example. As you can see, the second function is more than 10 times faster.

In [1]: lst = range(100)

In [2]: def f1(lst):
   ...:     out = []
   ...:     for item in lst:
   ...:         out.append(lst[lst.index(item)])
   ...:     return out

In [3]: def f2(lst):
   ...:     return [item for item in lst]

In [4]: %timeit f1(lst)
1000 loops, best of 3: 221 us per loop

In [5]: %timeit f2(lst)
100000 loops, best of 3: 9.61 us per loop