Python memory allocations seems pretty convoluted when I first began to wonder on what happens to memory when the object is not being used anymore. Well, memory allocation happens using malloc and free (Python is written in C ) and these operations are pretty expensive ( I will quantify that in a while).
Now, usually what you will think is when I allocate memory in delete it , the memory consumed by the objec t is freed. Guess what ? It is definetly not and I assure you thats not a bug in python. Lets take a closer look on how this works.
Until Python 2.3ish versions , python primarily handled memory management using reference counting (Google it :P) . Here is an interesting article (Its a must read) on how the memory handling happens, when only reference counting was used. The summaries, python holds the memory when the objects is deleted. Why? Cuz memory operations are expensive so if your program has a sudden burst of object creations, without pyMalloc you would suffer huge performance issues. Since python interpreter holds the memory for you (which any decent high level language should do) , you dont have to worry about it, unless you intend to write a long daemon process and have throw int, floats and lists all over your program. (ints / floats / lists are not handled via PyMalloc ) . But definitely a BIG reason to worry, if the code is running in the server and for very very long time.
So what do we do about it ?
Good news. On later versions (probably python 2.6 onwards , I am not too sure on that ) Python uses more reference counting and garbage collections. I am not getting into the debate on whether GC is a good thing or not. Hard-core C++ programmers will never agree with me, as GC is not suitable for real time systems. Since, they use significant amount of computing power to clean up memory.
Automatic garbage collection will not run if your Python device is running out of memory
In your python program, you can " import gc " (GC in python). By default GC is enabled in python , unless u manually disabled it during installation.
import gc
gc.get_threshold() # Will print the memory threshold limit after which GC will run in python
gc.collect() #will collect unwanted objects and freeup memory for you.
If you are running a long daemon process, you can call gc.collect() after a function call or could be time based. Its upto you.
A few Caveats to keep in mind:
del l
You may have deleted l , but all the memory pertaining to it is been help up by python interpreter
(What ?? I am still confused . Read the article I mentioned above). In such scenarios using xrange is
a better idea
Now, usually what you will think is when I allocate memory in delete it , the memory consumed by the objec t is freed. Guess what ? It is definetly not and I assure you thats not a bug in python. Lets take a closer look on how this works.
Until Python 2.3ish versions , python primarily handled memory management using reference counting (Google it :P) . Here is an interesting article (Its a must read) on how the memory handling happens, when only reference counting was used. The summaries, python holds the memory when the objects is deleted. Why? Cuz memory operations are expensive so if your program has a sudden burst of object creations, without pyMalloc you would suffer huge performance issues. Since python interpreter holds the memory for you (which any decent high level language should do) , you dont have to worry about it, unless you intend to write a long daemon process and have throw int, floats and lists all over your program. (ints / floats / lists are not handled via PyMalloc ) . But definitely a BIG reason to worry, if the code is running in the server and for very very long time.
So what do we do about it ?
Good news. On later versions (probably python 2.6 onwards , I am not too sure on that ) Python uses more reference counting and garbage collections. I am not getting into the debate on whether GC is a good thing or not. Hard-core C++ programmers will never agree with me, as GC is not suitable for real time systems. Since, they use significant amount of computing power to clean up memory.
Automatic garbage collection will not run if your Python device is running out of memory
In your python program, you can " import gc " (GC in python). By default GC is enabled in python , unless u manually disabled it during installation.
import gc
gc.get_threshold() # Will print the memory threshold limit after which GC will run in python
gc.collect() #will collect unwanted objects and freeup memory for you.
If you are running a long daemon process, you can call gc.collect() after a function call or could be time based. Its upto you.
A few Caveats to keep in mind:
- Its better off using xrange functions to populate a list than range. Lists are pretty tricky in python , they are immortal and unbound. For example .
del l
You may have deleted l , but all the memory pertaining to it is been help up by python interpreter
(What ?? I am still confused . Read the article I mentioned above). In such scenarios using xrange is
a better idea
- Debug your code with Garbage Collector
In here we are creating a single object . The following is the output
As you can see the GC has detected L dictionary to be unreachable and after we run gc.collect() the object is freed.