A guide to analyzing Python performance

Sunday, July 29, 2012

Copied From: http://www.huyng.com/posts/python-performance-analysis/

While it’s not always the case that every Python program you write will
require a rigorous performance analysis, it is reassuring to know that
there are a wide variety of tools in Python’s ecosystem that one can
turn to when the time arises.

Analyzing a program’s performance boils down to answering 4 basic

  1. How fast is it running?
  2. Where are the speed bottlenecks?
  3. How much memory is it using?
  4. Where is memory leaking?

Below, we’ll dive into the details of answering these questions using
some awesome tools.

Coarse grain timing with time

Let’s begin by using a quick and dirty method of timing our code: the
good old unix utility time.

$ time python yourprogram.py

real    0m1.028s
user    0m0.001s
sys     0m0.003s

The meaning between the three output measurements are detailed in this
but in short

  • real - refers to the actual elasped time
  • user - refers to the amount of cpu time spent outside of kernel
  • sys - refers to the amount of cpu time spent inside kernel specific

You can get a sense of how many cpu cycles your program used up
regardless of other programs running on the system by adding together
the sys and user times.

If the sum of sys and user times is much less than real time, then
you can guess that most your program’s performance issues are most
likely related to IO waits.

Fine grain timing with a timing context manager

Our next technique involves direct instrumentation of the code to get
access to finer grain timing information. Here’s a small snippet I’ve
found invaluable for making ad-hoc timing measurements:


import time

class Timer(object):
    def __init__(self, verbose=False):
        self.verbose = verbose

    def __enter__(self):
        self.start = time.time()
        return self

    def __exit__(self, *args):
        self.end = time.time()
        self.secs = self.end - self.start
        self.msecs = self.secs * 1000  # millisecs
        if self.verbose:
            print 'elapsed time: %f ms' % self.msecs

In order to use it, wrap blocks of code that you want to time with
Python’s with keyword and this Timer context manager. It will take
care of starting the timer when your code block begins execution and
stopping the timer when your code block ends.

Here’s an example use of the snippet:

from timer import Timer
from redis import Redis
rdb = Redis()

with Timer() as t:
    rdb.lpush("foo", "bar")
print "=> elasped lpush: %s s" % t.secs

with Timer as t:
print "=> elasped lpop: %s s" % t.secs

I’ll often log the outputs of these timers to a file in order to see how
my program’s performance evolves over time.

Line-by-line timing and execution frequency with a profiler

Robert Kern has a nice project called
line_profiler which I
often use to see how fast and how often each line of code is running in
my scripts.

To use it, you’ll need to install the python package via pip:

$ pip install line_profiler

Once installed you’ll have access to a new module called
“line_profiler” as well as an executable script “kernprof.py”.

To use this tool, first modify your source code by decorating the
function you want to measure with the @profile decorator. Don’t worry,
you don’t have to import anyting in order to use this decorator. The
kernprof.py script automatically injects it into your script’s runtime
during execution.


def primes(n):
    if n==2:
        return [2]
    elif n<2:
        return []
    mroot = n ** 0.5
    while m <= mroot:
        if s[i]:
            while j<half:
    return [2]+[x for x in s if x]

Once you’ve gotten your code setup with the @profile decorator, use
kernprof.py to run your script.

$ kernprof.py -l -v fib.py

The -l option tells kernprof to inject the @profile decorator into
your script’s builtins, and -v tells kernprof to display timing
information once you’re script finishes. Here’s one the output should
look like for the above script:

Wrote profile results to primes.py.lprof
Timer unit: 1e-06 s

File: primes.py
Function: primes at line 2
Total time: 0.00019 s

Line #      Hits         Time  Per Hit   % Time  Line Contents
     2                                           @profile
     3                                           def primes(n):
     4         1            2      2.0      1.1      if n==2:
     5                                                   return [2]
     6         1            1      1.0      0.5      elif n<2:
     7                                                   return []
     8         1            4      4.0      2.1      s=range(3,n+1,2)
     9         1           10     10.0      5.3      mroot = n ** 0.5
    10         1            2      2.0      1.1      half=(n+1)/2-1
    11         1            1      1.0      0.5      i=0
    12         1            1      1.0      0.5      m=3
    13         5            7      1.4      3.7      while m <= mroot:
    14         4            4      1.0      2.1          if s[i]:
    15         3            4      1.3      2.1              j=(m*m-3)/2
    16         3            4      1.3      2.1              s[j]=0
    17        31           31      1.0     16.3              while j<half:
    18        28           28      1.0     14.7                  s[j]=0
    19        28           29      1.0     15.3                  j+=m
    20         4            4      1.0      2.1          i=i+1
    21         4            4      1.0      2.1          m=2*i+3
    22        50           54      1.1     28.4      return [2]+[x for x in s if x]

Look for lines with a high amount of hits or a high time interval. These
are the areas where optimizations can yield the greatest improvements.

How much memory does it use?

Now that we have a good grasp on timing our code, let’s move on to
figuring out how much memory our programs are using. Fortunately for us,
Fabian Pedregosa has implemented a nice memory
modeled after
Robert Kern’s line_profiler.

First install it via pip:

$ pip install -U memory_profiler
$ pip install psutil

(Installing the psutil package here is recommended because it greatly
improves the performance of the memory_profiler).

Like line_profiler, memory_profiler requires that you decorate your
function of interest with an @profile decorator like so:

def primes(n):

To see how much memory your function uses run the following:

$ python -m memory_profiler primes.py

You should see output that looks like this once your program exits:

Filename: primes.py

Line #    Mem usage  Increment   Line Contents
     2                           @profile
     3    7.9219 MB  0.0000 MB   def primes(n):
     4    7.9219 MB  0.0000 MB       if n==2:
     5                                   return [2]
     6    7.9219 MB  0.0000 MB       elif n<2:
     7                                   return []
     8    7.9219 MB  0.0000 MB       s=range(3,n+1,2)
     9    7.9258 MB  0.0039 MB       mroot = n ** 0.5
    10    7.9258 MB  0.0000 MB       half=(n+1)/2-1
    11    7.9258 MB  0.0000 MB       i=0
    12    7.9258 MB  0.0000 MB       m=3
    13    7.9297 MB  0.0039 MB       while m <= mroot:
    14    7.9297 MB  0.0000 MB           if s[i]:
    15    7.9297 MB  0.0000 MB               j=(m*m-3)/2
    16    7.9258 MB -0.0039 MB               s[j]=0
    17    7.9297 MB  0.0039 MB               while j<half:
    18    7.9297 MB  0.0000 MB                   s[j]=0
    19    7.9297 MB  0.0000 MB                   j+=m
    20    7.9297 MB  0.0000 MB           i=i+1
    21    7.9297 MB  0.0000 MB           m=2*i+3
    22    7.9297 MB  0.0000 MB       return [2]+[x for x in s if x]

Where’s the memory leak?

The cPython interpreter uses reference counting as it’s main method of
keeping track of memory. This means that every object contains a
counter, which is incremented when a reference to the object is stored
somewhere, and decremented when a reference to it is deleted. When the
counter reaches zero, the cPython interpreter knows that the object is
no longer in use so it deletes the object and deallocates the occupied

A memory leak can often occur in your program if references to objects
are held even though the object is no longer in use.

The quickest way to find these “memory leaks” is to use an awesome tool
called objgraph written by Marius
Gedminas. This tool allows you to see the number of objects in memory
and also locate all the different places in your code that hold
references to these objects.

To get started, first install objgraph:

$ pip install objgraph

Once you have this tool installed, insert into your code a statement to
invoke the debugger:

import pdb; pdb.set_trace()
Which objects are the most common?

At run time, you can inspect the top 20 most prevalent objects in your
program by running:

(pdb) import objgraph
(pdb) objgraph.show_most_common_types()

MyBigFatObject             20000
tuple                      16938
function                   4310
dict                       2790
wrapper_descriptor         1181
builtin_function_or_method 934
weakref                    764
list                       634
method_descriptor          507
getset_descriptor          451
type                       439
Which objects have been added or deleted?

We can also see which objects have been added or deleted between two
points in time:

(pdb) import objgraph
(pdb) objgraph.show_growth()
(pdb) objgraph.show_growth()   # this only shows objects that has been added or deleted since last show_growth() call

traceback                4        +2
KeyboardInterrupt        1        +1
frame                   24        +1
list                   667        +1
tuple                16969        +1
What is referencing this leaky object?

Continuing down this route, we can also see where references to any
given object is being held. Let’s take as an example the simple program

x = [1]
y = [x, [x], {"a":x}]
import pdb; pdb.set_trace()

To see what is holding a reference to the variable x, run the
objgraph.show_backref() function:

(pdb) import objgraph
(pdb) objgraph.show_backref([x], filename="/tmp/backrefs.png")

The output of that command should be a PNG image stored at
/tmp/backrefs.png and it should look something like this:

back refrences

The box at the bottom with red lettering is our object of interest. We
can see that it’s referenced by the symbol x once and by the list y
three times. If x is the object causing a memory leak, we can use this
method to see why it’s not automatically being deallocated by tracking
down all of its references.

So to review, objgraph allows us to:

  • show the top N objects occupying our python program’s memory
  • show what objects have been deleted or added over a period of time
  • show all references to a given object in our script

Effort vs precision

In this post, I’ve shown you how to use several tools to analyze a
python program’s performance. Armed with these tools and techniques you
should have all the information required to track down most memory leaks
as well as identify speed bottlenecks in a Python program.

As with many other topics, running a performance analysis means
balancing the tradeoffs between effort and precision. When in doubt,
implement the simplest solution that will suit your current needs.



Sunday, February 12, 2012

Linux.conf.au 2012活动中介绍了一个很给力的工具safe-rm,
它重新封装了一下/bin/rm, 对Linux系统管理员很有用的,


apt-get install safe-rm


$ rm -rf /usr
Skipping /usr

3,通过配置/etc/safe-rm.conf~/.safe-rm 添加你的需要保护的路径或文件

Hidden tips 001

Tuesday, November 22, 2011

hidden tips

1, install ruby-1.9.3 on Mac OSX Lion via rvm

 rvm install 1.9.3 --with-gcc=clang

2, package name with underscore in buildout, should be replace underscore to DASH, for example pyramid_jinja2 in buildout.

 pyramid-jinja2 = 1.2

3, install proxychains on Mac OSX Lion via homebrew
download my Proxychains Formula, and run brew install proxychains

4, setting pyramid_debugtoolbar.
If the request’s REMOTE_ADDR is not, u should add config debugtoolbar.hosts in your .ini file, for example:

debugtoolbar.hosts =

Upgrade postgresql-8.4 to postgresql-9.1

Tuesday, November 22, 2011

install postgresql 9.1 on ubuntu via apt-get

1, back up your databases

 ~ pg_dumpall > outputfile

2, add postgresql apt repository

~ sudo add-apt-repository ppa:pitti/postgresql

3, remove postgresql-8.4

~ sudo apt-get remove postgresql-8.4

4, update apt source index

~ sudo apt-get update

5, install postgresql-9.1

~ sudo apt-get install postgresql-9.1

6, create new user for postgres

~ sudo -u postgres sh
[sudo] password for eric: 
$ createuser -P eric
Enter password for new role: 
Enter it again: 
Shall the new role be a superuser? (y/n) y
$ exit

7, restore your data from backup

~ psql -d postgres -f outputfile


Mac OS X 下无密钥方式连接基于L2TP协议的VPN

Monday, November 7, 2011

需要连接一个L2TP协议的vpn, 填好信息竟然报错“IPSec 共享密钥”丢失。请验证您的设置并尝试重新连接。 但是这个vpn不需要IPSec 共享密钥啊, google了一把发现需要打补丁来绕过它。

/etc/ppp目录下新建一个文件options, 写入下面的内容

plugin L2TP.ppp


In Pyramid-based website production environments, should generate a 404 NotFound Error instead of Server Error `URLDecodeError`.

Sunday, October 30, 2011

a unknown url
http://lxneng.com/sms/ests%C3%CD%E1%EF2.asp always request my
Pyramid-based website server, it will be raise a URLDecodeError.


Traceback (most recent call last):
File "/root/env/lib/python2.6/site-packages/repoze.tm2-1.0b2-py2.6.egg/repoze/tm/__init__.py", line 24, in __call__
    result = self.application(environ, save_status_and_headers)
File "/var/www/lxneng/src/lxneng/__init__.py", line 27, in __call__
    return self.application(environ, start_response)
File "/root/env/lib/python2.6/site-packages/pyramid/router.py", line 176, in __call__
    response = self.handle_request(request)
File "/root/env/lib/python2.6/site-packages/pyramid/tweens.py", line 17, in excview_tween
    response = handler(request)
File "/root/env/lib/python2.6/site-packages/pyramid/router.py", line 116, in handle_request
    tdict = traverser(request)
File "/root/env/lib/python2.6/site-packages/pyramid/traversal.py", line 610, in __call__
    vpath_tuple = traversal_path(vpath)
File "/root/env/lib/python2.6/site-packages/repoze/lru/__init__.py", line 96, in lru_cached
    val = f(*arg)
File "/root/env/lib/python2.6/site-packages/pyramid/traversal.py", line 486, in traversal_path
    raise URLDecodeError(e.encoding, e.object, e.start, e.end, e.reason)
URLDecodeError: 'utf8' codec can't decode bytes in position 4-5: invalid data

so this should generate a 404 instead of a 500 internal server error.

@view_config(context='pyramid.exceptions.URLDecodeError', renderer='404.html')
@view_config(context='pyramid.exceptions.NotFound', renderer='404.html')
def error_view(context, request):
    return {}

Pyramid project paster command line tools script template

Wednesday, October 26, 2011

1, new a commands script file commands.py

from paste.script.command import Command
import os.path
from pyramid.threadlocal import manager as threadlocal_manager
from paste.deploy import loadapp

class PyramidCommand(Command):

    group_name = "lxneng"
    min_args = 1
    max_args = 1
    parser = Command.standard_parser()

    def setupPyramid(self):
        app = loadapp('config:%s' % self.args[0], relative_to=os.path.abspath('.'))
        registry = app.registry

class DemoCommand(PyramidCommand):
    summary = "demo script"
    usage = "paster --plugin=lxneng demo_script development.ini"
    parser = PyramidCommand.standard_parser()

    def command(self):
        from lxneng.models.post import Tag
        tag = Tag.find_by_name('Python')
        print tag.name

2, add command to setup.py

  main = lxneng:main

  demo_script = lxneng.commands:DemoCommand

3, build

python setup.py develop

4, run

paster --plugin=lxneng demo_script development.ini

lxneng is my project name

Objective-C 扩展名

Tuesday, October 25, 2011

  • .h 头文件, 头文件包含类, 类型, 函数和常数的声明.
  • .m 源代码文件, 这是典型的源代码文件扩展名, 可以包含Objective-C和C代码.
  • .mm 源代码文件, 带有这种扩展名的源代码文件, 除了可以包含Objective-C和C代码以外还可以包含C++代码. 仅在你的Objective-C代码中确实需要使用C++类或者特性的时候才使用这种扩展名.

Setup ipython to work in a virtualenv

Monday, August 29, 2011

add this script to \~/.ipython/ipython_virtualenv.py

#Call this inside ~/.ipython/ipy_user_conf.py main()

import sys
import subprocess
from os import environ

if 'VIRTUAL_ENV' in environ:
    #This is kludgy but it works; grab the right sys.path from the virtualenv python install:
    path = subprocess.Popen(['python', '-c','import sys;print(repr(sys.path))'],
    sys.path = eval(path)
    del path

del sys, subprocess, environ

from https://gist.github.com/817737

using gettext convert .mo to .po, and .po to .mo

Monday, August 29, 2011

install gettext

on Mac OSX via Homebrew

brew install gettext

on Ubuntu via apt-get

apt-get install gettext

mo to po ###

msgunfmt xxx.mo -o xxx.po

po to mo

msgfmt xxx.po -o xxx.mo

Manage (Add, Edit, & Delete) cookies in jQuery

Monday, June 27, 2011

Setting and deleting cookies with jQuery is really easy (especially in
comparison to regular JavaScript) but this feature is not included in
the jQuery core. For this we need a plug-in. This post shows how to set
and get the value of cookies with jQuery.

First download the jQuery cookie plugin for here:

Set a cookie

Setting a cookie with jQuery is as simple as this, here
we are creating a cookie called “example” with a value “demo”:

$.cookie("example", "demo");

This is a session cookie and will be destroy when user close his/her
browser. To make the same cookie for suppose 7 days. We can do it like

$.cookie("example", "demo", { expires: 7 });

The above example will create the cookie at the root level. If you
wanted to make it apply only to e.g. “/admin” and make it for 7 days you
can do it like this:

$.cookie("example", "demo", { path: '/admin', expires: 7 });

Get the cookie’s value

Getting the cookie’s value is also very easy
in jQuery. The following would alert the value of “example” cookie:

alert( $.cookie("example") );

Delete the cookie

And finally, to delete a cookie set its value to
Note:Setting it to e.g. an empty string doesn’t remove it;
it just clears the value.

$.cookie("example", null);


Sunday, June 12, 2011

  • 好设计是简单的设计
  • 好设计是永远不过时的设计
  • 好设计是解决主要问题的设计
  • 好设计是启发性的设计
  • 好设计通常是有点趣味性的设计
  • 好设计是艰苦的设计
  • 好设计是看似很容易的设计
  • 好设计是对称的设计
  • 好设计是模仿大自然的设计
  • 好设计是一种再设计
  • 好设计是能够复制的设计
  • 好设计常常是奇特的设计
  • 好设计是成批出现的
  • 好设计常常是大胆的设计


UUID in SQLAlchemy

Friday, June 10, 2011

Add a Custom Types


# model/uuid.py
from __future__ import absolute_import
import uuid
from sqlalchemy.types import TypeDecorator, CHAR
from sqlalchemy.dialects.postgresql import UUID

class GUID(TypeDecorator):
    """Platform-independent GUID type.

    Uses Postgresql's UUID type, otherwise uses
    CHAR(32), storing as stringified hex values.

    impl = CHAR

    def load_dialect_impl(self, dialect):
        if dialect.name == 'postgresql':
            return dialect.type_descriptor(UUID())
            return dialect.type_descriptor(CHAR(32))

    def process_bind_param(self, value, dialect):
        if value is None:
            return value
        elif dialect.name == 'postgresql':
            return str(value)
            if not isinstance(value, uuid.UUID):
                return "%.32x" % uuid.UUID(value)
                # hexstring
                return "%.32x" % value

    def process_result_value(self, value, dialect):
        if value is None:
            return value
            return uuid.UUID(value)



# model/user.py
from you.app.model.uuid import GUID
from you.app.model import meta

class User(meta.BaseObject):

    __tablename__ = "user"

    uuid = schema.Column(GUID(), default=uuid.uuid4, nullable=False, unique=True)


# model/meta.py
from sqlalchemy import schema
from sqlalchemy.ext.declarative import declarative_base

__all__ = ["Session", "metadata", "BaseObject" ]

Session = None

metadata = schema.MetaData()
BaseObject = declarative_base(metadata=metadata)

reference: Backend-agnostic GUID

ajax get server side time string, pyramid example

Friday, June 10, 2011

ajax get server side time string, pyramid example

config.add_route("servertime", "/servertime")

#server side
@view_config(route_name="servertime", renderer='string')
def serverTime(request):
    return datetime.now()

var timeStr = $.ajax({url: "/servertime", async: false}).responseText;

UUID primary keys in Rails 3

Friday, June 10, 2011


# Gemfile
gem 'uuidtools'

# 1. :id => false
# 2. :uuid

class CreateSites < ActiveRecord::Migration
  def self.up
    create_table(:sites, :id => false) do |t|
      t.string :uuid, :limit => 36, :primary => true

  def self.down
    drop_table :sites


# app/models/site.rb
class Site < ActiveRecord::Base
  include Extensions::UUID


# app/models/extensions/uuid.rb
module Extensions
  module UUID
    extend ActiveSupport::Concern

    included do
      set_primary_key 'uuid'
      before_create :generate_uuid

      def generate_uuid
        self.id = UUIDTools::UUID.random_create.to_s

SQLAlchemy's not in

Friday, June 10, 2011

from sqlalchemy import not_ 
session.query(Region).filter(not_(Region.id.in_( (1,2,3) )))

div>img 底部对齐

Friday, June 10, 2011

外层div的位置设置成绝对的, img 设置成块级元素,位置设置成相对的,bottom设置成0

html code

<div class="bodyShape">
    <div class="barbara">
        <img src="${tools.static_url('media/images/figure-analysis-barbara-82x135.png')}" />

css code

.bodyShape{height:150px; width:200px;border-right: 2px solid #ccc;}
.bodyShape .barbara{position:relative;height:100%;}
.bodyShape .barbara img{bottom:0;display:block;position:absolute;}

© 2009-2013 lxneng.com. All rights reserved. Powered by Pyramid

go to Top