quora - python expertise levels

Python progression path-From apprentice to guru (13)

Learning algorithms/maths/file IO/Pythonic optimisation

This won't get you guru-hood but to start out, try working through the Project Euler problems The first 50 or so shouldn't tax you if you have decent high-school mathematics and know how to Google. When you solve one you get into the forum where you can look through other people's solutions which will teach you even more. Be decent though and don't post up your solutions as the idea is to encourage people to work it out for themselves.

Forcing yourself to work in Python will be unforgiving if you use brute-force algorithms. This will teach you how to lay out large datasets in memory and access them efficiently with the fast language features such as dictionaries.

From doing this myself I learnt:

  • File IO
  • Algorithms and techniques such as Dynamic Programming
  • Python data layout
    • Dictionaries/hashmaps
    • Lists
    • Tuples
    • Various combinations thereof, e.g. dictionaries to lists of tuples
  • Generators
  • Recursive functions
  • Developing Python libraries
    • Filesystem layout
    • Reloading them during an interpreter session

And also very importantly

  • When to give up and use C or C++!

All of this should be relevant to Bioinformatics

Admittedly I didn't learn about the OOP features of Python from that experience.

I've been learning, working, and playing with Python for a year and a half now. As a biologist slowly making the turn to bio-informatics, this language has been at the very core of all the major contributions I have made in the lab. I more or less fell in love with the way Python permits me to express beautiful solutions and also with the semantics of the language that allows such a natural flow from thoughts to workable code.

What I would like to know is your answer to a kind of question I have seldom seen in this or other forums. This question seems central to me for anyone on the path to Python improvement but who wonders what his next steps should be.

Let me sum up what I do NOT want to ask first ;)

  • I don't want to know how to QUICKLY learn Python
  • Nor do I want to find out the best way to get acquainted with the language
  • Finally, I don't want to know a 'one trick that does it all' approach.

What I do want to know your opinion about, is:

What are the steps YOU would recommend to a Python journeyman, from apprenticeship to guru status (feel free to stop wherever your expertise dictates it), in order that one IMPROVES CONSTANTLY, becoming a better and better Python coder, one step at a time. Some of the people on SO almost seem worthy of worship for their Python prowess, please enlighten us :)

The kind of answers I would enjoy (but feel free to surprise the readership :P ), is formatted more or less like this:

  • Read this (eg: python tutorial), pay attention to that kind of details
  • Code for so manytime/problems/lines of code
  • Then, read this (eg: this or that book), but this time, pay attention to this
  • Tackle a few real-life problems
  • Then, proceed to reading Y.
  • Be sure to grasp these concepts
  • Code for X time
  • Come back to such and such basics or move further to...
  • (you get the point :)

I really care about knowing your opinion on what exactly one should pay attention to, at various stages, in order to progress CONSTANTLY (with due efforts, of course). If you come from a specific field of expertise, discuss the path you see as appropriate in this field.

EDIT: Thanks to your great input, I'm back on the Python improvement track! I really appreciate!

Thoroughly Understand All Data Types and Structures

For every type and structure, write a series of demo programs that exercise every aspect of the type or data structure. If you do this, it might be worthwhile to blog notes on each one... it might be useful to lots of people!

Understand Introspection

  • write a dir() equivalent
  • write a type() equivalent
  • figure out how to "monkey-patch"
  • use the dis module to see how various language constructs work

Doing these things will

  • give you some good theoretical knowledge about how python is implemented
  • give you some good practical experience in lower-level programming
  • give you a good intuitive feel for python data structures

Download Twisted and look at the source code. They employ some pretty advanced techniques.

I learned python first by myself over a summer just by doing the tutorial on the python site (sadly, I don't seem to be able to find that anymore, so I can't post a link).

Later, python was taught to me in one of my first year courses at university. In the summer that followed, I practiced with PythonChallenge and with problems from Google Code Jam. Solving these problems help from an algorithmic perspective as well as from the perspective of learning what Python can do as well as how to manipulate it to get the fullest out of python.

For similar reasons, I have heard that code golf works as well, but i have never tried it for myself.

I recommend starting with something that forces you to explore the expressive power of the syntax. Python allows many different ways of writing the same functionality, but there is often a single most elegant and fastest approach. If you're used to the idioms of other languages, you might never otherwise find or accept these better ways. I spent a weekend trudging through the first 20 or so Project Euler problems and made a simple webapp with Django on Google App Engine. This will only take you from apprentice to novice, maybe, but you can then continue to making somewhat more advanced webapps and solve more advanced Project Euler problems. After a few months I went back and solved the first 20 PE problems from scratch in an hour instead of a weekend.

I'll give you the simplest and most effective piece of advice I think anybody could give you: code.

You can only be better at using a language (which implies understanding it) by coding. You have to actively enjoy coding, be inspired, ask questions, and find answers by yourself.

Got a an hour to spare? Write code that will reverse a string, and find out the most optimum solution. A free evening? Why not try some web-scraping. Read other peoples code. See how they do things. Ask yourself what you would do.

When I'm bored at my computer, I open my IDE and code-storm. I jot down ideas that sound interesting, and challenging. An URL shortener? Sure, I can do that. Oh, I learnt how to convert numbers from one base to another as a side effect!

This is valid whatever your skill level. You never stop learning. By actively coding in your spare time you will, with little additional effort, come to understand the language, and ultimately, become a guru. You will build up knowledge and reusable code and memorise idioms.

If you're in and using python for science (which it seems you are) part of that will be learning and understanding scientific libraries, for me these would be

  • numpy
  • scipy
  • matplotlib
  • mayavi/mlab
  • chaco
  • Cython

knowing how to use the right libraries and vectorize your code is essential for scientific computing.

I wanted to add that, handling large numeric datasets in common pythonic ways(object oriented approaches, lists, iterators) can be extremely inefficient. In scientific computing, it can be necessary to structure your code in ways that differ drastically from how most conventional python coders approach data.

One good way to further your Python knowledge is to dig into the source code of the libraries, platforms, and frameworks you use already.

For example if you're building a site on Django, many questions that might stump you can be answered by looking at how Django implements the feature in question.

This way you'll continue to pick up new idioms, coding styles, and Python tricks. (Some will be good and some will be bad.)

And when you see something Pythony that you don't understand in the source, hop over to the #python IRC channel and you'll find plenty of "language lawyers" happy to explain.

An accumulation of these little clarifications over years leads to a much deeper understanding of the language and all of its ins and outs.

Teaching to someone else who is starting to learn Python is always a great way to get your ideas clear and sometimes, I usually get a lot of neat questions from students that have me to re-think conceptual things about Python.

def apprentice():
  experiment(interpreter, modules/files)

def master():
  refer(PEPs/language reference)
  read(good_python_code) # Eg. twisted, other libraries
  write(basic_library)   # reinvent wheel and compare to existing wheels
  if have_interesting_ideas:

def guru():
  pass # Not qualified to comment. Fix the GIL perhaps?