Python Imports Can Be Tricky

Imports are based on the module path, so if you import the same object using different paths, Python will see them as different objects.

Okay, But Why Does That Matter?

At work I maintain an older Python application that runs 2.6.x and was written by people who probably did not write as much Python as they did other languages. This was easy for me, an accredited MD, to diagnose by looking at a lot of patterns used throughout the codebase, but it really stands out in the over use of global Singleton-like objects. Using these Singletons causes some erros in our system, but not in the way you may be thinking.

Yeah, there are situations where values are being set on the Singleton that are hard to, if not impossible, trace, or values exist in it that are not applicable in other situations, but my head-scratching issue actually had to do with Python and it handles imports.

What Happened To Me and How it Was Fixed

Long story short: one of these Singleton object had properties set during the execution stack that simply were not there later in the code-base. This was a major pain and I had do some weird development machine-level debugging on one of the higher environments. It was a mess and took way too much time to figure out. I don't want you to make the same mistake.

So I went straight to #python on freenode via IRC and asked a really long question involving ids being different for the same class that was being used like a Singleton. That is when Ned Batchelder chimed in and said something like "the only way a class will have a different id is if it were redefined." I did a quick search of the codebase to ensure that wasn't the case, but then I said "would it matter if they were imported differently?" Ned said yeah and shot me a link (which I don't have, atm. Ned shoot me the link) to a talk that he gave on a similar topic.

Lets Recreate the Bug

This is easy to understand once you've been through it like I have. Your app just needs to meet some simple criteria:

Have your PYTHONPATH include both the package and sub-package on the same path
Define a Singleton object in that path
And have your code import that object two different ways

The folder structure looks something like this:

Next we'll create our Singleton object and place it in wtf/lib/single/ton.py, meeting criteria #2:

class Singleton(object):

    prop = 'DEFAULT'
    i = None

    @classmethod
    def increment(cls):
        if not cls.i:
            cls.i = 0

        cls.i += 1

    @classmethod
    def info(cls):
        _id = id(cls)

        print('\tSingleton id: {} prop: {} i: {}'.format(_id, cls.prop, cls.i))

This class doesn't do much, but it will give us the coverage that we need to better understand things. The info method shows the state of the object at any given time by outputting its id, prop, and i members.

Add this code to one.py, it nests package paths and adds them to the PYTHONPATH, meeting the first point in our criteria defined above:

import os
import sys


here = os.path.dirname(__file__)
wtf = here
lib = os.path.join(here, 'lib')

sys.path.append(wtf)
sys.path.append(lib)

We'll be adding two functions to the file that will meet the final point; different imports for the same class.

def single_ton():
    from single.ton import Singleton

    Singleton.prop = 'THIS IS FROM single_ton'

    print('single.ton import')
    Singleton.info()
    Singleton.increment()


def lib_single_ton():
    from lib.single.ton import Singleton

    print('lib.single.ton import')
    Singleton.info()

Each function with its descriptive name, imports the Singleton class its own way and proceeds to call the info method.

We'll wrap up one.py by simply calling the functions in order.

single_ton()
lib_single_ton()
single_ton()

What do you expect the output to be? Well, if you read the first part of this post, you'll know that the Singleton classes are different objects.

And as you can see, the calls that imported Singleton with the same path share an instance, while the outlier used a different path.

What Is Happening?

When you import in Python the first thing that is checked is sys.modules and if the module that your object is apart of isn't there it will be added. In our case we first import Singleton form the module single.ton and that is added to sys.moduels. Then when we go to get the same object from lib.single.ton, a different module, it correctly says that module doesn't exist in sys.modules, the system will create a new one and add it to the stack. This is also why we are import the same objects over and over again without any real impact on performance.

Wrapping Up

This was an interesting error to track down in the code base because everything looked correct to the naked eye and then all of a sudden it would error out (I will admit that we only saw the error after I changed some imports, but whatever). In the future keep an eye out for those nested module additions to the PYTHONPATH and standardize the imports in your application.

@EmEhRKay