Using python suds in Multi Threaded and Multi Process Environments16 Sep 2014
Working on a team that primarily builds integrations between systems, you will inevitably end up writing integrations against SOAP based APIs. If, like me, your team uses python, you will likely end up using Suds. As you can see from this stackoverflow post, there aren’t a great deal of options for SOAP in python; and none are phenomenal.
While Suds is not actively maintained, it has wide adoption, and is fairly stable and mature. Suds is actually a pretty good abstraction around SOAP, and it encourages idiomatic python. I have my fair share of frustrations with Suds, but for the most part, it “just works”. It does, however, have two major flaws that make it unusable out of the box for my use-case.
Suds is not Thread safe
For me, this manifested only in production (wsgi is designed such that most development environments, including mine, are single threaded) with a SAXParseException:
Ideally this could be fixed in suds properly, but since the library is no-longer maintained, we opted for a simple workaround. Basically we use a thread safe queue, to add a lock around suds usage:
To Set Up:
This technique obviously requires additional resources compared to using a
Client instance, but for us that trade-off is worth it. If you
have flexible real-time requirements, or limited parallelism, you can make
MAX_THREADS = 1 to achieve thread safety without any overhead.
Suds is not Friendly in Multi-Process Environments
After we solved our thread safety problems, we ran into another exception coming
from scripts running on the same server as our wsgi application. The
exceptions were of the form:
There is an open ticket related to this bug in suds, but we didn’t want to fork suds to apply the patch provided. Instead, we found an alternative solution that works well for us.
To understand our solution to this problem, we need to look more closely at how
to reproduce it. First we must have a system in which there are at least two
users - in our case
app-prod. The first of these users (say,
apache) to use a suds client will create a directory at
/tmp/suds which it
will store the suds document cache. Assuming the
umask of this user is
configured in a standard way, and that
app-prod is not in the
the process run by
app-prod will suffer the dreaded Permission denied
Suds allows us to provide an
ObjectCache object to the client constructor, and
in turn the
ObjectCache constructor allows us to specify the path of the
We initially considered adding the
pid to the path to ensure permission safety,
but were concerned with the side effect of creating many folders as the
lifetime of the machine goes on (every script that uses suds would create a new
folder, and our default system only cleans up
/tmp on reboot. Instead we
decided to use the
uid of the process, which would protect us from permission
issues, while limiting the number of folders to the number of active users.
Check out the resulting class.
Hopefully these tricks will help you sucessfully use
suds in your production