Working on a team that primarily builds integrations between systems, you will inevitably end up writing integrations against SOAP based APIs. If, like me, your team uses python, you will likely end up using Suds. As you can see from this stackoverflow post, there aren’t a great deal of options for SOAP in python; and none are phenomenal.
While Suds is not actively maintained, it has wide adoption, and is fairly stable and mature. Suds is actually a pretty good abstraction around SOAP, and it encourages idiomatic python. I have my fair share of frustrations with Suds, but for the most part, it “just works”. It does, however, have two major flaws that make it unusable out of the box for my use-case.
Suds is not Thread safe
The Problem
For me, this manifested only in production (wsgi is designed such that most development environments, including mine, are single threaded) with a SAXParseException:
File "/venv_path/lib/python2.6/site-packages/suds/client.py", line 542, in __call__
return client.invoke(args, kwargs)
File "/venv_path/lib/python2.6/site-packages/suds/client.py", line 602, in invoke
result = self.send(soapenv)
File "/venv_path/lib/python2.6/site-packages/suds/client.py", line 649, in send
result = self.failed(binding, e)
File "/venv_path/lib/python2.6/site-packages/suds/client.py", line 702, in failed
r, p = binding.get_fault(reply)
File "/venv_path/lib/python2.6/site-packages/suds/bindings/binding.py", line 258, in get_fault
faultroot = sax.parse(string=reply)
File "/venv_path/lib/python2.6/site-packages/suds/sax/parser.py", line 136, in parse
sax.parse(source)
File "/usr/lib64/python2.6/xml/sax/expatreader.py", line 107, in parse
xmlreader.IncrementalParser.parse(self, source)
File "/usr/lib64/python2.6/xml/sax/xmlreader.py", line 123, in parse
self.feed(buffer)
File "/usr/lib64/python2.6/xml/sax/expatreader.py", line 211, in feed
self._err_handler.fatalError(exc)
File "/usr/lib64/python2.6/xml/sax/handler.py", line 38, in fatalError
raise exception
SAXParseException: <unknown>:15:2: mismatched tag
The Solution
Ideally this could be fixed in suds properly, but since the library is no-longer maintained, we opted for a simple workaround. Basically we use a thread safe queue, to add a lock around suds usage:
To Set Up:
from Queue import Queue
from contextlib import contextmanager
MAX_THREADS = 5
suds_service_queue = Queue(MAX_THREADS)
for n in range(MAX_THREADS):
suds_service_queue.put(construct_suds_client())
@contextmanager
def get_suds_client():
client = suds_service_queue.get() # blocks until client available
try:
yield client
finally:
suds_service_queue.put(client)
To Use:
from setup import get_suds_client
with get_suds_client() as client:
client.do_stuff
This technique obviously requires additional resources compared to using a
single suds Client instance, but for us that trade-off is worth it. If you
have flexible real-time requirements, or limited parallelism, you can make
MAX_THREADS = 1 to achieve thread safety without any overhead.
Suds is not Friendly in Multi-Process Environments
The Problem
After we solved our thread safety problems, we ran into another exception coming
from scripts running on the same server as our wsgi application. The OSError
exceptions were of the form:
File "/venv_path/venv/lib/python2.6/site-packages/suds/client.py", line 109, in __init__
options.cache = ObjectCache(days=1)
File "/venv_path/venv/lib/python2.6/site-packages/suds/cache.py", line 145, in __init__
self.checkversion()
File "/venv_path/venv/lib/python2.6/site-packages/suds/cache.py", line 277, in checkversion
self.clear()
File "/venv_path/venv/lib/python2.6/site-packages/suds/cache.py", line 251, in clear
os.remove(os.path.join(self.location, fn))
OSError: [Errno 13] Permission denied: '/tmp/suds/suds-6882323804701353659-document.px'
There is an open ticket related to this bug in suds, but we didn’t want to fork suds to apply the patch provided. Instead, we found an alternative solution that works well for us.
The Solution
To understand our solution to this problem, we need to look more closely at how
to reproduce it. First we must have a system in which there are at least two
users - in our case apache and app-prod. The first of these users (say,
apache) to use a suds client will create a directory at /tmp/suds which it
will store the suds document cache. Assuming the umask of this user is
configured in a standard way, and that app-prod is not in the apache group
the process run by app-prod will suffer the dreaded Permission denied OSError
Suds allows us to provide an ObjectCache object to the client constructor, and
in turn the ObjectCache constructor allows us to specify the path of the
document cache.
We initially considered adding the pid to the path to ensure permission safety,
but were concerned with the side effect of creating many folders as the
lifetime of the machine goes on (every script that uses suds would create a new
folder, and our default system only cleans up /tmp on reboot. Instead we
decided to use the uid of the process, which would protect us from permission
issues, while limiting the number of folders to the number of active users.
Check out the resulting class.
import os
import suds
from suds.cache import ObjectCache
class SudsApi(object):
_suds_client = None
def __init(self, wsdl_url):
self.wsdl_url = wsdl_url
@property
def client(self):
if self._suds_client is None:
cache_path = "/tmp/{0}-suds".format(os.getuid())
cache = ObjectCache(cache_path, days=1)
self._suds_client = suds.client.Client(self.wsdl_url,
cache=cache)
return self._suds_client
Conclusion
Hopefully these tricks will help you sucessfully use suds in your production
environment.