Home/ Questions/Q 97355
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-10T23:59:20+00:00

I’m writing a reasonably complex web application. The Python backend runs an algorithm whose

  • 0

I’m writing a reasonably complex web application. The Python backend runs an algorithm whose state depends on data stored in several interrelated database tables which does not change often, plus user specific data which does change often. The algorithm’s per-user state undergoes many small changes as a user works with the application. This algorithm is used often during each user’s work to make certain important decisions.

For performance reasons, re-initializing the state on every request from the (semi-normalized) database data quickly becomes non-feasible. It would be highly preferable, for example, to cache the state’s Python object in some way so that it can simply be used and/or updated whenever necessary. However, since this is a web application, there several processes serving requests, so using a global variable is out of the question.

I’ve tried serializing the relevant object (via pickle) and saving the serialized data to the DB, and am now experimenting with caching the serialized data via memcached. However, this still has the significant overhead of serializing and deserializing the object often.

I’ve looked at shared memory solutions but the only relevant thing I’ve found is POSH. However POSH doesn’t seem to be widely used and I don’t feel easy integrating such an experimental component into my application.

I need some advice! This is my first shot at developing a web application, so I’m hoping this is a common enough issue that there are well-known solutions to such problems. At this point solutions which assume the Python back-end is running on a single server would be sufficient, but extra points for solutions which scale to multiple servers as well 🙂

Notes:

  • I have this application working, currently live and with active users. I started out without doing any premature optimization, and then optimized as needed. I’ve done the measuring and testing to make sure the above mentioned issue is the actual bottleneck. I’m sure pretty sure I could squeeze more performance out of the current setup, but I wanted to ask if there’s a better way.
  • The setup itself is still a work in progress; assume that the system’s architecture can be whatever suites your solution.
Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. I think that the multiprocessing framework has what might be applicable here – namely the shared ctypes module.

    Multiprocessing is fairly new to Python, so it might have some oddities. I am not quite sure whether the solution works with processes not spawned via multiprocessing.

    • 0