Home/ Questions/Q 4028882
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-20T11:16:32+00:00

Current ZMI management Pack database functionality is little rough. 1) Could it be possible

  • 0

Current ZMI management “Pack database” functionality is little rough.

1) Could it be possible to have some kind of progress indicator for web UI? E.g. one telling how many minutes/hours are left giving at least some kind of estimate

2) How does ZODB packing affect the responsivity of the site? Are all transactions blocked?

3) Any command line scripts with progress indicator available so you could do this from a ZEO command line client?

4) At least some kind of log markers to logout output… [INFO] 30% done… 3:15 to go

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Packing Anatomy

    ZODB FileStorage packing is process of selective copying of data from one file to another one (only transactions that are “younger” then specified age). Before this copying starts some soft of index is built in memory to aid in process. Thus whole ZODB packing contains following steps:

    1. Building pack index
    2. Copying transactions to temporary file
    3. Appending transactions that were performed after packing started
    4. Replacing original FileStorage with packed one and reopening it in read/write mode

    I’m usually monitoring the process by combination of top, vmstat/dstat, watch ls -la var/filestorage.

    As Geir mentioned, you can have separate ZEO client dedicated to packing. This was reasonable as thread you invoked packing from blocked until packing finished. Now there is no need to if you use ZEO. ZEO server provides zeopack utility that connects directly to ZEO (no need for dedicated ZEO client) and initiates FileStorage packing. One of the benefits is no need for password, just proper permissions to access ZEO control socket.

    Packing progress

    As packing is performed by ZEO server (even not server but FileStorage itself), possibility of proper communication of progress to ZEO client is limited. ZEO protocol was not designed to communicate that type of information.

    IMHO FileStorage itself could be more verbose in communicating through log file what it is doing right now. Some kind of progress could be built in. And if you feel like need the progress indicator, then you can design some kind of feedback channel through logging module back to ZEO-client/Zope-instance to be communicated back to browser.

    Performance while packing

    As FileStorage packing is quite intensive disk operation, it reduces throughoutput of disk subsystem. Additionally it expunges disk cache (in case of larger FileStorage), that impact disk performance even after packing finished, as caches should be warmed up again. Possible improvements that lead to longer packing time but smaller impact on system in FileStorage are:

    • reverting to O_DIRECT operations (not to touch file cache)
    • reducing disk scheduling priority (ionice on Linux) for thread performing the packing
    • throttling packing speed
    • 0