Way of rebuilding only broken images / thumbnails?
|
chalsall
Joined: 2008-05-06
Posts: 5 |
Posted: Tue, 2008-05-06 14:09
|
|
I recently installed Gallery 2 on a running club's site (http://www.barbadoshash.com/gallery2/), and am quite impressed. However, I have found an issue generating thumbnails and scaled images. Some are randomly broken. I think this is because of 1and1 (the hosting site) killing processes when the load is high. My question is, is there a way to have the system regenerate only those scaled images and thumbnails which are broken, rather than all? What I am finding is if there are a number of broken thumbnails, and then I try to regenerate them via the admin area, another random set of thumbnails will be broken. Short of editing each image by hand, going to the "Crop Thumbnail" tab, and "Reset to Default", there is no way to ensure that all the thumbs will be generated. This is obviously very manually intensive. It would be very handy if in the "Edit Album" section, "Album" tab, that there was two additional check boxes in the "Apply settings to existing items": "Build broken thumbnails" and "Build broken sizes". Also, a report listing all broken thumbnails and resizes would help a great deal as well. I am a programmer, and have spent some time trying to get to the bottom of this. I have tried using GD, ImageMagick and NetPBM -- the latter both the 1and1 installed and the Mini Binary version. All exhibit the same issue. PHP Memory limit is set to 40 Megs. The issue is correlative only with server load. I assume others have experienced this issue, and so I also assume (or at least, hope) that a solution as above has received some attention. Rather than reinventing the wheel, I would appreciate any pointers anyone can provide. If not, I will go ahead and try to implement something which automates the above. Thanks for any assistance / insight anyone can provide. Chris Halsall |
|
| Login or register to post comments |
Posts: 9
If you think the processes are being killed, have you tried to lower the priority in the helper process section in the site admin panel?
Did you get error output from site admin > maintenance > rebuild thumbs for any items? Were they these items? Are these items using CMYK? After I built 139,000 thumbnails, I had about 20 that were actually bad jpgs, but would still show in a browser. IF you re-upload one of those pictures, does it generate a thumbnail then? If it doesn't, it could be that the helper processes just can't make a thumbnail from it for some reason.
Posts: 5
Thanks for your reply nichiyume.
Yes, I have tried lowering the priority -- no difference. I have, however, noticed a correlation between errors in thumbnail generation and requests for gallery views from other users. Very explicitly, not other administrators, simply users viewing. And, often during admin work, the viewers will receive "500 -- Internal Server Error" messages.
Also also, to answer your other questions... Yes, I am aware of the issue with image files with errors -- that is not the issue in this case; these are perfectly valid images. And no, none of them are using CMYK; they are all simple JPGs from digital cameras.
An example log section showing an example of a error during thumbnail generation is below. Please note that /homepages/13/d140744427/htdocs/ is *not* the web-server root; it is the top-most user-writable directory under the 1and1 system. The actual server-root is one level below, and the /homepages/13/d140744427/htdocs/bh3_g2data/ directory is *not* accessible from the web.
These errors occur randomly with one or more images during generation, again, correlative with server load and/or other users *viewing* the gallery. I suspect this is 1and1, but unfortunately, it is the hosting service we're stuck with (at least, for the moment). Again, work-around would be to simply be able to regenerate only those thumbs which are broken.
Any insight anyone might be able to provide would be most appreciated.
Thanks and regards.
-Chris
Posts: 5
Hey all...
This is really, really stupid, but...
I modified NetPbmToolkit.class, starting at line 571 thusly:
for ($uglyhack = 0; $uglyhack < 100; $uglyhack++) { list ($ret, $output) = $this->_exec($command); if ($ret == 0) break; sleep(1); }This is only temporary, but it has solved the problem. It will, of course, cause problems (or at least, a 100 second delay per resize) in the case where an image is truly broken.
A longer-term solution would be a means of attempting to regenerate broken derivatives. Based on other posts here, I'm not the only one to have encountered this type of issue.
Please feel free to point out if I'm being a complete idiot here...
Thanks.
-Chris
Posts: 5
My thanks to nichiyume for pointing an idiot (yours truly) in the right direction...
"Site Admin" ("Admin Options") > "Maintenance" ("System Maintenance") > "Build all thumbnails/resizes" ... appears to do the first of what I was needing, although the language is not entirely clear on what this function actually does. If I may, the language suggests that the function rebuilds all thumbnails and resize derivatives, although the "help" language does suggest it only rebuilds "broken ones." I assume this is clearly documented somewhere that I missed.
Do please note that I am having to re-run this function multiple times, because the web interface showing progress stops after a period of time. Again, this is likely 1and1 limiting the consumption of resources. However, each time it is run it makes speedy progress to the point where it previously stopped, which is why I'm inferring that it is only rebuilding derivatives which are "broken".
To the second part of my first post... does there exist a means (via the web interface) of getting a report of broken derivatives? If not, I may undertake this personally.
Thank you again nichiyume. And my apologies for asking a question for which a solution was available. Again, I'm sure this was clearly documented somewhere...
-Chris
Posts: 5
OK -- to share some more of what I've learnt...
The issue I've encountered is a process limit at 1and1. Their documentation says they limit processes to 6, but in fact it is 12.
Since a image resize consumes five processes (plus the one running main.php), all it takes is for two other users to be viewing an image which hasn't been processed yet, and one or more other processes to be underway (e.g. browsing other pages; an open shell or two open (possibly tailing a log file, editing files, etc) and derivate generation will fail.
As a suggestion -- what about making it an option in the Graphic Toolkits which fork to use temporary files rather than pipelining? Obviously it would be a bit slower, but would greatly increase the number of concurrent derivative generations which could occur?
This also points out that 1and1.com (at least, their shared servers) are not appropriate for popular Gallery 2s....
Best regards.
-Chris