Gallery 2 is fragile -- assumes access to infinite computing resources.
|
chalsall
Joined: 2008-05-06
Posts: 19 |
Posted: Thu, 2008-05-15 19:01
|
|
This posting is not meant as being negative, but fair warning for anyone who tries to install Gallery 2 on a shared server. In my particular case, a 1and1 shared server. This is also meant as constructive criticism to the developers, who might consider keeping in mind that not everyone have access to a dedicated server. 1. G2 assumes any required processes can run for as long as they need. Examples: 1. Admin Options -> System Maintenance -> Build all thumbnails/resizes 2. Watermark module. 3. Watermark module. 4. Often during the above (manually re-run) processes database errors would occur. It was often necessary to run (and occasionally re-run) the "Admin Options -> System Maintenance -> Delete database cache" function. 5. Thumbnails often require regeneration for no apparent reason. As might be obvious from the above, no one without software development skills would have been able to determine what was occurring, let alone find work-arounds. I therefore (again, meant as constructive criticism) argue that G2 is fragile, and is not appropriate for general usage. Obviously a lot of work has gone into this project, but assuming that no one will be using it in an unconstrained environment is unrealistic. Feedback expected and welcome. Kindest regards. -Chris |
|
| Login or register to post comments |
Posts: 1841
How would you yourself fix examples 1 and 2, so that these long operations run under a single script without timeout, given that the operation total time might be longer than 60 seconds, or whatever? Is there a way that a php script can tell it's about to be terminated by a zealous ISP's watchdog?
Posts: 19
I, myself, would...
For example 1:
1.1. Break down extremely extensive processes into manageable sub-sets, and launch a single sub-process to handle these. (This assumes a browser-only interface.)
1.2. Same as 1.1 above, but provide a means of specifying a "start" (read: offset) and limit (read: maximum number of images to process) to the user.
1.3. Provide a means of handling large transformations via crontab (or similar) launched sub-processes.
For example (for example 1 above), wouldn't it be useful for the administrator to be able to instruct the G2 system (in English) "I want to apply the following watermark to all my images, in all my albums, and all their (possibly recursive) sub-albums Please do this. Oh, and also please regenerate all the derivatives, so my users don't face delays when they view the images. (By inference, I, the user, don't wish to be intimately involved with what you (read: the G2 system) have to do to achieve this.)"?
Or, as another example, "I want you to build all the derivatives any of my users might ever request. I don't care how long this takes, so long as it's done."
For example 2: (sub-processes)
1.1. G2 currently pipelines it's generation of derivatives. There should be an option to constrain this to a single sub-process, which writes to a temporary file.
1.2. IPC between the G2 processes (or, at least, a single central process manager), such that background work (as detailed in my response to Example 1 above) is delayed in the case where real-time user(s)' requests come in.
Even under 1and1 (with shell access), crontabs are available. *Any* hosting environment can have implemented similar background functions by using web-access requests launch processes.
Let's please be perfectly honest -- currently if an Administrator is rebuilding their entire album's derivatives, and a single (let alone several) viewer(s) come by requesting an underived image, one, both, or many will receive "500 -- Internal Server Error" messages. Rarely comforting...
The above, and my original post, is meant and hoped to be constructive.
Kindest regards.
-Chris
Posts: 19
Oh, and for context...
I would have given up on G2 almost immediately after actually trying use it, if I hadn't found (and understood) the following:
http://codex.gallery2.org/Gallery2:How_can_I_call_a_View_or_Controller_from_the_Command_Line
Which reminds me... It's time to manually edit "gallery2/modules/core/classes/BuildDerivativesTask.class" with a new custom limit range, so my gallery-wide derivative generation can continue... (I have a local wget series currently running building all my derivatives...)
G2 is not for the faint of heart nor hairless of chest...
With regards...
-Chris
Posts: 19
Oh, and for further context...
Without manually editing php code (or manually going through each and every album, rather than using the "apply to all sub-albums fetcher"), it is impossible apply a watermark to a large set of images. Click "reload" after the "500 -- Internal Server Error" does not work because the Watermark module does not check to see if it has already applied the requested watermark to a derivative. Also -- importantly -- the watermark appears to "touch" (read: mark for regeneration) all derivatives of an image even if it hasn't been requested to apply the watermark to same. Or, at the very least, I have empirically determined this to be true for thumbnails.
Once again, I claim that G2 is fragile.
-Chris
Posts: 1841
Interesting stuff. Much of what you say I'm not qualified to comment on, but specifically...
Wouldn't the master process managing these sub-processes still time-out if all the sub-processes hadn't finished?
One already has the option to this album-by-album, which amounts to much the same thing. Although I don't disagree that a numeric limit would be interesting.
Is directing the user to crontab any more for the faint-of-heart than the things you already dislike?
Speaking personally (I'm not a Gallery core developer) I think there's a matter of expectation. If I'm a n00b with 25 out-of-focus pictures of my dog in a Gallery installation then these 'long' operations aren't long at all and will work just fine on any hosting package. But consider that when I try to watermark 1500 pictures in Photoshop, locally, I run a batch action and go eat dinner - I know I'm in for a 10 minute wait. That's on a 2.4GHz core 2 duo with 2 gig of ram *dedicated* to running that task. I think it's a little unfair to expect my $6 per month hosting package to watermark all 16,000 images in my Gallery installation (or rebuild three resizes each) - on a server shared with 6000 other websites - all from one page, and if it did then it would probably take about 24 hours to complete the job given its fair share of resources. Looking at it the other way, I don't want the other 5999 users of that server host rebuilding 60k derivatives from one page because nobody is going to have the processor time left to view my pictures. Splitting the task into sub-processes or running from a cron-tab isn't going to make it any less processor-intensive and if it runs quickly enough to be useful it's going to be unfair to the other users of the system (and likely to get pulled by my ISP for that reason.) One has to have reasonable expectations of what you get on a cheap, shared host. It's true that not everyone has access to a dedicated server but that doesn't mean Gallery can achieve the impossible.
Gallery's known to be 'heavy', and it doesn't advertise itself as either n00b-friendly, or for the faint of heart, as you put it. Anything other than a vanilla installation - say a neat integration or customisation even of just a few templates - takes time and knowledge. Which is not to say that your ideas aren't good and shouldn't be implemented (I'm not really qualified to say, sadly.)
I will decline your argument ("Once again I claim that G2 is fragile") though - saying it's meant to be constructive doesn't make it sound any less like an invitation to a fight!
Posts: 1841
Oh, and "for context"...
I guess what I'm saying is that to me it sounds like you're conflating two problems: firstly that some operations take a lot time (are processor intensive) and secondly, that Gallery doesn't give any advance warning that a requested operation will time-out or be pulled by the ISP.
There may be optimizations to be done in the first case but that's only a matter of degree. There will always be some number of pictures to resize/watermark that's going to take a very very long time, so essential this is insoluble.
For the second problem (which I agree is an issue) how would you detect this is about to happen so the user can be alerted in advance?
Posts: 19
Truly, I am not trying to start a fight. However, I *am* trying to point out a serious issue with G2 as it's currently implemented. And I could be wrong here, but no where on the G2 site do I see a big warning saying "Not appropriate for many shared serving environments".
No. A process doesn't "time out" -- instead (at least under 1and1) it is killed after consuming 10 seconds of CPU time. So, for example, if the master process simply launched a sequence of single sub-processes to build some number of derivatives, it would increase (by several orders of magnitude) the number of images which could be processed. However, you are correct when you point out that this doesn't scale infinitely -- there would be some point at which the processing could not complete.
This is where a cronjob'ed "batch" processing system, or at least, a continue from [offset] function would be useful.
But this is a manual effort. And it seems that when you ask for derivatives to be built from the "Edit Album" panel that it re-builds all the derivatives, not just the broken ones.
I don't think so. Many other php web applications ask the "administrator" to add a crontab entry. And again, the system could do it itself by way of launching a (semaphored) "maintenance" process after serving user's viewing requests.
I am *not* expecting a $6 a month shared server to process a tonne of images as quickly as a dedicated server or workstation. In my particular case, I have 10897 images in 170 albums, resulting in 44435 derivatives. With a watermark, it takes about 48 hours to reprocess all these images.
What I *am* expecting is that if the system presents the user with "apply this globally" options, that it actually be able to do the job that's been promised. Again, using my own example, the "Build all Derivatives" option in the Site Admin Maintenance area is only able to run through between 16000 to 20000 derivatives before the 10 second CPU limit is reached. This is not actually generating any derivatives, mind you, simply determining that they have already been processed. Thus, I cannot apply the function to the entire set without hand-editing PHP code.
This is why I again claim that G2 is fragile. It doesn't scale; it breaks easily.
As I'm trying to point out -- this doesn't *have* to be the case. There are several ways that G2 could deal with limited environments in an automatic manner.
I am going to post another message shortly with some additional detailed suggestions. Again, I am wishing to contribute here, not just complain. A lot of work has obviously gone into this system. It would benefit and expand the userbase if G2 was more robust.
With regards.
-Chris
Posts: 19
I don't think I'm conflating the two issues. What I'm pointing out is, given the above, G2 should be more resilient to failed processes.
Some suggestions (some are repeats from above, some are new ones I dreamt up last night while asleep):
1. Provide a batch subsystem, leveraging on crontab and/or web-access launched sub-processes.
2. Provide a configuration option which indicates to G2 that it is in a constrained environment, and so heavy modules should serialize functions rather than pipelining them. This could actually be determined heuristically by the system.
3. Provide an option such that if a user-view request triggered derivative generation fails, it will be retried some number of times (settable by the administrator). Currently if a user requests an image, and the generation fails, that derivative generation will never be tried again without administrative intervention, and the user is presented with the dreaded "92x92" broken image icon.
4. Provide a means of specifying an offset for global functions. This could actually be semi-automated with the ProcessBar display -- using the JavaScript onload() function. The processing script could provide information to the page as to what point it has last checkpointed (read: committed to the file store / database). If onload executes, and this number is less than the total, then the page could display a "Did not complete... Continue the process" link which would effectively leverage on the offset function.
5. Provide reporting function which lets the administrator receive a list of broken derivatives (globally and on a per album basis), so they can regenerate them themselves, and/or determine if the original image is in fact corrupt.
6. Provide an additional option in the Edit Album area which only regenerates broken derivatives. I could be proven wrong here, but based on my experiments it appears the "regenerate thumbs" and "regenerate resizes" does just that -- regenerates them all.
Lastly, I'd like to point out that the Watermark module could use some optimization work. It is extremely wasteful of resources, and "touches" derivatives it need not do.
I do hope this is taken constructively. Based on my research within the forums solving the issues I encountered, many many others have encountered such issues as well, and some have simply given up in frustration. There is no reason for this to be.
Best regards.
-Chris
Posts: 1841
Hi Chris,
Lots of constructive suggestions, indeed, and I'm sure the core developers will be interested to hear them - have you tried the dev (https://lists.sourceforge.net/lists/listinfo/gallery-devel) mailing list, it's a more sure-fire way to get your ideas under the right noses and they are quite open to new ideas.
Personally I disagree that gallery is "Not appropriate for many shared serving environments". It's very suitable, for a sufficiently reasonable number of images. That limit might be lower than you'd hope, but that's a different matter. 10897 images in 170 albums is way way beyond that level if you're expecting to rebuild all derivatives in one hit. I have roughly the same number of images on two installations, both on shared hosting (one on 1and1 by the way) and I just don't have an issue with these things - I'd never dream of expecting gallery to go away and successfully complete a task that would take 48 hours all in one hit. I wouldn't even dream of doing that on my desktop system, and that's a dedicated system.
And in one sense, Gallery is very resilient to failed processes. It certainly doesn't (in my experience) stop working, deliver the wrong images, or give up if a script dies. The database keeps its integrity and I don't get 500 errors or broken links.
I do hope some of your ideas get considered for future implementation though - maybe you'd have the time and interest to work on them yourself?
Posts: 19
Hi Alecmyers.
Thanks for this ongoing discussion.
I would like to simply counter argue with what is now entering a user interface philosophy debate.
I am of the school of thought which states that if a user interface gives the user the option of doing something, then the system behind that user interface should be prepared to deliver on what has been promised. I don't feel it is appropriate to say, to paraphrase, that "the user should have known it wouldn't work". The user interface is free to say "I can't do what you're asking." or "Are you aware that this is going to require two (or twenty, or a thousand) days of solid work to complete? [Cancel] or [Continue]" But if the UI accepts a request for a particular process, then it should be able to do what's been requested (one way or another).
For context, I personally have written web-based systems which has made graphical access to hundreds of millions of records to hundreds of concurrent users (one was a real-time stock market system, for example). There was a great deal of back-ground processing going on, naturally, but at no time was a user given the opportunity to ask for something which could not be delivered.
The fact remains that, currently, G2 can lead the unwary user into a situation where it *cannot* deal with the dataset the user has painstakingly imported into it. (Or, at least, cannot deal with any global processing / change requests.) This is not a good thing (IMHO).
Thank you for your suggestion of the devel list -- I have joined same, and will be taking these issues and my suggestions into that forum.
Kindest regards.
-Chris
Posts: 1841
I totally agree. It sounds like you have some good ideas about how to make Gallery deliver what it promises . But there are always going to be situations where it can't - the server crashed, the script got pulled by an Act of God, or an overzealous ISP's level 3 tech support who wondered what a process was doing using so much memory and killed it from a shell). Or it ran out of memory. Or something.
So two questions, surely (although you didn't agree with my distinction) - how to get long tasks to run more smoothly (a "small matter of programming" as perhaps you've outlined)- and how to deal with tasks that die before finishing (and I have no clue how one would tackle this.) Don't you agree?
Posts: 17
If the admins want to delete this then by all means do so but I would just like to say that G2 is excellent and will serve the purpose of most people using it. If people like Chris think that it is not good enough for their needs then why dont they go off and write their own app that does not have these problems. In my opinion, especially when something is free, people must accept that the app wont suit all the people and if it does not suit you then dont complain - create your own.
Posts: 6
BRAVO. G2 & the G2 team rocks.
Posts: 19
To jonathanb and pooter
From the mouths of babes... Nieive, ignorant, and possibly stupid...
I challenge you (and anyone else) to implement a G2 system with 12699 images (of 1600 by 1200 pixels, with watermarks and with derivatives of 640x, 1200x and 1600x), and actually be successful as the code is delivered...
Sure, if you are under your own environment (read: a dedicated server) or a VPS, you might pull this off.
But if you, like a vast majority of your users, have to suffer within an outsourced "shared server", you'll quickly be toast. (Things seem fine, initially, right up until you try to scale...)
It is only because I have somewhat rarefied skills that I am able to make G2 work within this environment. I have documented clearly within my posts how I have accomplished this. (Hint: the most important is SQL selects with "order by **** desc...)
Keep pounding the drum, boys....
Posts: 1841
Hi Chris,
Insulting contributors to this forum isn't going to improve the product, is it...
have you made your suggestions to the development team mailing list like you said you would? Did you get a response?
Posts: 19
alecmyers...
Oh, I don't know. Perhaps it might generate enough emotion to actually accomplish something...
Yes, I did submit my comments to the "development team". They were, as far as I am aware, ignored...
Posts: 11717
Flames don't accomplish much except destruction.
I looked through 3 other threads that you have participated in and don't see any comments about specific improvements.
The dev team occasionally frequents specific topics but not all. If they spent their time in the forums there would not be any time left for coding/improvements, would there?
I also searched the gallery-devel mailing list and did not find any either. Perhaps I missed something, please point me in the right direction.
Please point to specific improvements that we can make to make this free product better. The earlier the better as G2.3 is almost ready for release candidate status.
Pop into #gallery on IRC and join in the discussing the improvements that you have in mind.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 19
You see alecmyers..., sometimes causing emotion can result in results...
floridave, thank you for your response.
If you can't see tangible suggestions in my posts above, that you're not reading them closely enough. (Not meant as a flame.)
Please also see http://gallery.menalto.com/node/77393, where I give a specific (and, admittedly, stupid) code delta.
Please also note my e-mail to the e-mail list
, sent Mon, May 19, 2008 at 13:11, titled "A bit of a rant..." The content is below:
Please let me know how I might contribute and assist.
-Chris
Quoted e-mail:
"Good day everyone.
I went on a bit of a rant over on the support forums, expressing my
concerns about how Gallery2 can break easily under a constrained
serving environment such as 1and1. I got into a dialogue with
alecmyers, wherein he suggested I join this list as it was a better
place to get suggestions "under the right noses".
Please see:
http://gallery.menalto.com/node/77693
...for context.
In this forum I present several ideas on how to make G2 more usable
for those poor souls who are stuck with constrained server facilities.
As I expressed there, I was bring this forward with the hope and
intent of being constructive, but at the same time intending to warn
potential users that they could very easily encounter problems which
without programmatic skills they could not solve.
Now I'd like to put my money where my mouth is, and find out how I
might contribute to the development process.
The three most important, and easiest, ideas presented are:
1. To simply add a "desc" to the "ORDER BY [DerivativeImage::id]"
statement in the
gallery2/modules/core/classes/BuildDerivativesTask.class file, since
this would allow an administrator to [re]build all newly created
derivatives after adding images, since under 1and1 this function
cannot complete after a certain number of images are in the system.
2. Add an ability to have the system retry building derivatives upon
request some number of times. Currently if a user requests a
derivative, and it fails, the system will present the dreaded 92x92
icon and never try again.
3. Add an ability for the system to serialize image processing,
rather that the current pipelining. Yes, this would take more time,
but since 1and1 limit the total number of processes to 12, busy
servers very quickly have their processes killed, and the user is
presented with "500 Internal Server Error" messages.
How may I contribute? Is there someone to whom I should send DIFF files to?
Please advise.
Thanks and best regards.
Posts: 11717
Not taken as a flame.
Your suggestions, from what I can tell, are more specific to your host. Suggestions are great but they also need to come with backup as the team does not have the bandwidth/cycles to address each specialized request.
Remember that G2 has to run on a myriad of OSs php versions and other numerous parameters that have to be tested/unit tested and code reviewed.
I am not a developer so can't offer opinions on your code changes, if they would work in all situations.
I have:
"Pop into #gallery on IRC and join in the discussing the improvements that you have in mind."
You can also submit patches to:
http://sourceforge.net/tracker/?group_id=7130&atid=307130
@#2 I would like this feature as well. perhaps a module that searches for 92x92 derivatives and presents a user with a option to delete them and build them again.
@#3. Great suggestions but how do we go about implementing those changes? Some would argue against taking more time as they have a better host that can handle 'pipelining'. They would suggest that this would be 'code bloat' and not needed. We get it from both ends.
See you in IRC
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 1841
Hi Chris,
Just on the subject of the email: I am subscribed to that list, and I've just checked my archive; I wasn't delivered a copy of that email. And I don't see it in the archive here (http://www.nabble.com/gallery-devel-f3818.html) so for whatever reason, I don't think your email was ever received by the development team. I'm pretty sure that had it been, it wouldn't have been ignored.
Posts: 19
floridave...
I do appreciate that G2 has to run on many OSs and environments. And these suggestions are only needed in constrained environments.
However, I would further argue that many, many people will be working within such an environment. I am not a big fan of 1and1; however, you cannot deny that they are used by many people. And the biggest issue in my mind is the fact that G2 appears to work fine, but then starts breaking as the number of images increases. Most shared serving environments are similarly configured.
Because it is critical, please let me repeat the point: anyone in a constrained environment will find that G2 *stops* *working* after some finite number of images are in the system.
With regards to my suggestion 3, while it could be argued to result in code bloat, I would counter that you have to admit the code is already pretty huge, and simply adding an *option* for serialization would not be that big a deal. (I'm guessing perhaps 10 additional lines of code.) Further, please note that I am suggesting this as an *option*, such that those who are not constrained continue to enjoy the speed of a pipelined process, while those who are constrained have a system which works.
Best regards to all.
-Chris
Posts: 11717
Done:
http://gallery.svn.sourceforge.net/viewvc/gallery?view=rev&revision=17669
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 19
floridave...
Noted. Appreciated.
-Chris
Posts: 4
I just read through this entire thread and not being a programmer, some of that went completely over my head. But I do have to admit I see where Chris is going. I like the idea of having "options" to help G2 deal with hosting situations it's used it, that might not be up to the challenge of doing things the way they're currently designed. I have to say I wouldn't see that as code bloat, because it's code that would only be invoked IF that option was needed/wanted, correct?
Anyway. I think it's great to see community input, developer response, and actual changes coming out of conversations like this. Great stuff!