Thumbnail image maintenance module failure
|
damato
Joined: 2004-10-30
Posts: 29 |
Posted: Sun, 2005-07-10 13:10
|
|
Hi guys, I have about 13000 images/objects in my gallery2 and if I try to use the thumbnail recreation module it always returns me an error on image 4275 of 13373 according to the progress bar: -- cut here --
Verarbeite Bild 4275 von 13373
Voraussichtliche Restzeit: 50:51
Memory used: 22978272, total: 64M
Error (ERROR_MISSING_OBJECT)
* in modules/core/classes/GalleryStorage/DatabaseStorage.class at line 378 (MySqlDatabaseStorage::error)
* in modules/core/classes/GalleryStorage.class at line 118 (MySqlDatabaseStorage::loadEntities)
* in modules/core/classes/helpers/GalleryEntityHelper_simple.class at line 82 (GalleryStorage::loadEntities)
* in modules/core/classes/GalleryCoreApi.class at line 2168 (BuildDerivativesTask::loadEntitiesById)
* in modules/core/classes/helpers/GalleryDerivativeHelper_advanced.class at line 667 (BuildDerivativesTask::loadEntitiesById)
* in modules/core/classes/GalleryCoreApi.class at line 830 (BuildDerivativesTask::rebuildCache)
* in modules/core/classes/helpers/GalleryDerivativeHelper_simple.class at line 66 (BuildDerivativesTask::rebuildDerivativeCache)
* in modules/core/classes/GalleryCoreApi.class at line 817 (BuildDerivativesTask::rebuildCacheIfNotCurrent)
* in modules/core/classes/BuildDerivativesTask.class at line 92 (BuildDerivativesTask::rebuildDerivativeCacheIfNotCurrent)
* in modules/core/AdminMaintenance.inc at line 104 (BuildDerivativesTask::run)
* in ??? at line 0 (AdminMaintenanceController::runTask)
* in modules/core/classes/GalleryTemplateAdapter.class at line 642
* in /var/www/gallery.light-speed.de/g2data/smarty/templates_c/%%776776087/%%3A^3A9^3A9D1F36%%theme.tpl.php at line 81 (GalleryTemplateAdapter::trailer)
* in lib/smarty/Smarty.class.php at line 1251
* in modules/core/classes/GallerySmarty.class at line 90 (GallerySmarty::fetch)
* in modules/core/classes/GalleryTemplate.class at line 226 (GallerySmarty::fetch)
* in main.php at line 311 (GalleryTemplate::display)
* in main.php at line 87
* in main.php at line 78
-- cut here -- I am using the very latest CVS checkout version 10.7.2005 but the problem also occurred in earlier versions (some weeks ago). Anyone having any idea what this might be and how I can get more debug output or solve the problem so that I can recreate all thumbnails with the mainteneance tool? ---- |
|
| Login or register to post comments |
Posts: 32168
the problem is corrupted referential integrity in your database or something related.
how that could happen, i do not know. it's a very bad thing and i see missing object errors here in the forums from time to time. so it would be great if we knew whether it is a bug or what is causing the problem.
how to find the item (here a derivative) that is causing the problem?
we're talking here about 13'000 derivatives i guess, and one of them is causing the problem.
it isn't necessarily item 4275 of 13373, because the progress bar makes steps of 5 not 1.
first, clean your db cache in site admin maintenance -> delete db cache.
then try again? is it fixed?
if not:
activate buffered debug mode and edit modules/core/classes/GalleryStorage/DatabaseStorage.class
directly before the return statement on line 378 add:
$gallery->debug($query); $gallery->debug('ids: ' . implode(', ', $targetIds));and run the build task again. in the debug output, at the end, you should see the output we're looking for.
Posts: 29
I already tried that quite often but without any result.
Ok I did this and run the maintenance job once again.. I guess the following is the relevant part of the debug output:
Search for the "ids:" string we have inserted into the class code. It says it must be object 8877 that is missing/failing.
So how to proceed now that we know it is object 8877 that is missing/failing?
thanks for your help.
Posts: 32168
ah i see. the additional debug output we added wasn't necessary, it already had such an output on another level.
now that you know that 8877 is the problem, take a look at the g2_entity table and search for the g_entityId 8877, could you post the row?
basically you have to search all of the involved db tables for the id 8877 and report if the id is missing or not:
tables:
g2_Entity,
g2_ChildEntity,
g2_FileSystemEntity,
g2_Item,
g2_DataItem,
g2_PhotoItem
Posts: 29
sure. here it is:
Ok. your above SELECT statement returns an Empty set and if I search through each table I can see that all tables, except the "g2_DataItem" carries a reference to g_id=8877.
So somehow the entry in the g2_DataItem table seems to be missing?!?! So how should I proceed then?
Posts: 32168
backup your database now. (e.g.
mysqldump --opt --compress --user=foo --password=bar gallery2 > g2_dump.sql
)
have you ever had a g2 error before?
what operations have you tried before in g2? link item? delete item?
see:
http://gallery.menalto.com/modules/GalleryAPI/apidoc/GalleryCore/Classes/GalleryPhotoItem_core.html
there should be an entry in g2_ChildEntity, g2_FileSystemEntity , g2_Item , g2_DataItem, with g_id = 8877
and also in g2_PhotoItem.
can you double check that there's really just an entry in the entity and in the dataItem table?
again, i can't explain how this happened.
your problem is that a specific resize/thumbnail says "photo item 8877 is my original, if you want to generate me, you need 8877".
let's try to fix this.
search the table g2_Derivative for rows with g_derivativeSourceId = 8877:
SELECT * from g2_Derivative WHERE g_derivativeSourceId = 8877;
then we know which derivatives have an non existant source / original.
the question is then how do we fix this?
the best thing would be to delete these derivatives. until now, there exists only one kind of derivatives, derivativeImages. see:
http://gallery.menalto.com/modules/GalleryAPI/apidoc/GalleryCore/Classes/GalleryDerivativeImage.html
now we have to remove the entries of these corrupt derivatives (derivatives with a non existing source id) from all db tables involved:
g2_derivativeImage
g2_derivative
g2_childEntity
g2_entity
good luck.
Posts: 29
Ok. luckily, but especially with your help I was finally able to fix my gallery2 database so that the maintenance tool for regeneration of all thumbnails finally walked through.
In the end there has been 3 database inconsistency where 2 of them were just simple ones where the table entry in the g2_DataItem table was missing, but the actual image files and the rest of the database refering to those items were also fine. So I just had to add an entry for the respective object in the g2_DataItem table and everything was fine afterwards.
But then there was one item which was of type "derivativeImages" where I actually had to delete all other entries in the g2_derivatativImage, g2_childEntity and g2_Entity tables.
However now everything seems to be fine. So thanks A LOT for telling me where I have to look for solving the problems.
But please, keep on reading! Because there is another (possibly unrelated problem) I am facing since about one month with my latest gallery2 checkouts:
Please have a look at the following page:
http://gallery.light-speed.de/v/damato/2005/birthday_party/
It shows an album of my gallery where you can see that some thumbnails are actually missing. However if you click on the bottom right image:
http://gallery.light-speed.de/v/damato/2005/birthday_party/IMG_0016_002.JPG.html
you should see the larger image and you will notice that the gallery contains all the thumbnails and also the real image files. But why is gallery2 not showing these thumbnail in the album view anymore?
Of course I tried many time to force a recreation of the thumbnails in the Album edit mode but without any luck...
So do you have any idea how to solve that or how to debug that and find out where exactly the problem is?
Thanks again for your help !!
Posts: 32168
hmm, i don't see any problem.
on http://gallery.light-speed.de/v/damato/2005/birthday_party/
i see 9 thumbs in the album (plus the imageblocks)
and on http://gallery.light-speed.de/v/damato/ ... 2.JPG.html
i see the image.
exactly like it should be.
maybe a browser caching issue?
close this album page, then clear your browser cache.
Posts: 29
Ah ok. Then this seems to be a browser issue. I am using Camino (MacOSX) from the mozilla projet here and it only shows me the first 8 thumbs no matter if I refresh the cache or not.
But if I use Safari for example, everything is shown fine. Strange. Does anyone else with a running Camino browser can confirm this? Would be interesting to know where exactly the problem is (Camino or gallery?!?)
Posts: 32168
i'd say it's a browser cache issue. browse to another page (don't have your browser showing this album page) and then clear your browser cache.
Posts: 29
Ok, got it.
Played around a bit with the Camino preferences and after disabling the "Block web advertising" feature in Camino it perfectly shows all thumbnails.
So the Camino (and probably in all other mozilla browsers also) the web advertising blocking feature seems to hide images of exactly the size these thumbnails have.?
Sorry for the confusion, but I guess the default thumbnail size in gallery seems to be exactly the size for recognizing web advertising banners. Any chance to work around that or find out at which sizes the mozilla browsers are really recognizing an image as an advertising image?
jens.
Posts: 32168
no idea.
i guess it's impossible to find out how you ended up with corrupted referential integrity in the database. so this case is closed.
Posts: 3517
I believe that's a Camino only thing. I've never seen this "block web advertising" feature in Mozilla, Netscape or FireFox. Us Firefox users have Adblock which blocks ads more intelligently than just what size an image is ;)
You can change the thumbnail settings for Gallery to be larger or smaller than what Camino thinks is an ad becuase of it's size. Why would anyone block images based on size anyway?
Posts: 29
Yes I also don't know. So thank you once again for your superb help!
And to prevent future corruptions: Isn't it possible in MySQL to use primary key mechanism to link all table to eachother so that the referntial integrity is always persistent? I know that from other database systems. So with this mechanism the database would always be valid and if the software tries to modify the database such that the integrity would be violated the database denies these operations?!?
And if not, then I guess it is time to write another maintenance module which tries to analyze the data integrity of the database and report the results with some short suggestions to the user. Would that be possible/reasonable?
Posts: 32168
@referential integrity: nope, primary keys are somethings different. they too help keeping integrity, but what you mean are foreign keys.
foreign keys would complicate G2 but yes, i brought up this topic before when talking to other g2 devs.
@maintenance task: i too thought about a maintenance task / emergency tool task to check for integrity. but that would need a lot of maintenance to keep this task up to date.
Posts: 32168
i've created 2 tasks for these 2 ideas. i'm not sure if we will ever implement them, but they would be very nice to have.
Posts: 21
Hi,
hope valiant reads this
I have a similar problem... here's my error message:
I followed the instructions so far, but in neither of the MySQL-tables mentioned above exists an G2-item with ID 4260.
So, how do I go on from there...?
THX!
-----------------
Gallery-Version = 2.0-rc-2+ Kern 0.9.37
PHP-Version = 4.3.10 cgi-fcgi
Webserver = Apache/1.3.27 (Linux/SuSE) mod_fastcgi/2.4.2 FrontPage/4.0.4.3 PHP/4.3.10 mod_perl/1.27 mod_ssl/2.8.12 OpenSSL/0.9.6i
Datenbank = mysql 4.0.23-Max-log
Werkzeuge = ArchiveUpload, Exif, Gd, NetPBM, Thumbnail, ImageMagick
Betriebssystem = Linux dd1618 2.4.21-260-athlon #1 Wed Nov 17 20:54:44 UTC 2004 i686
Browser = Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Maxthon)
Posts: 32168
florre, somewhere in your database there is a reference to an object with g_id / sourceId / ...Id = 4260. probably in the g2_Derivative / g2_DerivativeImage table.
Posts: 21
yes, I have g_id = 4260 entries in g2_DerivativeImage and g2_Derivative...
here they are:
g2_DerivativeImage:
g2_Derivative:
thanks for your fast reply! :D
Posts: 32168
you have the choice between deleting this id from all tables or trying to fix the existing entries / adding some rows if necessary to fix the data corruption.
see:
http://gallery.menalto.com/modules.php?full=1&set_albumName=miscellaneous&id=aaf&op=modload&name=gallery&file=index&include=view_photo.php
if there's an entry in g2_DerivativeImage, there should also be an entry in g2_Derivative and in some other tables and in g2_Entity. It's complicating yes, but it was never meant to be done manually.
Posts: 21
valiant,
thanks for your help!
i decided to not do it the hard way and deleted the tables :-D
that fixed the error message, thumbnail maintainance runs as smooth as ever...
i think the entries got messed up when i imported an older version of the mysql database.
Posts: 17
I got something similar, for me it is not trigered by a bug, but rather by messing up tables. I used to have some troubles which needed manual editing, though I didn't do well obviously. This is the ERROR:
**
Error (ERROR_MISSING_OBJECT) : Missing object for 1657
* in modules/core/classes/GalleryStorage/DatabaseStorage.class at line 2124 (gallerystatus::error)
* in modules/core/classes/GalleryStorage/DatabaseStorage.class at line 305 (mysqldatabasestorage::_identifyentities)
* in modules/core/classes/GalleryStorage.class at line 118 (mysqldatabasestorage::loadentities)
* in modules/core/classes/helpers/GalleryEntityHelper_simple.class at line 82 (gallerystorage::loadentities)
* in modules/core/classes/GalleryCoreApi.class at line 2186 (galleryentityhelper_simple::loadentitiesbyid)
* in modules/core/classes/helpers/GalleryDerivativeHelper_advanced.class at line 673 (gallerycoreapi::loadentitiesbyid)
* in modules/core/classes/GalleryCoreApi.class at line 843 (galleryderivativehelper_advanced::rebuildcache)
* in modules/core/classes/helpers/GalleryDerivativeHelper_simple.class at line 66 (gallerycoreapi::rebuildderivativecache)
* in modules/core/classes/GalleryCoreApi.class at line 830 (galleryderivativehelper_simple::rebuildcacheifnotcurrent)
* in modules/core/classes/BuildDerivativesTask.class at line 92 (gallerycoreapi::rebuildderivativecacheifnotcurrent)
* in modules/core/AdminMaintenance.inc at line 104 (buildderivativestask::run)
* in ??? at line 0 (adminmaintenancecontroller::runtask)
* in modules/core/classes/GalleryTemplateAdapter.class at line 758
* in g2data/smarty/templates_c/%%3965159830/%%C7^C79^C790B080%%theme.tpl.php at line 75 (gallerytemplateadapter::trailer)
* in lib/smarty/Smarty.class.php at line 1251
* in modules/core/classes/GallerySmarty.class at line 90 (smarty::fetch)
* in modules/core/classes/GalleryTemplate.class at line 228 (gallerysmarty::fetch)
* in main.php at line 318 (gallerytemplate::display)
* in main.php at line 87
* in main.php at line 80
So far I figured out the error is trigered at line 2124, I checked out the code trying to find what tables are having issues, but so far i'm failing in that.
Thanks for your time.
gOochi
Posts: 32168
yep, obviously you didn't do that too well.
this will need some work.
canned response:
Error (ERROR_MISSING_OBJECT) : Missing object for ...
some part of your g2 database isn't 100% ok, the data integrity is a little corrupted.
reason: unknown. fix: it's not easy, but you can fix the database manually, someone else had the same problem and i gave a few instructions:
http://gallery.menalto.com/node/32741 (which is this thread, so read the instructions, can't help you more than that)
Posts: 7
I'm having a similar problem with my Gallery 2.0.2 release version. Visit http://www.drewvogel.com/gallery2/ to see it for yourself.
Here's the error I'm seeing (this is different from other errors because my "(ERROR_MISSING_OBJECT)" doesn't have anything after the last colon (for example, Goochi's error says, "Error (ERROR_MISSING_OBJECT) : Missing object for 1657"):
Error (ERROR_MISSING_OBJECT) :
in modules/core/classes/helpers/GalleryItemAttributesHelper_medium.class at line 58 (MatrixTheme::error)
in modules/core/classes/GalleryCoreApi.class at line 1178 (MatrixTheme::fetchViewCounts)
in modules/core/classes/GalleryTheme.class at line 1203 (MatrixTheme::fetchItemViewCounts)
in themes/matrix/theme.inc at line 75 (MatrixTheme::loadCommonTemplateData)
in modules/core/classes/GalleryTheme.class at line 900 (MatrixTheme::showAlbumPage)
in modules/core/classes/GalleryView.class at line 285 (MatrixTheme::loadTemplate)
in main.php at line 287 (ShowItemView::doLoadTemplate)
in main.php at line 87
in main.php at line 80
The only line that looks suspicious to my (non-technical) eyes in the debug output is:
(mysql): SELECT g2_ChildEntity.g_parentId, COUNT(g2_ChildEntity.g_id) FROM g2_ChildEntity, g2_AccessSubscriberMap WHERE g2_ChildEntity.g_parentId IN (7,4831,4954,4500,19,7879,5786,6501,6546,6904,7036,7551,4383,7622,203,7745) AND g2_AccessSubscriberMap.g_itemId = g2_ChildEntity.g_id AND g2_AccessSubscriberMap.g_accessListId IN (4388,8912,7749,8911,7751,7880) GROUP BY g2_ChildEntity.g_parentId
I have NO idea how to mend this error. I've tried completely re-installing the latest Gallery (2.0.2), I've cleared the cache on my browser, my server, and the Gallery cache (via lib/support/cache.php). No joy.
I'm running Gallery 2.0.2 on a fully-up-to-date Fedora Core 4, PHP 5.04, Apache 2.0.54, MySQL 4.1.16.
Someone please help!
-drew
www.drewvogel.com
Posts: 32168
if you did a clean reinstall or g2 (not just rerun the installer choosing the "reuse tables" option in the database step), then you'd deal with a fresh, empty g2 and there's no reason you'd get the error_missing_object then.
Posts: 6
Valiant, thanks! Managed to resolve my issues with corrupted database based on the information in this topic (even figured out how to write some very basic scripts to sae time searching (PHPMYADMIN takes a long time!).
Anyhow, i think i know why my database problems occured and thought i would post to let you know!In my case the images that i was having problems with where all uploaded at the same sort of time using REMOTE. When i uploaded the files i had to stop half way (using wireless from a bus, not a clever option), and while i had time to 'cancel' the upload in the remote program, i never came back and finished uploading the files. I suspect that this loss of connection while remote was uploading the files is what has caused my problem, though i don't know how remote works so it might be totally unrelated!
Anyways, thanks again, thus far thumbnails are creating normally!
Posts: 32168
we sure should experiment a little to see what exactly can happen with gallery when the connection is closed during an operation.
thanks for the information.