importing from G1 times out (no specific error)

Nobodys Hero

Joined: 2005-09-16
Posts: 25
Posted: Wed, 2005-09-21 02:06

I have a fairly large gallery 1.5.1-RC3 install at http://www.michaelmolino.com/gallery/ that I'm trying to upgrade to Gallery 2;
I installed G2 from CVS to http://www.michaelmolino.com/gallery2
I ran the find orphans script on g1 with no problems;
GD module is disabled in G2;

Every time I try to import G1 though, it just stops at some point... anytime from 30 seconds to 1 minute in. I checked and safe_mode is not enabled nor are any functions disallowed by php.ini.

I've enabled debug mode, but I don't get any debug messages or error messages when it stops... it's just the the progress bar stops moving, mozilla indicates that the page is fully loaded, and if I go to my gallery in a different browser window and keep refreshing it, I can see that nothing is happening

Here is a link to php_info():
http://www.michaelmolino.com/tmp/check_info.php

Here is my host's php.ini (my host is 1and1.com):

memory_limit = 40M;
upload_max_filesize = 20M;
max_execution_time = 300;
max_execution_time = 50000;
browscap = /usr/local/lib/browscap.ini
register_globals = on;
error_reporting = E_ALL & ~E_NOTICE & ~E_WARNING
url_rewriter.tags = "a=href,area=href,frame=src,form=fakeentry,fieldset=";

Here is the requested support info:

Gallery version = 2.0 core 1.0.0
PHP version = 4.3.10 cgi
Webserver = Apache/1.3.33 (Unix)
Database = mysql 4.0.25-standard-log
Toolkits = ArchiveUpload, Exif, NetPBM
Operating system = Linux infong 2.4 #1 SMP Mon Aug 9 10:21:08 CEST 2004 i686 unknown
Browser = Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8b4) Gecko/20050908 Firefox/1.4

I hope someone can help. I've tried reading as much documentation as I can and I tried searching the forums, but couldn't find anything relevant. If you need to, you can email me at

Thanks.
 
jpeadro

Joined: 2003-12-24
Posts: 198
Posted: Wed, 2005-09-21 02:46

I have about 15,000 photos on my gallery. I just finished migrating from G1 to G2 and saw the same issue you're having. What I did was import between 4-6 albums at a time. Give that a go and let me know how it goes.

Jeff
 
Nobodys Hero

Joined: 2005-09-16
Posts: 25
Posted: Wed, 2005-09-21 03:35

First of all, thank you very much for the quick response...

Unfortunately, that doesn't work though... I tried doing it 1 album at a time... some of them migrated ok (the smaller ones), but some didn't. I gave up a couple of albums in because I figured it wasn't worth migrating all the small albums and then having to re-upload all the large ones...

Also, some more details I thought of that might help... I've tried using both Firefox 1.5 beta 1 and IE 6.0 SP2. I've also tried both from home and at work (thinking maybe my cable modem was just crapping out).
 
robotphotos
robotphotos's picture

Joined: 2005-09-21
Posts: 111
Posted: Wed, 2005-09-21 05:16

I migrated over 17,000 images without the problem you are describing. Could you look up your image toolkit priority (under Site Admin -> Graphics Toolkits -> Toolkit Priority). I've read that Gd is a resource hog, so if it's first in the list, try making it last (assuming you have more than just that toolkit installed).
 
fryfrog

Joined: 2002-10-30
Posts: 3236
Posted: Wed, 2005-09-21 06:43

He said GD module is deactivated and stuff, plus his info implies that is true. I would try checking your apache's error_log and perhaps trying with GD *on* and NetPBM *off* (help eliminate the toolkit as the problem).

Like Gallery? Appreciate my help? Think I'm an asshole? Make your point by donating to the Gallery project! Or maybe just visit my website.
 
Nobodys Hero

Joined: 2005-09-16
Posts: 25
Posted: Wed, 2005-09-21 12:29

I disabled NetPBM and enabled GD.... same problem so I guess that means it's not the image toolkit.

I also noticed it gives the memory usage at the top... everything looks fine, so it's not a memory issue.
Memory used: 12295520, total: 40M (at the time when it stops).

Unfortunately, I don't think 1and1.com gives me access to the error logs... I'll look into that later though, right now I'm late for work!
 
robotphotos
robotphotos's picture

Joined: 2005-09-21
Posts: 111
Posted: Wed, 2005-09-21 12:38

Sorry - I missed that salient fact. FWIW, I migrated using the following priority: ImageMagick, NetPBM, Gd
 
Nobodys Hero

Joined: 2005-09-16
Posts: 25
Posted: Wed, 2005-09-21 17:22

1and1 doesn't have ImageMagick installed... when i get home from work, i'll look into seeing if i can install it and get it working under my home directory...
 
robotphotos
robotphotos's picture

Joined: 2005-09-21
Posts: 111
Posted: Wed, 2005-09-21 17:24

OK. I'd be very interested to hear if that makes a difference for you. To date, I've not found a conclusive pro/con list between those two toolkits.
 
Nobodys Hero

Joined: 2005-09-16
Posts: 25
Posted: Thu, 2005-09-22 03:11

It's been a while since I ran linux on any of my machines... I had forgotten how fun it was to install software from source, especially when you don't have root! ;)

Anyway, after some tinkering, I got ImageMagick installed using the folling configure parameters: --prefix=/kunden/homepages/46/d89726853/htdocs/local --exec-prefix=/kunden/homepages/46/d89726853/htdocs/local --without-ttf --without-perl

I'm having the exact same problem still with only ImageMagick activated... Considering I've tried using each of the 3 image toolkits at this point with the exact same results, I think it's fair to say it's not an image toolkit problem.

Any other suggestions? I'm nearing 3,000 photos in my G1 and I'd really like to be able to manage them better.
 
robotphotos
robotphotos's picture

Joined: 2005-09-21
Posts: 111
Posted: Thu, 2005-09-22 08:09

What's the "du -sk" of the albums directory in your G1 installation?
 
Nobodys Hero

Joined: 2005-09-16
Posts: 25
Posted: Thu, 2005-09-22 15:08

1957172

(my quota is 4 gigs... so i realize space might be a problem, but it "times out" even when i try importing just 1 gallery, so it's not a space issue.)
 
robotphotos
robotphotos's picture

Joined: 2005-09-21
Posts: 111
Posted: Thu, 2005-09-22 16:04

Yeah, but I wasn't asking that. ;-) I was going to offer to do it for you, but I need to know how much size we're talking about.
 
Nobodys Hero

Joined: 2005-09-16
Posts: 25
Posted: Thu, 2005-09-22 16:11

i don't understand how you would do it.

download my G1 to your server, migrate to G2, then reupload? yikes! i think i'm too stubborn to resort to something like that... I want to try and figure out why I can't do it myself!! :)

is it possible apache has a timeout set???

(thanks for all the help so far though... i really appreciate it).
 
Nobodys Hero

Joined: 2005-09-16
Posts: 25
Posted: Thu, 2005-09-22 16:49

I'd really like to see the apache error logs, but 1and1 doesn't provide them for shared hosting users such as myself... I found this in their FAQ though:

Quote:
Creating own error logs for debugging php scripts.

Since we dont provide access to Apache error logs on shared hosting packages for
technical reasons, you can create your own error logs for debugging PHP Scripts.

Please insert the following code in your PHP script (or create separate file and
and add the code in it. Include the file using "include()")

error_reporting(0);
$old_error_handler = set_error_handler("userErrorHandler");

function userErrorHandler ($errno, $errmsg, $filename, $linenum, $vars)
{
$time=date("d M Y H:i:s");
// Get the error type from the error number
$errortype = array (1 => "Error",
2 => "Warning",
4 => "Parsing Error",
8 => "Notice",
16 => "Core Error",
32 => "Core Warning",
64 => "Compile Error",
128 => "Compile Warning",
256 => "User Error",
512 => "User Warning",
1024 => "User Notice");
$errlevel=$errortype[$errno];

//Write error to log file (CSV format)
$errfile=fopen("errors.csv","a");
fputs($errfile,"\"$time\",\"$filename:
$linenum\",\"($errlevel) $errmsg\"\r\n");
fclose($errfile);

if($errno!=2 && $errno!=8) {
//Terminate script if fatal errror
die("A fatal error has occured. Script execution has been aborted");
}
}

Is there someplace I can add this code to have it globally affect all the PHP scripts in G2?
 
robotphotos
robotphotos's picture

Joined: 2005-09-21
Posts: 111
Posted: Thu, 2005-09-22 16:58

I would think that if you were running into an Apache limit, your browser would return some sort of indiciation. Have you tried from a machine that has a different ISP? I've run into something similar to what you are describing when trying to perform time-consuming processes from work (using a client at work, running the process on my server). When I did the same thing at home, I didn't have the issue. My thought, in my scenario, was that the proxy server through which I was connecting my client was causing the issue.
 
robotphotos
robotphotos's picture

Joined: 2005-09-21
Posts: 111
Posted: Thu, 2005-09-22 17:03

With respect to your question about the PHP error reporting, the import module already does error reporting. The way you are describing the problem, it's as if someone clicked the Stop button in your browser. It doesn't sound like the error is occuring on the server end.
 
Nobodys Hero

Joined: 2005-09-16
Posts: 25
Posted: Thu, 2005-09-22 22:35

Yes, I've tried it both from work (t1) and home (cable) using both IE and Firefox... although, i haven't tried all 12 permutations of ISP, browser, and image toolkit, i've tried enough to rule out those 3 as problems... I've also tried it from at least 2 different computers (addendum: and by at least 2 different computers, I mean exactly 2 different computers).

Your description is perfect - it's exactly as if someone hit the stop button.
 
jmullan
jmullan's picture

Joined: 2002-07-28
Posts: 974
Posted: Fri, 2005-09-23 01:45

Nobody's Hero:
Heh, 17 messages before I managed to intervene. The reason that I ask about your APACHE error log is that if php runs out of memory it dies instantly and silently except for an inscrutable message in the apache error log. You will not be able to catch this with any sort of pure php function.

First try shutting off the exif module. If you are not seeing the memory get used up, then exif may be exploding (as usual) and trying to read 4 gigabytes out of a 4 megabyte file. Yes, this happens.

If turning off the exif module doesn't help, please try alloting more memory -- like 64M or 128M. g1 is very inefficient, and the migration module has to load a bunch of g1 data all at once while running a bigger chunk of code than g1. Your g1 might be just squeaking by for global operations, but g2 might run out of memory for those same operations.

If you can't get a full migration to work, please try migrating in chucks of albums, like five or ten or twenty at a time. If you find that one album or image always fails, see if you can add just that album. If it's just one image that is failing, please try downloading that image and manually uploading it to g2. If it works then the migration module has some sort of problem. If not, the problem is probably somewhere else in your gallery.

Good luck. Now that I have responded to this thread I will be notified of any further updates.
 
Nobodys Hero

Joined: 2005-09-16
Posts: 25
Posted: Fri, 2005-09-23 13:52

1) I actually already tried disabling EXIF. I read about it in this thread: http://gallery.menalto.com/node/37016 ... I forgot to mention that in my last post.
2) My gut tells me it's a memory issue. Unfortunately, since I'm on a shared hosting plan with 1and1.com, I don't have access to modify my php.ini file. I did some googling to see if I could override the settings there, and I found this:

Quote:
Finally, a note on the ini_set() function. While PHP reads all its settings at startup from the php.ini configuration file, it also lets you override those settings on a per-script basis with the very cool ini_set() function. This function accepts two arguments: the name of the configuration variable to alter, and its new value. Here is an example, which increases the maximum execution time for the script in which it appears:

<?php
ini_set('max_execution_time', 600);

// more code
?>

The setting only affects the script in which it is set. Once the script has completed executing, the original value of the variable is restored automatically.

If your PHP applications are running on a shared server, it's unlikely that you will have access to the master php.ini configuration file. The ini_set() function can help significantly by allowing you to reconfigure PHP on the fly for your special needs.

Can I just add

Quote:
ini_set('memory_limit', 128);

to the beginning of main.php?

3) I already tried doing a couple galleries at a time (I mentioned this in a previous post)... I have at least 2 galleries that fail even when I try importing them 1 at a time. It's not always the same photo where it times out.

... and thanks for jumping in on this thread. i need all the help i can get ;)
 
Nobodys Hero

Joined: 2005-09-16
Posts: 25
Posted: Fri, 2005-09-23 14:01

I guess that doesn't work... Every time I add that ini_set line (i've tried adding it in main.php, bootstrap.inc, and Gallery.class, and all 3), I get a message similar to the following... I've even tried values other than 128 (such as 40, 39, and 1).

Quote:
Fatal error: Allowed memory size of 128 bytes exhausted (tried to allocate 84 bytes) in /homepages/46/d89726853/htdocs/michaelmolino.com/gallery2/main.php on line 25
 
Nobodys Hero

Joined: 2005-09-16
Posts: 25
Posted: Fri, 2005-09-23 14:05

ok, nevermind my last post... i forgot the 'M'

when I add

Quote:
ini_set('memory_limit', '128M');

to the top of main.php, I don't get that error... now for the moment of truth. i'm about to try the import... cross your fingers for me!
 
Nobodys Hero

Joined: 2005-09-16
Posts: 25
Posted: Fri, 2005-09-23 14:37

I tried the import twice with the new memory limit and it still fails... The first time it failed on IMGP1885.jpg and then on IMGP1883.jpg. I can tell the ini_set() function is working because it shows 128M at the top as the memory limit instead of 40M, so apparently that wasn't the problem.

I was beginning to think 1and1 (my host) just had their server set to kill any script after 30 seconds or something... but that can't be the problem because I've had no problem running G1 and I've uploaded some pretty large albums all at once.
 
robotphotos
robotphotos's picture

Joined: 2005-09-21
Posts: 111
Posted: Fri, 2005-09-23 14:53

Let me know if you change your mind on my offer. ;-) If bandwidth is an issue, and it's no larger than 4 GB, you might be able to snail me a DVD-R of the files, and I could send back the modifications the same way.
 
Nobodys Hero

Joined: 2005-09-16
Posts: 25
Posted: Fri, 2005-09-23 15:54

haha... i really appreciate it, but i want to try and figure this out...

if i can't get it to work, i'll do 1 of 2 things:

1) stick with G1 until G2.x comes out and fixes my problem...
2) i have a desktop that still has linux installed... i'll just rsync my G1 onto the local machine, migrate there, and rsync it back... but i'm not ready to give up yet!
 
jmullan
jmullan's picture

Joined: 2002-07-28
Posts: 974
Posted: Fri, 2005-09-23 17:56

Nobody's Hero:
Since I wrote the migration module and mindless is the only other person to work on it, your problem probably won't be solved for gallery 2.x unless we can figure out exactly what your problem is so we can fix it. Try visiting the failing albums in g1, maybe change the properies so they get resaved. Sometimes that helps.

You might want to extend the timeouts.
Look for these lines in modules/migrate/ConfirmImport.inc and replace the 30 with a larger number like 180. Just for fun.
$gallery->guaranteeTimeLimit(30);
 
Nobodys Hero

Joined: 2005-09-16
Posts: 25
Posted: Fri, 2005-09-23 18:12

still doesn't work.. i changed that line from 30 to 180 for 3 instances in ConfirmImport.inc and once in ChooseObjects.inc...

it still "timed out" at the same approximate place it always does.
 
Nobodys Hero

Joined: 2005-09-16
Posts: 25
Posted: Fri, 2005-09-23 18:26

and for the other suggestion about changing the properties... one of the failing galleries has had several updates (adding photos, resorting, moving sub albums around) since i first started this thread.
 
jmullan
jmullan's picture

Joined: 2002-07-28
Posts: 974
Posted: Fri, 2005-09-23 19:15

Can you move that album to the root of your gallery, tarball it, then move it back and send me a link to download that album? This is probably going to require me to investigate more directly. Let me know.
 
jmullan
jmullan's picture

Joined: 2002-07-28
Posts: 974
Posted: Fri, 2005-09-23 21:46

michalp, I moved your post into a new thread for your problem.

Read this:
http://codex.gallery2.org/index.php/Gallery2:migration

Go to your new post here:
http://gallery.menalto.com/node/37168
 
myn

Joined: 2003-01-23
Posts: 8
Posted: Sat, 2005-09-24 16:19

I am yet another person using 1and1 hosting, trying to migrate my g1 to g2 with the exact same results. Perhaps this PHP based migration solution that is dependant on defined memory contraints isn't the best choice. Anyone up for the task of writing a binary to perform the migration?
 
jmullan
jmullan's picture

Joined: 2002-07-28
Posts: 974
Posted: Sat, 2005-09-24 19:44

myn, open a new thread.

READ THE MIGRATION STICKY!
 
Nobodys Hero

Joined: 2005-09-16
Posts: 25
Posted: Sat, 2005-09-24 19:59
jmullan wrote:
Can you move that album to the root of your gallery, tarball it, then move it back and send me a link to download that album? This is probably going to require me to investigate more directly. Let me know.

Here's one of them...

http://www.michaelmolino.com/albums/Zakopane-Slovakia.tar.gz
 
myn

Joined: 2003-01-23
Posts: 8
Posted: Sat, 2005-09-24 21:47
jmullan wrote:
myn, open a new thread.

READ THE MIGRATION STICKY!

I'm going to follow along with this one. Since we are having the exact same issue.
 
robotphotos
robotphotos's picture

Joined: 2005-09-21
Posts: 111
Posted: Sun, 2005-09-25 03:53

If you do that, then they can't properly keep track of the metrics for problem resolution.
 
myn

Joined: 2003-01-23
Posts: 8
Posted: Sun, 2005-09-25 04:11
robotphotos.org wrote:
If you do that, then they can't properly keep track of the metrics for problem resolution.

I wasn't aware there were metrics on problem resolution. Is there paid staff? Why does this bureaucratic metrics policy exist? I feel like I am at work :(

Support is about giving people answers. It would be a waste of time for me to create another post and have more time wasted by more people reading it. Especially since we share the exact same issue and hosting provider.

Just my 2 cents.

I appreciate all and any help I receive in these forums.

Thanks for helping.
 
jmullan
jmullan's picture

Joined: 2002-07-28
Posts: 974
Posted: Sun, 2005-09-25 04:12

Myn, while your problems might seem similar, they might be caused by different things. Plus, if I ask Nobody's Hero a specific question, I may or may not want to ask you the same question. The first question that I want to ask you is if you have read the Migration guide. Have you read the migration guide? If not, please go read it. While you read it, take note of any questions that I ask in it. The more information that you give me, the faster I will be able to figure out if you are experiencing the exact same problem or just a very similar one.

The second question that I want to ask is if you can fill out the support template.
 
robotphotos
robotphotos's picture

Joined: 2005-09-21
Posts: 111
Posted: Sun, 2005-09-25 04:20

Why do you need paid staff to have metrics on problem resolution? It's just good quality assurance and it helps keep track of how many people are having problems in a specific area, so it can be determined if more time needs to be devoted to a particular area because of usability issues. ...and is it less of a waste of time to not follow the pre-written steps that the people giving you free help ask of you, only to have them ask you directly, because you didn't do it the first time? ;-)
 
jmullan
jmullan's picture

Joined: 2002-07-28
Posts: 974
Posted: Sun, 2005-09-25 06:24

Uh, I'm not paid staff and I wrote the migration module. I want it to work for you, but I need you to answer questions for me and I don't want you to mix your answers in with Nobody's Hero's answers. Now, once you start a new thread I will delete this extraneous discussion out of this thread.
 
Nobodys Hero

Joined: 2005-09-16
Posts: 25
Posted: Tue, 2005-10-04 14:07

hey jmullan... i've been looking through the posts over the past week, and i see a few people are having my same problem... i was wondering if you had gotten any closer to a solution... thanks for all your help so far.
 
darley
darley's picture

Joined: 2003-04-05
Posts: 28
Posted: Mon, 2005-11-07 16:42

bump...

I am having a smiliar problem, any resolution to this?

http://gallery.menalto.com/node/39530

Thanks for any info!
 
kevinprice

Joined: 2005-11-28
Posts: 1
Posted: Mon, 2005-11-28 22:49

I had what I suspect is the same problem with my install. What's going on appears to be completely unrelated to any memory management issues. Rather, on some web service providers the install script is messing up with activating the image processing modules. I took the steps at the bottom of this thread: http://gallery.menalto.com/node/40480

After I did that, the migration works fine.