Too many open files

samokk

Joined: 2003-06-25
Posts: 33
Posted: Sun, 2006-09-24 20:05

Hi,

I am trying to upgrade from Gallery 1.5.2-pl2 to Gallery 2.1.2.
ii apache2 2.0.55-4ubuntu2.1
ii libapache2-mod-php5 5.1.2-1ubuntu3.2
ii php5 5.1.2-1ubuntu3.2

(It was the same using php4, Ubuntu Linux Hoary)

Here is the error message I got :

Quote:
Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: require_once(/srv/http/pingoo.ath.cx/gallery2/modules/exif/lib/exifer/makers/canon.inc) [function.require-once]: failed to open stream: Too many open files in /srv/http/pingoo.ath.cx/gallery2/modules/exif/lib/exifer/exif.inc on line 606

Fatal error: require_once() [function.require]: Failed opening required '/srv/http/pingoo.ath.cx/gallery2/modules/exif/lib/exifer/makers/canon.inc' (include_path='.:/usr/share/php:/usr/share/pear') in /srv/http/pingoo.ath.cx/gallery2/modules/exif/lib/exifer/exif.inc on line 606

I also tried to drastically increase the max number of files that can be opened, it doesn't change anything...

Does anyone have an idea of how to solve that ?
Thanks,
Sami Daloche
Login or register to post comments
valiant

Joined: 2003-01-04
Posts: 32155
Posted: Mon, 2006-09-25 15:04

too keep the resource hunger down:
- disabling the exif module during import helps
- not generating thumbnails during import helps

but before, you should do the integrity checks for g1:
http://codex.gallery2.org/index.php/Gallery2:migration#Read_First

what's the current ulimit for open files?
Login or register to post comments
samokk

Joined: 2003-06-25
Posts: 33
Posted: Mon, 2006-09-25 17:45

Hi,

First of all, thanks for your help

yes, I did the 2 integrity checks, both orphan and the verification. Everything was OK.

I also disabled the EXIF module, and disactivated thumbnail generation.

Concerning the limits :

root@bluerock:/proc/sys/fs# cat file-max
100203

root@bluerock:/proc/sys/fs# su www-data
sh-3.1$ ulimit
unlimited

root@bluerock:/proc/sys/fs# echo 2000000 > file-max ( 2 million files !!)

I now get a slightly different error message, which mostly means the same...

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line
646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: Invalid argument supplied for foreach() in /srv/http/pingoo.ath.cx/gallery2/modules/migrate/ConfirmImport.inc on line 646

Warning: copy(/srv/data/webapps/pingoo.ath.cx/gallery2/albums/all/2005/200504/20050401as01/00012_G.jpg) [function.copy]: failed to open stream: Too many open files in /srv/http/pingoo.ath.cx/gallery2/modules/core/classes/GalleryPlatform.class on line 65

My gallery contains more than 17k items, with 200 or 300 albums.

Any idea ?...

Sami Dalouche
Login or register to post comments
valiant

Joined: 2003-01-04
Posts: 32155
Posted: Tue, 2006-09-26 00:48

ulimit -a
shows the actual limit. ulimit itself just reports the max filesize that can be written, i guess.

it's an interesting error. let's see how low your limit actually is

better do the check in a php script that you browse to, since the ulimit can be different for your account vs. the webserver user.
<?php system('ulimit -n'); ?>
Login or register to post comments
samokk

Joined: 2003-06-25
Posts: 33
Posted: Tue, 2006-09-26 06:30

Hi,

system('ulimit -n'); reports 1024....

So, the actual limit seems to be 1024, which should be enough, I think, for a simple import script...
However...

# ulimit -n 10000
# ulimit -n
10000
# /etc/init.d/apache2 restart

and the system('ulimit -n') from php now reports 10,000

I guess it will now be possible to run the import script correctly, even if I have to raise that limit again, or have to raise the memory limit too.. (it is currently 80M for php, I will increase it if needed)...

The script is currently running, will take something around 2 hours to run, so I'll post later to confirm that it worked..

But.. I'd still like to know why the script opens soo many files ? Shouldn't the script close them as they aren't needed anymore ?

Thanks for your help !
Sami Dalouche
Login or register to post comments
samokk

Joined: 2003-06-25
Posts: 33
Posted: Tue, 2006-09-26 12:36

Ok, so some news about the improt process :
- 10,000 was not enough, it resulted in the same error, so I set it to 100,000, with memory_limit=200M
=> The import process went fine..

I'm just still amazed that it is necessary to tweak the max # of open files just for a simple import script....

Anyways, thanks a lot for your help, my actual problem is solved :-)

Sami Dalouche
Login or register to post comments
valiant

Joined: 2003-01-04
Posts: 32155
Posted: Tue, 2006-09-26 17:33

Wow, and LOL. :)

It's more a sarcastic laughing over here. I didn't expect that the migrate module would keep so many files open, that's crazy.

you had to import 17k items in 200-300 albums.

setting ulimit to 10k wasn't high enough, 100k worked fine.

that's good debug information, thanks. we'll see if that helps to improve the process.
Login or register to post comments
valiant

Joined: 2003-01-04
Posts: 32155
Posted: Tue, 2006-09-26 17:56

Our guess is that you used file / flock based locking ("site admin", bottom of the page has locking options).
If you used database based locking, an open file limit of ~100 should have been sufficient.

i'll see if we can optimize the code for file based locking as well.
Login or register to post comments
samokk

Joined: 2003-06-25
Posts: 33
Posted: Tue, 2006-09-26 19:38

Ok, so to confirm the previous figures :
- 2 top-level albums (685 total), 17431 images (in other words, gallery1 is currently dying ;-) )
- I left the default locking, which is file / flock based locking
- I disabled thumbnail generation and the EXIF module during import

And thanks again for your help !
Keep up the good work, Gallery2 is awesome !!! I am amazed how much work has been achieved!!!

Sami Dalouche
Login or register to post comments