Managing really large numbers of files?

I have an issue that I'm trying to get past and I'm wondering what other people have done about it or if they have thoughts on it.

I have approximately 8 gigs of photos and it's growing fast. I was working on my own web based photo / media presentation app until I came across Gallery, now I'm transisitioning to Gallery for many reasons, including user management.

In the app I was working on, the entire site was built based on an existing directory structure of photo files. You point the app to your existing directory structure, and it internally generates and stores thumbnails etc, but it doesn't copy the full image file. There is no need to "import" your files into the app. Where as with Gallery, when you publish/import photos, you immediately double your storage requirements for your photos.

What if someone were to make Gallery more data driven in the sense that "galleries" were based on directories full of photos? Publishing would then consist of simply creating a new directory in that structure and dumping photos in it. Gallery would then dynamically pick up that change and reflect it on the web, at that point allowing the admin to tweak descriptions and such.

I realize hard drives are really cheap these days, but for me that is not really the point, it's more about managability, design, and to some degree disk usage.

My questions are:
- For people with REALLY large amounts of photos to publish, how do you manage that? Do you just "bite the bullet" and publish/import all of your photos, doubling the space you use to store photos?
- Has anybody thought about modifying Gallery (would be an optional feature) so that it automatically, at runtime, builds galleries based on an existing directory and file structure, eliminating the need to "import" or "publish" photos? If you wanted to publish photos in gallery you would just create a new directory in your dir structure, dump the photos in there and gallery would pick up the changes automatically.

I realize what I'm talking about here is a design decision and the two approaches, data drives vs. a publish model each have their pros and cons.

I've searched the archives but have failed to find any big discussions about this, sorry if it's a repeat.

Keep up the good work on Gallery, it rocks as is, these are just ideas. And thanks to any who have input on my questions.

Same thing here. I decided to hold off on Gallery inspite of the really cool features (v1.4.2) due to no bulk/batch import back that I could find in March 2004. I just looked at it again for grins to see if there were any updates and OMG I scrolled down to the bottom in the admin options "Add Photos" and found the XP Publishing Agent!! Very very cool.

While I would still love to see a way to handle existing data (like pointing at a root dir and doing some sort of auto import), I think that this makes it much more manageable.

Thanks!! :D