Integrating Gallery with Amazon S3
|
neilhand
Joined: 2006-03-27
Posts: 4 |
Posted: Mon, 2006-03-27 04:08
|
|
Hi all, I didn't see this mentioned on the site during my searching so hopefully this is not something already covered. I currently use Gallery to host an online photo album. Since space is becoming an issue i would like to store the images using the S3 web service from Amazon. My first thought was to create a quick and dirty PHP+flash website, but then i started to wonder if this is something that Gallery could be expanded for. So i have a few questions. 1) Is this something already being looked at Any pointer would be appreciated before i start digging through the code. Best Regards |
|
Posts: 32509
no, there's nothing available or planned for g2 <-> amazon s3.
s3 isn't that special. the general case of remote storage is sure interesting and has been discussed in these forums from time to time. there's no relazted solution there yet, but possible approaches have been discussed.
just a few notes without repeating myself too much:
GalleryUrlGenerator.class which generates all URLs -> your new core.Downloaditem urls should point to the remote storage
GalleryPlatform.class -> maybe add a new platform which stores entity related data on a remote host.
Posts: 1
I would like to see this kind of integration as well. I have well over 20GB of photos I'd like to post online. S3 appears to be an economical solution.
Please keep us posted on your progress, Neil.
Posts: 1
I too would like to see this kind of integration between Gallery and S3. In my mind, S3 IS something special in that it gives a really affordable remote storage option, much more affordable that most others.
I too would like to see any progress on that, Neil, if you are able to tweak gallery to configure it for remote storage.
Best,
~Evan
Posts: 9
has there been any work done on this?
Posts: 4
Unfortunately not by me.
My baby daughter is keeping me too busy, and any spare time has been used to do programming for my wifes design business. It is however something that i would like to get back to when the opportunity arises.
Posts: 40
I think that implementing a more general remote storage interface would be more valuable than one specifically for s3.
/carmi
Posts: 6
I was thinking about this myself until I came across this thread, just recently. My problem is that I like my hosting provider except that it's not cheap to get more storage. And any hosting provider with cheap storage either does not have the features I want or have enough reviewers posting how crappy the service is. So, Amazon S3 seemed to be a really good choice.
One problem with Amazon S3, or any remote storage, is that it costs money for the bandwidth. And any hosting provider also charges for the bandwidth. So, if Gallery handled the data transfers through redirection, it would cost double. So, ideally, the image transfers would interact directly with the remote storage provider than through Gallery. For downloading, a remote storage URL could be generated pretty easily. (Would IE complain about mixed URLs in an SSL environment?) For the tools which use the remote API, another API would have to be created to get the storage URL and then the images downloaded directly. I have to read up on the Amazon S3 protocol to see how easy it would be to use simple URL redirection.
However, upload seems like it would be harder to implement. Since every remote storage implementation would probably have a different API for storing the data, it probably would be hard to avoid using Gallery to redirect the image.
I would love to volunteer here at implementing this in the next month or so once things settle down during the holidays. I don't have any experience with touching Gallery code, but I'm sure I can push my way through and some up with some prototypes. Any suggestions on how to start?
Posts: 32509
i haven't thought seriously about remote storage for gallery yet, but i can give you some starting points.
- see my post earlier in this thread: http://gallery.menalto.com/node/46771#comment-177307
- all Gallery filesystem operations run through modules/core/classes/GalleryPlatform.class (and its child classes WinNt and UnixPlatform).
there's no feature in g2 to replace the whole platform implementation.
- all URLs are generated by the GalleryUrlGenerator.class.
any module can register a new GalleryUrlGenerator which is then responsible to generate all URLs. that's what the URL rewrite module does to generate the short URLs.
- there is an existing module which uses the Coral CDN (http://www.coralcdn.org/) to offload bandwidth / load from the Gallery server to a world-wide content distribution network. but this won't help you with saving disk space.
- disk space is cheap. if you have a webhost which doesn't give you enough disk space, you have the wrong one. more and more webhosts give you 10, 50, ... GByte disk space.
the problems you'll be facing:
- changing URLs to point to amazon or any other host will be easy
- but if you don't copy, but actually store the only original file on the remote host, you will need to control all disk-operations of Gallery.
e.g. if Gallery needs to regenerate thumbnails, you will need to tell it where to get the original file from (and then e.g. temporarily copy it to the Gallery server).
i haven't thought of a good way to deal with that yet.
but as i said. if diskspace is your problem, you probably have the wrong webhost.
Posts: 6
I did some research into web hosts that give 50-200 Gb. For everyone that looked reasonably cheap (less than $10 a month), I would easily find a website with all of the issues with that webhost. I even signed up for one only to find out that any of my request tickets took days to get resolved, which was pretty annoying. Second, alot of them didn't have the features that I was looking for. So, needless to say, the right webhost, IMHO, is hard to find. And I got really tired of looking around, especially when I was really happy with the webhost I was already using.
I forgot about the thumbnail regeneration issue. That's an interesting problem. But I would assume that it would have to be as you said, Gallery pulling the image to the local server. So, to solve this issue, it would require a cache on the local server of some arbitrary limited size, so that this is not always expensive. I'm guessing that when people generate thumbnails, they deal with a small set of images as they are adjusting the sizes, etc.
I would think that this remote storage solution should solve the bandwidth issues as well. Amazon S3 has the ability to use Bittorrent.
Thanks for the info. I'll take a look at the GalleryPlatform class. I am guessing that I would have to create a different interface anyways since there are cases where some of the files will need to be stored locally and some remotely.
Posts: 32509
tried dreamhost? they give you tons of webspace and i heard mostly good stuff about them.
http://www.dreamhost.com/hosting.html
they're also a certified gallery 2 webhost and as such they must be one of the better webhosts since we test their offer and their support.
Posts: 6
I looked at them. and then I found http://www.dreamhost-sucks.com/ then i decided to wait.
Posts: 32509
Thanks for bringing that site to our attention. We'll look into the issue and have our webhost affiliate manager deal with the situation.
EDIT:
Please note that the rating of dreamhost has been raised to 'B', second best and many of the complaints this site brings up are no longer valid. Anyhow, we'll look into it.
Posts: 2
One of the good things about S3 is that everything's stored in multiple data centres. You don't really get that if you've got your own server or a regular hosting provider. I like the idea of storing my photos on S3.
One thing I thought about was setting up s3infidisk on my local server, to store the data on S3, but have it mounted as a local drive (using fuse fs).
This is then symlinked to the gallery data directory. So to gallery, it looks as if all the photo files are local.
One disadvantage would be the double-bandwidth issue. However, you could rewrite the URLs to point to S3 easily, but then continue to use the g2 data directory when uploading photos and regenerating thumbs.
Tom
Posts: 344
Hi Guys,
I currently host over 2000 photographs and have near 10gb of space used up. I've been very happy with my host at www.doreo.com and they even used to host phpbb. They are not a big company, so you won't have to do the annoying "send a ticket and hope they answer" routine that usually lasts a week, if you are even lucky. Also the server guys are always on MSN and you can just contact them right away if you have any issues. I've been with them since 2000 and have not had any issues. They are the nicest people to work with. I know it sounds like a sales pitch, but give 'em a try! Tell them what your needs are and I'm sure they can work out something for you. Hosting is a very important service and it should be done right so you can sleep at night. Oh yea, and if you are going to be hosting tons of photos, you want to make sure you site is backed up automatically every few hours. Doreo does that for free and they've already saved my ass a few times after I deleted the wrong files
Hope that helps.
Posts: 3
throw me into the camp that would love to see s3 support in gallery. i don't want to go through the hastle of moving hosting providers when all i need is cheap storage. and s3 is cheap for hosted top tier storage. for work, i have used it before and it's a great solution. s3 support would definitely improved the commecial applications for gallery as well.
Posts: 4
Count me in as another person requesting that Gallery support Amazon S3!
Posts: 4
If you would like to add Amazon S3 support for Gallery, please vote for this feature here:
http://gallery.menalto.com/sfvote/
Click on "Vote by id". The ID is: 1646294
Posts: 1
I'd like to see this feature added as well.
Posts: 1
Me too. I work at Amazon and I love Gallery and S3, but I have not yet heard of an integrated system for them both. I will check on this from my end.
Tom Hubbard
206-266-3806
Posts: 1
I use JungleDisk, an disk mount / backup application that uses amazon S3.
I uses WebDAV to mount a disk which uses S3. <http://www.jungledisk.com/>
Is there any way to hook it up to gallery2 ??
Posts: 15
Not likely using JungleDisk since it uses a somewhat proprietary way of storing data (and it encrypts data).
But if gallery2 support s3, no reason why your gallery install couldn't hold things in a temp directory, and upload to s3 (then delete the temp). That way you can use any upload method supported by gallery2.
Posts: 18
I would love to see a amazon s3 module.
I was thinking about using it as a backup solution.
Options:
1: copy all albums to S3 automatically
2: copy selected albums to S3 manually.
I would love to see any integration w S3 tho!
Posts: 11
Any further developments on this front?
Looks like a great idea.
Posts: 2
I'd like to see S3 integration too.
The way I'd like to use it is; store the originals on S3 and the thumbnails and intermediate images on the local server.
I have two servers at Datapipe now but they charge $5/GB for backups which is cost prohibitive for large scale media storage.
Of course these are real off-site backups not what the cheap webhosts offer.
Those cheap webhosts IF they offer any backups they often backup on the same disc.
If you get hacked, or the disc becomes corrupted, what happens to your data?
Many people think RAID would protect them from disc failure but often disc corruption is incremental which means you'll end up with corrupt files in every disc on the raid.
Here is a glimpse to Dreamhost's "backup" procedure.
Not guaranteed and done on the same disc.
Does it make you feel secure?
http://wiki.dreamhost.com/Snapshot_data_restoration
Posts: 42
I'd also like to see S3 support. It really could cut costs, and give us many more choices for where to host our galleries. Right now we're limited to the companies who oversubscribe their services.
Posts: 42
Here's something interesting... a MySQL storage engine for Amazon S3.
http://fallenpegasus.com/code/mysql-awss3/
Posts: 11
Gallery2, s3fs and Amazon S3
http://gallery.menalto.com/node/77816
Posts: 1
I also want this feature.
Posts: 16504
pc2s, Read the full thread and links.
____________________________________________
Like Gallery? Like the support? Donate now!!! See G2 live here