Module: Coralialze - generates image urls that use Coral CDN
|
sandymac
Joined: 2005-09-15
Posts: 31 |
Posted: Thu, 2005-09-15 04:28
|
|
Linked is a tarball for G2 module that when activated rewrites image urls to use the Coral Content Distribution Network. It currently is smart enough to not coralize non-public images as Coral CDN doesn't support cookies and they would get a permission denied error anyways. This is my first coding for Gallery, feedback welcomed. http://Sandy.McArthur.org/gallery/modules/ Features:
|
|
Posts: 32509
sandymac, you rock!
this is a great idea and i'm impressed of the implementation. you had an excellent idea and obviously knew exactly how to implement it in g2
of course there's room for improvement
- make it compatible with the rewrite module. both modules rock, it would be a pity if you couldn't use both at the same time as a user. but that's not your problem, it's a shortcoming of the g2 framework or the rewrite module.
pelle is the author of the rewrite module.
one approach to make them work together:
1. the GalleryUrlGenerator.class needs to look for url generation plugins when generating urls. it would check for registered UrlGenrationPlugins and if there is one, it would call the plugin like $url = $plugin->handleUrl($url);
2. your module would register such a plugin instead of a whole Urlgenerator.
this is just one possible way to solve it.
- permissions
so you only want to coralize public files? that's a good idea.
your permissions check is slightly wrong though. core.view isn't the correct permission to check. core.view determines if you are allowed to see the ShowItem page.
you'll have to add some logic there.
if it's a GalleryUtilities::isA($item, 'GalleryDataItem'), check core.viewSource for $item->getId() (it's the original size photo)
else if GalleryUtilities::isA($item, 'GalleryDerivativeImage') { // resize or thumbnail
if $item->getderivativeType() == DERIVATIVE_TYPE_IMAGE_THUMBNAIL check core.view for $item->getderivativeSourceId()
else check core.viewResizes for $item->getderivativeSourceId()
}
- what if coral hasn't cached this item?? can you enforce coral to cache something?
- coding style:
it's not yet 100% g2 coding style, but very near to it. i've started much worse when coding for g2 for the first time
e.g. comments are always in /* */ style in g2 and tabulator is equal to 8 spaces, not 4. indent always by 4 spaces after a opening { and replace 8 spaces by a single tab
http://dev.gallery2.org/modules.php?op=modload&name=GalleryDocs&file=index&page=gallery2-devguide.coding-standards.php
[Moderator edit: new url is http://codex.gallery2.org/index.php/Gallery2:Coding_Standards]
i've added your module to the user contributions page:
http://codex.gallery2.org/index.php/Gallery2:UserContributions#Modules
you can edit this page yourself. just sign up to codex.gallery.org.
Posts: 31
Thank you, but all I really did was add stuff until I got it to work then removed everything I added that didn't break it when I removed it.
Okay, I'll look into that. Developer documentation is presently missing, and the current implementation reflects my understand from what I was able to infer from other modules.
I'll fix that today.
Coral responds to caching request as they come in. For example to coralize http://cnn.com/ just access the url http://cnn.com.nyud.net:8090/ . Coral with then either serve a cached version of the http request or fetch the original right then and serve that as it adds the content to it's cache.
No, but you can make it fail gracefully. You can control what Coral does when it chooses not to cache something by adding the X-Coral-Control header with the value "redirect-home" to the http request from Coral to your server. If that header is present then Coral will redirect the clients back to the origin server. Presently I'm using a .htaccess file to add the header but I hope to figure out how to make it so the module can do it if so desired. The default behavior doesn't failover back to the origin site because if a site is being slashdotted and coral cannot handle it, then your site probably doesn't want even more load that coral cannot handle.
I'll fix that in the next release.
Posts: 31
I just uploaded version 0.4.3 to http://sandy.mcarthur.org/gallery/modules/
The pervious version was broken with respect to it's permission checks and thus didn't actually coralize any images. That has been improved but there are still problems. For example if you have water marked your images the permission check fails and the image won't be coralized.
Posts: 31
I just uploaded version 0.4.4 to http://sandy.mcarthur.org/gallery/modules/
This included a work around for this bug I also found:
http://sourceforge.net/tracker/index.php?func=detail&aid=1292343&group_id=7130&atid=107130
Once that bug is fixed version 0.4.3 should be fine.
Posts: 32509
why does the permission check fail for watermarked images? because the original versio / source files are then preferred source files?
Posts: 31
Because of a bug in the permissions check code. version 0.4.4 which works around
Bug 1292343 makes Coralizing images with watermarks work fine.
Posts: 32509
nice finding. i guess you're right.
so you do a permission check with userId = anonymousUserId i guess. makes sense.
i'll look into fixing it.
Posts: 31
Yea, that bug wasted a lot of m time trying to figure out why my code wasn't doing what it looked like it should be doing.
Are those last two steps how things are now or are you suggesting that the two modules could be made compatible by improving the core classes like you describe?
Posts: 32509
these are just wild ideas how it could be done, but we'd have to change at least the rewrite module or more probably the original urlgen too.
for now, your module is incompatible with the rewrite url gen, as you already found out.
Posts: 174
I was wondering why the Album thumbs arn't coralized. I looked at your class but I dont know enough php to add the album thumbs. Wondering if this will ever be a part of gallery rather than a third party addon. Seems like a great idea.
Posts: 31
The module needs a lot more work. I get all sorts of weird behavior in weird corner cases. Since my last revision I found when using Coralize:
* The Java applet for cropping thumbs fails because of Java security
* Images that are linked so they appear in more than one gallery may not be coralized based on which gallery they originiate from and how it's access controls are set up.
Once I get it to what I call a 1.0 release I will suggest making it a part of the stock G2 download.
Posts: 32509
we will offer downloadable modules soon, then you can download new modules directly to your server in the site admin section of g2.
and to add something to the official g2 package, you need to include unit tests and you'll have to go through some code review passes such that the source code is 100% compliant with our coding guidelines.
but for now, it's a good idea to concentrate on bugs and correct functionality
Posts: 174
Well Its up and running on my server http://www.robplatt.com/gallery2, be happy to test.
Posts: 31
I just uploaded version 0.4.7 to http://sandy.mcarthur.org/gallery/modules/
This has a configuration settings for what type of images you want to coralize. Now you should be able to coralize the originial image, resized images, thumb nails, or static theme related images or any combination you choose.
It still has problems with the permissions check. If someone could explain why the following fails the permission check for anonynmous users I'd be grateful.
First upload an image to a private album. Then create a link for that image to a public album. Viewing the image in the private album is not coralzied which is correct. Viewing the image in the public album is still not coralized which is wrong.
Posts: 174
Looks great. Although, coralizing the logo in the corner and the couple icons i have is actually slower. So...
I'm still waiting to be able to coralize my album thumbnails
Posts: 174
hey sandymac. I set it up to coralize thumbs and resized images. However, i dont believe its coralizing my resized images.
heres just one example
http://www.robplatt.com/gallery2/main.php?g2_view=core.ShowItem&g2_itemId=22091
but if you browse my gallery, it goes for the rest as well.
Posts: 31
I just uploaded version 0.5.0 to http://Sandy.McArthur.org/gallery/modules/
I think this fixes all known (to me) permissions check problems. While this will still have problems for administrators doing some admin tasks when enabled, this should be problem free now for "normal" viewing activity. Upgrades are recommended.
Posts: 174
Looks good sandymac.
Posts: 174
sanymac I have a request for you if it wouldnt be too difficult to implement.
can you give us an option to put our local ip subnet in and if gallery detects the browser ip from that subnet it wont coralize the urls?
i really really like this module you built, but it sure makes it a pain to browse my gallery locally as my server is here on my own network. and im sure many users are like that.
just a thought.
i would like to add 192.168.1.1-255 and if any ip is detected in that range it will temp disable it.
Posts: 31
Yea, sounds reasonable. I'll make it not coralize requests from all private ip addresses since they won't benefit from Coral:
127.0.0.0/8; // loopback
10.0.0.0/8; // RFC 1918 private
172.16.0.0/12; // RFC 1918 private
192.168.0.0/16; // RFC 1918 private
169.254.0.0/16; // link-local
192.0.2.0/24; // test network
Posts: 174
thanks
Posts: 31
I just uploaded version 0.6.0 to http://Sandy.McArthur.org/gallery/modules/
This version does two new things:
1. Doesn't coralize requests coming from the local machine or from a private IP address (an IP that isn't on the internet).
2. Doesn't coralize content on pages that are something other than "view=core.ShowItem". This is how I'm fixing the problem where some admin pages break when requesting image content from a domain other than the local one.
This module is pretty much at a point where it does everything I want it to do. Unless I find some bugs I don't really plan on improving it any more. Someone else is more than welcome to take over this module if they want.
Posts: 52
Coral module DOES really rocks guy!
I'm using it and i love it. Very useful for people with G2 installed on their lowbandwidth domestic webserver
--------------------------------------------------
Gallery 2.0.1 core 1.0.0.1 http://www.mcgrelio.com
PHP 4.3.2 apache2filter
Apache/2.0.46 (Red Hat)
Mysql 3.23.58
Browser Mozilla/5.0 Gecko/20051107 Firefox/1.5
Posts: 3236
I really wish this worked with URL rewrite... I suppose you can turn off the rewrite for item downloads, but leave it on for pages and album views? (the /d/ links, not the /v/ ones). Does that allow it to work?
_________________________________
Support & Documentation || Donate to Gallery || My Website
Posts: 31
The problem is that you cannot chain Gallery URL Generators but you can inherit from another URL Generator. A solution is make another internal URL Generator that inherits from the Rewrite URL Generator and load that version instead when both Coralize and Rewrite are enabled. After I'm done with exams I'll see if I can do that.
I also have one other feature I want to implement: Coral doesn't always work for everyone. It should be possible to detect if a chunk of JavaScript is successfully loaded via Coral and if that is the case then Coralize any future requests for that user.
So I guess I'm not as done as I've previously stated.
Posts: 32509
i recently thought about a solution for short urls + coralize.
it's not just coralize, if we ever want to add mirroring or using a remote image server, then we would want to use 2 url generators too.
and i had three alternative ideas:
- as you describe it, having a second url generator in your coralize url Generator.
- or introduce UrlGeneratorTransformer or UrlGeneratorPlugins, maybe you can think of a better name...
your module would then register such a urlgeneratorPlugin instead of a urlgenerator and the original urlGenerator would check for urlgenerator plugins when generating urls.
e.g. before returning the result it could check $url $plugin->transformUrl($url, $params);
then you could check $params whether to apply your coralize stuff or not and return a coralized url, or the original
- or we introduce an admin panel where you can chain url generators. so all generateUrl calls would first run through rewrite url gen, then the result would run through your url gen
not sure what i prefer, but you should definitely do the first thing (nested url gen) for the time being since the other solutions require some changes.
Posts: 31
@valiant: Any clue how one module can detect state changes in another module? The problem I'm having is that while I can register the right URL Generator at the time the Coralize is activated, if the URL Rewrite module is activated or deactivated after that then the Coralize module won't notice the change.
Posts: 32509
sure and good question
see modules/rewrite/module.inc
on how to register an event listener. in the module constructor, $this->setCallbacks('getSiteAdminViews|registerEventListeners');
and add a function registerEventListeners
also, you'll have to add a handleEvent function somewhere.
Posts: 31
I just uploaded version 0.6.2 to http://Sandy.McArthur.org/gallery/modules/
This version now plays nice with the URL Rewrite module.
Posts: 32509
fantastic job!
great solution to extend RewriteUrlGenerator!
a few notes:
preferred derivatives are derived, web-viewable images for non-webviewable data items (tiff images, other weird image formats, PDFs, ...).
it's also explained here:
http://gallery.menalto.com/node/36885#comment-134279
- http://sourceforge.net/tracker/index.php?func=detail&aid=1292343&group_id=7130&atid=107130 has been fixed
- function handleEvent($event) {
you shouldn't ignore the return value of
CoralizeModule::performFactoryRegistrations(); (error checking)
all in all it looks excellent. add a little more comments please, e.g.
'modules/coralize/classes/CoralizeUrlGenerator.class', 'coralize', null, 2);
you should comment why you set the priority to 2 vs. 1
Posts: 32509
important note (for january 2006):
the rewrite module will be changed considerably in a week or so (in BRANCH_API_DEV, so the change will be in the nightlies starting around new year).
class RewriteUrlGenerator will have 3 classes that extend it:
- ModRewriteUrlGenerator extends RewriteUrlGenerator
- PathInfoUrlGenerator extends RewriteUrlGenerator
- IsapiRewriteUrlGenerator extends RewriteUrlGenerator
(only one of them will be registered at a time)
so you'll face a little problem there, since you can't extend RewriteUrlGenerator anymore.
ideas:
- maybe have 3 COralizeRewrite classes then instead of just 1, one for each specific rewrite url gen
- Instead of using "class CoralizeUrlGenerator extends RewriteUrlGenerator ", do "class CoralizeUrlGenerator extends GalleryUrlGenerator ", register your url generator as top priority implementation in all cases
and in function init of your url gen, do list ($ret, $urlGeneratorList) = GalleryCoreApi::getAllFactoryImplementationIds('GalleryUrlGenerator'); $urlGeneratorList is an array ('CoralizeUrlGenerator' => 'coralizeurlgenerator', 'ModRewriteUrlgenerator', ...) then do list ($ret, $this->_rewriteUrlGenerator) = GalleryCoreApi::newFactoryInstanceById('GalleryUrlGenerator', $id); and then call $this->_rewriteUrlGenerator->init(....args);
in generateUrl you would also first call $this->_rewriteUrlGenerator->generateUrl and then do something with the result
that's just an idea...maybe there are better alternatives.
Posts: 31
I just uploaded version 0.6.3 to http://Sandy.McArthur.org/gallery/modules/
This version fixes a silly bug introduced in 0.6.2 that breaks the permission check for non-public images such that non-public images won't load at all. That sounds worse that it is but an upgrade is encouraged if you are using 0.6.2.
Posts: 31
@valliant: I would like to insert a template into every page view and I'm having trouble finding the right hook. I don't quite want a block as I don't have anything to display. I just want to add some javascript to do detection if the user can access the coral servers.
Posts: 32509
g2 doesn't yet have such a hook yet.
what you'll have to do is to add a block to your module (templates/blocks/ + Callbacks.inc) and then you can use
there you can call $template->javascript('lib/javascript/BlockToggle.js'); (there are other methods too, see modules/core/classes/GalleryTemplateAdapter.class
Posts: 31
I just uploaded version 0.7.0 to http://Sandy.McArthur.org/gallery/modules/
This version introduces a feature to detect if the user's browser can access the Coral servers. Once enabled and configured this feature requests some JavaScript through coral before trying to coralize images. If that JavaScript is successfully requested, then a cookie is set indicating that a user can connect to Coral.
The Coral servers may be unavailable for a number of reasons including:
When Coral is unavailable for a user all the images should load correctly. The browser's throbber will spin for a while as the browser tries to load the javascript. It's annoying but this should not prevent the page from fully loading otherwise.
Posts: 174
Not working for me. I'll leave it on for awhile so you can chekc it out. Both IE and firefox dont load images
http://www.robplatt.com/gallery
Posts: 174
Nevermind. It wasnt working at first. Then it started to work after a few minutes. I'm running it on IIS6 so I dont know of a fall back like you suggested for apache.
Guess I'll just leave it on for now and see what people say..
By the way... what was the timeout for the coral cache? say i view an image, how long will it remain in their cache?
Posts: 31
I will refer you to the CoralCDN website: http://wiki.coralcdn.org/wiki.php/Main/FAQ#expiry
Posts: 174
So the way I read that, unless my data is being accessed more than every 5 minutes. It wont remain on their cache, and can assume it'll be faster to leave coral turned off....?
Posts: 31
No, they say they follow the spec with the exception of not respecting cache expires of less than 5 minutes. This means they will use a cached version for at least 5 minutes before considering requesting an image again from your server. Other than the 5 minute minimum, it will respect the cache control headers. On my server Gallery says images should be cached until the year 2038. (Ignoring that this violates the HTTP spec on the max time in the future an expires header should be) This means that the images will effectively never be expired from a cache because they are "stale". Coral may choose to discard a cached copy for other reasons but not that one.
Using Coral does increase page load time. Only use it if you are hurting for bandwidth. In my galleries, most all bandwidth is used the few days after party pics are posted. If you want, just turn Coralize on when you may need it. If you want to add a feature to detect when the hits per minute crosses a boundary and auto enable Coralize that would rock too.
Posts: 31
I just uploaded version 0.7.1 to http://Sandy.McArthur.org/gallery/modules/
Another brown bag release. I can never get a new release right the first try. :-/ This fixes a bug in the auto-detection code so that it actually tries to detect that a Coral server is available instead of pretending it did. Otherwise it is the same as the previous release.
Posts: 389
@valiant and BRANCH_API_DEV:
There's two new features in rewrite 1.1.0 that might intrest you. First off, a module may now register a function to parse keywords and secondly another function to be run at urlGenerator::init time.
http://codex.gallery2.org/index.php/Gallery2:URL_Rewrite#Developer_Reference
See keywords => function and the onLoad settings.
Posts: 6
I'm using coralize with url rewrite + formatted urls.
I notice my urls obtained via formatted urls do not have the coralize :8090 thingy.
example: this shows links to
href= http://chrisloup.blogdns.com/gallery2/v/public/xmas+2005+stuff/roybdayparty+_18_000474.jpg.html
img = http://chrisloup.blogdns.com/gallery2/d/15505-2/roybdayparty+_18_000474.jpg
<a href="http://chrisloup.blogdns.com/gallery2/v/public/xmas+2005+stuff/roybdayparty+_18_000474.jpg.html"><img src="http://chrisloup.blogdns.com/gallery2/d/15505-2/roybdayparty+_18_000474.jpg"></a>
Posts: 31
This is probably because Coralize doesn't coralize anything when viewing non-normal-image-viewing-pages. It used to break some of the admin features when I had it coralize everything. I'll add "making it work with Formatted URLs" to the TODO list but no promises since I don't use that module myself.
Posts: 6
ok: but in the interim,
http://chrisloup.blogdns.com.nyud.net:8090/gallery2/d/15505-2/roybdayparty+_18_000474.jpg
also doesn't seem to work
(it will change the address to http://chrisloup.blogdns.com.80.nyud.net:8090/gallery2/d/15505-2/roybdayparty+_18_000474.jpg)
Forbidden
You don't have permission to access /gallery2/main.php on this server.
---if I turn off url rewrite, the url given is
http://chrisloup.blogdns.com/gallery2/main.php?g2_view=core.DownloadItem&g2_itemId=15561
which gives the same error... I think this has nothing to do with coral per se... (maybe its a friendly referer thing??)
anyway, its just a fyi
Posts: 5
i just installed that module, and working fine, just a little bit slow on the first pageload.
nice job SandyMac!
much thanks
dy
test: http://kiccsaj.hu
Posts: 5
i have to override myself. at the moment coral cdn does not working (it means 30 seconds to catch 1 piece jpg file). but coral check does not recognize this problem. is there any way to ignore coral cdn if slower then a snail?
dy
Posts: 31
The Coralize module is about reducing your bandwidth. It doesn't make any claims of improving the performance of your site. There are about 260 different Coral CDN servers. They make no claims of quality of service.
If you come up with some clever way to detect that Coral is too slow to meet your wants, send it to me and I'll probably add it to the Coralize module.
Posts: 132
How about instead of just Coral CDN server... make it possible to use any form of Mirror server with a standard mirroring method? (ie: FTP upload)
I had made mirroring with this script http://www.hotten.dds.nl/gallery/index.html with my Gallery 1.51. Now that I updated to Gallery 2.0.2 nightly. I certainly miss it. Not only it have the potiental to save bandwidth and money, it also serves much much faster to the user if the files are downloaded from many server at once.
Can this Coral CDN server module possilby be extended to include other type of server?
Thank you.
Posts: 31
No.
Coral CDN will not let you to prefill mirrors with the images. This makes the module simple. Images are cached as the clients request them. Handling the work of duplicating images on a set of mirrors is definitively desired by some people but not me. Someone else would have to create that module.