notification module (with scheduler discussion)
|
cainlevy
Joined: 2005-09-21
Posts: 60 |
Posted: Thu, 2005-10-20 07:16
|
|
I've decided to learn how to create G2 modules by creating a Notifications module. My idea is that it'll listen for various events, look for users that want notification, log the events, then periodically check the log and send an group of notifications by email. I'm just getting started. So far it looks like I'll need to insert some events into other modules, like for adding a comment and registering a new user. Then I'll need to figure out how to use database storage ... how to send emails, etc. That's all fine, I can look around for examples of how to do those things. But what about periodic checks for whether emails need to be sent? Suppose, for instance, that I set it to send notifications every 30 minutes, tops. How can I make that happen? I basically need cron ... |
|
Posts: 32509
there's no scheduler available in G2 yet.
i've recently thought about how it could be integrated since the use of such a scheduler would be great, e.g. also for scheduled content publishing etc.
cron would be the most reliable and probably the easiest solution. but an external dependency on cron should be the last resort, if everything else fails. so let's think of solutions in g2.
- a G2 PHP shutdown handler that uses set_time_limit() to live forever
This won't work since apache or other resource management tools will stop it after some time
- code that checks on each request if there's something scheduled for the next few seconds or something, that should already have been executed
This could be very lightweight, since we can cache the schedule on disk, so no extra db queries would be necessary.
It would work for scheduled content publishing, since as long as noone requests from G2, it doesn't matter if it's not already published. E.g. say we scheduled publishing of album X for 12.00h. But from 10h until 14.00h there were no HTTP requests to G2, noone even looked at our Gallery. On 14.00h there's a request, we publish immediately in the background and the user will see album X. So noone cares if album X hasn't been published exactly at 12.00h as long as it's published if someone looks at G2 at 12.00h or after 12.00h.
But this won't work as well for scheduled emails. Since there's no external signal that G2 could listen on. The probability is small, but if there's no HTTP request to G2 for a day, noone will get an email notification for a day, even if you have scheduled to send them every 30 minutes, if necessary.
- a poor solution, but what a few scripts do: ask you (the admin) to keep a browser window open on your own desktop. the displayed page is a self-refreshing request to G2 which will make sure that G2 receives a request every x seconds and then G2 can check the schedule etc.
This would work, but then I'd even prefer an external dependency on cron (this could be a fallback solution if cron is unavailable).
The email notification problem could be solved differently:
On a change event, send an email to user X immediately. Unless you've already sent an email to user X before and user X hasn't visited your G2 yet since the last notification.
It's not the same as what you suggested, but it would work.
If you have other ideas how a scheduler could be implemented, let us know. But the simple fact that G2 isn't a daemon / relies on external requests to run is quite a limiting factor.
Posts: 60
Hmm.
I have had some success creating PHP daemons ... mine is a queue that uses sockets to listen for requests. The only time it ever really shuts down is when Apache shuts down. I think something similar could work for G2, the question is what tools are available. Can we assume, for instance, that there's a CLI? Can we assume socket support?
What I'm imagining is a socket server that dynamically sets socket timeouts based on how long it needs to sleep before the next event. Then, if anything wants to schedule a new event, it would log that event in the database, send a message to the daemon that says "check again", which would interrupt the daemon's sleep cycle, at which point the daemon could re-calculate the appropriate timeout/sleep.
It's all very low CPU. After days of life, it might use 5 seconds of CPU time. But it really does require the ability to spawn PHP processes. For instance, not only would you have to spawn the daemon, but the daemon itself would want to spawn a handler for each event so that if some part of the schedule event crashes the daemon continues to live.
The daemon itself could be monitored and started through page loads. Granted, page loads are unreliable, but they should be enough to once in a while check up on the daemon and make sure it's running. And if it's not, well, they spawn it. There could even be an admin page where you can send control messages to the daemon, telling it to shutdown, start up, re-check the schedule, etc.
Thoughts?
Posts: 32509
- well, we need a platform independent solution.
- you can assume we have CLI access through php functions like system, exec, ...
- we have a platform abstraction in G2 (GalleryPlatform, UnixPlatform extends GalleryPlatform, WinNtPlatform extends GalleryPlatform).
so if you have a solution that works only on unix/linux, we have to find an alternative for windows.
i don't think we can have a PHP / any process running 24x7 on the webserver. we have to code for shared webhosting solutions.
at least some of them have monitor programs / limitations that processes can't run longer than x seconds. or at least i'd think they have something like that.
but without such a background daemon / process, it's seems to be infeasible.
what we should do:
explore the design space of alternative solutions. and for each approach, write down the external dependencies / assumptions / requirements.
your approach doesn't sound bad, but would it run on a shared webhosting plan? maybe we can use it and have to find a less reliable alternative for cheap shared webhosting plans.
Posts: 60
Ok, so we've got:
Third-Party Daemon: rely on cron. Could try to find and support various alternatives for different platforms/installs. Assumes that shared webhosting allows cron jobs.
G2 Daemon: run a custom PHP daemon. Current daemon idea requires sockets, and any daemon requires the absence of anything trying to kill long-running (time-wise, not cpu-wise) processes.
User Daemon: make people do the work. Check schedule on page loads, have optional admin page that self-refreshes to ensure frequent page loads. Not very reliable, schedule-wise, but very cross-platform.
I really have no experience with shared webhosting. Do they monitor and kill idle but long-running processes? Do they allow cron jobs? Can anyone with experience confirm either of these?
Posts: 60
Incidentally, your proposal for how to handle notifications (immediate notification unless user hasn't visited since last email) misses one of my design goals - I want the user to know exactly what has changed since their last visit. Well, what has changed within the scope of what they care to know about. That's why I was hoping to periodically send an email listing all relevant changes within the last 30 minutes, if such changes exist.
Another design goal is to provide an alternative to RSS feeds. RSS feeds are great and all, I love 'em, but there's no way I can rely on users having RSS readers, whereas I can definitely rely on them having email clients. So I believe that an email notification module is more urgent and central than any RSS module. Which is why I'm starting work on one.
Posts: 32509
you still can present a changes since last visit page for each user... no need to send this list in an email.
Posts: 60
Oh, right. So I could send an email saying "there be new stuff0rz" and just link them to a "since your last visit ..." page. That's probably what I'll do, since a scheduler is unlikely to appear in the near future.
Posts: 32509
yep, a scheduler is a lot more difficult than the suggested solution.
but still, at some point we need it
Posts: 27300
My previous host did not provide cron jobs. My new host does. But there is 100s or even thousands of instals at my old host.
Dave
____________________________________________________
Gallery Frames / Mods || G1 Test Gallery
Posts: 3236
It could be something as simple as having a "cron.php" file (or maybe a main.php thingie) that expects to be loaded every 30 minutes or something. This way, the cron job can be setup *anywhere*. You can use a little windows tool or script to load that page every 30 minutes, you can use a linux/unix cron deamon on the server... or on another server... or anywhere in the world. You could combine that with a schedule check at every page load to get even more down and dirty.
___________________________
Like Gallery? Appreciate my help? Think I'm an asshole? Make your point by donating to the Gallery project! Or maybe just visit my website.
Posts: 8601
what fryfrog describes is how drupal works.. cron.php
Whenever we do a scheduler, I think there should be a cron.php and that should be the recommended method. Perhaps a site admin control could select alternatives, like poor man's solution of checking on every http request.
cainlevy, feel free to suggest additional events you'd like to see so you can record notifications.
How will you configure notifications? On/off switches for different events? Select different groups of events for different users/groups? Allow users to select their own notifications? How will you know what events are available? Hardcode a list of known events? Publish an interface and let other modules announce they have an event users can be notified about? Lots of choices in how notifications can work... feel free to start small, of course, but give a little thought to how it might grow..
Thanks for stepping up!
Posts: 60
Some events I'd like to see:
* A user just finished registration. (happens after email confirmation, if appropriate)
* A comment was just added.
* An item (album/pic/etc) was just added.
Those're the only ones I can think of, and they seem like a good starting point.
For configuration, I figure there'll eventually be a Site Admin page where administrators can determine which notifications are available to end users and how often notifications would get sent. Then, for logged in users, there'd be an ItemLink for adding an album/pic/item to a notifications (subscriptions) list. And hopefully there's a way to provide the users with a tab on their "Your Account" pages where they can see and manage their subscriptions. If not, I might have to be satisfied with a dedicated page.
So yeah, a lot like Drupal.
Even though I've never really used Drupal, that's at least the system I've noticed in use on this site.
Not sure how to have a user opt-in or opt-out of notifications when leaving a comment. Main problem is that having a post-time setting would require a tie-in with the comments module.
Haven't given much thought to how I'd know what events were available for notifications. Figured I'd probably hard-code the available events for now. Eventually a method like you described where other modules can register notifiable events would be preferable. I'll give it more thought when my brain isn't complaining because my body is still vertical.
Posts: 32509
* user registration -> no need for a new event.
you can call it "user registration" in your module options, but technically it's a GalleryEntity::save event with persistent storage flag NELY_CREATED for GalleryUser (register without confirmation) or GalleryPendingUser (register with confirmation needed) entity
* comment added -> again, no new event needed.
listen for GalleryEntity::save events with NEWLY_CREATED flag set for entity type GalleryComment
* item / album was added
listen for GalleryEntity::save events with NEWLY_CREATED flag set for entity type GalleryAlbumItem and GalleryDataItem (photos, movies, unknown, ...)
ah, you not only want global "notify me on ... changes" option, you'd like the option to be able to just to be notified for updates on comments of item x?
Posts: 60
Cool. GalleryEntity::save is much more powerful than I was expecting.
And yes, I think notification for updates on item X is important. I'm essentially imagining four common notification scenarios:
Should I be making a new thread for discussion on the notifications module? This one's theoretically about scheduling
Posts: 32509
this topic already contains a lot information about the notification module. and this poor forum software doesn;t allow splitting topics. so please continue with this topic.
you may want to change the topic title correspondingly...
and if you choose to begin a new topic, add a link to this topic in your first post there, just to link the two.
@subscribing for notifications for specific items:
ya, it's definitely a good thing. you'll need a class / db table to map userId and itemId, maybe add other table columns to save options for each subscription.
- the admin can then enable / disable offering notifications for certain categories in site admin.
- users can change their notification defaults somehwere (best would be in the user profile i guess, or maybe add another system link which shows right next to "your profile", e.g. "your subscripitions").
- and users could change their subscriptions for each item/album (override for the defaults) e.g. in a view which is accessed through a item link (see module.inc) and there they could change the settings in a dedicated view
just brainstorming.
Posts: 60
Another option: only email me for these types of events ... for the rest I'll log in and check my "what's changed" page.
Any email would link to that same "what's changed" page, and anytime a user viewed that page, those items would be deleted from the notifications table. This probably means that if the page links to the changed items, it should link via new windows.
So if notifications persist until the page is viewed, then the notifications table will need a flag for whether an email has been sent (if email should be sent).
So we're looking at two tables here?
One to keep track of subscriptions, with roughly these fields:
userId
itemId (may be null)
onEvent (this is the meat ... itemId would only matter for certain events)
sendEmail
And another to keep track of notifications:
userId
itemId (may be null)
event (same content as subscriptions.onEvent)
emailSent
Feel free to propose different names that better fit the G2 database paradigm.
Posts: 7
Hello,
I wrote a custom mod for G1 for notifications and it worked pretty well for me. I have been wanting to move my G1 to G2 but definately want notifications working in G2 before I do this. So I began writing my own G2 notification module a while back. Your ideas are much more in depth then what I had working in G1, but basically the same idea.
My first goals:
1) Add a simple global notification on/off to the Album properties page. COMPLETED
2) Add a simple notification on/off to the user profile properties. STARTED
3) Send test email whenever a new album or photo is added. COMPLETE
4) Send email to users with notification flag on and album globabl notifications are on when new photo is added. STARTED
Perhaps we can join forces and work together on this module.
Posts: 60
Ah, excellent. Yeah, let's see what we've got put together. I'll start by describing what my final product is going to be, then point out what isn't working yet.
goal
Four notification scenarios:
New User Registration - a setting on the Site Admin page enables/disables this notification. Email is sent to all Site Admins.
New Gallery Item - a registered user can add an album to their subscriptions. When a GalleryItem they are permitted to view is added beneath that album, a notification event is logged. Setting on Site Admin page enables/disables this type of notification.
New Comment on Item - a registered user can add a GalleryItem to their subscriptions. When a GalleryComment is saved for that GalleryItem, a notification event is logged. Setting on Site Admin page enables/disable this type of notification.
New Comment on Owner's Item - When a GalleryComment is saved for a GalleryItem, the owner of that GalleryItem gets notified. Setting on Site Admin page enables/disables this globally, and setting on user profile page enables/disables this per-user.
So, some notifications require subscriptions, others do not.
Then, users get emailed regarding their recent notifications (for now, it'll just be "there's something new, check this page here"). The page they link to will display their recent notifications (pulled from table) and let them cancel any subscriptions or change any settings (like the "notify owner on new comment" setting). When a user clicks on a notification, that notification is deleted in the table and the user is directed to the appropriate item.
status
site admin page exists and is basically done.
subscriptions are being created and removed for gallery items.
notifications are being logged for new user registration.
remaining
need email class/template/whatever
need to log notifications for rest of events (this is my next step, I expect it to go quickly)
need to email users selectively (without a scheduler my best idea is to add a "notified" flag for each user. when they visit the notifications page, set it to false. when a notification is logged, and flag is false, send email. when an email is sent, set it to true.)
need user profile settings
need a view for the subscriptions/notifications page.
need to develop a version that only requires CoreApi 2.0 ... mine requires 2.2 so far, which is only in CVS. only difference is in sql files.
It sounds like you already know how to handle the email and the user profile stuff. That would be a major help, since that's the major portion of what I haven't researched/implemented yet.
Here's my module so far:
http://www.cainlevy.net/gcvs/modules/notifications.zip
edit by valiant: changed bbcode (made link work)
Posts: 60
I fixed a couple bugs and added logging of notifications for new items and new comments. Still haven't handled logging new comments for owners of the item ... that's next.
The linked file above is (and will remain) my latest version.
Posts: 3
Cainlevy~
So I am a little new to Gallery but want use your feature. I d/l'd the zip file but it doesn't have any READMEs or INSTALL instructions unless I am clueless and just missed it.
Can you help?
Thanks,
Bart :-ß
Posts: 60
Oh, sorry 'bout that. It's not actually ready for general use yet ... that's a development version. So far it doesn't email anyone or show recent notifications ... which is kind of crucial.
Incidentally, you need to unzip it into modules/ directory. Every module is just a sub-directory there, which makes them pretty, well, modular.
Posts: 7
Cainlevy,
I have downloaded your module and got a new development environment setup for it. I was wondering if you have manually extending the database schema at this point because I get errors when I try to turn on a notification and click save. If so can you supply the DB information?
Once I get this information I will see if I can get the user profile stuff and email working.
Also my email address is
if you want to contact me outside of this forum.
Posts: 8601
cainlevy, looks like you've no shortage of testers
Posts: 3
DOH!!!! Should have realized I was in the "Development" forum. Sorry 'bout that! I will however, keep checking back to see how it is going. I look forward to using your module.
Thanks for the help!
Posts: 9
And how about email notification of zipDownload?
I'd need to have admin notified whenever anybody downloads zipCart (and what's in it).
Any simple ideas? Thanks!
Posts: 32509
you can file a feature request for that. that would be a zipcart feature request, unrelated to this module, since there's no download zip event in g2.
or ask in the customization forum on how to hack your g2 to do that.
Posts: 81
cainlevy,
You still working on this emailing function when new comments are added?
Just checkin, I'd die for this module/feature.
thanks
brett
Posts: 60
Well, I've been distracted for a while and haven't put much effort into this module. And now that I want to finish it up I can't see how, because of a problem I'd thought solved: when is it the right time to notify users? With comments that's pretty easy: whenever a comment gets posted. But for albums and pictures, which is the bulk of work in this module, it's much trickier.
Consider two users with different workflow patterns. User Joe tends to create a new album, then upload a bunch of pictures into that album with proper names, maybe delete a few pictures, etc. User Jill tends to create a new album, upload a bunch of pictures, then go through and caption them. With Joe, people should be notified that there are new pictures just as he saves them. But with Jill, they shouldn't be notified until after she captions them all.
Or consider a situation where someone makes a mistake: Joe creates a new album, uploads a bunch of pictures, discovers that they're all duplicates, and erases everything. This particular example has some possible solutions (not complete solutions, but pretty good), but it helps to illustrate the overall problem: at what point in the album or picture's life is it complete enough to actively notify users?
Now, the need to know when something is complete only matters for active notification, because it's very possible to frivolously notify users at multiple steps in the contributor's workflow. [note: I consider RSS to be active notification] And if users start to feel that notifications aren't important, that sometimes they're not worth checking, then users will start to ignore them. And then the module (at least, the module that I want to build) has failed.
It's almost as though, instead of automatically notifying people that there's a new picture or album, contributors need to manually trigger notification. A sort of "ok, I've got everything I want up, so send an email to everyone who's waiting for more of my pictures". And this sort of solution only has a very slight advantage to the contributor just sending an email directly, because G2 would do the work of maintaining the mailing list.
The other solution (or solution area) is passive notification, where a user requests an update. For example, the user visits a web page that lists the sum total of changes since his last visit.
Maybe if someone could help me see how this should work, I'll be able to finish it. Until then...
Posts: 32509
yep, i see that some album creators don't want to notify the users of the new album until it's ready / finished.
e.g.
- they change the permissions of the album to non-public until it's finished, then they change the permissions to public (i'd do it that way on a high-traffic g2)
- or they just edit / add items and then want to send the notification (no hiding during construction phase needed, low traffic site = it's not very likely that someone visits the page during the album construction phase)
you could add an option that is shown in the add album / add item page which says: "notify now" vs "notify in 1 hour" vs "don't notify".
and if one chooses "don't notify", there should be an option in "edit album / item" that shows "notify now".
and your notifications should be sent as a digest or for each album / item (mailing lists have similar features ).
Posts: 35
I for one would be one of those people that would like a manual notification option. It would be great if a user could subscribe to one parent album and have all the sub-albums inherit the subscription unless blocked. I usually upload about 200 photos per update (about once every 2 months) but when I do I move them, add titles and such. I don't want ppl to get notified until I'm done. I would only want them to receive one email about the all the changes since the last time I ran the "notification".
What I would REALLY love is some kind of virtual album that contains only updates. I know they have a most recent on 1.5 but I don't see that on 2.x. Even on 1.5 thought you couldn't do a slide show of those pictures and it was based on time not user. I don't know if it is possible to have a album that links to a picture in another with out having two copies. If so I could see it being able to do a dynamic album that utilizes the permissions module.
Right now when you load an album it checks the permissions on all pictures (per that page) and will only display if you have permissions to see them.
If you could have one album that is virtually linked to every picture on the site and then remove the view permissions as they are viewed from the user logged in (within THAT virtual album only) you would get basically a view once permission. If you want to take it one step farther you could have a virtual album for each album that re-curses all sub-albums. That way if you want you could link to updates of a specific album and it's sub-albums rather than the whole site.
So to recap my wish-list is as follows:
*Virtual Albums for each album that link to the original album and all sub albums with a view once only permission. (hmm here's a thought use the virtual albums and have a "view updates only" mode for users.. that way the only thing they would see is the updates)
*Manual notification of updates when users are subscribed to update notification. (this could also utilized the read once permission. ex. if a user doesn't have any permissions on the virtual albums OR isn't subscribed to the album the user doesn't get notified and vice versa the user gets notified and then linked to the virtual album containing the updates)
*Inherited subscription of sub-albums with a disable feature
That's my two cents. I know my idea really isn't part of the scope you currently have laid out. I just feel that this would not only give you and easy way to update users but also make it easier for the user to see all the updates at one time. I don't think you would need a cron job to do it my way either. I don't know about you but I don't want 15 emails (or one email with 15 links) for each of the updated albums. I want to hop on a site and click a button and have it show me all the updates I haven't seen before. The trickiest part will the the virtual albums. I don't know if it can even be done but man would it would be useful, not to mention using it for other applications.
Stephen
Posts: 60
Good ideas. I'd actually like to see a lot of that incorporated into a more advanced Advanced Search. Imagine being able to say "only search items (pics/movies/comments) from this (or these) users" and "posted between these dates" and "only items I haven't seen yet".
With functionality like that, I wouldn't even care about this notification module anymore. Not as much need to actively email someone if they can just come and check what new pics a contributor has put up.
The latest incarnation of the email I've been imagining, Stephen, is actually just something like "there's new stuff, come check it out, there's a complete list on <this page>". Then that linked page would show them every notification that had accumulated since they left.
Posts: 35
That would be awesome. If you could search like that you would be able to send out email notifications with the correct search already populated for them. After I posted on this thread I created a new thread http://gallery.menalto.com/node/42801 which is more of an expansion on my idea for the vitural albums. I hope some replies with either a possible way to do what I'm thinking or a better way.
Posts: 1
I think this brings up a good point. Rather Than notifications, you might want to think about this as a subscription. Similar to a forum subscription feature. Members subscribe to an album as they would a forum in which they have at least two choices: immediate email notifications; or delayed and then compiled notifications (at daily, weekly intervals). Considering the medium, daily would probably be the best interval if you coded for only one.
I just recently began using Gallery and this subscription feature is something I thought for sure would be a core feature in Gallery2, but was a little disturbed to find that it wasn't. If you think about it, it could greatly increase the amount of activity (and exposure) a gallery project gets. Without something bringing you back to a gallery, people could easily forget about them. I would definitely put this Subscription feature at the top of the todo list, it is very important!
Posts: 60
Hmm, well, I was already treating album and item notification on a subscription basis, but if I understand you correctly anastas you're suggesting that the whole question of "when" should be decided by the subscriber? How would that let an album creator decide when an album was ready for viewing? The difference between forums and galleries, or at least the one that creates the most problems here, is knowing when content is ready for viewing. With forums it's when the creator clicks "submit" (an active action) but with galleries it may (or may not) be when the creator stops working (a passive action).
Posts: 20
Hi,
I would like to test this module, but the link provided above doesn't exist...do you have a new link?
thank you,
tom
Posts: 132
I would love to test an e-mailer and subscribion program as well. I have been looking for a replacement for the e-mail feature avilable in Gallery 1.x.
e-mail list is never going to be the best solution for everything, but for updating user when the album is update, it is still one of the best idea. Cron feature is starting to get popular even for shared hosting these day. and if come to worst, they can always schedule a spider crawler from search engine or statistic engine to runs the cron.php file. I have not seem anyone else who had a better solution beside perhaps using a perl script as daemon, but even that, have limitation.
Can someone post the file? The one in the original topic is not working anymore
Posts: 60
Sorry, I probably wasn't very clear: when I abandoned the module it wasn't even in a testable state.
I could probably dig up the code I ended up with, but it's both outdated and incomplete. My guess is that if you (or someone) wanted to proceed with the module, you'd be better off doing it from scratch. But if you're interested in taking over development, let me know and I can dig through my code and notes and pass it on.
Posts: 24
I really REALLY wish someone could get this up and running. I'm no developer and if I was, I'd be working on this now. This even used to be a module you could vote on but has mysteriously disappeared of the poll. If anyone else as interested in seeing this work as I am?
Posts: 60
Me!
The biggest problems I ran into trying to create this module were all theoretical "how it should work" questions. It would greatly help the development of this module if someone, anyone, could simply clarify the solution. Don't really need to be a techie for that.
Posts: 27300
it was a suggestion in our Google SOC ideas:
http://codex.gallery2.org/index.php/Gallery:Summer_of_Code#Notification_Module
and is being worked on as we speek by Zimzat with mindless as the mentor:
http://code.google.com/soc/gallery/appinfo.html?csaid=39E6FA0BFC5C785F
with some code alread done:
http://gallery-contrib.svn.sourceforge.net/viewvc/gallery-contrib/trunk/gallery2/modules/notification/
So the answer is: please standby
Dave
Posts: 24
/bookmarks thread
Thanks for the updated info Dave!
Posts: 24
I haven't seen any updates in quite a while. Was this dropped? (I hope not!)
Posts: 32509
there's a notification module in the community repository. you can use it with gallery 2.2.
there's no scheduler module yet, probably we'll take care of it in g2.3.
Posts: 17
I love this module but was wondering if there is a plan to incorporate localization support for non-English sites?
Posts: 32509
this module is sure not localized yet since it's not an official module yet. but i guess the code is internationalized (ready for localization).
i guess we'll move this module to an official status in g2.3. that's when you can expect localizations for it.
--------------
Enter the Gallery 2 Theme Contest today!