Transcoding all uploaded videos to FLV
|
danneh3826
Joined: 2007-08-18
Posts: 290 |
Posted: Thu, 2010-09-23 18:32
|
|
Hi, Continued in this thread: |
|
|
danneh3826
Joined: 2007-08-18
Posts: 290 |
Posted: Thu, 2010-09-23 18:32
|
|
Hi, Continued in this thread: |
|
Posts: 290
in anticipation that such module doesn't exist yet for G3, i've started to write one. it'll behave somewhat like a youtube clone, taking your video and converting it to one (or a selection of) different flv resolution/formats (240p, 360p, etc..).
unless anyone else already has one either in progress, developed, or otherwise, i'll post it here shortly if others are interested
Dan
Posts: 27300
Please do!
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 25
this is EXACTLY what I am looking forward to.
Can;t wait
Dave
Posts: 290
lol i actually forgot a lil bit about this. been working on a couple of other projects recently (paid work > free work).
anyhoo, i have actually gotten this working. first version's here:
http://www.danneh.org/files/transcode.tar.gz
drop it in the modules directory, extract, and enable it in the modules section. then go to the 'video transcode' settings page, select the quality(qualities) you want to transcode to, and save changes. also check the path to ffmpeg is correct. you might find that, when you upload videos, the tasks appear in the maintenance page as 'stalled'. this most likely means ffmpeg died either transcoding, or doesn't support some of the parameters. the command that's run is spit out into the g3_logs table. pull it out of the data and run it manually on the command line (if you have access to it) and see what it says.
it currently requires/is hard coded to use the aac codec for audio and flv for video (seems to be the most common combination that doesn't contain the mp3 codec).
i envisage there being issues, being the first version and all. i also can't work out how to get the auto-updater/version detection working yet, so any new versions you'll have to check for manually either here or my website (http://www.danneh.org). i'll eventually get round to writing a blog post for it and a category for subscription.
in the meantime, if someone from the gallery team (or anyone else who's written a g3 module) can tell me how to get the auto-updater going, that'll most likely be the next thing i implement.
Dan
Posts: 290
blog post here: http://www.danneh.org/2010/10/youtube-esq-module-for-menalto-gallery-v3/
updates here: http://www.danneh.org/category/gallery3/
or here (rss): http://www.danneh.org/category/gallery3/feed/
Dan
Posts: 23
hi, i have tried to install this module and got "The transcode module requires the php5-ffmpeg extension." not sure what that means, but i do have ffmpeg installed on the server. any help appreciated
Posts: 7994
diverselab: if you're on debian or ubuntu, try doing "aptitude install php5-ffmpeg" then restart Apache2.
---
Problems? Check gallery3/var/logs
bugs/feature req's | upgrade to the latest code | use git
Posts: 23
thanks for your reply, i'm using XAMPP installed on a local server (mac os x), not sure how i would go about doing that. sorry don't really know a lot about this stuff.
Posts: 337
I cannot get it working because Gallery 3 uploader doesn't allow to select unsupported formats in the uploading window. So any video format except flv and mp4 is unclickable (grayed out).
P.S I've tried to edit the file /modules/gallery/views/form_uploadify.html.php replacing:
fileExt: "*.gif;*.jpg;*.jpeg;*.png;*.GIF;*.JPG;*.JPEG;*.PNG<? if ($movies_allowed): ?>;*.flv;*.mp4;*.m4v;*.m4v;*.FLV;*.MP4;*.M4V<? endif ?>",with:
fileExt: "*.gif;*.jpg;*.jpeg;*.png;*.GIF;*.JPG;*.JPEG;*.PNG<? if ($movies_allowed): ?>;*.*<? endif ?>",Now I can select any format. However, after the uploading completes I get Server Error 400. Any help?
Thanks!
Posts: 337
Hmm, the content of all log files I found in /var/modules/transcode/log is:
P.S Okay, I am getting in. Actually the one in the log here seems to be a different issue. It happened with the mp4 file I tried to upload after giving up with the server error problem which happened with the avi file I was trying to get it converted. It seems there is a problem with the spaces in the file names. The mentioned file in the log above has the name 'Bla Bla', but the module ignores anything after the space.
I get a different error log now, after renaming the file to 'Bla.mp4':
FFmpeg version SVN-r14473, Copyright (c) 2000-2008 Fabrice Bellard, et al. configuration: --prefix=/usr/local/cpffmpeg --enable-shared --enable-nonfree --enable-gpl --enable-pthreads --enable-liba52 --enable-libamr-nb --enable-libamr-wb --enable-libfaac --enable-libfaad --enable-libmp3lame --enable-libtheora --enable-libvorbis --enable-libx264 --enable-libxvid --extra-cflags=-I/usr/local/cpffmpeg/include/ --extra-ldflags=-L/usr/local/cpffmpeg/lib --enable-cross-compile libavutil version: 49.7.0 libavcodec version: 51.62.0 libavformat version: 52.18.0 libavdevice version: 52.0.0 built on Apr 8 2010 19:28:07, gcc: 4.1.2 20080704 (Red Hat 4.1.2-46) Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/home/****/public_html/gallery3/var/albums/Bla.MP4': Duration: 00:01:01.02, start: 0.000000, bitrate: 134 kb/s Stream #0.0(und): Video: h264, yuv420p, 320x240, 24.00 tb(r) Stream #0.1(und): Audio: libfaad, 32000 Hz, stereo Unknown encoder 'aac'Posts: 290
yep, as i thought - your ffmpeg installation doesn't have support for the aac codec. now, if you have root access, you can re-compile and re-install ffmpeg with aac enabled. if not, you can run your own ffmpeg version in your home directory, which is actually what i did.
(this is also assuming you have ssh access to the server)
note: the 'make' process takes quite a while, so go make a coffee or something while this happens. once it's complete, go into the transcode module's settings and change the path for ffmpeg from what it is now to the location you just compiled your own ffmpeg in, (i.e; instead of the default /usr/bin/ffmpeg (for example), if you compiled in /home/dan/ffmpeg0.6, the path would be /home/dan/ffmpeg-0.6/ffmpeg).
this should at least get the module working. i'll fix the issue with spaces in the filenames later on today and roll a new version.
Posts: 337
I have no root access to my server (shared hosting), but I do have SSH. However, the compile doesn't work for me. I got:
I tried to temporarily change the tmp directory to an accessable folder I've created under my home directory, but this doesn't work also. Now I get a different error:
Any thoughts...?
Thanks!
Posts: 290
hmm, i have vague nightmares about trying to do anything productive in a jailed shell. mostly because most everything you can do normally isn't possible.
i've tried to build a statically linked ffmpeg binary, download this, put it in your home directory, chmod 755 it and run ./ffmpeg and see if it runs. with a bit of luck, all the libraries that it needs are compiled into the executable, so it *should* just work.
ffmpeg here: http://www.danneh.org/ffmpeg.gz
(note: this was built on ubuntu 8.04 32-bit on gcc v4.3.2. i don't have any resources to compile a 64 bit binary, but this is worth a shot anyway).
Posts: 337
Thanks for your time, Dan.
Posts: 337
Hmm, I've just found a pre-compiled FFMpeg version here. I am going to give it a try. Will keep you posted.
Posts: 337
It seems it's also compiled without aac enabled:
Posts: 290
hmm, run
DEA because:
D = Decoding supported (need that)
E = Encoding supported (definately need that)
A = Audio codec
it's easier to filter the list of ones that will definately work, than trying to sift through an even longer list
look at the list that's spit out and pick something from the list. in order of preference, depending on what's compiled in, i'd choose
1. aac
2. vorbis
3. mp2
4. flac
then navigate to the gallery's modules/transcode/helpers directory, open transcode_event.php in your favorite text editor, and go to line 61:
$srcACodec = "aac";
change to:
$srcACodec = "<your choice>";
replace <your choice> with one of aac, vorbis, mp2 or flac (lower case, in these exact letters) that your ffmpeg binary supports.
save and re-upload a video and see if it goes through.
note to self: add detection routines for audio input, populate select box in module's configuration settings and allow choices of only audio codecs which are supported by chosen ffmpeg binary.
Posts: 130
Hi,
I've installed this module, but when trying to activate it I get the "The transcode module requires the php5-ffmpeg extension." I told my host -LunarPages- about this, and asked them to install the extension and I got this response:
FFMPEG installed. Its path is /usr/local/bin/ffmpeg
This is the path already setup in my Advanced settings in Gallery3. What should my next step be if I don't have have shell access?
Thanks,
-Marc
Posts: 290
php5-ffmpeg is different to ffmpeg. what they're telling you is the path to the actual ffmpeg binary/application. php5-ffmpeg is a php extension which provides php with easy access to a handful of ffmpeg routines for detecting and setting various things to do with videos. a wrapper, if you will. if makes available an class called ffmpeg_movie() which has various methods to achieve certain tasks without having to run ffmpeg on the command line/exec and parsing it's output.
given the responses i've received today about this module, i'm going to re-work a couple of things as it seems that ffmpeg doesn't come compiled with certain dependencies as standard.
this is my plan:
1. remove dependency for ffmpeg php extension. it was just easier at the time to utilise this class, mainly to get it working. now it seems that, from a compatibility point of view, it's easier to just parse out the command line response to get the basic info it needs about the source video.
2. package a pre-compiled ffmpeg application which is known to work correctly. this should hopefully aid those that either don't have ffmpeg installed, or don't have the relevant codecs compiled in.
3. allow a dropdown selection of compiled in (and flv compatible) audio codecs, removing it's static dependency on the aac codec and allowing you to select a preferred audio codec which is known to work.
this will, eventually, work out of the box. i've just never submitted anything open-source like this before, so bear with me
Dan
Posts: 25
I get the "The transcode module requires the php5-ffmpeg extension."
You say above that you will remove the dependency for ffmpeg php extension. Any idea when? (I appreciate the module and am not trying to ruch you in any way
Dave
Posts: 290
workin' on it
hope to have something to post in a couple of hours
Dan
Posts: 290
ok, version 2's here:
http://www.danneh.org/files/transcode-2.tar.gz
this one fixes:
- dropped dependency for php5-ffmpeg extension
- detect list of supported audio codecs, cross referencing them against supported flv audio codecs to provide a selection. unfortunately flv only supports aac, mp3 and adpcm_swf. i believe ffmpeg supports adpcm out of the box, but it doesn't support mp3 encoding normally unless you install a specific library for it, and the other option's aac.
- use ffprobe to detect video information to perform aspect scaling, fps and bitrate detection for smaller/larger files
hopefully this time things should go a little smoother
Dan
Posts: 25
I copies the transcode directory to the Modules directory on my server
When I try to add it in the admin panel, it does not load (i.e., I check the box but on reload of the page, it remains unchecked)
Dave
Posts: 290
blah, i'm such a clutz
i forgot to remove the dependency in the installer which checks for php5-ffmpeg. that, and there was a syntax error too. boo.
re-download it, i re-uploaded the tarball
Dan
Posts: 25
Ok. It installed. Now for my next dumb question.
How do I upload files? The upload option only allows pics and .flv files.
Does this mod work with the Video mod?
And last question...
the only setting that is filled is Path to ffmpeg binary: which is set to /usr/local/bin/ffmpeg
do the other fields have to be filed in. Any idea what they should be set to?
Posts: 337
I get the same error message:
./ffmpeg: /lib/libc.so.6: version `GLIBC_2.7' not found (required by ./ffmpeg)Well. I think, with the auto detection of supported audio codecs feature of version 2, there is no need to do that manually anymore.
I've upgraded the module to version 2 and re-configured it to use the FFMpeg that is installed in my server. However, the available audio codecs drop-down menu is empty. Is that mean that the FFMpeg version that is installed in my server have no support for anyone of these codecs?
Posts: 337
Hmm, the really strange thing is that ClipBucket script says that most of those codecs are really installed:
[img]http://grab.by/6M99[/img]
Any clue...?!
Posts: 130
My host, Lunarpages, doesn't support FFPROBE. is there anyway around this? what's the best host for a gallery install?
Posts: 290
if you're getting an empty list of audio codecs, if you can, via ssh run this:
that's the exec command the detections function runs, and cross-references it against supported flv audio codecs. if you can paste that here, i can maybe see why the list is empty.
shecter; that's really strange that they don't have ffprobe installed but do have ffmpeg - as ffprobe comes (and is built) as standard out of the ffmpeg box. ffprobe provides a better statistic output of the source video, whereas the same routine in ffmpeg doesn't provide the same level of detail. in fact, the only statistic missing from ffmpeg's statistical analysis is the fps value, which is shows on some videos but not others. ffprobe, in all my test cases, shows an fps value all the time. try running one of these on the shell (if you can) and see if it turns anything up:
(just incase the auto detection doesn't work correctly, which would be a little weird, as it runs the 'whereis' variation).
as far as hosts for gallery installs go, i'm only one for trying to sell my own hosting services, and supposing you'd like 100% compatibility with this module (and seemingly every other gallery3 module available so far), my hosting solution would be the best way to go </shameless_plug> lol. (since that's also the same server i developed this module on). but seriously, send me a PM if you're interested, and i'll match what you're paying now for your existing host, and maybe dump a 10% discount on there or something
anyone else having issues with ffprobe not being available?
Posts: 337
Do you think it's because of the jailed shell?
P.S With the pre-compiled version I mentioned earlier, I get:
Posts: 290
i see now. the ffmpeg you got there (precompiled) is quite a historic one. that's revision 9332, whereas my box is running 19441 and locally for development i'm running 25200. it seems that, since at least 19441, the -codecs option isn't available (which may explain why your auto detection doesn't exactly work). try, on your precompiled version;
i'll have to find out what version that was introduced in and change the command according to the ffmpeg version installed. the output from both commands differ, -codecs gives a better list of what's availble, -formats lists everything but doesn't list the codec as video and/or audio codec.
the jailed shell probably doesn't help matters, as if it wasn't jailed you'd be able to compile a more recent version in your home directory for your own use. it seems that the installed ffmpeg is trying to access the shared libraries, most likely in /usr/lib, which you subsequently can't see because you're jailed, so it's falling over.
note to self: don't develop with the idea that everyone has access to a decent shell!
Posts: 337
What about the fact that ClipBucket detects those codecs in FFMpeg that is installed in the hosting server?
Posts: 337
Well. I've just asked my hoster now and he told me that FFMpeg that is installed on my server DOES support all of those codecs (aac, vorbis, mp2, and flac).. so the question now is why the module doesn't detect any one of them? :S
Posts: 290
ok, i did a bit of research, and it seems that everything i was depending on to make this module work was introduced into the ffmpeg package from version 0.6 (which is quite recent).
so, i did some more mods, as it turned out my hosting server doesn't have that version either (unbeknownst to me earlier), so i used that as an example of a server that doesn't have ffprobe, and has an older version of ffmpeg that doesn't support -codecs.
try this version: http://www.danneh.org/files/transcode-2.tar.gz (still version 2, i know. next one i'll bump up to v3, i promise lol)
overwrite the transcode folder with what's in that tarball and try again. it no longer requires ffprobe, and i've improved the audio codec detection routine work with both older and newer versions of ffmpeg.
let me know how it works out. if it's still not picking up the audio codecs correctly from ffmpeg (use the ffmpeg path it auto-detects rather than any other custom/precompiled version you have installed which it now lists below the text box in the admin area, just as an fyi), i'll knock together a quick diagnostic script so i can find out more.
Dan
Posts: 337
I cannot change the path of FFMpeg anymore. When I save my changes the page get refreshed with the new value. However, when I go to some admin page and return again I see the old value for the path again. Unfortunately, I cannot test the new version with the FFMpeg that is installed on the server until getting that fixed.
By the way, I still see a field for FFProbe path, although you said that you removed the dependency of it.
Posts: 290
hm, something doesn't add up there. the ffprobe field isn't there anymore.
delete the transcode folder entirely, check it's gone by trying to get to it from the admin area, then re-download and re-extract the tarball.
this would be so much easier if i could figure out how the version detection and auto-updater worked
Posts: 337
I already did all of that, exactly. The same behavior still happens + The field is still there!
Posts: 337
Could you please try that with your version and see if the problem exists for you or not?
Steps to reproduce:
(1) Go to the transcode admin page
(2) Change the FFMpeg path to something else, and save your changes
-> You should see the new path in the field
(3) Go to module admin page, then go to transcode admin page again
-> You will see that the FFMpeg path wasn't actually changed
Posts: 290
sounds like the server's caching the php files or something.
http://www.danneh.org/files/transcode-3.tar.gz
rinse & repeat with above tarball. version number incremented, just incase gallery's doing something clever with opcode/output caching or something.
Posts: 290
changes to the ffmpeg path on both my production server and development environment locally persist across page loads.
let me know if v3 behaves the same or not. push comes to shove you'd need to give me admin access to your gallery/ftp for a short period while i figure this one out.
Posts: 337
Well. The FFProbe field is gone. However, the same issue with changing the path still exists. Could you please try the steps above to see if you have the problem?
P.S Will PM you the details shortly.
Posts: 290
for anyone else following this module's progress, a small update;
i'm working with mamouneyya on his server/account to get this working in his particular scenario. the situation is there that aac is not installed (at least, ffmpeg doesn't support encoding with aac), but mp3 encoding is. whereas my development areas have aac and no mp3. there's also a couple of other issues i've located and squashed, and whereas it's not quite ready to provide an updated version yet, i should have something tomorrow night to fix all these horrid issues. ffmpeg to blame, mostly!
it's getting closer, i promise
like i said, i'm new to this open-source release-to-the-community stuff, i only usually develop code that will only usually ever sit on one particular server, that's either mine, i have root access to, or have access to in advance to work around it's limitations. i'm learning though
i'm also writing an updater/helper module that i'll ship with all of my modules in the future to provide version checking and auto-updating (from my repository) when stuff becomes available. at least, until someone from g3 tells me how their updating system works
Dan
Posts: 130
Hi Again,
Using version 3 of this module. I'm going to get this working. I know it.
There are currently no audio codecs to select. and where do we go to add videos? when using the standard 'add photos' option, my AVI files don't show up in the pick list.
-Marc
Posts: 23
Hi, i uploaded version 3 and when i go to settings, i added the ffmpeg path and clicked save but doesn't seem to be able to save. when i go back to settings again path field is blank. really excited to use this module hope i can get this working. thanks so much.
also like to add that after activating the module, i was not able to play all the videos untill i deactivate the module. also notice there is a Alternative Resolutions select box under the video but its empty.
Posts: 12
after upload a (MPEG) nothing shows up and nothing is in the album folder
using version 3 and did this change to select a avi file
P.S I've tried to edit the file /modules/gallery/views/form_uploadify.html.php replacing:
fileExt: "*.gif;*.jpg;*.jpeg;*.png;*.GIF;*.JPG;*.JPEG;*.PNG<? if ($movies_allowed): ?>;*.flv;*.mp4;*.m4v;*.m4v;*.FLV;*.MP4;*.M4V<? endif ?>",
with:
fileExt: "*.gif;*.jpg;*.jpeg;*.png;*.GIF;*.JPG;*.JPEG;*.PNG<? if ($movies_allowed): ?>;*.*<? endif ?>",
edit: now i did upload a flv file and get a error message
test.flv - Unable to process this file
Posts: 23
anyone got this working? i've tried fresh install with this module still no go =(
Posts: 290
there is a couple of issues with this at present. i have access to a server which is fails miserably attempting to run, and it's getting there. unfortunately not having much free time available doesn't help in getting this working.
@shecter; this is an issue with running older (pre 0.6) versions of ffmpeg. i've resolved this in an uncommitted development version i have on my server.
@diverselab; i'm aware of that issue and also have a fix in place in the development version.
@swissphoenix; i'm currently only working with the built-in facilities which gallery provides. this includes, at the moment, only allowing file extensions which gallery allows to be uploaded. i'll hook in eventually to allow more file types to be uploaded, but this would have to be on the basis that the installed ffmpeg version supports the given file type.
i'll post a new version in the next couple of days.
Posts: 23
thanks so much for working on this module, really appreciate you taking the time to do this.
Posts: 25
Any update on this module?
Posts: 290
Hihi;
Sorry about the delay. Paid work takes priority unfortunately. Anyhoo, had a little bit of spare time so I thought I'd take a crack at this again. I've got a new version for those who want to try it. This has been tested on my server that hasn't got a later version of ffmpeg installed (svn rev 194xx) which doesn't support -codecs and all that other fancy stuff that 0.6 (svn rev 250xx) does.
get it here: http://www.danneh.org/files/transcode-5.zip
might help to completely remove the transcode directory beforehand, unzip into the modules dir, visit admin -> modules and "upgrade" if requested. make sure the path to ffmpeg is set fine in admin -> settings -> video transcoding, and give it a whirl.
it spits a *lot* of information into var/modules/transcode/log/transcode.log about what it's doing and any problems it incurs along the way. there's a debug script as well that's web-accessible for me to be able to see a handful of things/settings/stats if things don't work right. if you have problems, pass me the transcode.log file, and the url to your gallery and hopefully we can get this working once and for all
Dan