IPTC/EXIF and Keywords

I noticed in the EXIF/IPTC module that there is an option to look and see if there are IPTC keywords and if so add them to the database.

Does this mean the same KEYWORDS that will then be searchable?

Thanks in advance,
Shawn

----

Gallery URL (optional):
Gallery version:
Webserver (with version):
Datatabase (with version):
PHP version (eg 4.2.1):
phpinfo URL (optional):
Graphics Toolkit(s):
Operating system:
Web browser/version:
G1 version (for migration bugs):

We plan to allow you to extract all the EXIF/IPTC data into the database (even that data which isn't put into the general item fields) and make it searchable. It's not that hard, we just need more dev resources.

I've gone through the code forwards and backwards and I think I've found some clue as to how I can get the EXIF data into the database. It appears that it would require modification of handleRequestAfterAdd(...) in ExifDescriptionOption.inc. The workhorse code is:

                    ExifExtractor::getMetaData(
                        array($itemId),
                        array('IPTC/Caption',
                              'ImageDescription',
                              'UserComment',
                              'IPTC/Keywords',
                              )
                        );

Am I right that populating that array with my custom EXIF fields would cause the extractor to return the data for those fields? If so, great but the current set of properties is simply assigned to the item description:

$itemDescription = $exifData[$itemId]['UserComment']['value'];

I traced the setDescription(...) calls back hoping to eventually find a database query to insert/update the description but its a long and winding trail of code that appears to include caching and other things before it reaches the database. I thought I would add a new method to GalleryEntity that would eventually store the data persistently but it seems very complicated. I'm not sure what has to change beyond GalleryEntity to get the data into the database. Anyone?

As a test I modified handleRequestAfterAdd(...) in ExifDescriptionOption.inc to support the fields of interest to me. I created a database table to store the itemID, profile, field, and value and cheated by directly connecting to the database to populate the fields. The test it, I added an image with the GPS fields to a test gallery. But nothing happened. The fields were not populated, I assume, because the code was never called to populate them. No errors were given and the image was added to the gallery without a problem. So my question is, when is handleRequestAfterAdd(...) called? Once I get that figured out and the database populating when images are added I will trudge through the source and do it the right way by adding the appropriate methods to entity, storage, etc.

Yes, if you want to add methods to the objects you will have to update a long chain of classes to get from the initial creation of the data to the database. I myself have not figured that chain out fully yet except that it must require changes to the GalleryStorage class, the implementation classes for GalleryStorage, and at least some of the GalleryItem and GalleryEntity type classes. I wasn't too worried about that part just yet because I figured one of the active developers could illuminate that for us. My test was simply to see if the handler was getting called and, as you know, it is apparently not. It would be useful to have a UML sequence diagram of the item addition process but I doubt such a thing exists. If it comes to it, I will make one but I have done no other development on gallery and so am a complete noob to the source. I'll poke around in there more tonight and let you know what I find out.

wait a momement

1.
all files in modules/*/classes/interfaces/* are generated automatically.
also all CREATE TABLE sql statements in modules/*/classes/GalleryStorage/DatabaseStorage/schema/platform/*/*.sql is generated automatically.

how?
you create a class in modules/*/classes/ClassNameMap.class and add some markup (take a look at the existing *Map.class files.
we use php to parse our .class files for XML markup and generate from the markup the interfaces/*.inc files and the sql files for all databases (mysql, postgresql, ..).

2.
do no add columns to existing tables etc. e.g. do not add something to g2_Item.

first: get the latest G2 version. we changed some important things since 2.0. get the nightly snapshot.

you should create a new *Map.class file in modules/exif/classes/ which has the members g_id, profile (exif or iptc), field, value.
then run gmake in modules/exif/classes/exif/ . this will generate the corresponding interface file and the SQL table definition.

then add your code to add and query data into / from the map table.

finished.

OK, so I installed the latest CVS nightly and created my own *Map.class file called ExifFieldsMap.class. This I based directly on ExifPropertiesMap.class. It is as follows:

<?php
/**
 * @g2 <class-name>ExifFieldsMap</class-name>
 * @g2 <schema>
 * @g2   <schema-major>1</schema-major>
 * @g2   <schema-minor>0</schema-minor>
 * @g2 </schema>
 *
 * @package Exif
 * @subpackage Classes
 */

class ExifFieldsMap_core {

    /**
     * @g2 <map>
     * @g2   <member>
     * @g2     <member-name>itemId</member-name>
     * @g2     <member-type>INTEGER</member-type>
     * @g2     <member-size>LARGE</member-size>
     * @g2   </member>
     * @g2   <member>
     * @g2     <member-name>profile</member-name>
     * @g2     <member-type>STRING</member-type>
     * @g2     <member-size>MEDIUM</member-size>
     * @g2   </member>
     * @g2   <member>
     * @g2     <member-name>field</member-name>
     * @g2     <member-type>STRING</member-type>
     * @g2     <member-size>LARGE</member-size>
     * @g2   </member>
     * @g2   <member>
     * @g2     <member-name>value</member-name>
     * @g2     <member-type>STRING</member-type>
     * @g2     <member-size>LARGE</member-size>
     * @g2   </member>
     * @g2 </map>
     */

    /**
     * Return the target properties for the given view mode
     *
     * @param int the view mode (EXIF_SUMMARY, etc)
     * @return array object GalleryStatus a status code
     *               array logical exif property names
     * @static
     */
    function getProperties($itemId) {
        global $gallery;

        $query = '
        SELECT
          [ExifFieldsMap::itemId], [ExifFieldsMap::profile], [ExifFieldsMap::field], [ExifFieldsMap::value]
        FROM
          [ExifFieldsMap]
        WHERE
          [ExifFieldsMap::itemId] = ?
        ORDER BY
          [ExifFieldsMap::field] ASC';
        list ($ret, $searchResults) = $gallery->search($query, array($itemId));
        if ($ret->isError()) {
            return array($ret->wrap(__FILE__, __LINE__), null);
        }

        $data = array();
        while ($result = $searchResults->nextResult()) {
            $data[] = $result[0];
        }

        return array(GalleryStatus::success(), $data);
    }

    /**
     * Set the target properties for the given item ID
     *
     * @param int the item ID
     * @param array logical property key/value pairs
     * @return object GalleryStatus a status code
     * @static
     */
    function setProperties($itemId, $properties) {

        /* Remove all old map entries */
        $ret = ExifPropertiesMap::removeMapEntry(array('itemID' => $itemId));
        if ($ret->isError()) {
            return $ret->wrap(__FILE__, __LINE__);
        }

        for ($i = 0; $i < sizeof($properties); $i++) {
            $ret = ExifPropertiesMap::addMapEntry(array('property' => $properties[$i], 'itemId' => $itemId));
            if ($ret->isError()) {
                return $ret->wrap(__FILE__, __LINE__);
            }
        }

        return GalleryStatus::success();
    }
}

include(dirname(__FILE__) . '/interfaces/ExifFieldsMap.inc');
?>

I simply made changes for the appropriate database columns and reflected those changes in the methods that already existed.

There is no ExifFieldsMap.inc in interfaces, so I run gmake to generate everything. It completes without error:

cd interfaces && gmake -
gmake[1]: Entering directory `/gallery2/modules/exif/classes/interfaces'
perl ../../../../lib/tools/bin/extractClassXml.pl --dtd=../../../../../lib/tools/dtd/GalleryClass2.0.dtd --stub-ok --quiet --out-dir=tmp ../*.class
php ../../../../lib/tools/bin/generate-interfaces.php Exif
Content-type: text/html
X-Powered-By: PHP/4.4.0

gmake[1]: Leaving directory `/gallery2/modules/exif/classes/interfaces'
cd GalleryStorage/DatabaseStorage/schema && gmake -
gmake[1]: Entering directory `gallery2/modules/exif/classes/GalleryStorage/DatabaseStorage/schema'
perl ../../../../../../lib/tools/bin/extractClassXml.pl --dtd=../../../../../../lib/tools/dtd/GalleryClass2.0.dtd --quiet --out-dir=tmp/classxml ../../../*.class
php -f ../../../../../../lib/tools/bin/generate-dbxml.php
php -f ../../../../../../lib/tools/bin/generate-sql.php tmp/classxml
gmake[1]: Leaving directory `/gallery2/modules/exif/classes/GalleryStorage/DatabaseStorage/schema'

But after this, there is no matching ExifFieldsMap.inc in interfaces. I expected it because the ExifPropertiesMap.inc says its automatically generated. Additionally there is no matching database table in the database. Does something else have to be done to generate them? Or perhaps I have done something wrong?

After this I optimized the database, removed the caches, etc as listed under maintanence thinking maybe it had something to do with that but no such luck. Is this process documented somewhere? I don't want you to have to repeat information that already exists.

I am getting the following error when I try and run the make. Any Ideas?

The java class is not found: com/icl/saxon/StyleSheet
make[3]: *** [iptcPropertyValuesMap.sql] Error 1
make[3]: Leaving directory `/srv/www/htdocs/gallery2/modules/exif/classes/GalleryStorage/DatabaseStorage/schema/platform/postgres'
make[2]: *** [sql] Error 2
make[2]: Leaving directory `/srv/www/htdocs/gallery2/modules/exif/classes/GalleryStorage/DatabaseStorage/schema/platform'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/srv/www/htdocs/gallery2/modules/exif/classes/GalleryStorage/DatabaseStorage/schema'
make: *** [all] Error 2

I just did another update.
cvs -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/gallery checkout -P gallery2
And I am still getting the same error.

I forgot to add the Debugging module. Now the script looks like it is running but the .inc file is empty.
I am attaching a gmake.txt file with what all happens when I run the make.

jrmint, it's still using g 2.0 stuff, not the current cvs version.
delete your g2 and get the nightly snapshot http://galleryupdates.jpmullan.com/G2/gallery-2.0+.tar.gz . or install it as a second g2 on your box.
obviously you're doing something wrong with cvs or it doesn't work as it should or you have used a release / date flag in prior cvs checkouts / updates.
just get the nightly snapshot.

2 questions.
1. I would like to extend the GalleryItem.inc and GalleryItem.class. These files are doing what I need them to do, except I want to point the IPTC metadata to a new database table I created. Can I just create a new .inc and .class with the code I need and do a requireonce in the original files?

2. If I do create a new module How do I get Gallery2 to run the new module?

you can register your own entity type in a module, which is then used instead of GalleryItem.

What are the steps to register my own module?

What I am trying to do is make a module that will add a table called g2_ItcpItems with some indexed fields called headline, city, byline, objectname ext ext.
The module should look at the jpgs during upload and populate these fields in the database.

Then I will need to be able to search by one or more of these fields.

Thanks you again for your help.

I tryed to do the gmake *.map.class but it never finishes. I keep getting Java errors.
So I thought that maybe I should look into doing this with a differant table.

If I can get a module that will add the IPTC fields into g2_Item then populate those fields I will use it..

I am probably doing this wrong.

Here is my last map.class file
What am I doing wrong? I think that I don't have the information to point to the g2_Item table..

<?php
/**
* @g2 <class-name>IptcItemMap</class-name>
* @g2 <schema>
* @g2 <schema-major>1</schema-major>
* @g2 <schema-minor>0</schema-minor>
* @g2 </schema>
*
* @package Exif
* @subpackage Classes
*/

class IptcItemMap_core {

/**
* @g2 <map>
* @g2 <member>
* @g2 <member-name>headline</member-name>
* @g2 <member-type>TEXT</member-type>
* @g2 </index>
* @g2 </member>
* @g2 <member>
* @g2 <member-name>city</member-name>
* @g2 <member-type>TEXT</member-type>
* @g2 </member>
* @g2 <member>
* @g2 <member-name>byline</member-name>
* @g2 <member-type>TEXT</member-type>
* @g2 </member>
* @g2 <member>
* @g2 <member-name>objectname</member-name>
* @g2 <member-type>TEXT</member-type>
* @g2 </index>
* @g2 </member>
* @g2 </map>
*/

/**
* Return the target properties for the given view mode
*
* @param int the view mode (EXIF_SUMMARY, etc)
* @return array object GalleryStatus a status code
* array logical exif property names
* @static
*/
function getProperties($itemId) {
global $gallery;

$query = '
SELECT
[IptcItemMap::itemId], [IptcItemMap::profile], [IptcItemMap::field], [IptcItemMap::value]
FROM
[IptcItemMap]
WHERE
[IptcItemMap::itemId] = ?
ORDER BY
[IptcItemMap::field] ASC';
list ($ret, $searchResults) = $gallery->search($query, array($itemId));
if ($ret->isError()) {
return array($ret->wrap(__FILE__, __LINE__), null);
}

$data = array();
while ($result = $searchResults->nextResult()) {
$data[] = $result[0];
}

return array(GalleryStatus::success(), $data);
}

/**
* Set the target properties for the given item ID
*
* @param int the item ID
* @param array logical property key/value pairs
* @return object GalleryStatus a status code
* @static
*/
function setProperties($itemId, $properties) {

/* Remove all old map entries */
$ret = ExifPropertiesMap::removeMapEntry(array('itemID' => $itemId));
if ($ret->isError()) {
return $ret->wrap(__FILE__, __LINE__);
}

for ($i = 0; $i < sizeof($properties); $i++) {
$ret = ExifPropertiesMap::addMapEntry(array('property' => $properties[$i], 'itemId' => $itemId));
if ($ret->isError()) {
return $ret->wrap(__FILE__, __LINE__);
}
}

return GalleryStatus::success();
}
}

include(dirname(__FILE__) . '/interfaces/IptcItemMap.inc');
?>

I knew I was doing this wrong.
I am learning this stuff as I go. as you can see..

What would you use to put the IPTC fields into the g2_item table.

I tryed this and I got an error Notice: Undefined index:

<?php
/**
* @g2 <class-name>IptcItemMap</class-name>
* @g2 <schema>
* @g2 <schema-major>1</schema-major>
* @g2 <schema-minor>0</schema-minor>
* @g2 </schema>
*
* @package Exif
* @subpackage Classes
*/

class IptcItemMap_core {

/**
* @g2 <map>
* @g2 <member>
* @g2 <member-name>itemId</member-name>
* @g2 <member-type>INTEGER</member-type>
* @g2 <member-size>LARGE</member-size>
* @g2 </member>
* @g2 <member-name>headline</member-name>
* @g2 <member-type>TEXT</member-type>
* @g2 </indexed>
* @g2 </member>
* @g2 <member>
* @g2 <member-name>city</member-name>
* @g2 <member-type>TEXT</member-type>
* @g2 </member>
* @g2 <member>
* @g2 <member-name>byline</member-name>
* @g2 <member-type>TEXT</member-type>
* @g2 </member>
* @g2 <member>
* @g2 <member-name>objectname</member-name>
* @g2 <member-type>TEXT</member-type>
* @g2 </indexed>
* @g2 </member>
* @g2 <key>
* @g2 <member-name>itemId</member-name>
* @g2 <member-name>headline</member-name>
* @g2 <member-name>city</member-name>
* @g2 <member-name>byline</member-name>
* @g2 <member-name>objectname</member-name>
* @g2 </key>
* @g2 </map>
*/
}
include(dirname(__FILE__) . '/interfaces/IptcItemMap.inc');
?>

Once I know what we are working on works. I will add the other fields into the module. I am just using a few as a test. I need more than just these 4.

I just cvs'ed a fresh checkout of gallery 2 to try valiant's ExifPropertyValuesMap.class. After running through the standard install procedure and making sure G2 was functional, I copied the map class into modules/exif/classes and ran gmake with this result:

$ gmake
cd interfaces && gmake -
gmake[1]: Entering directory `/cvs/gallery2/modules/exif/classes/interfaces'
perl ../../../../lib/tools/bin/extractClassXml.pl --dtd=../../../../../lib/tools/dtd/GalleryClass2.0.dtd --stub-ok --quiet --out-dir=tmp ../*.class
php ../../../../lib/tools/bin/generate-interfaces.php Exif
Content-type: text/html
X-Powered-By: PHP/4.4.0

<br />
<b>Warning</b>:  main(../../../../lib/smarty/Smarty.class.php): failed to open stream: No such file or directory in <b>/cvs/gallery2/lib/tools/bin/generate-interfaces.php</b> on line <b>29</b><br />
<br />
<b>Fatal error</b>:  main(): Failed opening required '../../../../lib/smarty/Smarty.class.php' (include_path='.:/usr/local/share/pear:/usr/local/lib/php') in <b>/cvs/gallery2/lib/tools/bin/generate-interfaces.php</b> on line <b>29</b><br />
gmake[1]: *** [inc] Error 255
gmake[1]: Leaving directory `/cvs/gallery2/modules/exif/classes/interfaces'
gmake: *** [all] Error 2

The error is because the relative directory goes up one directory too far. So I changed '../../../../lib/smarty/Smarty.class.php' to '../../../lib/smarty/Smarty.class.php' in both files. Then gmake ran fine:

$ gmake
cd interfaces && gmake -
gmake[1]: Entering directory `/cvs/gallery2/modules/exif/classes/interfaces'
perl ../../../../lib/tools/bin/extractClassXml.pl --dtd=../../../../../lib/tools/dtd/GalleryClass2.0.dtd --stub-ok --quiet --out-dir=tmp ../*.class
php ../../../../lib/tools/bin/generate-interfaces.php Exif
Content-type: text/html
X-Powered-By: PHP/4.4.0

gmake[1]: Leaving directory `/cvs/gallery2/modules/exif/classes/interfaces'
cd GalleryStorage/DatabaseStorage/schema && gmake -
gmake[1]: Entering directory `/cvs/gallery2/modules/exif/classes/GalleryStorage/DatabaseStorage/schema'
perl ../../../../../../lib/tools/bin/extractClassXml.pl --dtd=../../../../../../lib/tools/dtd/GalleryClass2.0.dtd --quiet --out-dir=tmp/classxml ../../../*.class
php -f ../../../../../../lib/tools/bin/generate-dbxml.php
php -f ../../../../../../lib/tools/bin/generate-sql.php tmp/classxml
gmake[1]: Leaving directory `/cvs/gallery2/modules/exif/classes/GalleryStorage/DatabaseStorage/schema'

At least that looks fine to me. I don't really know what to expect. But the result is the same as before. There is no ExifPropertyValuesMap.inc in interfaces and no database was created. So I am in essentially the same position as jrmint.

sorry that we can't solve this immediately. I'm still hoping for assistance by bharat, since he's the author of the generate-*.php scripts.
Thanks for your patience. I'm sure we get this fixed soon.

I'm confident that its a minor hurdle to getting this feature implemented. I'll keep poking at it to see if it moves. As jrmint said, many thanks for the help you've given us thus far.

sorry for the wait, but you have to be a little more patient :/

@make:
it sure works.
see:

modules/core/classes/interfaces/GNUmakefile is:

so it does a lot more than you think
maybe the scripts aren't robust enough yet since we (bharat) changed it since september 12th.
until 2.0, you needed java, saxon and rxp, plus perl, to generate the interface and sql files.
now, you only need php and perl. but for some reason it doesn't work on all machines yet.

bharat, this is a great idea. based on that we can expand the wizard etc. cool

but unless you have changed the existing xml parsing scripts etc. it probably won't help these two guys.

e.g. jrmint gets this output on gmake:

cd interfaces && make -
make[1]: Entering directory `/srv/www/htdocs/gallery2/modules/exif/classes/interfaces'
perl ../../../../lib/tools/bin/extractClassXml.pl --dtd=../../../../../lib/tools/dtd/GalleryClass2.0.dtd --stub-ok --quiet --out-dir=tmp ../*.class
php ../../../../lib/tools/bin/generate-interfaces.php Exif

Notice: Undefined index: content in /srv/www/htdocs/gallery2/lib/tools/bin/generate-interfaces.php on line 75
XML error: mismatched tag at line 17make[1]: Leaving directory `/srv/www/htdocs/gallery2/modules/exif/classes/interfaces'
cd GalleryStorage/DatabaseStorage/schema && make -
make[1]: Entering directory `/srv/www/htdocs/gallery2/modules/exif/classes/GalleryStorage/DatabaseStorage/schema'
perl ../../../../../../lib/tools/bin/extractClassXml.pl --dtd=../../../../../../lib/tools/dtd/GalleryClass2.0.dtd --quiet --out-dir=tmp/classxml ../../../*.class
php -f ../../../../../../lib/tools/bin/generate-dbxml.php
XML error: mismatched tag at line 17php -f ../../../../../../lib/tools/bin/generate-sql.php tmp/classxml
make[1]: Leaving directory `/srv/www/htdocs/gallery2/modules/exif/classes/GalleryStorage/DatabaseStorage/schema'

@jrmint:
please CVS update -dPC to get the new files and try gmake again. if it doesn't work, create a zip file of your (exif) module and i'll try gmake for the same module on my computer.
we tried this before based on the single file you edited, but let's try it again / make sure your source files are ok.

Gallery2 will have fulltext boolean search, probably in the 2.1 release. The main problem is figuring out the proper way to cross-database support of creating the indexes and doing the matching.

The advantage of the sample module generator is that it creates something that should definitely work (barring some slight bugs). If that code gives you the same error then we should figure out what's wrong with your environment. Otherwise, we should be trying to figure out what's wrong with the code that you wrote (so zipping up what you have like valiant suggests will probably help us figure this one out).

after gmake etc. you need to uninstall and reinstall the module in question (site admin -> modules ).
only installing a module creates new tables.

(alternatively, increase the module version, then you can just upgrade the module in site admin -> modules).