My first run with G2 - migrating from G1 to G2

Lapinoo
Lapinoo's picture

Joined: 2004-05-08
Posts: 378
Posted: Mon, 2004-05-10 19:24

After hearing Pierre-Luc saying for days how great G2 was :-) I decided to give it a try.

I downloaded a complete built from Chetan Sarva (alas, this one was not working) and then retrieved the CVS tree (I reinstalled everything sunday morning since my CVS installation would not update anymore cvs.GALLERY.sourceforge.net -> cvs.sourceforge.net).

I easily installed this release (much easier to do than G1 that checks all parameters on page 3 instead that doing it whenever I move from one page to another), and then I migrated my complete gallery (1800 photos), tens of albums.

Here are the problems I encountered :

Migration

  • Users should be imported before albums so that the interface does not say it cannot attribute albums to users before the migration.
  • The import of so many albums may fail (it failed on my machine), the process should be cut into smaller actions so that it doesnot fail while processing (can be applied to G1 process when updating a big album).
  • High ANSI characters are not migrated correctly (é, è, ç, à...).
  • Order is not fully migrated (some elements' position is modified).
  • When no title has been defined in G1, G2 shows the filename instead of nothing. This is nasty since some pictures have very long file names with no space, it produces some odd presentation when displaying albums.
  • Comments are not migrated.
  • I got a few broken images (the last one from one album).
  • Slideshow uses only mostly thumbnails (from times to times I got the big picture) ? For instance the pictures where 640 x 427, and G2 just displayed the big one when 1024x768 or no limit was selected.

Matrix layout

  • When a page contains 1 (or more) album and many images (more than col * rows), the layout will just present col * rows - 1 images (the last one is displayed on the next page).
  • If the thumbnail size is not "150", but "200" for example, then the presentation is messy in the "reorder items" for an album (1st column is not wide enough).
  • Album descriptions are not displayed ?

Locking

  • The locking mechanism is not based on the database. As a result, some actions may leave some elements locked. I encountered the problem : I could no longer erase one album. A "remove all locks" feature is welcome in the Admin interface as long as the locking mechanism has not been fully debbuged.

Commands

  • Command texts should be coherent: if albums actions are presented with a verb and a noum, then "permissions" should be "edit permissions" and "slideshow" would be "view slideshow". It's not a big issue, but it's so easy to fix !

Other

  • the "latest album display" feature does not display the title of the album.

Features

  • Most useful features are available.
  • Image linking is great, but I was looking for album linking (right now the feature I really look after).
  • I'd like to have the 'spiral book' and 'shadow' effects available in the default presentation (see Gallery 1.4.x).

All in all, this is quite an achievement. :D

I plan to migrate to G2 as soon as :

  • the database structure is stable, so that I can upgrade G2 without losing the database,
  • the comments are migrated,
  • the locking mechanism is stable enough of G2 provides some feature to remove all locks in the admin interface.

[/]

[/]

[/]

[/]

[/]

[/]

[/]

 
jmullan
jmullan's picture

Joined: 2002-07-28
Posts: 974
Posted: Mon, 2004-05-10 20:09
Lapinoo wrote:
Migration
  • Users should be imported before albums so that the interface does not say it cannot attribute albums to users before the migration.

I will have the module check the album owner against the list of users that will be imported. That should make the warning go away. If you are really worried, you could import your users first, then go back and import your albums.

Lapinoo wrote:
  • The import of so many albums may fail (it failed on my machine), the process should be cut into smaller actions so that it doesnot fail while processing (can be applied to G1 process when updating a big album).

Well, you can always choose fewer albums and do them in batches. I will be trying to improve this process, but it probably will not ever be automatically broken into batches.

Lapinoo wrote:
  • High ANSI characters are not migrated correctly (é, è, ç, à...).

I won't care about this until I start getting some sample albums:
http://gallery.menalto.com/index.php?name=PNphpBB2&file=viewtopic&t=14703

I mean it. I don't give a crap about anyone who wants their language imported who can't provide me with a G1 sample album. I will be making my own ISO-8859-(something) album for US English if no one else sends me one, but in the meantime I'm not even thinking about it.

Oh yeah, there is a bug for this:
https://sourceforge.net/tracker/?func=detail&aid=933435&group_id=7130&atid=107130

Lapinoo wrote:
  • Order is not fully migrated (some elements' position is modified).

This is most likely the albums. I'm not sure that I care about the album order, but I am working on fixing it anyway. If actual images are out of order, please let me know, as that would be a new bug.

Album order bug: https://sourceforge.net/pm/task.php?func=detailtask&project_task_id=94875&group_id=7130&group_project_id=14056

Lapinoo wrote:
  • When no title has been defined in G1, G2 shows the filename instead of nothing. This is nasty since some pictures have very long file names with no space, it produces some odd presentation when displaying albums.

Hmm. I think this is a G2 setting. I don't think that the migration module pushes anything into the caption field that wasn't there in the first place. I will look into it.

Lapinoo wrote:
  • Comments are not migrated.

Yep. There is a task for it, but I haven't made any progress. https://sourceforge.net/pm/task.php?func=detailtask&project_task_id=95407&group_id=7130&group_project_id=14056

Lapinoo wrote:
  • I got a few broken images (the last one from one album).

Try adding those images directly to your G2. If they work, I will try to figure out why they wouldn't migrate. If not, we will try to figure out why G2 doesn't like them.

Lapinoo wrote:
  • Slideshow uses only mostly thumbnails (from times to times I got the big picture) ? For instance the pictures where 640 x 427, and G2 just displayed the big one when 1024x768 or no limit was selected.

This isn't a migration issue, this is a slideshow issue.[/]

[/]

[/]

[/]

[/]

[/]

[/]

[/]

 
Lapinoo
Lapinoo's picture

Joined: 2004-05-08
Posts: 378
Posted: Mon, 2004-05-10 20:56
jmullan wrote:
Lapinoo wrote:
Migration
  • Users should be imported before albums so that the interface does not say it cannot attribute albums to users before the migration.

I will have the module check the album owner against the list of users that will be imported. That should make the warning go away. If you are really worried, you could import your users first, then go back and import your albums.

I am not worried, since I discovered that everything was imported correctly even if there was a warning. :-)
Some users might think there might be a problem, it should be as user friendly as possible.

jmullan wrote:
Lapinoo wrote:
  • The import of so many albums may fail (it failed on my machine), the process should be cut into smaller actions so that it doesnot fail while processing (can be applied to G1 process when updating a big album).

Well, you can always choose fewer albums and do them in batches. I will be trying to improve this process, but it probably will not ever be automatically broken into batches.

OK by me.
I'd suggest the import tool invites the user not to import all albums at once, but to chose some albums so that the number of photos does not exceed XXX (to be defined for most configurations).

jmullan wrote:
Lapinoo wrote:
  • High ANSI characters are not migrated correctly (é, è, ç, à...).

I won't care about this until I start getting some sample albums:
http://gallery.menalto.com/index.php?name=PNphpBB2&file=viewtopic&t=14703

I mean it. I don't give a crap about anyone who wants their language imported who can't provide me with a G1 sample album. I will be making my own ISO-8859-(something) album for US English if no one else sends me one, but in the meantime I'm not even thinking about it.

Oh yeah, there is a bug for this:
https://sourceforge.net/tracker/?func=detail&aid=933435&group_id=7130&atid=107130

I can create such an album. I'll send you a message when done.

jmullan wrote:
Lapinoo wrote:
  • Order is not fully migrated (some elements' position is modified).

This is most likely the albums. I'm not sure that I care about the album order, but I am working on fixing it anyway. If actual images are out of order, please let me know, as that would be a new bug.

Album order bug: https://sourceforge.net/pm/task.php?func=detailtask&project_task_id=94875&group_id=7130&group_project_id=14056

I didn't check the order of the images. It seamed OK to me.
To be honest, most albums were in order. The first sub-album from the first album was in second position, that's what I saw.

jmullan wrote:
Lapinoo wrote:
  • When no title has been defined in G1, G2 shows the filename instead of nothing. This is nasty since some pictures have very long file names with no space, it produces some odd presentation when displaying albums.

Hmm. I think this is a G2 setting. I don't think that the migration module pushes anything into the caption field that wasn't there in the first place. I will look into it.

That's possible.

jmullan wrote:
Lapinoo wrote:
  • Comments are not migrated.

Yep. There is a task for it, but I haven't made any progress. https://sourceforge.net/pm/task.php?func=detailtask&project_task_id=95407&group_id=7130&group_project_id=14056

So it's being taken care of, I'll wait :-)

jmullan wrote:
Lapinoo wrote:
  • I got a few broken images (the last one from one album).

Try adding those images directly to your G2. If they work, I will try to figure out why they wouldn't migrate. If not, we will try to figure out why G2 doesn't like them.

Funny detail, although I was the owner of the album (my user account was migrated) I could not add or remove any picture in this album... I'll check that later.

I used the admin account to upload the picture and it didn't work...

I checked : the picture got corrupted ! (the column of the first 8 pixels was displaced at the last column, funny...)

jmullan wrote:
Lapinoo wrote:
  • Slideshow uses only mostly thumbnails (from times to times I got the big picture) ? For instance the pictures where 640 x 427, and G2 just displayed the big one when 1024x768 or no limit was selected.

This isn't a migration issue, this is a slideshow issue.

Yes, this is not related to the migration module.

I just wanted to give a global overview of a migration attempt on a real gallery (a 2 year old one). So I can provide many developers with some information.[/]

[/]

[/]

[/]

[/]

[/]

[/]

[/]

 
bharat
bharat's picture

Joined: 2002-05-21
Posts: 7994
Posted: Tue, 2004-05-11 23:07

Thanks for the feedback, Lapinoo! I'll address your points in-line, below, omitting the ones that Jesse has already addressed.

Lapinoo wrote:
When no title has been defined in G1, G2 shows the filename instead of nothing. This is nasty since some pictures have very long file names with no space, it produces some odd presentation when displaying albums.

I don't know what else to put there. We can't leave it blank. What do you suggest? One thing we can do is to truncate it to 32 characters or something similar. How does that sound?

Lapinoo wrote:
Slideshow uses only mostly thumbnails (from times to times I got the big picture) ? For instance the pictures where 640 x 427, and G2 just displayed the big one when 1024x768 or no limit was selected.

I can't reproduce this. I just imported an album that had 46 images, all had 2304x1728 full sized images and 600x450 resized images and when I set the slideshow threshold to 1024x768 it showed me all the 600x450 images. Can you reproduce it reliably, and if so can you tell me the steps so that I can see it?

Lapinoo wrote:
When a page contains 1 (or more) album and many images (more than col * rows), the layout will just present col * rows - 1 images (the last one is displayed on the next page).

Albums count in the (col * rows) total. So if you have a 3x3 grid you may see 7 albums and 2 images on the first page. Is that what you meant? That's by design.

Lapinoo wrote:
If the thumbnail size is not "150", but "200" for example, then the presentation is messy in the "reorder items" for an album (1st column is not wide enough).

I'm confused. The "Reorder Album" page shows three dropdowns and no images. How does the thumbnail size come into play here? Am I loking at the wrong page?

Lapinoo wrote:
Album descriptions are not displayed ?

I fixed it so that we show the album description on the album page. It does weird things to the CSS in FireFox right now; I'll get our CSS guys to look at it.

Lapinoo wrote:
The locking mechanism is not based on the database. As a result, some actions may leave some elements locked. I encountered the problem : I could no longer erase one album. A "remove all locks" feature is welcome in the Admin interface as long as the locking mechanism has not been fully debbuged.

The locking system isn't row level database locking because that's not supported in all databases. We do store lock information in the database, however. But that lock information should never persist at the end of a request. Do you have any idea why it might be sticking around in your case? For now just execute this SQL:
delete * from g2_Lock in your database to get rid of the locks. We will have an administrative page that will allow this kind of maintenance in the future.

Lapinoo wrote:
Command texts should be coherent: if albums actions are presented with a verb and a noum, then "permissions" should be "edit permissions" and "slideshow" would be "view slideshow". It's not a big issue, but it's so easy to fix !

I agree. I fixed those two, let me know if you find other inconsistencies like that.

Lapinoo wrote:
the "latest album display" feature does not display the title of the album.

Are you sure? I just checked and it looks like it's showing the title for me.

Lapinoo wrote:
Image linking is great, but I was looking for album linking (right now the feature I really look after).

That's a much harder problem to solve than image linking. It's on my list, but it's not a high priority.

Lapinoo wrote:
I'd like to have the 'spiral book' and 'shadow' effects available in the default presentation (see Gallery 1.4.x).

Yep, that's coming. I think that we'll probably do it as part of the theme, instead of making it a separate capability (but the jury is still out on that).

Lapinoo wrote:
I plan to migrate to G2 as soon as :

  • the database structure is stable, so that I can upgrade G2 without losing the database,
  • the comments are migrated,
  • the locking mechanism is stable enough of G2 provides some feature to remove all locks in the admin interface.

Cool. We're getting pretty close![/]

 
SHS`

Joined: 2002-12-10
Posts: 30
Posted: Wed, 2004-05-12 08:47
bharat wrote:
Lapinoo wrote:
I plan to migrate to G2 as soon as :

  • the database structure is stable, so that I can upgrade G2 without losing the database,
  • the comments are migrated,
  • the locking mechanism is stable enough of G2 provides some feature to remove all locks in the admin interface.

Cool. We're getting pretty close!

Excellent, though I'm more than comfortable coping with DB data/schema changes now in running a live G2 install, there aren't unfortunately more than 24 hours in a day so I can just spare a few minutes to keep such and install inline with CVS by hand. :D[/]

 
Lapinoo
Lapinoo's picture

Joined: 2004-05-08
Posts: 378
Posted: Wed, 2004-05-12 19:48

A few screenshots to illustrate :

  • Clip_01 : the title for the newest album is "gallery", it should be "Thailand..."
  • Clip_02 & Clip_03 : view of an album that contains a subalbums and many pictures : the subalbum is displayed on one row, but on the last row of the same page, one more picture could be displayed. Please note also the title of the pictures that were generated while importing (there were none), they are so long and contain no space that they "disturb" the presentation

FYI, my browser is Mozilla 1.7 RC1[/]

 
Lapinoo
Lapinoo's picture

Joined: 2004-05-08
Posts: 378
Posted: Wed, 2004-05-12 20:32

Here is what I mean regarding the problem involving big thumbnails and the "reorder items".

You can also see what occurs when someone migrates elements containing special characters like 'é', 'à', 'ç'... :-?

 
bharat
bharat's picture

Joined: 2002-05-21
Posts: 7994
Posted: Sun, 2004-05-16 22:52

Lapinoo, I fixed the problem with the extra-large thumbnails. We now constrain that thumbnail to be no greater than 135px which lets it fit into the sidebar. I'll let Jesse handle the special characters issue (since he's on that already).

 
Lapinoo
Lapinoo's picture

Joined: 2004-05-08
Posts: 378
Posted: Mon, 2004-05-17 08:46

Hi bharat,

I just checked : it now works OK.

However, I found a new occurence : I use the Matrix layout, 3 photos per row. When thumbnail size is 200, when the last photo of a row is horizontal, a few pixels disappear under the column with the commands.

This is probably no side effect : it was probably there several days ago, but I didn't noticed.

 
Lapinoo
Lapinoo's picture

Joined: 2004-05-08
Posts: 378
Posted: Mon, 2004-05-17 08:56

Ooops, wrong description...

In fact, if I enlarge the browser window, the problem dissapears ! ;-)

But there is something that I find strange : why is there a gap between the rightest columns (commands) and the border of the windows (ca 50 pix) ?
Whithout this gap, the 3 pictures per row at 200pix size would display nicely.

 
Lapinoo
Lapinoo's picture

Joined: 2004-05-08
Posts: 378
Posted: Tue, 2004-05-18 20:40

Another one for jmullan :

I saw that you commited some changes into CVS... So I decided to test it again...

I have a strange problem : I try to import (as before) several albums at once.

After a few hundred pictures, G2 tries to import the last album : the biggest one in fact (more than 400 pictures). At some point during this album migration, G2 slows down dramatically : soon it requires around 30s to import one picture. Apache is looking for 85% of the CPU time, MySQL for the remaining 15%. Memory consuption is normal.

2 weeks ago I managed to import those albums in a few minutes, now several hours are required.

Update

:cry: AAAAAAAAAHHHHHHHHHHHHHHH... It's even worse... My machine rebooted for the second time (yesterday I wasn't sure it was G2's fault)...

This machine (Redhat 9 plus latest patches) is usually running very fine. I did not know one could make Linux reboot so easily !

 
bharat
bharat's picture

Joined: 2002-05-21
Posts: 7994
Posted: Sun, 2004-05-23 16:24

Lapinoo, that's really weird. Apache/PHP should never be able to get your machine to reboot. The only way that's possible is if there's a bug in the kernel since it's the kernel's job to prevent userland processes from getting out of control. :-?

I'm seeing similar slowdown when using the CreateRandomTree test. I tried adding 100K images on my dual 600Mhz FreeBSD box and by the time it got close to 50K images it was adding one image every 3 seconds (whereas at the start it was adding about 4 images a second). I'm looking into that now.

 
Lapinoo
Lapinoo's picture

Joined: 2004-05-08
Posts: 378
Posted: Tue, 2004-06-01 21:29

I got the problem : I removed the fan over the heatsink CPU a long time ago... I must admit that I wanted my Eden board to be as quiet as possible...

The thing is that when I import a lot of pictures from G1 to G2, it computes for more than one hour... CPU temp reached more than 140°C... then the machine crashes...

Nothing related to G2/PHP/Apache !

 
bharat
bharat's picture

Joined: 2002-05-21
Posts: 7994
Posted: Wed, 2004-06-02 04:46

Ha! Well the good news is that your issue prompted me to do some performance enhancements in the caching layer so if G2 doesn't cause your machine to melt down (which sounds like a distinct possibility) it should at least run a littel faster :-)

 
jmullan
jmullan's picture

Joined: 2002-07-28
Posts: 974
Posted: Wed, 2004-06-02 15:47

Lapinoo, I would be interested to know the results of your next migration. If you can import your entire G1 I will feel much better. :)

What motherboard are you using? I have a secret interest in those tiny systems.

 
Lapinoo
Lapinoo's picture

Joined: 2004-05-08
Posts: 378
Posted: Wed, 2004-06-02 20:23

jmullan,

I'll reload the whole G1 gallery tomorrow, so stay tuned !

This machine uses a Via Epia M9000 with 256 GB of RAM. OS is now Fedora Core 2 since this weekend.
I was looking for a small a quiet multipurpose Linux box since I want it to be online 24x7 in a (small) flat in Paris. Since it sits under my bed, I want it quiet so I removed the fan over the heatsink of the CPU. It is OK most of the time : it didn't crash even during the heat wave last summer (god that was hot... it reminded me death valley :o)) but migrating from G1 to G2 or BZip2ing a huge file was able to make it reboot.

I'll try to find a bigger copper heatsing this WE so that I can disconnect the fan once again !

 
Lapinoo
Lapinoo's picture

Joined: 2004-05-08
Posts: 378
Posted: Thu, 2004-06-03 20:02

Yahoo !

That was much faster !
I updated my g2 code base, reinitialized the database, reimported all my albums... All in less than 10 minutes (around 2 hours were necessary the last time).

Great work !

 
Lapinoo
Lapinoo's picture

Joined: 2004-05-08
Posts: 378
Posted: Thu, 2004-06-03 21:08

I think there is a little problem with permission migration : most of them are declared twice. E.g.:

Group Permissions
Group name 	Permission 	
Everybody 	[core] View all versions 	
All Users 	[core] View all versions 	
All Users 	[comment] Search comments 	
All Users 	[comment] Add comments 	
All Users 	[core] View all versions 	
All Users 	[comment] Search comments 	
All Users 	[comment] Add comments 	
Site Admins 	All access

User Permissions
User name 	Permission
Arnaud 	All access
Arnaud 	All access
 
jmullan
jmullan's picture

Joined: 2002-07-28
Posts: 974
Posted: Thu, 2004-06-03 22:07

Lapinoo, I'm trying to clear up the permissions stuff - I'm refactoring it and will be writing additional unit tests to make sure that the same permissions aren't added twice, or three times. This is the luxury of being pre-alpha - things can still go fairly wrong at this point. :)