Implement a 'keyword search' new search mode

[stuarth]

From the Slim Devices Developers Forum (reported by rutt@…):

Hi, it's been a long time since I opened the hood of the slimserver. In the early days I did a number of things. Looks like a lot of progress has been made, but something I did and liked a lot has been apparently lost.

I'd like a keyword search in addition to the current artists, albums, genres, and songs searches. This would be more like a google search than anything we have now. For example:

mozart solti

would find all the albums with where the words mozart and solti appear in any of the artist or album or song title fields.

dylan highway

finds Highway 61 Revisited and not any other highway albums.

This would be especially useful from the remote where I really don't seem to be able to recover this functionality. But even from the web page, it allows for search refinement.

OK, so here are the real questions.

1. Is this a FAQ?
2. Does anyone else think it's a good idea?
3. Has anyone implemented it, perhaps as a plugin?
4. Could it play with LazySearch??
5. Thoughts on partial keywords, especially with lazySearch? (especially useful from the remote, but not as important with lazySearch

[stuarth]

That's an interesting idea. I think I could definitely get that going with the Lazy Search plugin, and I'm sure it could go into the normal Search button handling, too. Two points, though:

1. It's going to slow down searching since you need to issue three separate searches (and the 'songs' one tends to be slow), unless there is some change to the schema for a combined content table that can be searched. Maybe you could do some clever joining in DBIx, but I'm not sure that would be any quicker depending on how clever the index processing in the database is. I'd need to experiment to know for sure.
2. I wonder how we would display results to the user so he didn't confuse the different types of result? Perhaps the results are a submenu with "Artists", "Albums", "Songs" that can be browsed into for the search results, with any submenu that was empty hidden (so if it only found matching artists then only the artists submenu would be present).

That's definitely worth pursuing, I think. I'll note it as an enhancement for my plugin.

Some additional ideas whilst I'm thinking about it:

• This would be a new top-level Lazy Search mode menu item (Artists, Albums, Genres, Tracks, Keywords - if you wanted a keyword search you can quickly get at that with the UP button on the menu).
• I'll need to think about how you would enter multiple words (ie for AND searches). Perhaps they could be separated by the '0' button that is used to represent spaces?
• It would be good if the 'results category' menu included a count of the number of each type that was found (eg "Artists (1)", "Albums (2)", "Songs (7)").
• I'm assuming this wouldn't be applied to genres.
• A set of checkbox preferences (on the plugin settings page), could be used to control which types would be searched for keywords - for example "Arists", "Albums", "Songs". If the user turned all of these off we could just hide the keywords search menu item altogether.

I will consider this for v2.x of the plugin (ie for SlimServer? 6.5) only, since the database work is considerably different between pre-6.5 and 6.5 and I don't really want to write it twice.

[stuarth]

I have made the following test releases: source:tags/2.2b1 and source:tags/2.2b2. Follow the links and read the changelogs for details.

[stuarth]

Just so that I don't lose it, here is a further description of the desired functionality from Rutt:

1. How to search

Search what the user tells you to search, but require all keywords to match in at least one field. User checks Artists, Albums, Songs, then you search a conjunction like:

(keyword0 in Artist OR keyword0 in Albums OR keyword0 in Songs) AND (keyword1 in Artist OR keyword1 in Albums OR keyword1 in Songs)

...

If the user doesn't check one of the boxes, don't add the related field to the disjuncts.

2. How to organize search results

I might not be really with the program here. I'd like to see all search results presented:

Artist

Album

Song

If the user wants song titles searched, then that's what should happen. Then you do get more hits than if they are not searched. I think that's the expected behavior. Perhaps you want to make it easy to do either kind of search from the remote?

I'd love to see words which match one of the keywords highlighted in the results. That would make it really easy to see.

There is a subtlety here that we might be glossing over. Just what are you going to show? I suppose it's all the

artists - albums - songs

such that there is a complete keyword match at any level. So searching for "The Beatles" gets you all their songs arranged in the above hierarchy.

Adding "Love" to the search adds the albums with songs that match, for example, "Meet the Beatles" because of "She Loves You" and "Revolver" because of "Love You To". But it still shows all the songs in those albums. Now I'm wondering if that is right. Perhaps the rule should be this:

Artist - Albums - Songs is a tree Prune the tree to show items which contain at least one keyword which matches AND everything below.

So with this rule, only the song that match the word "love" are shown because the search is incomplete at both the artist and album level. But if you searched for "Beatles Revolver" the match would be complete at the Artist - Album level and you'd show:

Beatles

Revolver

[All the songs]

I don't know. If one of these two options is easier to implement, I think that's the one you should do.

[stuarth]

Here's what I'm planning to do to address this:

1. Build more appropriate custom search results whilst scanning.
2. Change existing lazy song search.
3. Perform keyword search.
4. Display search results.

In turn...

1. Build more appropriate custom search results whilst scanning.

Currently the song table's CUSTOMSEARCH column is populated with just the lazy-encoded song title in this column. I intend to expand that with the following syntax:

<lazy song title>|<lazy song title><lazy song album title><lazy song artist>

This will be controlled by the three extra plugin settings that indicate which of these parts will be considered during keyword searches - if any of those parts is not enabled then it won't be added to the "right-hand" part of the CUSTOMSEARCH value.

Note that this means that if the user changes those preferences then they won't take effect unless a database clear-and-rescan is performed. I might improve that in the future as that's pretty inconvenient for large libraries, but that will do for now as I don't expect the user will want to change these settings once they've decided how they want the plugin to behave.

For example, the song "Gone Fishing" from "Auberge" by "Chris Rea" is currently encoded as:

0466303474464

I'm planning on now encoding that as follows, assuming that lazy keyword searching has been enabled for all of songs, albums and artists:

0466303474464|0466303474464028237430247470732

I expect this will make the song table lazification slower, but I'm not too concerned how slow as it only happens on rescan and happens in the background anyway.

The lazy encoding for albums and artists will be unchanged.

2. Change existing lazy song search

Having added to the CUSTOMSEARCH column for songs, the existing song search will be broken because it will now match any part of the encoded keywords - not what is required. Instead, the normal lazy song search will be modified to search for a match before the "|" separator. For example, searching for "GONE" would use the SQL search "%04663%|%". That should ensure that the results are as before and the keyword part of the CUSTOMSEARCH column is ignored.

3. Perform keyword search

Keyword searching is now rather simple and proceeds pretty much like the song search other than it's now searching the right-hand part of the CUSTOMSEARCH column. For example, searching for "FISH,REA" as keywords (expecting that to find songs about fishing by Chris Rea, or albums about Reading by Fish etc), would use the SQL WHERE clause as follows:

WHERE CUSTOMSEARCH LIKE "%|%03474%"
  AND CUSTOMSEARCH LIKE "%|%0732%"

ie all keywords must be found somewhere to the right of the "|" character and therefore as part of the encoded keywords.

4. Display search results

The keyword search will only have found songs (but might have found all songs on an album if the keywords were the album name, or all songs by the artist if the keywords were the artist name). Those results will be presented in the normal Squeezebox browse-type hierarchy:

• ARTIST
  • ALBUM
    • SONG

The user can navigate this hierarchy and use PLAY/ADD/INSERT as normal on any level in the hierarchy. For example, he can press PLAY on an album name and have all tracks for that album that were found for the keyword search played. However, because this is a normal browse display he'll be able to browse right into a track and hence access the whole album for playing/inserting/browsing from there anyway, so I think that's the most flexible.

I might enhance this a later stage to make it honour the "play other songs in album" server preference, but not initially (it isn't done for normal lazy song searches either).

Unfortunately, SlimServer? doesn't support much font control for a Squeezebox, and in particular won't support adding emphasis to individual words. This means I can't highlight matching keywords, I'm afraid. If that's important to you (and arguably it could be added to the standard Squeezebox search function as well), then I suggest you raise an enhancement request on  http://bugs.slimdevices.com and if that's implemented I could enhance the plugin to make use of it. If you do raise the bug then let me know in this ticket and I'll also vote for the SlimServer? enhancement.

[stuarth]

Just a quick update so you don't think I've disappeared...

I've just about got the queries for this done now - the delay has been due to trying to get them efficient and other things I've been busy with for the last couple of weeks.

I've a couple of minor things still to work out with them then it's just the browse navigation to do (which is reasonably straightforward, I think).

I'll post back more here when I've made some progress.

[stuarth]

OK, sorry for the delay. I've put together a new release here:

 http://hickinbottom.demon.co.uk/lazysearch/browser/tags/2.2b3

The description of the changes are available here:

 http://hickinbottom.demon.co.uk/lazysearch/browser/tags/2.2b3/LazySearch2.zip?rev=133

There is quite a bit of tidying-up still to do before a full release, but this should work well enough to test properly.

Can you give it a thorough run and see whether it's now doing what is required? Thanks.

[stuarth]

A small update, I've put together source:tags/2.2b4 that fixes a problem I noticed on my system where it would never find any tracks to lazify (so track and keyword searches wouldn't work).

You can get the new release here:

 http://hickinbottom.demon.co.uk/lazysearch/browser/tags/2.2b4

The description of the changes are available here:

 http://hickinbottom.demon.co.uk/lazysearch/changeset/146

I'm still waiting for any reports of success (or otherwise) of the latest beta. I'll probably release it anyway in a week or so.

[stuarth]

Implemented in v2.2 of the plugin.