Foo Bulk Tagger (Foobar plugin)

Seltzer · 07-03-2010, 07:44 AM

Currently, processing a new album for me involves using Tag and Rename to handle most things, Album Art Downloader to find the album art, Mp3Tag to embed it in all of the mp3s and tore's genre script to set genres. And there are misc. tasks like cleaning out bs like m3u files etc. Since this process is becoming quite tedious, I've decided I'm going to write a Foobar plugin in C++ to help out.

My original premise for the Foobar plugin was to fetch tags from Musicbrainz and apply them to albums on a batch basis. I know there are a few in existence already but this one would work exactly as I wanted; i.e. it would preview all changes to existing tag data, alert me about radical suggested changes (which could be indicative of a really badly tagged album) and above all, make the whole process highly customisable.

Then I started thinking about what else it could do and I came up with this feature set. Some of these features are already implemented in other programs I use, but it's easier to have them all in one place.

* Search for album art and embed it in files:
This is mainly a matter of finding the most reputable AA sources and perhaps using an image recognition algorithm to pick out the most common result.

* Warnings about:
Overly large album art, possibly incomplete albums and low bitrates... list goes on.

* Erase various standard/custom ID3 tags -
Some of these custom tags *are* MusicBrainz tags and so this would perhaps best be performed after trying to match music with a MusicBrainz release (as MusicBrainz ID tags help with this process). I would use this mainly to erase garbage metadata like torrent tracker URLs in comments etc.

* Check style conformance of tags:
One example of style might be punctuation - the user would be able to set up capitalisation rules such as "the first letter of 'of' should never be capitalised in a song title unless it's the first word". A rule like this should not be applied without previewing changes, as punctuation rules especially are not black and white.

* A feature to generate filenames from tags:
This has of course been done in many programs ad nauseam using masking (including Tag & Rename) but this would allow for custom mappings; i.e. colons cannot be used in Windows filenames, therefore colons in tags require a custom mapping to some other character.

* Garbage removal feature:
For removing unwanted files from albums (based on file extension)

* Copy/swap tags feature with rules:
I always copy Artist -> Album Artist unless it's a compilation for example, in which case, Album Artist should be "Various Artists"

* Genre Finder:
Same idea as tore's script except it would be based on using a data source API if possible, rather than web scraping. It would be nice to allow for rules too; e.g. if a band is "frostbitten dire wolf industrial black metal", they are also black metal, industrial and metal. And I think that "Abstract" is bollocks as a genre descriptor, but discogs.com seems to like it... so some genre exclusion rule would be handy there. I've hacked these things into that script in the past btw.

* Compilation Finder:
I dislike having single songs floating around but I hate downloading whole studio albums just to have a single song (especially when it's a one hit wonder and the rest of the album is shite). It seems that the ideal option is to delete the orphan song and download a compilation which includes that song. A useful feature would be to automatically find the compilation which maximises the coverage of single tracks.

So I'll reiterate that I know much of this has been done before. It'd just be nice to have it all encapsulated in one batch process.

Guybrush · 07-03-2010, 08:22 AM

Oh man, Seltzer! If you could make something like that that I could use, man .. That would be awesome

Sorry if I sound a little *** here, but you would be my hero for sure!

Like you, I spend quite a bit of time meticulously tagging my music and getting album art etc.

Some suggestions I can think of though I would likely come up with more, but .. The ability to save album art in the folder containing the files with a custom name such as "folder" would be nice. I prefer it to embedding.

Although it was way beyond my current abilities as a programmer, I thought once about the possibility of fetching information from wikipedia. It would definetly be a bit tricky. I figured a possible way to get it though. Musicbrainz contains information about links and also usually a link to a band's wikipedia article if it has one.

You could use that information perhaps to construct an URL for lookup on dbpedia. I've used U2 as an example .. A problem with looking up wikipedia info is of course that you don't know if it should be "U2" or "U2 (band)", but a musicbrainz query could answer that question and then you could construct a dbpedia link. Dbpedia is a database dump from wikipedia which you can get access to and fetch data from.

Check this out for an example of information you could fetch about U2 -> About: U2

There may not be anything particularly interesting in there, but they often do have information on the origin of the band at least which is something I don't get from discogs. Also - information about the musicians playing on that album is something one can typically find on wikipedia, but not on discogs or musicbrainz I believe. For example, I would find it interesting that Al Di Meola plays guitar on Lenny White's solo release Venusian Summer or that Chick Corea plays keyboards on a Bèla Fleck and the Flecktones live album, so that kind of information is valuable to me, but these days it's real tedious getting that info and writing it to tags.

Anyways, I don't use musicbrainz much - I use discogs instead and the reason is really that there's a rather excellent tagging script for foobar which uses the discogs database (foo_discogs). It presents search results for releases for you so that you can review the search results and choose the one you think is correct. Then if you see the track listing or something else is slightly different from your files, you can go back and choose another one in a couple of clicks. It also lets you choose which tags to write from a range of choices, for example the members of the band, the format of the release (CD, Vinyl, etc), discogs genre and styles which I greatly appreciate and so on. You should check it out for inspiration if you haven't.

foo_discogs - Hydrogenaudio Forums

(it can also download album / sleeve / artist art)

Seltzer · 07-03-2010, 09:18 AM

Hmm I didn't know about dbpedia but it sounds like it could definitely be useful. Anything that provides an interface and supports queries is preferable to web scraping. I'm not settled on MusicBrainz as I haven't done much research into the range of music databases available - I just figured it must be pretty decent if last.fm relies on it. I'll definitely look into foo_discogs too.

This is pretty ambitious, I hope I can get started on it soon. And all suggestions/recommendations are welcome

Guybrush · 07-03-2010, 10:02 AM

It would also be cool if one could tag songs with singles, for example if those songs were singles from an album

Then it would be simple to for example set up an autoplaylist for singles if you for some reason only want to listen to popular music.

Seltzer · 07-06-2010, 03:07 AM

Development has begun!

I'm already proficient in C++ and I've been familiarising myself with the Foobar/MusicBrainz APIs today. The only learning curve will be using C++ to create user-interfaces as I normally use C++ for the guts of a system and another language like Python for creating a GUI to run over the top of it.

I've had a cursory glance at foo_discogs and it looks quite good for tagging single albums but I can't see a way to use it to efficiently tag albums in bulk (unless I'm missing something) - it's not automated enough for that anyway. It's fairly customisable with its tag mappings but there doesn't seem to be a way to map both <GENRE> and <STYLE> to <GENRE>, let alone add unwanted genres to a taboo-list.

So I guess the primary difference is that my plugin will be intended for bulk tagging of albums and automate the process as much as possible. This means it'll require some clever heuristics for matching albums in Foobar to releases in some music database. Some of the more difficult cases include albums which are devoid of any tag data, tags with funny characters, albums with 1000 versions, releases which appear as several albums in Foobar due to bad tagging, and albums whose tags are plainly wrong.

noise · 07-06-2010, 05:07 AM

what database do you intend to use?

Guybrush · 07-06-2010, 05:52 AM

Brilliant Seltzer

And I think you're right about foo_discogs. I love it, but it's definetly not a bulk tagger.

Something that could be cool would be custom field tagger. For example you could define some custom fields and possible values they could have in a config page (a bit like foo_quicktagger?) and then when you bulk tag something, you can choose from dropdown menus or something which custom tags you want to add and what values they should have, like a custom value or a predefined one.

Another idea at least

That way, I could bulk tag what I know to be studio albums and also add the custom tag I use to indicate studio albums.

edit :

Is this discussion perhaps more appropriate for the foobar thread? Or it's own thread, Seltzer?

Seltzer · 07-07-2010, 01:39 AM

Quote:

Originally Posted by noise

what database do you intend to use?

At the moment I'm using MusicBrainz for most of the data acquisition and discogs for style/genre. It was a bit of a pain to integrate the MusicBrainz code since it seems to have been geared towards Linux (and I'm using Win7) but it's working now.

Quote:

Originally Posted by tore

Brilliant Seltzer

And I think you're right about foo_discogs. I love it, but it's definetly not a bulk tagger.

Something that could be cool would be custom field tagger. For example you could define some custom fields and possible values they could have in a config page (a bit like foo_quicktagger?) and then when you bulk tag something, you can choose from dropdown menus or something which custom tags you want to add and what values they should have, like a custom value or a predefined one.

Another idea at least

That way, I could bulk tag what I know to be studio albums and also add the custom tag I use to indicate studio albums.

edit :

Is this discussion perhaps more appropriate for the foobar thread? Or it's own thread, Seltzer?

I intend to allow for the definition of standard and custom tags from user-selected values as well as combinations of existing tags (maybe using masks). And it's a good idea to add provision for limiting tags to a set of legal values - it's the natural complement to being able to ban certain values; i.e. banning Abstract as a genre in my case

And I've moved all of this here so that it doesn't clog up the music sorting / Foobar threads. Foo Bulk Tagger is a pretty crappy name for the plugin but I'll change it later

Seltzer · 12-22-2010, 08:06 PM

Well, here's a screenshot of my plugin in its current (and very early state). I'm focussing on functionality at the moment, hence the UI is a bit cruddy - gotta love the Do other stuff button

This is the main (and incomplete) dialog where tags are fetched from MusicBrainz and applied to your music. The screenshot indicates how I spawned the dialog, through the right click context menu of selected music in my library.

Its first job is to split up the selected music into 'units', each of which should be matched with a MusicBrainz release, and displaying them as a tree. Currently it's doing this in a fairly naive manner by sorting by artist and then by album (or by directory if the album tag is missing). There are obvious potential flaws with this approach and I can alter this so that it divides the selection according to the current library view (i.e. Folder View, Artist, Artist/Album etc.)... or according to something completely different. How I should do this really depends on other things (such as the heuristic mentioned below), so I haven't pre-emptively implemented anything major yet.

When selecting one of these units, the current metadata is displayed - this works very much like the Foobar Properties screen and allows you to write new tags to the tracks.

Now I'm starting to work on the heuristic for matching the units with MusicBrainz releases. It occurs to me that I can use audio fingerprinting for popular releases and my own heuristic wherever fingerprinting fails. And if the plugin is unsure about which possible release to choose, it'll present the the user with the candidate releases (and affixed relevant data for each one), and the user will be able to pick one... ideally this will rarely happen.

This dialog is the crux of the plugin but there will of course be other dialogs, most of which will be in support of this one; e.g. a dialog which allows you to specify tag mappings for mapping the MusicBrainz data to track metadata.

Guybrush · 12-23-2010, 02:33 AM

Oh man, Seltzer! Sweet job

I'm so much looking forward to trying this out. Lemme know if you need some alpha/beta testing done.

07-03-2010, 07:44 AM	#1 (permalink)
Seltzer Fish in the percolator! Join Date: Dec 2005 Location: Hobbit Land NZ Posts: 2,870	Foo Bulk Tagger (Foobar plugin) Currently, processing a new album for me involves using Tag and Rename to handle most things, Album Art Downloader to find the album art, Mp3Tag to embed it in all of the mp3s and tore's genre script to set genres. And there are misc. tasks like cleaning out bs like m3u files etc. Since this process is becoming quite tedious, I've decided I'm going to write a Foobar plugin in C++ to help out. My original premise for the Foobar plugin was to fetch tags from Musicbrainz and apply them to albums on a batch basis. I know there are a few in existence already but this one would work exactly as I wanted; i.e. it would preview all changes to existing tag data, alert me about radical suggested changes (which could be indicative of a really badly tagged album) and above all, make the whole process highly customisable. Then I started thinking about what else it could do and I came up with this feature set. Some of these features are already implemented in other programs I use, but it's easier to have them all in one place. * Search for album art and embed it in files: This is mainly a matter of finding the most reputable AA sources and perhaps using an image recognition algorithm to pick out the most common result. * Warnings about: Overly large album art, possibly incomplete albums and low bitrates... list goes on. * Erase various standard/custom ID3 tags - Some of these custom tags are MusicBrainz tags and so this would perhaps best be performed after trying to match music with a MusicBrainz release (as MusicBrainz ID tags help with this process). I would use this mainly to erase garbage metadata like torrent tracker URLs in comments etc. * Check style conformance of tags: One example of style might be punctuation - the user would be able to set up capitalisation rules such as "the first letter of 'of' should never be capitalised in a song title unless it's the first word". A rule like this should not be applied without previewing changes, as punctuation rules especially are not black and white. * A feature to generate filenames from tags: This has of course been done in many programs ad nauseam using masking (including Tag & Rename) but this would allow for custom mappings; i.e. colons cannot be used in Windows filenames, therefore colons in tags require a custom mapping to some other character. * Garbage removal feature: For removing unwanted files from albums (based on file extension) * Copy/swap tags feature with rules: I always copy Artist -> Album Artist unless it's a compilation for example, in which case, Album Artist should be "Various Artists" * Genre Finder: Same idea as tore's script except it would be based on using a data source API if possible, rather than web scraping. It would be nice to allow for rules too; e.g. if a band is "frostbitten dire wolf industrial black metal", they are also black metal, industrial and metal. And I think that "Abstract" is bollocks as a genre descriptor, but discogs.com seems to like it... so some genre exclusion rule would be handy there. I've hacked these things into that script in the past btw. * Compilation Finder: I dislike having single songs floating around but I hate downloading whole studio albums just to have a single song (especially when it's a one hit wonder and the rest of the album is shite). It seems that the ideal option is to delete the orphan song and download a compilation which includes that song. A useful feature would be to automatically find the compilation which maximises the coverage of single tracks. So I'll reiterate that I know much of this has been done before. It'd just be nice to have it all encapsulated in one batch process. __________________ In the Court of King Crimson \| Last.fm \| RYM

07-03-2010, 08:22 AM	#2 (permalink)
Guybrush Juicious Maximus III Join Date: Nov 2008 Location: Scabb Island Posts: 6,525	Oh man, Seltzer! If you could make something like that that I could use, man .. That would be awesome Sorry if I sound a little *** here, but you would be my hero for sure! Like you, I spend quite a bit of time meticulously tagging my music and getting album art etc. Some suggestions I can think of though I would likely come up with more, but .. The ability to save album art in the folder containing the files with a custom name such as "folder" would be nice. I prefer it to embedding. Although it was way beyond my current abilities as a programmer, I thought once about the possibility of fetching information from wikipedia. It would definetly be a bit tricky. I figured a possible way to get it though. Musicbrainz contains information about links and also usually a link to a band's wikipedia article if it has one. You could use that information perhaps to construct an URL for lookup on dbpedia. I've used U2 as an example .. A problem with looking up wikipedia info is of course that you don't know if it should be "U2" or "U2 (band)", but a musicbrainz query could answer that question and then you could construct a dbpedia link. Dbpedia is a database dump from wikipedia which you can get access to and fetch data from. Check this out for an example of information you could fetch about U2 -> About: U2 There may not be anything particularly interesting in there, but they often do have information on the origin of the band at least which is something I don't get from discogs. Also - information about the musicians playing on that album is something one can typically find on wikipedia, but not on discogs or musicbrainz I believe. For example, I would find it interesting that Al Di Meola plays guitar on Lenny White's solo release Venusian Summer or that Chick Corea plays keyboards on a Bèla Fleck and the Flecktones live album, so that kind of information is valuable to me, but these days it's real tedious getting that info and writing it to tags. Anyways, I don't use musicbrainz much - I use discogs instead and the reason is really that there's a rather excellent tagging script for foobar which uses the discogs database (foo_discogs). It presents search results for releases for you so that you can review the search results and choose the one you think is correct. Then if you see the track listing or something else is slightly different from your files, you can go back and choose another one in a couple of clicks. It also lets you choose which tags to write from a range of choices, for example the members of the band, the format of the release (CD, Vinyl, etc), discogs genre and styles which I greatly appreciate and so on. You should check it out for inspiration if you haven't. foo_discogs - Hydrogenaudio Forums (it can also download album / sleeve / artist art) __________________ Something Completely Different

07-03-2010, 09:18 AM	#3 (permalink)
Seltzer Fish in the percolator! Join Date: Dec 2005 Location: Hobbit Land NZ Posts: 2,870	Hmm I didn't know about dbpedia but it sounds like it could definitely be useful. Anything that provides an interface and supports queries is preferable to web scraping. I'm not settled on MusicBrainz as I haven't done much research into the range of music databases available - I just figured it must be pretty decent if last.fm relies on it. I'll definitely look into foo_discogs too. This is pretty ambitious, I hope I can get started on it soon. And all suggestions/recommendations are welcome __________________ In the Court of King Crimson \| Last.fm \| RYM

07-03-2010, 10:02 AM	#4 (permalink)
Guybrush Juicious Maximus III Join Date: Nov 2008 Location: Scabb Island Posts: 6,525	It would also be cool if one could tag songs with singles, for example if those songs were singles from an album Then it would be simple to for example set up an autoplaylist for singles if you for some reason only want to listen to popular music. __________________ Something Completely Different

07-06-2010, 03:07 AM	#5 (permalink)
Seltzer Fish in the percolator! Join Date: Dec 2005 Location: Hobbit Land NZ Posts: 2,870	Development has begun! I'm already proficient in C++ and I've been familiarising myself with the Foobar/MusicBrainz APIs today. The only learning curve will be using C++ to create user-interfaces as I normally use C++ for the guts of a system and another language like Python for creating a GUI to run over the top of it. I've had a cursory glance at foo_discogs and it looks quite good for tagging single albums but I can't see a way to use it to efficiently tag albums in bulk (unless I'm missing something) - it's not automated enough for that anyway. It's fairly customisable with its tag mappings but there doesn't seem to be a way to map both <GENRE> and <STYLE> to <GENRE>, let alone add unwanted genres to a taboo-list. So I guess the primary difference is that my plugin will be intended for bulk tagging of albums and automate the process as much as possible. This means it'll require some clever heuristics for matching albums in Foobar to releases in some music database. Some of the more difficult cases include albums which are devoid of any tag data, tags with funny characters, albums with 1000 versions, releases which appear as several albums in Foobar due to bad tagging, and albums whose tags are plainly wrong. __________________ In the Court of King Crimson \| Last.fm \| RYM

07-06-2010, 05:07 AM	#6 (permalink)
noise thirsty ears Join Date: Sep 2009 Location: Boulder Posts: 742	what database do you intend to use? __________________ my flac collection

07-06-2010, 05:52 AM	#7 (permalink)
Guybrush Juicious Maximus III Join Date: Nov 2008 Location: Scabb Island Posts: 6,525	Brilliant Seltzer And I think you're right about foo_discogs. I love it, but it's definetly not a bulk tagger. Something that could be cool would be custom field tagger. For example you could define some custom fields and possible values they could have in a config page (a bit like foo_quicktagger?) and then when you bulk tag something, you can choose from dropdown menus or something which custom tags you want to add and what values they should have, like a custom value or a predefined one. Another idea at least That way, I could bulk tag what I know to be studio albums and also add the custom tag I use to indicate studio albums. edit : Is this discussion perhaps more appropriate for the foobar thread? Or it's own thread, Seltzer? __________________ Something Completely Different

12-22-2010, 08:06 PM	#9 (permalink)
Seltzer Fish in the percolator! Join Date: Dec 2005 Location: Hobbit Land NZ Posts: 2,870	Well, here's a screenshot of my plugin in its current (and very early state). I'm focussing on functionality at the moment, hence the UI is a bit cruddy - gotta love the Do other stuff button This is the main (and incomplete) dialog where tags are fetched from MusicBrainz and applied to your music. The screenshot indicates how I spawned the dialog, through the right click context menu of selected music in my library. Its first job is to split up the selected music into 'units', each of which should be matched with a MusicBrainz release, and displaying them as a tree. Currently it's doing this in a fairly naive manner by sorting by artist and then by album (or by directory if the album tag is missing). There are obvious potential flaws with this approach and I can alter this so that it divides the selection according to the current library view (i.e. Folder View, Artist, Artist/Album etc.)... or according to something completely different. How I should do this really depends on other things (such as the heuristic mentioned below), so I haven't pre-emptively implemented anything major yet. When selecting one of these units, the current metadata is displayed - this works very much like the Foobar Properties screen and allows you to write new tags to the tracks. Now I'm starting to work on the heuristic for matching the units with MusicBrainz releases. It occurs to me that I can use audio fingerprinting for popular releases and my own heuristic wherever fingerprinting fails. And if the plugin is unsure about which possible release to choose, it'll present the the user with the candidate releases (and affixed relevant data for each one), and the user will be able to pick one... ideally this will rarely happen. This dialog is the crux of the plugin but there will of course be other dialogs, most of which will be in support of this one; e.g. a dialog which allows you to specify tag mappings for mapping the MusicBrainz data to track metadata. __________________ In the Court of King Crimson \| Last.fm \| RYM

12-23-2010, 02:33 AM	#10 (permalink)
Guybrush Juicious Maximus III Join Date: Nov 2008 Location: Scabb Island Posts: 6,525	Oh man, Seltzer! Sweet job I'm so much looking forward to trying this out. Lemme know if you need some alpha/beta testing done. __________________ Something Completely Different