Ironic Support Forum - Developer thoughts on adopting OpenMeta2010-09-08T22:43:39-07:00http://ironicsoftware.com/community/
Lussumo Vanilla & Feed Publisher
Developer thoughts on adopting OpenMetahttp://ironicsoftware.com/community/comments.php?DiscussionID=755&Focus=4561#Comment_45612009-03-10T14:13:25-07:002010-09-08T22:43:39-07:00Gotowhttp://ironicsoftware.com/community/account.php?u=578
I'm Jon Gotow, the developer of Default Folder X. There's been an ongoing debate about whether OpenMeta is a safe standard to adopt for associating tags with files. I'd like to address the issues ...
I've chosen to start out by recapping the issues that were raised in the "OpenMeta is a Hack" discussion so everyone can get up to speed without reading all 122 of those posts :) If you're already familiar with it, skip the first few postings here (I'm posting this in parts due to length limits on individual posts).]]>
Developer thoughts on adopting OpenMetahttp://ironicsoftware.com/community/comments.php?DiscussionID=755&Focus=4562#Comment_45622009-03-10T14:13:41-07:002010-09-08T22:43:39-07:00Gotowhttp://ironicsoftware.com/community/account.php?u=578
Some history:
Until now there has been no good, searchable, system-wide method of associating "tags" with files. Some people have been using a special character (such as '&') in ...
Until now there has been no good, searchable, system-wide method of associating "tags" with files. Some people have been using a special character (such as '&') in front of words in Spotlight comments to indicate that they're tags. Spotlight searches can then be performed with queries like "&photo &2008 &vacation &bahamas". However, using a special character is a nuisance, and this method isn't particularly convenient or reliable because Spotlight comments themselves have some significant drawbacks. Unfortunately Apple has not provided developers with any supported method to allow users to explicitly tag files.
As an alternative, Ironic Software proposed OpenMeta as an Open Standard for tag storage. The key is for all applications to use the same standard so tags can be read and written by any application and can be found using Spotlight. Tom Andersen of Ironic Software contributed open source code to implement the proposed standard so that developers could easily use the same code, making OpenMeta broadly available to the Mac community.]]>
Developer thoughts on adopting OpenMetahttp://ironicsoftware.com/community/comments.php?DiscussionID=755&Focus=4563#Comment_45632009-03-10T14:14:03-07:002010-09-08T22:43:39-07:00Gotowhttp://ironicsoftware.com/community/account.php?u=578
How OpenMeta stores tags:
The code that Tom contributed via OpenMeta uses a built-in feature of Mac OS X's HFS+ filesystem called Extended Attributes (which I'll refer to henceforth as 'xattrs'). ...
The code that Tom contributed via OpenMeta uses a built-in feature of Mac OS X's HFS+ filesystem called Extended Attributes (which I'll refer to henceforth as 'xattrs'). Xattrs allow developers to associate any arbitrary data with any given file. Xattrs have two parts: (1) a name and (2) the data.
Xattr names: Any given file on a disk can have many Xattrs, each with a different name. Names can be any text string. The name of an Xattr is used to access its data. There are a number of names currently in use in Mac OS X. Some of them are:
Xattr data: Each Xattr name has associated with it a blob of data. The format of this data is typically undocumented. Some of them are binary formats, some of them are text. All of the data formats for com.apple xattrs were designed by and are controlled by Apple, and as far as I know, none of them are public (Apple does not expect developers to read or write them.)]]>
Developer thoughts on adopting OpenMetahttp://ironicsoftware.com/community/comments.php?DiscussionID=755&Focus=4564#Comment_45642009-03-10T14:14:15-07:002010-09-08T22:43:39-07:00Gotowhttp://ironicsoftware.com/community/account.php?u=578
The OpenMeta debate:
The debate about OpenMeta has arisen because there's a key 'gotcha' in the way xattrs work with Spotlight (recall that the point of this exercise is to associate tags with ...
The debate about OpenMeta has arisen because there's a key 'gotcha' in the way xattrs work with Spotlight (recall that the point of this exercise is to associate tags with files AND be able to search for them with Spotlight). As of Mac OS 10.5, only xattrs whose names begin with "com.apple.metadata" are indexed and searchable with Spotlight.
As a matter of fact, Spotlight will index /any/ xattr that begins with "com.apple.metadata", not just those defined by Apple. Moreover, Developers have reverse engineered the format of the data and discovered that it is stored in plists, a standard Mac OS X data format. OpenMeta takes advantage of this by storing tags as a plist in an xattr named "com.apple.metadata:kOMUserTags", making them automatically searchable with Spotlight.
Now the 'gotcha': Programming conventions on Mac OS X dictate that when a name like "com.apple" is used, the owner of that internet domain has control over the named resources. This means that "com.apple.metadata" is "owned" by Apple. At any time, Apple could decide that the format of data blobs associated with those xattrs are no longer plists. Or it might make Spotlight stop searching them. Apple might do this as an optimization, to close a security hole, or for some other reason. The key point is that OpenMeta does not control the format of the blobs associated with com.apple.metadata tags. Apple does.
So OpenMeta is relying on Spotlight's indexing of com.apple.metadata tags, which hasn't been formally documented by Apple and uses a data format that isn't offically specified. Another developer, Michael Tsai, and I have both expressed worries about doing this. Apple could change this at any time. Is OpenMeta's tag data really safe?]]>
Developer thoughts on adopting OpenMetahttp://ironicsoftware.com/community/comments.php?DiscussionID=755&Focus=4565#Comment_45652009-03-10T14:14:49-07:002010-09-08T22:43:39-07:00Gotowhttp://ironicsoftware.com/community/account.php?u=578
The practical reality:
At first I was very skeptical about using OpenMeta because it saves data in a format not under OpenMeta's control. However, after Tom wrote some code to illustrate how ...
At first I was very skeptical about using OpenMeta because it saves data in a format not under OpenMeta's control. However, after Tom wrote some code to illustrate how widely the Xattrs are used, I've become convinced that putting data in an Xattr named com.apple.metadata:kOMUserTags isn't really much of danger. Here's why:
1) Xattrs are being used to store the data itself. This is a documented and supported mechanism. Xattrs are a completely generic storage system - blobs of data are simply filed under a specific name. HFS+ has no deeper knowledge of what the names mean or what the format is for the associated data, so the data can be stored and retrieved reliably regardless of what name or format is chosen.
2) OpenMeta's choice of Xattr names (com.apple.metadata:kOMApproved, kOMHidden, kOMUserTags, etc.) are different from names that Apple has chosen for its own xattrs, so data from OpenMeta will not be commingled with data from, say, Time Machine. The data is stored completely separately, so any changes Apple makes to their data formats will not directly affect OpenMeta's data and vice versa.
3) Because xattrs are already used frequently, it is unlikely that Apple will stop using them or that existing xattr entries will see their data formats change. Tom wrote an application that scanned every file on his Mac and listed the number of files having each unique xattr name it found. Here are the results:
com.apple.backup.SnapshotNumber had 5 entries com.apple.backup.SnapshotVersion had 5 entries com.apple.backupd.BackupMachineAddress had 1 entries com.apple.backupd.SnapshotCompletionDate had 5 entries com.apple.backupd.SnapshotStartDate had 5 entries com.apple.backupd.SnapshotState had 5 entries com.apple.backupd.SnapshotType had 5 entries com.apple.backupd.SnapshotVolumeFSEventStoreUUID had 15 entries com.apple.backupd.SnapshotVolumeLastFSEventID had 15 entries com.apple.backupd.SnapshotVolumeUUID had 15 entries com.apple.backupd.VolumeIsCaseSensitive had 15 entries com.apple.decmpfs had 236266 entries com.apple.diskimages.fsck had 32 entries com.apple.diskimages.recentcksum had 809 entries com.apple.FinderInfo had 449696 entries com.apple.metadata:_kTimeMachineNewestSnapshot had 4829796 entries com.apple.metadata:_kTimeMachineOldestSnapshot had 4817907 entries com.apple.metadata:backupdExcludeItem had 6 entries com.apple.metadata:com_apple_backup_excludeItem had 58 entries com.apple.metadata:FileSyncAgentExcludeItem had 9 entries com.apple.metadata:kBackupStampOM had 3626 entries com.apple.metadata:kMDItemFinderComment had 40867 entries com.apple.metadata:kMDItemIsScreenCapture had 518 entries com.apple.metadata:kMDItemLastUsedDate had 12 entries com.apple.metadata:kMDItemScreenCaptureType had 518 entries com.apple.metadata:kMDItemSupportFileType had 6413 entries com.apple.metadata:kMDItemUserTags had 6 entries com.apple.metadata:kMDItemWhereFroms had 34052 entries com.apple.metadata:kOMApproved had 463 entries com.apple.metadata:kOMBookmarks had 26 entries com.apple.metadata:kOMHidden had 65 entries com.apple.metadata:kOMInitialUse had 5 entries com.apple.metadata:kOMPalette had 410618 entries com.apple.metadata:kOMStarRating had 576 entries com.apple.metadata:kOMUserTagApplication had 10 entries com.apple.metadata:kOMUserTags had 14265 entries com.apple.metadata:kOMUserTagTime had 46 entries com.apple.quarantine had 1137753 entries com.apple.ResourceFork had 128252 entries com.apple.TextEncoding had 4069 entries com.apple.xcode.PlistType had 90 entries com.apple.XcodeGenerated had 485 entries fsid had 5 entries kOMApproved had 463 entries LAST_UPLOADED had 30 entries
You can see that Apple uses xattrs a lot for Safari (kMDItemWhereFroms), the Finder (kMDItemFinderComment), Time Machine (_kTimeMachineNewestSnapshot), disk images (com.apple.diskimages.recentcksum), and screen capture (kMDItemScreenCaptureType). To a lesser extent, they're used in many other places too.]]>
Developer thoughts on adopting OpenMetahttp://ironicsoftware.com/community/comments.php?DiscussionID=755&Focus=4566#Comment_45662009-03-10T14:14:56-07:002010-09-08T22:43:39-07:00Gotowhttp://ironicsoftware.com/community/account.php?u=578
These xattrs are attached to the files on your hard drive, and will remain on your hard disk even after a system upgrade. That means that any changes that come with Mac OS 10.6, 10.7, and beyond ...
4) As a worst-case scenario, it's possible that Apple might delete the OpenMeta xattr data altogether. (I view this as extremely unlikely, as Apple would have to explicitly look for and delete non-Apple uses of "com.apple.metadata" xattrs or target OpenMeta directly). But even in this scenario, OpenMeta has a built in backup system that stores copies of its data in the user's Library folder so that it can recover any lost xattrs. This was originally done to deal with Adobe Photoshop and a few other applications that use file I/O methods that don't preserve xattr data, but it also serves as a backup for any other loss of xattr data.]]>
Developer thoughts on adopting OpenMetahttp://ironicsoftware.com/community/comments.php?DiscussionID=755&Focus=4567#Comment_45672009-03-10T14:15:03-07:002010-09-08T22:43:39-07:00Gotowhttp://ironicsoftware.com/community/account.php?u=578
Conclusion:
I initially had reservations about the way OpenMeta stored tag data. However, given Apple's prevalent use of the com.apple.metadata labelling to store searchable data, the fact that ...
I initially had reservations about the way OpenMeta stored tag data. However, given Apple's prevalent use of the com.apple.metadata labelling to store searchable data, the fact that xattrs provide a well-defined and atomic storage mechanism, and the existence of a backup system built into OpenMeta, I think it's a reasonable and relatively safe mechanism.
Note that I'm saying "relatively safe." At it's core OpenMeta is still storing data using a name that's reserved for Apple, and is counting on Spotlight indexing that data, a capability that Apple could change or remove in the future.
While OpenMeta is a practical solution to a problem we have now, this should be something that evolves with additional support from Apple. To that end, Tom has filed a bug report requesting support for tags in the future, has contacted Apple via email to start a dialog, and has opened an incident with Apple DTS (Developer Technical Support) to get clarification from Apple on the best way to proceed.
OpenMeta will continue to evolve based on what Apple does and says in the future, but it's a good choice for the present, and something that I feel we can continue to work with in the long run. I plan on completing Default Folder X's support for OpenMeta and releasing it shortly.]]>
Developer thoughts on adopting OpenMetahttp://ironicsoftware.com/community/comments.php?DiscussionID=755&Focus=4568#Comment_45682009-03-10T14:39:58-07:002010-09-08T22:43:39-07:00BLUEFROGhttp://ironicsoftware.com/community/account.php?u=6
Jon:
Thanks for your thorough report. Since I am not a high-level programmer I don't know many of the details of the magic of OpenMeta but you laid it out in a very easy to understand way. ...
Thanks for your thorough report. Since I am not a high-level programmer I don't know many of the details of the magic of OpenMeta but you laid it out in a very easy to understand way.
Anxiously awaiting my OpenMeta version of DefaultFolder X.
Sincerely, Jim The Ironic Software Team]]>
Developer thoughts on adopting OpenMetahttp://ironicsoftware.com/community/comments.php?DiscussionID=755&Focus=4570#Comment_45702009-03-10T15:22:34-07:002010-09-08T22:43:39-07:00Jonohttp://ironicsoftware.com/community/account.php?u=251
Thanks for this. I'm glad you've taken a proper look at it, & come to the conclusion that you'll support it.
Like Jim, I'm eagerly awaiting an OM version of Default Folder X.
Like Jim, I'm eagerly awaiting an OM version of Default Folder X.]]>
Developer thoughts on adopting OpenMetahttp://ironicsoftware.com/community/comments.php?DiscussionID=755&Focus=4571#Comment_45712009-03-10T16:50:37-07:002010-09-08T22:43:39-07:00Michael Tsaihttp://ironicsoftware.com/community/account.php?u=278
Jon:
What are your thoughts on my suggestion to use org.openmeta as the primary storage location and to mirror the xattrs to com.apple.metadata in order to get the Spotlight indexing? I think that ...
What are your thoughts on my suggestion to use org.openmeta as the primary storage location and to mirror the xattrs to com.apple.metadata in order to get the Spotlight indexing? I think that this is would make OpenMeta more solid; we define the storage format, so it can remain stable. If Spotlight changes, we don't have to worry about what happens to com.apple.metadata; our data is pristine and we can simply mirror it in the new format (which might be an API rather than an xattr). The current OpenMeta approach, I think, is potentially painting us into a corner by needlessly combining the storage and the Spotlight indexing.
I see the plist backups as a solution to a different problem, and one that will become more rare as xattrs are better supported. Plus, as I mentioned in the other thread, I have concerns about the plist backups, e.g. complexity, efficiency, and how they handle multiple volumes/users.
Tom:
Could you let us know what Apple says in response to your DTS incident?]]>
Developer thoughts on adopting OpenMetahttp://ironicsoftware.com/community/comments.php?DiscussionID=755&Focus=4572#Comment_45722009-03-10T19:29:17-07:002009-03-10T19:43:21-07:00hmurchisonhttp://ironicsoftware.com/community/account.php?u=520
Well Jon just sold a few copies of Default Folder X and probably doesn't even know it. Fantastic post!
P.S - I just harassed the Devon Think guys once again. Hell if I'm spending $150 Dead ...
P.S - I just harassed the Devon Think guys once again. Hell if I'm spending $150 Dead Prez on their product I damn sure want it to give me some tagging latitude.
Jon...big big big news here. Your product and the future revisions to Leap and Yep will allow me to organize my files around data and not folder structure alone.
Plus I'm trying to work out a system that may work for my mothers sole practice law firm. OpenMeta and a sane tagging system could make case management so much easier.]]>
Developer thoughts on adopting OpenMetahttp://ironicsoftware.com/community/comments.php?DiscussionID=755&Focus=4578#Comment_45782009-03-11T01:11:20-07:002010-09-08T22:43:39-07:00rlfsosohttp://ironicsoftware.com/community/account.php?u=508
Hi Jon,
that is great news. I am anxiously to test out a (beta-)version of DFX with openmeta support. Some thing to put into consideration (as we did with spotlight-comment-tagging): ...
that is great news. I am anxiously to test out a (beta-)version of DFX with openmeta support. Some thing to put into consideration (as we did with spotlight-comment-tagging): autocompletion (where is it collected – I mean not app-centric but user-centric (opposed to system-centric)), recent tags…
looking forward to it,
Rolf]]>
Developer thoughts on adopting OpenMetahttp://ironicsoftware.com/community/comments.php?DiscussionID=755&Focus=4582#Comment_45822009-03-11T01:27:12-07:002010-09-08T22:43:39-07:00Jonohttp://ironicsoftware.com/community/account.php?u=251
Yea, I'd definitely like to see autocompletion of OpenMeta tags.
I know Jon Stovell of Tag Folders was looking into how/where OpenMeta stores recent tags, & it would be really useful if ...
I know Jon Stovell of Tag Folders was looking into how/where OpenMeta stores recent tags, & it would be really useful if Default Folder X (all OpenMeta apps) would autocomplete tags that weren't originally created with that app. (I.e. all OpenMeta apps autocomplete tags by reading the central OpenMeta tag store)]]>
Developer thoughts on adopting OpenMetahttp://ironicsoftware.com/community/comments.php?DiscussionID=755&Focus=4588#Comment_45882009-03-11T06:29:14-07:002010-09-08T22:43:39-07:00sherkanerhttp://ironicsoftware.com/community/account.php?u=52
Seconded!
And looking forward to DFX with OM tags ... thanks for the update.
And looking forward to DFX with OM tags ... thanks for the update.]]>
Developer thoughts on adopting OpenMetahttp://ironicsoftware.com/community/comments.php?DiscussionID=755&Focus=4594#Comment_45942009-03-11T08:46:07-07:002010-09-08T22:43:39-07:00Gotowhttp://ironicsoftware.com/community/account.php?u=578
Michael:
As you know, I was initially in favor of storing all tag data in an xattr named "org.openmeta:kOMUserTags" and mirroring that data into ...
As you know, I was initially in favor of storing all tag data in an xattr named "org.openmeta:kOMUserTags" and mirroring that data into "com.apple.metadata:kOMUserTags" attributes to get Spotlight indexing. However, after surveying Apple's use of extended attributes and thinking about the fact that xattrs are distinct and not attached to any other xattrs (so "com.apple.metadata:kMDItemFinderComment" has no relationship to "com.apple.metadata:kOMUserTags", for instance), I came to the conclusion that also storing tags in an org.openmeta:kOMUserTags xattr would result in extra data shuffling without a real benefit.
I don't view the "com.apple.metadata:kOMUserTags" xattr as being an inherently risky place to store tag data. Regardless of whether the xattr name starts with "com.apple" or not, Apple won't start intentionally deleting users' data. And as I said in my long explanation above, Apple is currently storing their own plist data in a lot of "com.apple.metadata" xattrs - this is persistent, long term data, so it's unlikely that the format will change. Apple would have to convert all of their own stored data and that wouldn't make sense, especially since plists are inherently extensible anyway (though if they do choose to change formats, we just have to deal with it via a converter app or other mechanism).
I agree that storing tags in an xattr with an "org.openmeta" prefix would be "cleaner" in principle, but in practice I don't think there's a real risk that requires it.]]>
Developer thoughts on adopting OpenMetahttp://ironicsoftware.com/community/comments.php?DiscussionID=755&Focus=4606#Comment_46062009-03-11T18:13:31-07:002010-09-08T22:43:39-07:00rickdudehttp://ironicsoftware.com/community/account.php?u=419
Jon, thank you so much for posting this explanation. Why Ironic couldn't do this before is one of the great unsolved mysteries of the universe, but I guess we can all move on now.
Developer thoughts on adopting OpenMetahttp://ironicsoftware.com/community/comments.php?DiscussionID=755&Focus=4609#Comment_46092009-03-11T21:43:29-07:002010-09-08T22:43:39-07:00sherkanerhttp://ironicsoftware.com/community/account.php?u=52
Well in their defense, rickdude, this is mostly what Tom has posted here before. But it is certainly good to hear that there is consensus in at least one important part of the wider developer ...
Developer thoughts on adopting OpenMetahttp://ironicsoftware.com/community/comments.php?DiscussionID=755&Focus=4614#Comment_46142009-03-12T06:17:57-07:002009-03-12T06:23:02-07:00tom.andersenhttp://ironicsoftware.com/community/account.php?u=3
On the tech support incident I filed: I requested that they document this 'spotlight importing xattr'. I also included information on open meta, etc. So far just the generic 'we will get back to you ...
I was hoping that most of the discussion on actual API, etc (which this discussion mostly isn't) would take place on the OpenMeta source site, where there is a google group:
--Tom]]>
Developer thoughts on adopting OpenMetahttp://ironicsoftware.com/community/comments.php?DiscussionID=755&Focus=4632#Comment_46322009-03-13T02:07:42-07:002009-03-13T02:23:01-07:00jlnhttp://ironicsoftware.com/community/account.php?u=515
Ah... How exciting it is to read this while having my morning coffee !
http://stclairsoft.com/blog/2009/03/13/openmeta-progress/
Thank you very much !
Julien
http://stclairsoft.com/blog/2009/03/13/openmeta-progress/
Thank you very much !
Julien]]>
Developer thoughts on adopting OpenMetahttp://ironicsoftware.com/community/comments.php?DiscussionID=755&Focus=4689#Comment_46892009-03-18T06:01:49-07:002010-09-08T22:43:39-07:00tom.andersenhttp://ironicsoftware.com/community/account.php?u=3
Well - Apple got back to me about the tech support incident. They basically declined to offer an answer, and refunded my tech support incident. They just said to keep an eye on Bug ID ...
They also said to file an enhancement request in addition to the bug, so that is one thing that I will do.