Not signed in (Sign In)

Home | Yep | Leap | About us | Support | Buy

Vanilla 1.1.4 is a product of Lussumo. More Information: Documentation, Community Support.

  • Search


  • Adding Images in Posts

      In order to add an image to a discussion, you need to first have a url to the image. This means that you have to upload the image to somewhere on the internet.


      One easy way to do this is to use www. www.postimage.com.


      You need to upload your image (remember anyone can see the image by reading a posting here, or other methods). After you upload the image you need to copy the 'html' line and paste it into a discussion comment.


      Here is the format of an image in html. (only needed if you do not use postimage.org)


      <img src="http://site.com/pictlink.jpg"/>


      This forum does not use BBCode - rather it uses html.


      --Tom

  • Adding Link

      You can add a link to a posting: Copy the template below, and replace xxx with the URL, and yyy with what you want to call it (which can be the same as the URL).

      <a href="xxx">yyy</a>

      --Tom

Ironic's community site has been set up for support and discussion. Please feel free to join and offer your ideas and solutions. Anyone can add a comment or question to a discussion topic that is already listed. In order to start a new discussion, you need to be member. Membership is easy.

OpenMeta: [Closed] Developer thoughts on adopting OpenMeta

Bottom of Page

1 to 20 of 20

  1.  
    • CommentAuthorGotow
    • CommentTimeMar 10th 2009
     
    I'm Jon Gotow, the developer of Default Folder X. There's been an ongoing debate about whether OpenMeta is a safe standard to adopt for associating tags with files. I'd like to address the issues that have been under discussion and give you my (evolving) take on this as a developer who's likely to adopt OpenMeta.

    I've chosen to start out by recapping the issues that were raised in the "OpenMeta is a Hack" discussion so everyone can get up to speed without reading all 122 of those posts :) If you're already familiar with it, skip the first few postings here (I'm posting this in parts due to length limits on individual posts).
    • CommentAuthorGotow
    • CommentTimeMar 10th 2009
     
    Some history:

    Until now there has been no good, searchable, system-wide method of associating "tags" with files. Some people have been using a special character (such as '&') in front of words in Spotlight comments to indicate that they're tags. Spotlight searches can then be performed with queries like "&photo &2008 &vacation &bahamas". However, using a special character is a nuisance, and this method isn't particularly convenient or reliable because Spotlight comments themselves have some significant drawbacks. Unfortunately Apple has not provided developers with any supported method to allow users to explicitly tag files.

    As an alternative, Ironic Software proposed OpenMeta as an Open Standard for tag storage. The key is for all applications to use the same standard so tags can be read and written by any application and can be found using Spotlight. Tom Andersen of Ironic Software contributed open source code to implement the proposed standard so that developers could easily use the same code, making OpenMeta broadly available to the Mac community.
    • CommentAuthorGotow
    • CommentTimeMar 10th 2009
     
    How OpenMeta stores tags:

    The code that Tom contributed via OpenMeta uses a built-in feature of Mac OS X's HFS+ filesystem called Extended Attributes (which I'll refer to henceforth as 'xattrs'). Xattrs allow developers to associate any arbitrary data with any given file. Xattrs have two parts: (1) a name and (2) the data.

    Xattr names: Any given file on a disk can have many Xattrs, each with a different name. Names can be any text string. The name of an Xattr is used to access its data. There are a number of names currently in use in Mac OS X. Some of them are:

    com.apple.TextEncoding
    com.apple.backup.SnapshotNumber
    com.apple.backupd.SnapshotCompletionDate
    com.apple.diskimages.recentcksum
    com.apple.FinderInfo
    com.apple.metadata:_kTimeMachineNewestSnapshot
    com.apple.metadata:kMDItemFinderComment
    com.apple.metadata:kMDItemIsScreenCapture

    Xattr data: Each Xattr name has associated with it a blob of data. The format of this data is typically undocumented. Some of them are binary formats, some of them are text. All of the data formats for com.apple xattrs were designed by and are controlled by Apple, and as far as I know, none of them are public (Apple does not expect developers to read or write them.)
    • CommentAuthorGotow
    • CommentTimeMar 10th 2009
     
    The OpenMeta debate:

    The debate about OpenMeta has arisen because there's a key 'gotcha' in the way xattrs work with Spotlight (recall that the point of this exercise is to associate tags with files AND be able to search for them with Spotlight). As of Mac OS 10.5, only xattrs whose names begin with "com.apple.metadata" are indexed and searchable with Spotlight.

    As a matter of fact, Spotlight will index /any/ xattr that begins with "com.apple.metadata", not just those defined by Apple. Moreover, Developers have reverse engineered the format of the data and discovered that it is stored in plists, a standard Mac OS X data format. OpenMeta takes advantage of this by storing tags as a plist in an xattr named "com.apple.metadata:kOMUserTags", making them automatically searchable with Spotlight.

    Now the 'gotcha': Programming conventions on Mac OS X dictate that when a name like "com.apple" is used, the owner of that internet domain has control over the named resources. This means that "com.apple.metadata" is "owned" by Apple. At any time, Apple could decide that the format of data blobs associated with those xattrs are no longer plists. Or it might make Spotlight stop searching them. Apple might do this as an optimization, to close a security hole, or for some other reason. The key point is that OpenMeta does not control the format of the blobs associated with com.apple.metadata tags. Apple does.

    So OpenMeta is relying on Spotlight's indexing of com.apple.metadata tags, which hasn't been formally documented by Apple and uses a data format that isn't offically specified. Another developer, Michael Tsai, and I have both expressed worries about doing this. Apple could change this at any time. Is OpenMeta's tag data really safe?
    • CommentAuthorGotow
    • CommentTimeMar 10th 2009
     
    The practical reality:

    At first I was very skeptical about using OpenMeta because it saves data in a format not under OpenMeta's control. However, after Tom wrote some code to illustrate how widely the Xattrs are used, I've become convinced that putting data in an Xattr named com.apple.metadata:kOMUserTags isn't really much of danger. Here's why:

    1) Xattrs are being used to store the data itself. This is a documented and supported mechanism. Xattrs are a completely generic storage system - blobs of data are simply filed under a specific name. HFS+ has no deeper knowledge of what the names mean or what the format is for the associated data, so the data can be stored and retrieved reliably regardless of what name or format is chosen.

    2) OpenMeta's choice of Xattr names (com.apple.metadata:kOMApproved, kOMHidden, kOMUserTags, etc.) are different from names that Apple has chosen for its own xattrs, so data from OpenMeta will not be commingled with data from, say, Time Machine. The data is stored completely separately, so any changes Apple makes to their data formats will not directly affect OpenMeta's data and vice versa.

    3) Because xattrs are already used frequently, it is unlikely that Apple will stop using them or that existing xattr entries will see their data formats change. Tom wrote an application that scanned every file on his Mac and listed the number of files having each unique xattr name it found. Here are the results:

    com.apple.backup.SnapshotNumber had 5 entries
    com.apple.backup.SnapshotVersion had 5 entries
    com.apple.backupd.BackupMachineAddress had 1 entries
    com.apple.backupd.SnapshotCompletionDate had 5 entries
    com.apple.backupd.SnapshotStartDate had 5 entries
    com.apple.backupd.SnapshotState had 5 entries
    com.apple.backupd.SnapshotType had 5 entries
    com.apple.backupd.SnapshotVolumeFSEventStoreUUID had 15 entries
    com.apple.backupd.SnapshotVolumeLastFSEventID had 15 entries
    com.apple.backupd.SnapshotVolumeUUID had 15 entries
    com.apple.backupd.VolumeIsCaseSensitive had 15 entries
    com.apple.decmpfs had 236266 entries
    com.apple.diskimages.fsck had 32 entries
    com.apple.diskimages.recentcksum had 809 entries
    com.apple.FinderInfo had 449696 entries
    com.apple.metadata:_kTimeMachineNewestSnapshot had 4829796 entries
    com.apple.metadata:_kTimeMachineOldestSnapshot had 4817907 entries
    com.apple.metadata:backupdExcludeItem had 6 entries
    com.apple.metadata:com_apple_backup_excludeItem had 58 entries
    com.apple.metadata:FileSyncAgentExcludeItem had 9 entries
    com.apple.metadata:kBackupStampOM had 3626 entries
    com.apple.metadata:kMDItemFinderComment had 40867 entries
    com.apple.metadata:kMDItemIsScreenCapture had 518 entries
    com.apple.metadata:kMDItemLastUsedDate had 12 entries
    com.apple.metadata:kMDItemScreenCaptureType had 518 entries
    com.apple.metadata:kMDItemSupportFileType had 6413 entries
    com.apple.metadata:kMDItemUserTags had 6 entries
    com.apple.metadata:kMDItemWhereFroms had 34052 entries
    com.apple.metadata:kOMApproved had 463 entries
    com.apple.metadata:kOMBookmarks had 26 entries
    com.apple.metadata:kOMHidden had 65 entries
    com.apple.metadata:kOMInitialUse had 5 entries
    com.apple.metadata:kOMPalette had 410618 entries
    com.apple.metadata:kOMStarRating had 576 entries
    com.apple.metadata:kOMUserTagApplication had 10 entries
    com.apple.metadata:kOMUserTags had 14265 entries
    com.apple.metadata:kOMUserTagTime had 46 entries
    com.apple.quarantine had 1137753 entries
    com.apple.ResourceFork had 128252 entries
    com.apple.TextEncoding had 4069 entries
    com.apple.xcode.PlistType had 90 entries
    com.apple.XcodeGenerated had 485 entries
    fsid had 5 entries
    kOMApproved had 463 entries
    LAST_UPLOADED had 30 entries

    You can see that Apple uses xattrs a lot for Safari (kMDItemWhereFroms), the Finder (kMDItemFinderComment), Time Machine (_kTimeMachineNewestSnapshot), disk images (com.apple.diskimages.recentcksum), and screen capture (kMDItemScreenCaptureType). To a lesser extent, they're used in many other places too.
    • CommentAuthorGotow
    • CommentTimeMar 10th 2009
     
    These xattrs are attached to the files on your hard drive, and will remain on your hard disk even after a system upgrade. That means that any changes that come with Mac OS 10.6, 10.7, and beyond must deal gracefully with the xattrs that are already assigned to your files. This is a very strong incentive for Apple to continue using the same format. However, in the event that Apple does change or abandon formats, OpenMeta can still provide an upgrade utility to migrate its data to whatever new format or storage method is necessary.

    4) As a worst-case scenario, it's possible that Apple might delete the OpenMeta xattr data altogether. (I view this as extremely unlikely, as Apple would have to explicitly look for and delete non-Apple uses of "com.apple.metadata" xattrs or target OpenMeta directly). But even in this scenario, OpenMeta has a built in backup system that stores copies of its data in the user's Library folder so that it can recover any lost xattrs. This was originally done to deal with Adobe Photoshop and a few other applications that use file I/O methods that don't preserve xattr data, but it also serves as a backup for any other loss of xattr data.
    • CommentAuthorGotow
    • CommentTimeMar 10th 2009
     
    Conclusion:

    I initially had reservations about the way OpenMeta stored tag data. However, given Apple's prevalent use of the com.apple.metadata labelling to store searchable data, the fact that xattrs provide a well-defined and atomic storage mechanism, and the existence of a backup system built into OpenMeta, I think it's a reasonable and relatively safe mechanism.

    Note that I'm saying "relatively safe." At it's core OpenMeta is still storing data using a name that's reserved for Apple, and is counting on Spotlight indexing that data, a capability that Apple could change or remove in the future.

    While OpenMeta is a practical solution to a problem we have now, this should be something that evolves with additional support from Apple. To that end, Tom has filed a bug report requesting support for tags in the future, has contacted Apple via email to start a dialog, and has opened an incident with Apple DTS (Developer Technical Support) to get clarification from Apple on the best way to proceed.

    OpenMeta will continue to evolve based on what Apple does and says in the future, but it's a good choice for the present, and something that I feel we can continue to work with in the long run. I plan on completing Default Folder X's support for OpenMeta and releasing it shortly.
    •  
      CommentAuthorBLUEFROG
    • CommentTimeMar 10th 2009
     
    Jon:

    Thanks for your thorough report. Since I am not a high-level programmer I don't know many of the details of the magic of OpenMeta but you laid it out in a very easy to understand way.

    Anxiously awaiting my OpenMeta version of DefaultFolder X.

    Sincerely,
    Jim
    The Ironic Software Team
    •  
      CommentAuthorJono
    • CommentTimeMar 10th 2009
     
    Thanks for this. I'm glad you've taken a proper look at it, & come to the conclusion that you'll support it.

    Like Jim, I'm eagerly awaiting an OM version of Default Folder X.
  10.  
    Jon:

    What are your thoughts on my suggestion to use org.openmeta as the primary storage location and to mirror the xattrs to com.apple.metadata in order to get the Spotlight indexing? I think that this is would make OpenMeta more solid; we define the storage format, so it can remain stable. If Spotlight changes, we don't have to worry about what happens to com.apple.metadata; our data is pristine and we can simply mirror it in the new format (which might be an API rather than an xattr). The current OpenMeta approach, I think, is potentially painting us into a corner by needlessly combining the storage and the Spotlight indexing.

    I see the plist backups as a solution to a different problem, and one that will become more rare as xattrs are better supported. Plus, as I mentioned in the other thread, I have concerns about the plist backups, e.g. complexity, efficiency, and how they handle multiple volumes/users.

    Tom:

    Could you let us know what Apple says in response to your DTS incident?
    • CommentAuthorhmurchison
    • CommentTimeMar 10th 2009 edited
     
    Well Jon just sold a few copies of Default Folder X and probably doesn't even know it. Fantastic post!

    P.S - I just harassed the Devon Think guys once again. Hell if I'm spending $150 Dead Prez on their product I damn sure
    want it to give me some tagging latitude.

    Jon...big big big news here. Your product and the future revisions to Leap and Yep will allow me to organize my files around data
    and not folder structure alone.

    Plus I'm trying to work out a system that may work for my mothers sole practice law firm. OpenMeta and a sane tagging system could make
    case management so much easier.
    • CommentAuthorrlfsoso
    • CommentTimeMar 11th 2009
     
    Hi Jon,

    that is great news. I am anxiously to test out a (beta-)version of DFX with openmeta support. Some thing to put into consideration (as we did with spotlight-comment-tagging): autocompletion (where is it collected – I mean not app-centric but user-centric (opposed to system-centric)), recent tags…

    looking forward to it,

    Rolf
    •  
      CommentAuthorJono
    • CommentTimeMar 11th 2009
     
    Yea, I'd definitely like to see autocompletion of OpenMeta tags.

    I know Jon Stovell of Tag Folders was looking into how/where OpenMeta stores recent tags, & it would be really useful if Default Folder X (all OpenMeta apps) would autocomplete tags that weren't originally created with that app.
    (I.e. all OpenMeta apps autocomplete tags by reading the central OpenMeta tag store)
    •  
      CommentAuthorsherkaner
    • CommentTimeMar 11th 2009
     
    Seconded!

    And looking forward to DFX with OM tags ... thanks for the update.
    • CommentAuthorGotow
    • CommentTimeMar 11th 2009
     
    Michael:

    As you know, I was initially in favor of storing all tag data in an xattr named "org.openmeta:kOMUserTags" and mirroring that data into "com.apple.metadata:kOMUserTags" attributes to get Spotlight indexing. However, after surveying Apple's use of extended attributes and thinking about the fact that xattrs are distinct and not attached to any other xattrs (so "com.apple.metadata:kMDItemFinderComment" has no relationship to "com.apple.metadata:kOMUserTags", for instance), I came to the conclusion that also storing tags in an org.openmeta:kOMUserTags xattr would result in extra data shuffling without a real benefit.

    I don't view the "com.apple.metadata:kOMUserTags" xattr as being an inherently risky place to store tag data. Regardless of whether the xattr name starts with "com.apple" or not, Apple won't start intentionally deleting users' data. And as I said in my long explanation above, Apple is currently storing their own plist data in a lot of "com.apple.metadata" xattrs - this is persistent, long term data, so it's unlikely that the format will change. Apple would have to convert all of their own stored data and that wouldn't make sense, especially since plists are inherently extensible anyway (though if they do choose to change formats, we just have to deal with it via a converter app or other mechanism).

    I agree that storing tags in an xattr with an "org.openmeta" prefix would be "cleaner" in principle, but in practice I don't think there's a real risk that requires it.
    • CommentAuthorrickdude
    • CommentTimeMar 11th 2009
     
    Jon, thank you so much for posting this explanation. Why Ironic couldn't do this before is one of the great unsolved mysteries of the universe, but I guess we can all move on now.
    •  
      CommentAuthorsherkaner
    • CommentTimeMar 11th 2009
     
    Well in their defense, rickdude, this is mostly what Tom has posted here before. But it is certainly good to hear that there is consensus in at least one important part of the wider developer community.
  18.  
    On the tech support incident I filed: I requested that they document this 'spotlight importing xattr'. I also included information on open meta, etc. So far just the generic 'we will get back to you soon' message.

    I was hoping that most of the discussion on actual API, etc (which this discussion mostly isn't) would take place on the OpenMeta source site, where there is a google group:

    http://groups.google.com/group/openmeta

    --Tom
    • CommentAuthorjln
    • CommentTimeMar 13th 2009 edited
     
    Ah... How exciting it is to read this while having my morning coffee !

    http://stclairsoft.com/blog/2009/03/13/openmeta-progress/

    Thank you very much !

    Julien
  20.  
    Well - Apple got back to me about the tech support incident. They basically declined to offer an answer, and refunded my tech support incident. They just said to keep an eye on Bug ID 6043965.

    They also said to file an enhancement request in addition to the bug, so that is one thing that I will do.

    --Tom

1 to 20 of 20

  1.