Jono:

Tom has worked out a workaround that includes backing up OpenMeta Tags on-the-fly. We should be seeing something with it soon (updated Tagit).

Yea, this problem particularly affects me as I 'live' in Illustrator & Photoshop. I filed a bug report a few days ago, but don't really expect Adobe to do anything about it (at least not in the near future).

I understand your cynicism but this also affects users not using OpenMeta. It is a BUG in every sense of the word. I would love to start a "class action bug report" with them - shake 'em up so they recognize the seriousness of the issue. I encourage you to spread the word (and the URL) - the more we file, the harder it is for them to ignore. 8^)

Tom,

The advantage of using a month by month system is that old months could be heavily optimized. Also a file by file system is friendly to Time Machine, is easily readable by simple Cocoa programs and is fairly easily understood. As for disk space, 100,000 4k attribute backups use up 400MB of disk space, which is about 6 cents worth of file space on a new desktop, or perhaps 50 cents on a laptop. The space used is trivial.

I think it's more like 60 cents of disk space, not 6. (A bare 1 TB drive is around $100 now.) Anyway, it's not really the dollar cost that bothers me. Disk space is super valuable on a laptop because it's so limited. Also, there's the overhead of having so many extra files (especially in the same folder), which slows down the file system as well as backups and synchronizations.

It sounds like you have a plan to eventually compact or delete the old backups, although this isn't implemented yet. I wonder, though, why we need multiple backups in the first place. Under what circumstances would we not use the most recent backup?

This is a backup - if there is any OpenMeta data on a file already, then that is for the most part considered to be accurate. The backup is only used if there is no meta data found on a file.

Then it sounds like a Time Machine restore of a file could bring back old tags in the xattrs, and these would be used even though the backup has newer tags?

The xattr system is designed to hold meta data - old programs that use 'old apis' will almost always work properly with xattrs. In order to not work with them, you need to go out of your way. Which I guess Adobe does, and is a huge bug on their part for all sorts of reasons.

I agree that it's a bug if Adobe's software doesn't preserve xattrs. However, I think you and Bluefrog are mistaken about having to go out of one's way to make it not work. The problems probably don't come from deleting the metadata, but rather from not actively doing something to preseve it when saving a new file. My understanding is that an application needs to go out of its way to call an API such as FSReplaceObject (added in 10.5) in order to preserve the metadata.

When someone writes a better backup system for extended attributes, for OpenMeta or not, then OpenMeta could use it. The present backup system is designed to be simple to change, expand or replace.

I want to make sure that the backup system really is a backup that's independent from the core OpenMeta functionality. This was not totally clear as I tried to explain in the previous post. What I'm getting at is that I don't want there to be any problems caused by having multiple applications, some using the original OpenMeta code that doesn't do backups (or Gravity's similar code), some using the current code, and some using a hypothetical newer backup system.

Bluefrog:

It's not a question of removal because Photoshop uses safe saving (as all apps should). The new file doesn't have any metadata, and it isn't supposed to. The metadata gets added back when the app calls the API to swap the new file with the old one. Until 10.5, the proper way to do this was using FSExchangeObjects. However, FSExchangeObjects does not handle xattrs or hard links (or, at least doesn't do so consistently--some consider this a bug in the OS). So now apps are supposed to be modified to use FSReplaceObject, but this API did not exist until 10.5, which shipped after CS 4.

I don't like the Photoshop engineer's attitude, but he or she is correct on the facts. Apple designed xattrs in such a way that applications must take positive steps to preserve them. They actually are "separate from the file," in some respects. Applications that have not been updated to use the latest APIs, or which use the normal Unix APIs, will not preserve the xattrs. (Even the Finder did not preserve them under Tiger.) Application that want to use xattrs must take this unfortunate reality into account.

Bluefrog:

My guess is that Photoshop CS 4 is not doing a safe save for the JPEG in that situation, and that's why the xattrs are preserved. (I'm using Photoshop Elements 6. In this situation, it does not preserve the xattrs.)

By the way, if the inodes are different, you can conclude that the file is new. However, if the inodes are the same you cannot conclude anything, because one of the features of the metadata preservation API is that it assigns the old inode to the new file.

Welcome, Colin!

Your vision is just fine, 20/10 possibly! >.<

Actually that is on the 1000 yard view for this whole thing. The ideal is dissolution of application / operating system boundaries for metadata. It is a grand vision indeed but we need to start somewhere and that's in our own neighborhood. (Think globally, act locally - right? ;^)

We welcome any and all comments (though we should have new threads going for new ideas - this one's getting a little long in the tooth).

Thanks for your comments.

Just to be clear in my suggestion, I wasn't suggesting moving the meta data anywhere other than xattr. NTFS supports attributes sufficiently to implement a Leap-type app on Windows. I agree that the best thing to do is get widespread adoption on OS X and then try to expand globally, but, and I believe I am making the same point as Mr. Tsai, although for different reasons, you don't want to start out by choosing a namespace that isn't going to be portable (to be fair, their is no reason you couldn't use a com.apple namespace on Linux, but it makes little sense for a number of reasons, the biggest being the fact that OpenMeta neither owns nor controls that domain).

As for preserving/copying attributes, it's a bit of a chicken and egg problem. Nobody is going to worry about preserving attributes across file systems until there is widespread need to do so and nobody is going to create apps with need for attribute preservation without support (witness the Adobe problems, which I do sympathize with Adobe over).

My proposal is that OpenMeta should own the OS-level meta data specification on the OS X platform, and when it has begun to really take off, start approaching the Linux community, maybe even setting up a working group to manage the specification (partnering with freedesktop.org would be a good idea here). Then you go after Windows. Contrary to popular belief, Microsoft will support community-lead initiative when it is in their interests to do see (e.g. recent integrated support for JQuery in their flagship development IDE).

Personally, as a professional software developer, I think the correct approach is to store OpenMeta tags in an OpenMeta namespace, and, to achieve the Spotlight integration, keep a shadow copy of the tags wherever Spotlight is expecting to find them. That way you have a future-proof primary storage, and you can react to whatever changes are introduced by future versions of Spotlight (e.g. if Apple were to provide a supported extension mechanism).

Since other OSes store xattrs in different places and different ways, I think trying to solve things for the world is premature and overly ambitious, and simply won't work at this time. I think we need to solve real problems that exist now, and KISS - keep it simple. If there are compelling reasons why com.apple makes things simpler, I think we should go with that and worry about what Apple may or may not do next year when that happens. We can always write a utility to update things later. I don't think we should complicate things for things that in all likelyhood won't happen.

Welcome, matthewb (your account is active now)…

Thanks for your comments.

You're all set, charles. Welcome and thanks for your comments.

Thanks for the heads-up, sjk.

<strong>Guest:</strong><br /><br />I'm a late comer to this discussion (via Daring Fireball), and I am quite intrigued. The two main points of discussion seem to be: 1) using com.apple.metadata: as the xattr prefix for non-Apple xattrs; and, 2) Spotlight's (in)ability to index xattrs that are _not_ prefixed by com.apple.metadata.

I come from a background of xattr from BeOS (and its BFS, which was designed by the same person who is now employed at Apple working on filesystem and Spotlight-related work), so I have a different perspective on xattrs. HFS+ has had xattrs for quite some time, but only recently received OS level support for them. With BFS files, xattrs did not have to have any specific &quot;prefix&quot;, they were indexed by default, and if a particular filetype had a set of xattrs associated with it, those xattr fields would be present on newly created files of that type.

Back to OpenMeta: Why do the xattr fields have to prefixed at all? Is that just for the ease of programatically reading them? If that's the case, I personally see no reason _not_ to use com.apple.metadata. Spotlight is an Apple program, and if it can only read metadata prefixed in that manner, then that's OK. It's similar to other programs that perhaps want to access to read (and possibly modify) your Safari bookmarks (say for iPhone syncing). Apple &quot;owns&quot; Safari's bookmarks, but access to the bookmarks file must conform to what Safari expects. In this same manner, access to indexed metadata must be what Spotlight expects.

I'm not up to par on the limitations of Spotlight plug-ins, but as I understand from this discussion, mdimporter will only recognize xattrs prefixed with com.apple.metadata. Is that correct? If that is correct, then I fully support OpenMeta using this method. However, if it is also correct, then it is a distinct limitation of Spotlight, and should be changed and filed as a bug. If Apple's search and index mechanism for their filesystem does not let me search or index the things I put there (such as custom xattrs), then this is an artificially imposed limitation. Apple has hindered the state of HFS+'s metadata on OS X for *years*, and it's time they finally let users have control of their own metadata. If OpenMeta can do this, then I fully support their effort. It does not appear they are doing anything in a &quot;hack&quot;-like manner, but rather working within the artificially-created limitations that Apple has imposed on developer's because of the lackluster implementation of xattr indexing.

-- R. Patrick Cameron
rpcameron at rpcameron dot net

<strong>Guest:</strong><br /><br />Let me add the perspective of someone who manages application supportability and risk for a living.

Whenever you write an application that makes use of data structures, APIs, etc. that are provided by another system -- in this case the OS-X file system and Spotlight -- you are making certain choices about what to rely on. Apple provides documentation that essentially says &amp;quot;if you want to add xattr metadata in your own namespace, we promise to respect that.&amp;quot; They don't make any promise to respect data other people put in com.apple.* namespaces.

OpenMeta wants to accomplish two main goals:
1. tag files in a standard way that other applications can use and manipulate
2. make those file tags searchable in Spotlight

Goal 1 can be accomplished using either an org.openmeta namespace or a com.apple namespace. All other things being equal, if this were our only goal it would be silly not to use org.metadata. After all, Apple has made an implicit promise to leave that data alone.

Goal 2 can only be accomplished if OpenMeta uses the com.apple.metadata namespace.

So here comes the first decision: do we use this item even though Apple hasn't promised that it will work in reasonable ways? If we do, we gain a great feature, but at the risk that it might break something unexpectedly either now or in the near future (not to mention that the feature might disappear entirely without any kind of warning). At least with org.openmeta.* we have some assurance that Apple won't just yank the feature without warning, and some assurance that doing this isn't supposed to break anything.

Now, OpenMeta has chosen to use com.apple.metadata and accept that risk.

**HOWEVER** the claim that OpenMeta &amp;quot;doesn't use any secret API&amp;quot;, which is *technically* true, is a little misleading. There may not be a secret API in use, but something unsupported is being used, which means the user is taking a risk -- even if it's a small one -- by trusting tag data to OpenMeta. At the very least, OpenMeta should be, well, more open about that. Users can decide whether they're willing to accept that risk.

Mr. Tsai has made an additional suggestion: OpenMeta can achieve Goal 1 in a way that's supported by Apple (using the org.openmeta namespace), and supply an application *option* to the user which will achieve Goal 2 by mirroring that data to the unsupported com.apple.metadata area.

This way, the user can get a lot of good out of OpenMeta even if they don't want to accept the risk in using an unsupported file system feature. User choice is a good thing. The approach has the additional advantage that if Apple dramatically changes how the com.apple.metadata feature works (say, by altering its data format), then all users lose is the Spotlight-searching feature -- their tag data is still safe and can be accessed by OpenMeta and other applications. Resilient software is *also* a good thing.

--Darren Meyer

This thread may be too long to read everything. The fact is that OpenMeta now backups all the tags in ~/Library. This is repeated in this thread.

Rolf,

My personal opinion is simple and I've already talked this issue with Michale and I think he understood my point. Using org.openmeta as a primary place and let developers/users choose to use com.apple.metadata as an option just brings chaos. I've wrote how such a world brings confusion to users. At least, I think Michael understood this point.

Of course, I know the superiority of org.openmeta. So, I personally prefer mirroring both com.apple.metadata and org.openmeta.

However, a person who is in charge of Openmeta does not seem to take this route. Instead, this person decided to store backup in ~/Library. It is not exactly what I prefer but this is much better than nothing. Furthermore, this is much better than a chaotic world where tags are primarily stored on org.openmeta and only the wishful developers use com.apple.metadata.

I strongly fear to lose tags. But, I strongly believe the solution should not destroy usability and I personally believe it is possible - just mirroring com.apple.metadata and org.openmeta. It looks just simple to me. I, an end-user, wonder why we need theological arguments.


Below is what I wrote in this thread two weeks ago. Who wants to live in such a messy world?
----
Imagine that X.app saves tags only on org.openmeta and Y.apps saves on both places and uses spotlight for searching tags. It is not difficult to imagine an end-user who does not know the exact mechanisms but know both apps use OpenMeta starts to wonder why s/he can't search a tag added by X with Y. S/he wonders why only the tags added by Y can be searched from both X and Y. What if he uses many more OpenMeta applications? Furthermore, what if users can choose whether to use com.apple.metadata when they first install applications? Do standard end-users remember their choice of configurations? It is almost obvious that allowing users to make a choice makes the situation much more complicated. I recommended you to write demonstration partly because I didn't have any idea how you are to dissolve this problem.

<strong>Guest:</strong><br /><br />With regards to the 'cost' of 400MB...I pay for continuous online backup. I pay per GB uploaded, and per month it's stored. So the 'cost' is not 6 cents, nor 60 cents. If all my applications made 400MB of backups, I'd be broke.

Aside, I think Michael Tsai made an excellent point, which has yet to be refuted (except by logical fallacies; 'show me where it says we can't do this'). Rather than address what he raised we've had doubt in his credibility; 'show me the Apple email', which he disproved, to be met with silence (not even a thank-you!). We've had 'why are you attacking only us', followed by an explanation, again met with silence. We’ve had a suggestion he would contribute if his work would be accepted followed by...silence. And then we've had a 'theological' discussion as to the whys and wherefores of making a decision which made this so pointlessly academic (I use that word very loosely) it was, at times, painful.

It's your project and you can do what you want, but please just man up and say 'we're doing it this way because...'. Going for the defensive 'well Apple didn't say we can't, ok they did but forget that for a minute, blah blah blah' looks really shoddy. As does the 'secret api' rubbish. People want transparency.

Guest

Michael,

>So I still think the primary storage should be in org.openmeta, with mirroring to com.apple.metadata.

If you mean that org.openmeta should be forcefully mirrored to com.apple.metadata as a default setting, I am perfectly with you. I don't care which is called primary or secondary. I believe (perfect real-time) mirroring is the best option.

I think many people concerning this problem want to achieve two goals. Use org.openmeta and allow users use spotlight by default. Only the solution I can think that can achieve both goals is mirroring. Honestly speaking, I don't understand why this option is not chosen by those who are in charge of OpenMeta. It seems to be so simple and user don't need to change anything in their daily usage.

Making backups in ~/Library is just the second best. As I asked the question in another thread, I feel that this second option may cause some problems aside from what you pointed out. However, it is much better than nothing. If the best option is implemented, we don't need to consider the necessity of this second option. I support this second best option only for assuring users who worry the tags they are currently adding may disappear. I am not perfectly happy with this backup solution. However, I think some people are getting too nervous given that the current OpenMeta system has a backup system.

<strong>Guest:</strong><br /><br />sherkaner here is the problem, the standard is that the domains are owned by the developer, whether it's Apple or Whacky Company. So com.whackycompany is a unique identifier that they can use. Now since Apple writes the OS it's tempting to use com.apple.metadata. But Apple still owns that data space. It's not that they didn't commit to protecting the data. They said in dev speak that it should not be used. While you may think using org.openmeta and mirroring it to com.apple.metadata seems dirty it is really the safest thing to do. Apple can remove com.apple.metadata at any time whenever they do an update. Mirroring the data means that user information is still preserved even if Spotlight search is impacted if Apple changes things. From a dev perspective the fact that they said that using com.apple.metadata is unsupported means they are going to change Spotlight searching of meta data at some point. Doing things the &quot;dirty&quot; way means that OpenMeta can be more flexible in the future. I'm really quite stunned that there is so much push back on this.

Michael Krzyzek

<strong>Guest:</strong><br /><br />countach first of all the revers domain is a convention used so that each developer can have a unique namespace. It's the equivalent of application creator codes from back in the day. Take a look at the Preferences folder for bloody sake. No proper developer is ever going to use a domain they don't own, seriously get real. Secondly the openment.org domain was used as an example, Michael Tsai even offered to buy it so perhaps he did. Thirdly read the whole thread. Do you see the comment from Michael Tsai where he asked Apple about using com.apple.metadata and they said it wasn't supported? That means that they do not want people using it. That is their way of saying it is reserved. Lastly the idea that you should wait for something to break and then fix it is just laughable. Why not be proactive and avoid the whole problem in the first place? That is the whole point of this discussion that keeps being missed over and over again.

At this point it seems like OpenMeta is a good idea with a flawed implementation. As it stands I don't see it getting much uptake from the rest of the developer community.

Michael Krzyzek

Jon Gotow from St. Clair Software here - I'd like to add my two cents to the discussion. This matters quite a bit to me because I've got a build of Default Folder X sitting here with OpenMeta support built in, and don't want to release it until we get this sorted out.

First, I'd like to get past the semantic quibbling - by convention, namespaces and files tagged with a company's domain are reserved for that company. That IS the convention and I don't think we need to argue about it. Writing a com.apple.metadata prefix in a file's extended attributes is writing to a data store that you don't own, regardless of whether you're technically using a private api or not.

I'll summarize the issues that have been raised:

1. Extended attributes on the filesystem are not always preserved because some applications use old API's that are not EA-preserving. Photoshop and Illustrator's 'safe save' procedures are among these. In my mind, this is essentially an early-adoption problem in which we're relying on a filesystem service that is not fully supported yet.

2. Writing extended attributes to the com.apple.metadata space works now, but because the format and use of this data is controlled by Apple, that may change or conflict with future formats and/or uses.

3. Spotlight is hard-wired to only index extended attributes prefixed with com.apple.metadata.

Now ideally we'd like to address all of these - doing so would make things as safe as possible, though still not guaranteed since we're fiddling with features and data that Apple owns and has not documented for public use. The goal here is to make OpenMeta work for the end-user. To that end, we need to:

a. Patch up the incomplete extended attirbute support in the filesystem. If EA's worked completely, they would be the ideal way to attach metadata to files, since the data is paired explicitly with the files themselves (as opposed to keeping it in separate parallel storage, as Spotlight comments and OpenMeta backups do). Unfortunately, the data can currently be wiped out by certain apps, or by transferring files to a non-EA-aware filesystem.

b. Provide a safety net that protects user data as best we can from changes to the com.apple.metadata changes in the future, while allowing indexing in Spotlight as it functions today.

As far as proposed solutions, we have:

A. OpenMeta now creates backups to ~/Library. This takes care of issue #1 and is already a done deal (though we can still argue about the implementation - I haven't looked at the code in enough detail to do that yet).

B. Michael has proposed that OpenMeta store tags in an org.openmeta labeled extended attribute, then 'mirror' those tags into the com.apple.metadata prefixed attribute. That makes complete sense to me and is the best we can do with issue #2. We should have OpenMeta write to a data store that it owns, and then push the tags to Spotlight as a separate 'mirroring' operation. com.apple.metadata should not be the primary storage space, since Apple owns that, not OpenMeta. Ideally, tag 'mirroring' should merge OpenMeta tags with whatever already exists in the com.apple.metadata store so that we don't stomp on any other data added by Apple or others. Yes, the logic is messy and not completely deterministic (because we don't know what others are doing, we have to 'play it safe' and may end up leaving extra tags there in certain exceptional cases - if that's the price of ensuring that we don't mess up the user's data, so be it).

I personally think that BOTH of these are necessary. Just writing to org.openmeta and mirroring to com.apple.metadata does not take care of issue #1. Issue #2 cannot be solved perfectly, since we don't know what Apple will do in the future. What we do know is that com.apple.metadata is currently a property list - we can check for that and not touch the data there at all if we see an unexpected format - that will 'future proof' things to the extent that we won't be corrupting data if things change. And by storing OpenMeta tags in org.openmeta as the primary store, we're preserving user data so that a utility can be written in the future, if necessary, to deal with changes from Apple when they occur.

Pretty compelling argument Jon makes there (even more so as the developer of a utility that I really want to see OpenMeta support in :)

Hopefully Jon and Tom will be having some discussion :)

<strong>Guest:</strong><br /><br />I've spend a significant portion of my day reading through this entire thread. I work for a company that really needs a metadata solution like openmeta. That said, as an IT person, I need to make sure that our data is future proof (as much as possible). That is why I support using org.openmeta as the primary storage location and mirroring data to com.apple.metadata (I don't care if the mirroring is forced or an option the user can turn off and on). Given the current implementation, I am reluctant to adopt the standard. Please make this change for the good of this very promising standard.

- RainDelay

I'm glad to see a weight of opinion turning to using org.openmeta as the primary store & mirroring tags to com.apple.metadata. I think that's a cleaner solution, and less risky in the long-run.

My experience was that I went through and tagged a huge number of files in YEP. Subsequently all the tags went missing, leaving me perplexed & irritated :D
I really don't want that to happen again because my tags were slid in under the spotlight radar. I've given up using YEP for tagging my files, because I don't trust its tagging mechanism (spotlight comments). I came on here to check progress towards YEP using OpenMeta, which I thought would be more robust. Now I'm cautious about any app using OpenMeta (or any other way of writing tags exclusively to com.apple.metadata)

I'm not sure where that leaves me, but I'm currently building a file tagging and storage workflow utility, and am now less convinced of using OpenMeta.

A further concern I have, is that the ~/Library backup solution is, in my eyes stupid (sorry ;-) The bulk of my files are on an external drive, which I share between 2 Macs. What will the behaviour of the Backup folder be? I dunno! I really hope that it's not just going to force create a /Volumes/ExtDisk/Library/OpenMeta Folder, akin to what YEP did, which I hated.

My 2c...

May I respectfully suggest that Ironic post an "official" response at about this point, summarising the issues raised in this thread and explaining their plan moving forward? Ideally, they could then close down this thread, while inviting people to continue this discussion on other threads. If we continue like this, I'm worried that this long and unsatisfactory thread will be the first thing that prospective OpenMeta users or developers see, coming here from Daring Fireball, and also the last, as they get a false impression of stubbornness and unresponsiveness.

But Jono, maybe instead of replying here the guys are working hard to please everyone ;)

Who knows! In irony, uh, ironic I trust ;) ^^

Am I right in thinking that perhaps this is supposed to be the answer to the issues raised in the current thread?

The most thorough answer seems to be here, posted by Jon of Default Folder fame. Poor Michael Tsai had to come back and ask again whether using org.openmeta wouldn't be better, and the answer is that "also storing tags in an org.openmeta:kOMUserTags xattr would result in extra data shuffling without a real benefit". If Jon's satisfied, I guess I am, too.

I hope the developers of BibDesk, Skim, DevonThink, Bookpedia (probably lots of others, too) are alerted to the new thread.

I’ve posted additional thoughts in these threads: 1, 2, 3. Christiaan Hofman of Bibdesk/Skim agrees with me that Jon Gotow is mistaken. I respect Jon, but keep in mind that his main product is a haxie, so he’s probably more accustomed to relying on unsupported OS behavior.

This thread is now closed. Official discussions have been taken to a more appropriate venue. - Jim