Put simply, OpenMeta is a hack. I say this not a value judgement but as statement of fact. It's clever, and it's useful, and it tricks Mac OS X into doing something it wasn't designed to do.

On Leopard, at least, Apple stores some metadata in a certain place (extended attributes prefixed with "com.apple.metadata:") and using a certain format (binary plist). Spotlight expects this and indexes those xattr values for searching. What OpenMeta does is store its data there, too. Spotlight, happily, seems to index it as well.

Unfortunately, Apple does not support this "feature." The storage location is reserved for Apple's use. The OS does not expect other applications to write data there, and the results are undefined. It might all work as desired, or the OpenMeta data might be deleted without warning, or it might not be consistently or promptly indexed, or it might work now but stop working in 10.5.7, or there might be some other side effect of mucking up Spotlight or other OS functions. We simply don't know.

OpenMeta is really two separate features combined into one. It's:

a) a common area for applications to share metadata
b) a way to make Spotlight index additional metadata, beyond what the Spotlight importer plug-ins produce

(a) is useful by itself, and it could be done in an Apple-supported way. Just choose an xattr prefix such as "org.openmeta:". But to do (b), OpenMeta needs to piggyback on undocumented and unsupported Leopard behaviors. Aside from the Apple xattrs mentioned above, the "ShortName" in schema.strings that's used for "tag:" searches is also unsupported, as far as I know. The Web page describes OpenMeta as not using any "secret API." This is true but misleading. There are many ways of doing unsupported things without using private API. For example, an application could modify or remove files in /System.

I think the developers should be up front about what OpenMeta is actually doing:

First, users and developers should know what they're getting into. I don't mean to spread FUD, and my personal belief is that it's probably safe, but we don't actually know. When I asked Apple for comment, they said that they do not support third-party software doing this. Some users like to live on the edge, and that's fine. Others like to play by the rules, run clean systems with no haxies, etc., and they may choose not to participate at this time. Perhaps there should be a way for such users to get (a) independently of (b).

Second, we need Apple's involvement. OpenMeta-type functionality is useful, and it would be better for all concerned if there were an Apple-blessed way to do (b). This might also solve the current problem that Time Machine only backs up the OpenMeta data if you change the modification date. People shouldn't think that OpenMeta is the solution to the metadata problem. Rather, it should whet their appetites so they encourage Apple to provide a real solution.

Bluefrog:

1. The main point of my post is that "no secret API" doesn't mean "does what you expect" or "safe" or "will continue to work." You could reverse-engineer the iTunes database format, figure out its "conventions," and modify it without iTunes' knowledge. You don't need any secret API to do this--just open the file and write to it. That doesn't make it supported or a good idea, though. And it would be misleading to say that you weren't doing anything secret.

2. I agree that it has great promise. That's why I'm trying to start a discussion. As to the code, I would prefer that it follow standard Cocoa naming and error handling conventions. These are amply documented by Apple. I've also found some bugs and will write them up on Google Code shortly.

3. No, you shouldn't sit idly by, but I think you should be open about what you are doing. I think that will increase the chances of Apple moving on this. Secondly, I think you should consider not tying the sharing functionality (a) to the secret stuff (b). I discussed this with Quinn at DTS, who talked to the Spotlight team.

I don't understand why you think my post was a flame. That was certainly not my intent. I'm talking with you guys because you're the ones developing and promoting this as a standard.

My thoughts as a simple user are that OpenMeta needs to exist and flourish if only
to provide the proof of concept that Apple needs . Since the underlaying structure native
OS X I don't think Ironic is going to be pissed at the Pope if they eventually come out with a
clear and defined method for assigning and managing system-wide metadata tagging.

Chances are developers would work out some way of taking OpenMeta data and importing it
into whatever tool Apple delivers.

As a consumer I'm willing to take the risks. It may not work forever or there may be issues but
keeping dialogue with Apple is a way to ameliorate anything bad from happening.

I think many of us realize this is a bit off the beaten path but there have been worse hacks IMO.

Jono: I'm sorry if it came off that way. I was trying to be super clear that I was using the word in the original sense of "clever but unsupported" without implying that it's bad.

hmurchison: Judging by the wiki and Bluefrog's comments, I didn't think it was clear to most people that this is off the beaten path.

rlfsoso: The point is, it's not under our control whether this will work reliably or in future versions of Mac OS X. No matter how hard the developers work, it's up to Apple.

And I apologize if I have not been clear. I have made a very concerted attempt to be very clear about this project and I would be interested to know which statements I made that may be "less than clear" so I can avoid future misunderstandings.

Jim

I did not mean to imply it was "supported behavior for a 3rd party" but I still maintain that the mechanism itself is not secret. Unless you're meaning it in the sense of the Google app on the iPhone using the proximity sensor. In that case I may be able to agree with you semantically.

BTW, Spotlight Comments are an official feature of OS X but their use as "Tags" (per current convention) is also NOT supported behavior. ;^)

Bluefrog:

Apple does not document the mechanism. You discovered it by reverse-engineering. Now that you know about it, it's not secret anymore. This just goes to show that "secret" has no useful meaning. There is supported (Apple documents it and promises to make it work and continue to work) and unsupported (all bets are off). The Google case is similar, although I think listening for an undocumented notification is a different level of risk compared to (a) storing user data, and (b) modifying data owned by other software.

Regarding Spotlight comments, using them for tags is supported! You can put whatever you want in the comments, and Apple agrees to preserve them and make them searchable. You don't need special license to put tags in the comments, just like you can create a text file and write whatever you want.

Similarly, as I said in the original post, it would be supported if OpenMeta stored the tags in an extended attribute called "org.openmeta:kOMUserTags". This would provide all the same benefits as the current mechanism as far as letting applications share metadata with one another.

Bluefrog:

The Spotlight comments are an area where the user can do whatever he wants. There is no prescribed way to use them. Therefore, I do not think that using them to store tags is a hack. What are tags if not user data that you want to be searchable? As I see it, the two main problems with using the Spotlight comments for tags are (1) the API for accessing the comments is slow, and (2) if you use the comments field for tags, you can't use it for something else. These have nothing to do with support, though.

The confusion seems to be over the meaning of the word "support." I'm using it in the developer sense of, "Does Apple guarantee that the mechanism will work and continue to work, modulo bugs and broken hard drives." In this sense, the Spotlight comments and text files are both supported, as would be "org.openmeta:kOMUserTags".

Michael,

Some of the points in here have already been made, but I reiterate them here, with my take.

Apple certainly could stop indexing openmeta data if they wanted, but the data would still be there, and it is not impossible to create a task similar to mds that listens to file system changes and creates an OpenMeta data store. Why not, though use what works?

xattrs are fully supported or documented, and the 'short names' can be found in Apple's sample code for Spotlight importers. /Developer/Examples/Metadata/ImporterExample/MyCustomImporter/English.lproj/schema.strings (spotlight search shortName).
Where is the documentation that says that you can't use 'com.apple.metadata'?

It is of course ok to use com.apple.metadata as a prefix - I want to mirror this data in the Apple metadata store! What other prefix would make sense? My spotlight data resides on my computer - not Apple's.

A hack is using a hammer to nail in a screw. What _is_ a hack is using spotlight comments prefixed with strange characters to store meta data - how do you store authors, workflows, ratings, times, etc in a space reserved for user editing? - You can't. Not to mention how does one handle multiple word tags, other languages and character sets, etc, etc. A hack is using something in a way it was not designed to be used. Comments were never designed to hold tags, urls, or other metadata. How big can a comment be? Is that bytes, characters or glyphs? Apple in fact guarantees nothing.

And I guess it would be fine in theory to store tags in comments if somehow everyone agreed to use the same prefix character, and have the same parsing rules, and had the same way of allowing user added 'real' comments to appear mixed in with the tags. (how do you add a comment about spotlight tagging in a spotlight comment without adding a tag?). But I have tried all of that and found that when you set spotlight comments by applescript, appleevents, editing the .DS_Store files, etc - all those methods lose data and only haltingly get synchronized to the spotlight database. So then I spend weeks trying to manually synchronize the comment data into spotlight. Tricky. At the end of all of this, all you get is lackluster slow support for one type of metadata in one character set.

Fundamentally, 'extended attributes' - which tags, authors, ratings, and other data are, should be stored with the xattr mechanism. The OS is designed to deal with them. By using com.apple.metadata in front of an extended attribute (only certain kinds), we are telling Apple to mirror my data in my spotlight data store. Certainly makes life simpler for users and programmers. But if Apple changes how it works, then the data is still there, and it still means the same thing, and building an sql db to search through it all and keep updated is not a big problem.
The real problem is agreeing on standard keys, and the formats for those keys. That is what OpenMeta is all about.

As far as getting Apple to do something like this: I have stood up and told them at appropriate times at at least 3 WWDCs (to more than scattered applause - developers want to do this.), filed bugs against it, etc. If this makes someone at Apple wake up and smell the coffee, then so be it. I would be very glad to write software that uses a meta data api that Apple wrote. But they don't appear ready to do it.

--Tom Andersen

Michael,

Combined with Tom's last comment, I think everything gets clear to me. Thanks.

I then think that, as people wrote, this is a simple trade-off problem. We have two possible futures - one in which Apple does not interrupt developers to use com.apple.metadata and the other in which Apple actively prevents them to use it. We know we cannot exclude the possibility that the second world can happen but we don't know the exact probability it occurs. The question is whether the cost for changing the current system is worth to pay.

Given that 1) moving from com.apple.metadta to org.openmeta requires extra effort to develop a new search engine / program and 2) we're rather confident that Apple does not intentionally delete com.apple.metadata:kOMUserTags from all the files on Finder without any warning, sticking the current OpenMeta system seems to be very reasonable to me.

Tom:

Apple certainly could stop indexing openmeta data if they wanted, but the data would still be there

You don't know that the data would still be there, because OpenMeta is writing it into an area that it doesn't own. The OS would be well within its rights to delete any "com.apple" xattrs that it doesn't recognize. Similarly, you could figure out the format of the iTunes database and add a column, and it might work, but there's no guarantee that it won't be lost if iTunes ever rebuilds its database. And there's no guarantee that it won't break iTunes because it isn't expecting its database to be modified in that way.

the 'short names' can be found in Apple's sample code for Spotlight importers. /Developer/Examples/Metadata/ImporterExample/MyCustomImporter/English.lproj/schema.strings (spotlight search shortName).

Interesting. They are not mentioned in the documentation, and they don't appear in the Xcode template. But, assuming it isn't a mistake, I guess the example code counts as official.

Where is the documentation that says that you can't use 'com.apple.metadata'?

It should be obvious from the "com.apple" that this is Apple's area of the extended attributes. This isn't explicitly documented, which is why I asked Apple to clarify. And they said that they don't support this.

It is of course ok to use com.apple.metadata as a prefix - I want to mirror this data in the Apple metadata store! What other prefix would make sense? My spotlight data resides on my computer - not Apple's.

"Of course it's OK to use private API - I want to use the motion sensor! It's my phone, not Apple's."

I sympathize with this line of reasoning. But if that's what you're going to do, don't advertise your software as not relying on secret API.

how do you store authors, workflows, ratings, times, etc in a space reserved for user editing?

I never suggested that one should.

Not to mention how does one handle multiple word tags, other languages and character sets, etc, etc. A hack is using something in a way it was not designed to be used. Comments were never designed to hold tags, urls, or other metadata. How big can a comment be? Is that bytes, characters or glyphs?

I agree that a string is not a ideal way to store tag data. Nevertheless, the Finder's dictionary says that I can set a file's comment to a Unicode string. That string can contain whatever I want. Presumably, if the string is not acceptable to the OS it will report an error.

But I have tried all of that and found that when you set spotlight comments by applescript, appleevents, editing the .DS_Store files, etc - all those methods lose data and only haltingly get synchronized to the spotlight database.

I think it's fair to say that Spotlight comments are an Apple-supported technology that's buggy.

Fundamentally, 'extended attributes' - which tags, authors, ratings, and other data are, should be stored with the xattr mechanism. The OS is designed to deal with them.

Agreed. And the convention for extended attributes is that you prefix them with your own domain name, not someone else's.

Hoge:

I don't think anyone is worried about Apple actively breaking the "com.apple.metadata" trick, just because they're mean. However, it's possible that in the course of improving the OS they will change something that causes it to stop working. Secondly, since they never said it would work in the first place, I'm not sure I want to rely on it today. Who's to say whether I can search for a tag and actually find a particular file with that tag? The best I can do is say that I tried it with some other files and it seemed to work.

And the convention for extended attributes is that you prefix them with your own domain name, not someone else's.

If this is true, please point to the official documentation that supports this claim. Also, please point to the official documentation that prohibits use of com.apple.metadata.

However, it's possible that in the course of improving the OS they will change something that causes it to stop working.

Possible? This does happen. Just look at all the deprecated / changed functions in Leopard. Apple never promises to keep the status quo (and that's part of what we expect from them.) Did OS X screw up things for OS9 devs? Yes. Did going Intel screw up things for PPC only devs? You betcha! If everyone took the stance that Apple may not support a given procedure in the future nothing would ever get written.

Secondly, since they never said it would work in the first place

But they never said it wouldn't work either.

If we have the privilege of having a discussion with someone from the Spotlight / Metadata Team at Apple (which we hope and look forward to, especially outside WWDC) and they say, "Hey! You can't do that!!" or "Spotlight indexing or the filesystem is going to be changing in such a way that this is not a reliable mechanism." then we will have a nice talk about what can be done to remedy the current problems.

Michael,

It is not essential. Anyway, the biggest difference between you and Tom seems to be subjective estimation to what extent com.apple.metadata: is secure. That's it. It may be secure or may not. Basically, what you're suggesting is to avoid the worst scenario without paying attention to the benefit of taking a risk (and Tom and Jim estimate the risk is smaller than what you estimate).

Michael,

I think we are trapped in never-ending cycle.

Validity of your argument totally depends on how likely one thinks com.apple.metadata disappears in future. You think it likely to happen. Because of such your belief, it is rational for you to develop a system for storing and searching metadata in org.openmeta.

I (Jim and Tom) think that it is less likely. Furthermore, the benefit of using com.apple.metadata is that we can use familiar spotlight. It is really, really useful. Throwing away this mega benefit is totally ridiculous to me. There is a reason to use com.apple.metadata.

If only the current benefit of using org.openmeta is that it is less likely to disappear than com.apple.metadata, the rational solution seems to just make a backup of com.apple.metada:kOMUserTags on org.openmeta:kOMUserTags and synchronize them perfectly. That's it.If com.apple.metadata is actually deactivated, then I will wait for Tom to write sql database or whatever as he wrote in the last comment. You may say that we should not wait but should move to the new system earlier. I don't buy that argument just because spotlight is so handy and useful.

So, my final question is; is there anything you recommend other than that keeping information on both places (com.apple.metadata and org.openmeta)? Technically speaking, it seems to be a tiny change on a source code. I'm pretty confused why you need to make this problem so big. To me as an end-user, wording does not matter at all. As long as OpenMeta is more robust than spotlight comments, it is fine. What you suggest at the end is just to make a backup of com.apple.metadata on org.openmeta so that OpenMeta gets more robust. That's a simple suggestion that does not require any big story. (I'm sorry if making a backup on org.openmeta is only the technical suggestion you made from the beginning of this thread.)

I feel that another difference between you and Tom (& Ted) may be a stance as a programmer. You try to build a software under the strict constraints of specifications. I appreciate that stance. Another stance may be to pursue practical benefit at the sake of going across the border which I also appreciate in some cases.

Anyway, as an end user, I hope that the bottom line of this controversy is to improve OpenMeta and make it more robust and useful.

I can't say for sure but I would guess that they hadn't even considered using com.apple.metadata.

Thank you, Jim and Tom.

>It only makes sense to use com.apple.metadata for data you want mirrored in spotlight.

Actually, annotations can be searched by spotlight if a file is saved as "pdf bundle", a skim's original package format where annotations are saved as *.txt/rtf inside of a package. So, I had just wondered why they did make skim so that annotations can be searched only when it is saved as a pdf bundle (or, when annotaions are saved as a separate file).


Just let me allow murmuring some thoughts of a non-professional.

Honestly speaking, I still don't get the logic why we can logically conclude that org.openmeta is safer than com.apple.metadata. I had relied on -f option of mdimport on Tiger for my shell scripts. It was an official option even described on the documents. But, it is obsolete on Leopard which annoyed me for a while. This is just a tiny example. But, it seems to me, a non-professional end user, that even what is explicitly supported on the Apple's official document is mortal. I may be missing something important but can't stop thinking in this way.

Michael,

Thanks for your reply. I wrote the last comment hoping that I can learn something new what I don't know yet.

>Because the former is owned by OpenMeta and the latter is owned by Apple. Do you think it's safer to store your data in your own file or inside com.apple.safari.plist, for example?

What does it mean com.apple.metadata is *owned* by Apple? As long as I understand it's just EA. Even I can make an arbitrary EA with com.apple.metadata prefix using xattr on terminal (thought it is useless). I understand that nobody can deny the possibility that such an arbitrary EA may be excluded from spotlight search in future. That's certainly a possibility. But, how about preservation of an arbitrary EA with com.apple.metadata prefix? If Apple moves to a new file system that does not support EA at all, it can happen that my arbitrary EA is lost together with all the rest. What kind of story can we think of in which my arbitrary com.apple.metadata is *deleted* but my org.openmeta EA is not deleted?

As an end-user, I'm confused about the meaning of *owned* by Apple. Sorry for asking question that may be too obvious to you.

Michael,

Yeah, assuming that something is *owned* by somebody, s/he has a right to do anything on his/her property.

In the first comment, Tom wrote that com.apple.metada is just a prefix. When having read it, I thought com.apple.metadata is just a name that can be used for spotlight-indexing all the attributes under it. Then, you wrote;

> It should be obvious from the "com.apple" that this is Apple's area of the extended attributes. This isn't explicitly documented,

So, now I understood that why people can see different meanings in com.apple.metadata because it is not explicitly documented. You continued;

> which is why I asked Apple to clarify. And they said that they don't support this.

This sounds very critical as this seems to be only the explicit information that can be used for defining the meaning of com.apple.metadata. I'm curious what Apple told to you. What statements made you convinced of that com.apple.metadata is owned by Apple but not just a prefix?

Hoge:

Tom didn't say it was just a prefix. He said it gave him his desired result, so he was going to use it.

Prefixes are a common pattern in software that are used to indicate ownership. This is what's obvious and doesn't need to be documented. The question I had for Apple was not whether Apple owned it but whether they would support developers using it. For example, "com.apple.TextEncoding" is an extended attribute that Apple owns, but they document it and explain how it should be used. I was hoping that, even though "com.apple.metadata" isn't documented, maybe Apple was OK with developers using it. Maybe they would give permission and make it official in the future. This is not uncommon, to have Apple say that something is not official yet, but go ahead and do it because it's in line with what they are planning. However, that was not the case this time.

step21:

The OS can't choose to overwrite anything. If it deleted user data (that wasn't stored in one of Apple's reserved areas), that would be a bug, and the users would have a legitimate complaint. Certainly, one can do lots of cool stuff with hacks. I'm just saying, (1) don't be under the illusion that this is supported, and (2) half of OpenMeta can be easily implemented in "blessed" fashion, so I think it should be.

Let me join this discussion. I, as a potential OpenMeta developer, think the point that Michael has brought up is very important.

BLUEFROG,

The issue does not depend on what the Apple guys actually told to Michael. Of course, Michael can provide such information and it is somewhat interesting, but the issue exists without such evidence. We know that if Apple officially states that com.apple.metadata can be used, then the issue is gone. However the issue does not disappear even if Apple does not explicitly state that com.apple.metadata must not be used.

Michael's suggestion is, as far as I understand, that 1) OpenMeta should use org.openmeta (for example) for the primary storage and 2) a developer can also chose to use com.apple.metadata to receive benefit from spotlight.

I think this is a practical and reasonable solution, which gives both security and usability. A drawback is that an application that wants to utilize spotlight needs to maintain two storage. But I think it is trivial. Does anyone have a better solution? Also is there any other drawbacks with this solution?

I don't think we should not use com.apple.metadata. Personally I like hacks very much. But if OpenMeta offers a choice that does not depend on the undocumented behavior of the OS as well, then it will be more attractive to every developer.

Welcome, shg and thanks for your comments.

Personally I think the opposite is more desirable (1) com.apple.metadata as primary so Spotlight immediately indexes without intervention, and (2) org.openmeta as the secondary. As our method continues to work… fantastic. If it doesn't then it would be fairly easy to switch over to the org.openmeta setup. (Bear in mind that I am not doing the programming.)

BTW, I hope no one is taking this discussion as antagonistic or mean spirited. Passion is good and it's obvious that there are passionate people involved here. The point is a Dialogue has been started. This is all very good stuff. - Jim

Michael,

Thank you for your patient reply.

> I agree with shg that this is not very relevant.

Your new information was valuable as it tells lay people a tone of the issue.

In principle, you're right - it is hack if we make a binary judgment. However, I had imagined that you heard kind of very strong warning from Apple as you frequently used a phrase such as owned by Apple. You know that property or possession give very strong prohibitive meaning to lay people. With the reference you indicated, I think I can understand it is avoidable people keep having different attitudes to com.apple.metadata. Concerning the future of com.apple.metadata, the reference sounds rather neutral. As I can't stop recalling several instances that were on the official documents but suddenly became obsolete without warning, I don't know how much negative value this reference indicates. But, I have been sure that you indicated us, end-users, very important point that anything can happen on com.apple.metadata. I appreciate it.

-------------------------
Aside from semantics, practical solution seems to be a bit complicated.

I think that you recommend to use org.openmeta for two reasons. First, it won't disappear. Second, OpenMeta should provide developers freedom not using unsupported feature of OS. I perfectly agree that we had better to make backup of com.apple.metadata given its unclear future. I'm concerned the second reason.

Just because spotlight is useful, I think nobody wants to stop using com.apple.metadata for a while. So, if OpenMeta applies your recommendation, we will have the same information on two places. Then, my question is; who has an incentive not to access com.apple.metadata but only org.openmeta?

First case: We heard that Skim will access OpenMeta tags and show them on its information panel. What's an incentive for them to access org.openmeta but not to com.apple.metadata?

Second case: Imagine a developer like Gravity decided to develop a new tagging/searching software. If they decide to use only org.openmeta, they need to develop their own search program as spotlight is not available. Who wants to choose org.openmeta instead of com.apple.metadata in this case? Maybe those who have a strong belief that com.apple.metadata will have no future and want to avoid rewriting a software when it is abandoned. Where is such a developer?

I understand the following logic - there may appear who desires a feature X. So, we had better implement such a feature so that we can include them. This logic gets more valid as such a demand increases. I just want know how likely such a developer appears because I literally have no idea how strong such a demand is. Do you have any idea?

If such a demand is not so strong, then, the reason to have org.openmeta is mainly for backup.

In general, keeping the information in different formats (xattr & files in ~/Library) seems to be more robust than keeping the information in the same format and in close places. Given that com.apple.metadata's future is unclear and file contents are more robust than xattr, Tom's solution seems to be reasonable one.
------------------------

I think that I'm irritating you but I'm just curious. And, it is my style to take extremely opposite position when trying to understand a person who has a strong opinion. Believe it or not, I think I've never rejected an idea of org.openmeta. I just want to know how valid the argument is.

Hoge:

Your new information was valuable as it tells lay people a tone of the issue.

I think, as a lay person, you don't have experience with the vocabulary that Apple and developers use. So the "tone" can be misleading. Saying that something is not supported is the normal way that Apple indicates prohibition. I would not expect them to say something stronger unless they felt like being verbose and happened to have specific knowledge about a particular problem that would be caused.

Just because spotlight is useful, I think nobody wants to stop using com.apple.metadata for a while.

Please don't speak for everybody. I don't use Spotlight, and I know I'm not the only one.

So, if OpenMeta applies your recommendation, we will have the same information on two places. Then, my question is; who has an incentive not to access com.apple.metadata but only org.openmeta?

Anyone who wants to write reliable code and not have to worry about what Apple does with "com.apple.metadata" or what the unknown side effects are of using it.

First case: We heard that Skim will access OpenMeta tags and show them on its information panel. What's an incentive for them to access org.openmeta but not to com.apple.metadata?

See above. Secondly, for accessing and showing tags in the information panel there is no benefit to using "com.apple.metadata".

Second case: Imagine a developer like Gravity decided to develop a new tagging/searching software. If they decide to use only org.openmeta, they need to develop their own search program as spotlight is not available. Who wants to choose org.openmeta instead of com.apple.metadata in this case?

A developer (such as myself) may already have their own search engine, so they don't need Spotlight. Or maybe their files aren't accessible to Spotlight, anyway (DEVONthink, for example) so there is no benefit to using "com.apple.metadata". I would like my customers to be able to choose whether they want to use both "org.openmeta" and "com.apple.metadata" and use Spotlight, or stay with what Apple supports and only use "org.openmeta". The current design of OpenMeta means that if they don't use the unsupported "com.apple.metadata", they can't share tags with other applications. And yet, nothing is gained by imposing this requirement except the exclusion of those who don't want to rely on hacks.

Honestly, I thought I made a reasonable and practical suggestion, and I don't understand why there's opposition. Would it help if I registered the domain and wrote the code?

Bluefrog:

My understanding is that Gravity has agreed to follow OpenMeta. Since Ironic owns OpenMeta, I thought it made sense to discuss it here. So, no, I am not seeking out all the developers that are using OpenMeta to repeat what I've said here. Presumably anyone who is interested in using OpenMeta is following the discussions about the project.

Bluefrog:

When it says "the toolkit will probably be incorporated in the future, according to the developers," doesn't that mean that they're planning to use OpenMeta?

"the toolkit will PROBABLY be incorporated in the future, according to the developers," (emphasis mine) doesn't that mean that they're planning to use OpenMeta?

I don't think this says anything concrete about their plans.

As I said previously…

I am also not aware of any communication between us and them concerning the extent of any cooperation (though Tom may have contact outside the channels I have access to).

… so I, unfortunately, cannot shed any more light on this right now.

Hoge:

It is you who has a vision. It is you who is ready to do it. Then, why don't you demonstrate it by yourself?

I don't think any demonstration is necessary. I've said what I want to do. shg understands it. If Tom doesn't understand it, I hope he will ask some questions. If it would help Tom to see the code, then I will write it, but I imagine that he understands what I have in mind. If Tom doesn't like the idea and is going to reject it, anyway, then I'm not going to waste my time writing the code.

Bluefrog:

I don't think this says anything concrete about their plans.

I don't understand why you are parsing the words so carefully. Gravity said they will probably do it. Nothing is for sure until they ship. Do you want them to specify a release date?

rlfsoso:

if org.openmeta would be used instead of com.apple.metadata how would spotlight search be enabled?

The user would have a choice of just "org.openmeta" (with no Spotlight search) or both (with Spotlight search working the same way it does now). If a developer doesn't want to ask the user, he could make his app always use both so that Spotlight search would always be enabled. So, if you want to use Spotlight, nothing would change for you, except that it would be a little more robust.

I don't understand why you are parsing the words so carefully. Gravity said they will probably do it. Nothing is for sure until they ship. Do you want them to specify a release date?

Because, to not read carefully is to risk misunderstanding. "Probably" is not a firm commitment so I would not assume they will do it. Things change, plans change - just look at push notifications for a current example… things change.

Either way, the salient point in my previous post was: Gravity put out their application before we debuted OpenMeta. We did not give them the com.apple.metadata code - they discovered what we discovered independently and make no mention of OpenMeta on their website (as of today - 01.30.2009). Therefore, since they are using the same mechanism discovered and used by them in their applications why aren't you contacting them? And they are, according to your logic, using an unsafe application and not telling anyone that it is unsafe… and charging for it too?!? (Note: I disagree with this line of logic and in no way mean to imply anything unethical or malicious on the part of Gravity Apps, LLC. - Jim)

Your argument is against the mechanism being unsafe, not against using kOM structured metadata so you should be contacting them with your concerns as well.

We appreciate the concern and welcome all opinions and suggestions (if we didn't we would close threads we didn't agree with - which we have never done) but you seem determined to single us out and I can't understand why.

Michael,

You (and shg) think it is the best to use org.openmeta as a primarily place and let users/developers choose whether to use com.apple.metadata. You said that the cost is trivial. You also think demonstration is not necessary.

I don't think so. From the perspective of user experience, demonstration is not trivial.It is a MUST.

Imagine that X.app saves tags only on org.openmeta and Y.apps saves on both places and uses spotlight for searching tags. It is not difficult to imagine an end-user who does not know the exact mechanisms but know both apps use OpenMeta starts to wonder why s/he can't search a tag added by X with Y. S/he wonders why only the tags added by Y can be searched from both X and Y. What if he uses many more OpenMeta applications? Furthermore, what if users can choose whether to use com.apple.metadata when they first install applications? Do standard end-users remember their choice of configurations? It is almost obvious that allowing users to make a choice makes the situation much more complicated. I recommended you to write demonstration partly because I didn't have any idea how you are to dissolve this problem.

Demonstration may be trivial only when tags are forced to be saved in both places. Only in that case, an end-user is not confused. Both X and Y always return the same search results. In that case, there is no distinction which is primary or secondary. In this case, it becomes almost meaningless to ask which EA an application uses. Both contains identical information. Choice is made only by a programmer's preference what search engine s/he wants to use. But, this is not what you (and shg) argued. Then, demonstration is indispensable.

Remember that we must assume users use multiple applications. OpenMeta enables it and that is THE power of OpenMeta.

I am an end-user and I don't have sufficient ability to make a judgment whether org.openmeta should be used. I think, if it is easy and does not provide any cost, it makes sense to implement org.openmeta together with com.apple.metadata. At least, Tom did not reveal explicit opinion about whether to use org.openmeta. Then, only what is logically concluded from this situation is that it is you who has a responsibility to show us how another world looks like.

I believe you made very important point on com.apple.metadata. I really appreciate that you patiently answered to almost all my questions. I really learned lots of new things from you. But, at least, it seems to be that your suggestion is not the only solution. It may turn out to be the best but only after somebody demonstrate it in front of the other developers (or, end-users). Let me repeat one thing. I don't (and, cannot) reject the possibility that your suggestion is the best one at the moment.

Do you think that system can be readily managed by end-users?

I feel you (and shg) tend to assume enlightened users - who know the logic of system perfectly and have a capacity to make a correct choice receiving all the necessary information. Not all the users who need tagging applications are like that. A system must be designed so that end-users can use it without thinking so much. In the world you envisioned, users may be able to get used to the system at the end but learning curve may be flat.

The world of Tagit.app. Tags.app. OpenMetaWizard.app, omtool and spotlight does not require any consideration. Copy apps on the application folders and that's it. I could tag with omtool and immediately search it with Tags.app. I synchronized two Macs with rsync 3.0 that detects changes in xattr and I could immediately search tags on the second machine because tags are promptly indexed by the spotlight on the second Mac after syncing. No thinking. No configuration.

User experience is important. That determines whether the system can attract users. It is always possible to provide a complex logic for solving a particular problem but it is not necessarily be user friendly. I believe that people are so much excited at OpenMeta mainly because of its seamlessness.

As I wrote, I know logical validity of your argument after asking so many questions. I thought it now is a time to see if it actually works. Mental simulation did not seem to be sufficient as, for instance, your solution in the last comment may work for some people but may be too much for the rest. What's solution can compromise multiple demands - security, robustness, transferability and user experience? I honestly think that you may be able to provide a solution. I'm not a perfect person. I make a misjudgment. I don't want to reject the possibility that something that seem to be bad can be actually good. That's why I think you need demonstration or something what is sufficient to give a flavor of another world to end-users.

At least, I personally think, forcing syncing com.apple.metadata and org.openmeta is one possible solution for providing assurance to end-users who are informed that com.apple.metadata is not officially supported.

Bare minimum of what we learned from Michael is that OpenMeta needs backup of tags for assuring users that it is secure. Now, we have one backup system. I can think of another system forcefully storing tags in both org.openmeta and com.apple.metadata. Which is better? Having both may be a foolproof way. Just my guess.

I'm also happy if we can hear at some point in near future (maybe not soon) from Tom himself how OpenMeta intends to solve the problem Michael pointed and can reassure end-users. I feel no word other than yours make end-users feel confident about OpenMeta.

I remain as excited as ever about OpenMeta, just as it is.

I see the merits of also writing copies of the tags to an org.openmeta namespace, so if anyone wants to take the time to write and submit code to do so, I say by all means do so.

But regardless whether anyone does that or not, OpenMeta is great, and is perfectly reliable as it is.

I am not concerned that Apple may somehow break OpenMeta later. It isn't impossible, but the risk is negligible. They spent a long time developing the current system, after all, and there is a lot built on top of it, so it should be around for a while yet. And if they do change it one day, OpenMeta can be rewritten to use the replacement system. Having implemented a metadata system of the scale they have, Apple isn't just going to drop it. Spotlight is here to stay, so even if they change the way some parts of the underlying system work one day, OpenMeta can be updated. And for those concerned that Apple might overwrite the com.apple.metadata extended attributes of files, it is inconceivable that Apple would do anything so drastic for anything less than a full point upgrade to the OS (if even then), which means that Ironic would have plenty of opportunity to test things on developer builds and update OpenMeta long before the hypothetical problem version of OS X were ever to be released.

So the long and the short of it is, while having a redundant copy of the tags in an org.openmeta namespace isn't a bad idea per se, all the doom-and-gloom in this thread is a bunch of FUD.

Sesquipedalian:

It does me no good to have my own system that's different from what other applications are using.

Tom:

Would you care to describe how your plist backup system is meant to work? The original OpenMeta was very easy to understand. It simply wrote the metadata into the xattrs. Now, it's much more complex, with more code and disk space devoted to the backups and their maintenance than to the actual metadata. Further, this is not a backup in the traditional sense, since the act of reading the metadata accesses the backup and can result in the data from the backup replacing the data in the xattrs.

Reading the code, there seem to be quite a lot of design decisions and subtleties, e.g. the format and location of the backup, the number of backups to keep, what happens to orphaned backups for files that were moved or deleted, which metadata is automatically backed up, how to detect when to restore, what happens when the xattrs are present but don't match the backup, how to handle long filenames and moved files, backups for files not on the boot volume, race conditions between different applications reading/writing to the xattrs and backup, what happens when the user Time Machine restores the file but not the backup or vice-versa, what happens when a file is in a shared location and multiple users have differing backups of its metadata, etc.

Some more specific questions:

What does kBackupStampOM do for us--is it necessary or an optimization? If the latter, how much faster does it make the system and what are the tradeoffs?

Why one file per backup? Is it not needlessly wasteful to store multiple 4 kilobyte files to backup perhaps a couple hundred bytes of metadata?

Won't performance degrade because HFS+ gets slow when there are many files in the same folder?

There are certain applications (photoshop, other CS4, subversion in some circumstances, etc) that can strip off all xattr - so a backup/restore solution is needed, not only for Apple wiping the disk of xattrs, but also the much more likely scenario of hard drive disasters, etc.

We've gone from me saying that "com.apple.metadata" is not to be relied upon to you worrying about all the xattrs being stripped off. And I agree with you that this is a real risk, especially with older applications that don't use metadata-preserving APIs.

OpenMeta now backs up all metadata set with it (all using kOM* as a key). These backups are in turn backed up by TimeMachine. This gets around having to twiddle with the mod date on a potentially HUGE file in order to back up a very small amount of metadata.

I don't like backing up large files, either, but one side effect of this is that Spotlight searches will return incorrect results after restoring from a backup. Perhaps there's no good way around that, but in any case it should be documented that this is what happens.

Slightly OT (but following a statement Tom made…)

I am documenting the Adobe Bug and have gotten the brush off by a Photoshop Engineer on their forums (of course, it's Apple's fault according to them). They are deleting data that is not their own (and not just OpenMeta, ANY EAs). If anyone else want to they can file a bug with Adobe at Feature Request/Bug Report Form

Jim

PS: This affects Adobe Illustrator as well as Photoshop. (Early tests show inDesign behaving as expected (another nod toward Adobe) but I haven't tested any other products yet). I encourage people to file a report for each product they're seeing the issue with.