Ironic Support Forum - OpenMeta Is a Hack

OpenMeta Is a Hack

2009-01-27T09:01:58-07:00

Put simply, OpenMeta is a hack. I say this not a value judgement but as statement of fact. It's clever, and it's useful, and it tricks Mac OS X into doing something it wasn't designed to do.

On Leopard, at least, Apple stores some metadata in a certain place (extended attributes prefixed with "com.apple.metadata:") and using a certain format (binary plist). Spotlight expects this and indexes those xattr values for searching. What OpenMeta does is store its data there, too. Spotlight, happily, seems to index it as well.

Unfortunately, Apple does not support this "feature." The storage location is reserved for Apple's use. The OS does not expect other applications to write data there, and the results are undefined. It might all work as desired, or the OpenMeta data might be deleted without warning, or it might not be consistently or promptly indexed, or it might work now but stop working in 10.5.7, or there might be some other side effect of mucking up Spotlight or other OS functions. We simply don't know.

OpenMeta is really two separate features combined into one. It's:

a) a common area for applications to share metadata
b) a way to make Spotlight index additional metadata, beyond what the Spotlight importer plug-ins produce

(a) is useful by itself, and it could be done in an Apple-supported way. Just choose an xattr prefix such as "org.openmeta:". But to do (b), OpenMeta needs to piggyback on undocumented and unsupported Leopard behaviors. Aside from the Apple xattrs mentioned above, the "ShortName" in schema.strings that's used for "tag:" searches is also unsupported, as far as I know. The Web page describes OpenMeta as not using any "secret API." This is true but misleading. There are many ways of doing unsupported things without using private API. For example, an application could modify or remove files in /System.

I think the developers should be up front about what OpenMeta is actually doing:

First, users and developers should know what they're getting into. I don't mean to spread FUD, and my personal belief is that it's probably safe, but we don't actually know. When I asked Apple for comment, they said that they do not support third-party software doing this. Some users like to live on the edge, and that's fine. Others like to play by the rules, run clean systems with no haxies, etc., and they may choose not to participate at this time. Perhaps there should be a way for such users to get (a) independently of (b).

Second, we need Apple's involvement. OpenMeta-type functionality is useful, and it would be better for all concerned if there were an Apple-blessed way to do (b). This might also solve the current problem that Time Machine only backs up the OpenMeta data if you change the modification date. People shouldn't think that OpenMeta is the solution to the metadata problem. Rather, it should whet their appetites so they encourage Apple to provide a real solution.

OpenMeta Is a Hack

2009-01-27T09:47:56-07:00

Michael:

1. We aren't using a "secret API". We are using Apple's own convention - no secret. Maybe it's surprising that it works like it does but it's not a secret.

2. We are not claiming OpenMeta is the Messiah of metadata. We are saying that we have stumbled on something that holds great promise to allow assignment of user-defined, arbitrary, shareable metadata. Also, this is an open source project. We encourage discussion and contribution on the matter (including yours) 8^).

it would be better for all concerned if there were an Apple-blessed way to do it would be better for all concerned if there were an Apple-blessed way to do…

3. Agreed, but Apple also has not found it a very important subject to deal with. We want to have these discussions and we want an airtight system. I know Tom and Ted have discussed this at WWDC (and probably more outside the conference that I am not privy too). So until something more "official" comes out should we sit idly by and not try and fix the problem ourselves? (If I may ask, who at Apple have you discussed this with? We would me more than happy to have a dialogue with the Spotlight people.)

Jim

(BTW, GravityApps says they arrived at essentially the same solution as we did and are selling Tags.app. They're getting a ton of press and accolades for their app. I'm just wondering if you will publicly flame them as well? 8^(

OpenMeta Is a Hack

2009-01-27T10:10:39-07:00

Bluefrog:

1. The main point of my post is that "no secret API" doesn't mean "does what you expect" or "safe" or "will continue to work." You could reverse-engineer the iTunes database format, figure out its "conventions," and modify it without iTunes' knowledge. You don't need any secret API to do this--just open the file and write to it. That doesn't make it supported or a good idea, though. And it would be misleading to say that you weren't doing anything secret.

2. I agree that it has great promise. That's why I'm trying to start a discussion. As to the code, I would prefer that it follow standard Cocoa naming and error handling conventions. These are amply documented by Apple. I've also found some bugs and will write them up on Google Code shortly.

3. No, you shouldn't sit idly by, but I think you should be open about what you are doing. I think that will increase the chances of Apple moving on this. Secondly, I think you should consider not tying the sharing functionality (a) to the secret stuff (b). I discussed this with Quinn at DTS, who talked to the Spotlight team.

I don't understand why you think my post was a flame. That was certainly not my intent. I'm talking with you guys because you're the ones developing and promoting this as a standard.

OpenMeta Is a Hack

2009-01-27T13:48:46-07:00

As an outsider looking in your tone comes across as arrogant/bullish.

Put simply, OpenMeta is a hack. I say this not a value judgement but as statement of fact.

If you're here to help then great, but opening a thread like this isn't the politest way to start off (even if it's true).

Hoping this leads to a friendly & productive outcome where everyone will benefit :)

OpenMeta Is a Hack

2009-01-27T14:03:11-07:00

My thoughts as a simple user are that OpenMeta needs to exist and flourish if only
to provide the proof of concept that Apple needs . Since the underlaying structure native
OS X I don't think Ironic is going to be pissed at the Pope if they eventually come out with a
clear and defined method for assigning and managing system-wide metadata tagging.

Chances are developers would work out some way of taking OpenMeta data and importing it
into whatever tool Apple delivers.

As a consumer I'm willing to take the risks. It may not work forever or there may be issues but
keeping dialogue with Apple is a way to ameliorate anything bad from happening.

I think many of us realize this is a bit off the beaten path but there have been worse hacks IMO.

OpenMeta Is a Hack

2009-01-27T14:04:18-07:00

Hi,

what concerns me is the question if this is going to keep working or not? For some time I thought spotlight-comments-based Tags would be a solution – and later have run into growing difficulties, Quicksilver not 100% working any more with Tags, TagBot occasionally overwriting Tags, the TagBot people simply going away etc. etc.

There was SpotMeta (which, if I recall right did use a similar "Xattr"-solution ) but with Leopard that was not compatible and not developed anymore. I intend to go with Apples further development of OSX (as long as my machine is going to support this) but I want to be at least assured that the people who implement this new scheme will make everything possible to make it likely that it wil continue to work in the reasonably future. Michael didn't start fear in my heart, he just gave me a starting point to ask these questions. (By the way I have asked a similar question the Tags-people, a question based on my past experiences with tagging solutions)

Greetings,

Rolf

OpenMeta Is a Hack

2009-01-27T14:36:48-07:00

Jono: I'm sorry if it came off that way. I was trying to be super clear that I was using the word in the original sense of "clever but unsupported" without implying that it's bad.

hmurchison: Judging by the wiki and Bluefrog's comments, I didn't think it was clear to most people that this is off the beaten path.

rlfsoso: The point is, it's not under our control whether this will work reliably or in future versions of Mac OS X. No matter how hard the developers work, it's up to Apple.

OpenMeta Is a Hack

2009-01-27T15:08:20-07:00

No problem Michael. Glad to hear your interested in helping :)

OpenMeta Is a Hack

2009-01-27T15:57:44-07:00

And I apologize if I have not been clear. I have made a very concerted attempt to be very clear about this project and I would be interested to know which statements I made that may be "less than clear" so I can avoid future misunderstandings.

Jim

OpenMeta Is a Hack

2009-01-27T16:50:15-07:00

Bluefrog:

I think the whole "no secret API" thing is unclear. It's in the wiki and in the PDF, and you brought it up as if to rebut my description of what OpenMeta is actually doing. It implies that OpenMeta is relying on supported/documented OS features, which is not the case. I haven't seen anyone from Ironic acknowledge this. On Twitter, you seemed to think that it was just as safe as using Spotlight comments, which for all their warts are an official feature of OS X.

OpenMeta Is a Hack

2009-01-27T17:17:47-07:00

I did not mean to imply it was "supported behavior for a 3rd party" but I still maintain that the mechanism itself is not secret. Unless you're meaning it in the sense of the Google app on the iPhone using the proximity sensor. In that case I may be able to agree with you semantically.

BTW, Spotlight Comments are an official feature of OS X but their use as "Tags" (per current convention) is also NOT supported behavior. ;^)

OpenMeta Is a Hack

2009-01-28T01:15:43-07:00

Quite the discussion! :) I am not as technical as the many who are commenting in this thread, but I would just like to add my perspective that although the openmeta may not be the final solution, it is a current solution. The various mediums that music came on could be a comparable example to tagging. We are past vinyls, cassettes, cd's and onto digital storage. So perhaps, openmeta is the 8-track of tagging. Sure, it may have some limitations and some bugs, it could be a bit like "hacking", but the developers are simply working with what they have available (and doing what I feel is a great job!).

I'm sure as things progress that they will provide ways to improve the tagging system, move onto new ways of keeping it, and provide bridges for us to convert our old metadata to new standards. I would suggest, like Michael Tsai suggested himself, that if anyone has qualms about this, they wait for several years until we get past the "8-track" phase. I just think it you may as well enjoy the benefits of using it now instead of being bound to folders (ooohhh! shiver!) :)

OpenMeta Is a Hack

2009-01-28T07:54:08-07:00

Bluefrog:

Apple does not document the mechanism. You discovered it by reverse-engineering. Now that you know about it, it's not secret anymore. This just goes to show that "secret" has no useful meaning. There is supported (Apple documents it and promises to make it work and continue to work) and unsupported (all bets are off). The Google case is similar, although I think listening for an undocumented notification is a different level of risk compared to (a) storing user data, and (b) modifying data owned by other software.

Regarding Spotlight comments, using them for tags is supported! You can put whatever you want in the comments, and Apple agrees to preserve them and make them searchable. You don't need special license to put tags in the comments, just like you can create a text file and write whatever you want.

Similarly, as I said in the original post, it would be supported if OpenMeta stored the tags in an extended attribute called "org.openmeta:kOMUserTags". This would provide all the same benefits as the current mechanism as far as letting applications share metadata with one another.

OpenMeta Is a Hack

2009-01-28T08:40:51-07:00

Regarding Spotlight comments, using them for tags is supported! You can put whatever you want in the comments, and Apple agrees to preserve them and make them searchable.

I disagree with this statement. The key word is "Tags".

Apple does not officially support using the Spotlight Comments as Tags. They support having a user editable text field in the GetInfo window that will be indexed and searchable by Spotlight: that is merely a mechanism. They do not advocate (or prohibit) the implementation. That is an incredibly HUGE difference. If you store all your "tags" in Spotlight Comments and lose them a call to Apple will not result in a "Let me transfer you to the Spotlight Comments Tags desk." In fact, I would say that such a call would most likely be answered with "We don't support using Spotlight Comments in that way." That whole idea (also a Hack) is merely another unsupported exploitation of a provided mechanism.

You don't need special license to put tags in the comments, just like you can create a text file and write whatever you want.

Very true, but that does not imply support of the use. If I store all my serial numbers and personal information in a text file and it is stolen or destroyed Apple is not going to say "We'll have to fix that since that's what we made TextEdit for." They made TextEdit for whatever a User wants to do with it and only support and are responsible for the mechanics of the product… creating a document, displaying characters, changing fonts, saving files, etc. If they choose to help a User solve a problem that's just good Customer Service and PR, not official support.

OpenMeta Is a Hack

2009-01-28T09:13:12-07:00

Bluefrog:

The Spotlight comments are an area where the user can do whatever he wants. There is no prescribed way to use them. Therefore, I do not think that using them to store tags is a hack. What are tags if not user data that you want to be searchable? As I see it, the two main problems with using the Spotlight comments for tags are (1) the API for accessing the comments is slow, and (2) if you use the comments field for tags, you can't use it for something else. These have nothing to do with support, though.

The confusion seems to be over the meaning of the word "support." I'm using it in the developer sense of, "Does Apple guarantee that the mechanism will work and continue to work, modulo bugs and broken hard drives." In this sense, the Spotlight comments and text files are both supported, as would be "org.openmeta:kOMUserTags".

OpenMeta Is a Hack

2009-01-28T12:52:16-07:00

Michael,

I have just one question. As long as I understand, you seem to suggest two things.

(1) Change a place of OpenMeta attributes from com.apple.metadata: to org.openmeta: or somewhere like this.
(2) Rewrite mdimporter only with a method officially approved by Apple.

(1) seems to be technically easy.It also seems to be both technically and practically easy to copy values stored in com.apple.metadata:kOMUserTags to org.openmeta:kOMUserTags. However, values stored under org.openmeta are not automatically detected / imported to spotlight database. Here comes the problem (2).

My question: How technically challenging do you think to solve the problem (2)? Is it technically easy, possible, difficult or impossible?

If it is easy / possible, I think we had better transit from com.apple.metadata to org.openmeta (given that what you said are all true). If it is difficult or impossible under the current methods officially approved from Apple, we may better stick to com.apple.metadata.

As some people have already pointed out, it seems to be a very simple trade-off problem. In my view, the most critical issue is whether one can write the code that work as current OpenMeta system only with the methods officially approved by Apple.

How do you think about the problem (2), Michael?

OpenMeta Is a Hack

2009-01-28T13:14:40-07:00

Michael,

Some of the points in here have already been made, but I reiterate them here, with my take.

Apple certainly could stop indexing openmeta data if they wanted, but the data would still be there, and it is not impossible to create a task similar to mds that listens to file system changes and creates an OpenMeta data store. Why not, though use what works?

xattrs are fully supported or documented, and the 'short names' can be found in Apple's sample code for Spotlight importers. /Developer/Examples/Metadata/ImporterExample/MyCustomImporter/English.lproj/schema.strings (spotlight search shortName).
Where is the documentation that says that you can't use 'com.apple.metadata'?

It is of course ok to use com.apple.metadata as a prefix - I want to mirror this data in the Apple metadata store! What other prefix would make sense? My spotlight data resides on my computer - not Apple's.

A hack is using a hammer to nail in a screw. What _is_ a hack is using spotlight comments prefixed with strange characters to store meta data - how do you store authors, workflows, ratings, times, etc in a space reserved for user editing? - You can't. Not to mention how does one handle multiple word tags, other languages and character sets, etc, etc. A hack is using something in a way it was not designed to be used. Comments were never designed to hold tags, urls, or other metadata. How big can a comment be? Is that bytes, characters or glyphs? Apple in fact guarantees nothing.

And I guess it would be fine in theory to store tags in comments if somehow everyone agreed to use the same prefix character, and have the same parsing rules, and had the same way of allowing user added 'real' comments to appear mixed in with the tags. (how do you add a comment about spotlight tagging in a spotlight comment without adding a tag?). But I have tried all of that and found that when you set spotlight comments by applescript, appleevents, editing the .DS_Store files, etc - all those methods lose data and only haltingly get synchronized to the spotlight database. So then I spend weeks trying to manually synchronize the comment data into spotlight. Tricky. At the end of all of this, all you get is lackluster slow support for one type of metadata in one character set.

Fundamentally, 'extended attributes' - which tags, authors, ratings, and other data are, should be stored with the xattr mechanism. The OS is designed to deal with them. By using com.apple.metadata in front of an extended attribute (only certain kinds), we are telling Apple to mirror my data in my spotlight data store. Certainly makes life simpler for users and programmers. But if Apple changes how it works, then the data is still there, and it still means the same thing, and building an sql db to search through it all and keep updated is not a big problem.
The real problem is agreeing on standard keys, and the formats for those keys. That is what OpenMeta is all about.

As far as getting Apple to do something like this: I have stood up and told them at appropriate times at at least 3 WWDCs (to more than scattered applause - developers want to do this.), filed bugs against it, etc. If this makes someone at Apple wake up and smell the coffee, then so be it. I would be very glad to write software that uses a meta data api that Apple wrote. But they don't appear ready to do it.

--Tom Andersen

OpenMeta Is a Hack

2009-01-28T13:23:44-07:00

Hoge:

(2) is impossible.

In my first post I mentioned two separate features (a) applications reading each other's tags and (b) searching those tags using Spotlight. Storing the tags at "org.openmeta:kOMUserTags" would enable (a) to work in a fully supported way. I think tag sharing has great potential, and I suggest that since it's trivial to do this in an Apple-approved way, perhaps that's what we should do. Why step off the beaten path unnecessarily?

(b) is obviously also useful. The current approach of using "com.apple.metadata:kOMUserTags" is the only way to do it, as far as I know. I suggest that the developers be up-front about this fact. Instead of saying that OpenMeta "uses no secret API" they could say that the searching feature relies on unsupported Spotlight behavior and that they hope Apple will make it supported in the future.

Then everyone could take advantage of (a) without any risk, and they could opt into (b) if they wanted. Presumably, Apple would receive lots of encouragement from users to make (b) an official Spotlight feature.

OpenMeta Is a Hack

2009-01-28T13:35:55-07:00

Michael,

Combined with Tom's last comment, I think everything gets clear to me. Thanks.

I then think that, as people wrote, this is a simple trade-off problem. We have two possible futures - one in which Apple does not interrupt developers to use com.apple.metadata and the other in which Apple actively prevents them to use it. We know we cannot exclude the possibility that the second world can happen but we don't know the exact probability it occurs. The question is whether the cost for changing the current system is worth to pay.

Given that 1) moving from com.apple.metadta to org.openmeta requires extra effort to develop a new search engine / program and 2) we're rather confident that Apple does not intentionally delete com.apple.metadata:kOMUserTags from all the files on Finder without any warning, sticking the current OpenMeta system seems to be very reasonable to me.

OpenMeta Is a Hack

2009-01-28T14:17:08-07:00

Well said, Hoge.

OpenMeta Is a Hack

2009-01-28T14:36:42-07:00

Tom:

Apple certainly could stop indexing openmeta data if they wanted, but the data would still be there

You don't know that the data would still be there, because OpenMeta is writing it into an area that it doesn't own. The OS would be well within its rights to delete any "com.apple" xattrs that it doesn't recognize. Similarly, you could figure out the format of the iTunes database and add a column, and it might work, but there's no guarantee that it won't be lost if iTunes ever rebuilds its database. And there's no guarantee that it won't break iTunes because it isn't expecting its database to be modified in that way.

the 'short names' can be found in Apple's sample code for Spotlight importers. /Developer/Examples/Metadata/ImporterExample/MyCustomImporter/English.lproj/schema.strings (spotlight search shortName).

Interesting. They are not mentioned in the documentation, and they don't appear in the Xcode template. But, assuming it isn't a mistake, I guess the example code counts as official.

Where is the documentation that says that you can't use 'com.apple.metadata'?

It should be obvious from the "com.apple" that this is Apple's area of the extended attributes. This isn't explicitly documented, which is why I asked Apple to clarify. And they said that they don't support this.

It is of course ok to use com.apple.metadata as a prefix - I want to mirror this data in the Apple metadata store! What other prefix would make sense? My spotlight data resides on my computer - not Apple's.

"Of course it's OK to use private API - I want to use the motion sensor! It's my phone, not Apple's."

I sympathize with this line of reasoning. But if that's what you're going to do, don't advertise your software as not relying on secret API.

how do you store authors, workflows, ratings, times, etc in a space reserved for user editing?

I never suggested that one should.

Not to mention how does one handle multiple word tags, other languages and character sets, etc, etc. A hack is using something in a way it was not designed to be used. Comments were never designed to hold tags, urls, or other metadata. How big can a comment be? Is that bytes, characters or glyphs?

I agree that a string is not a ideal way to store tag data. Nevertheless, the Finder's dictionary says that I can set a file's comment to a Unicode string. That string can contain whatever I want. Presumably, if the string is not acceptable to the OS it will report an error.

But I have tried all of that and found that when you set spotlight comments by applescript, appleevents, editing the .DS_Store files, etc - all those methods lose data and only haltingly get synchronized to the spotlight database.

I think it's fair to say that Spotlight comments are an Apple-supported technology that's buggy.

Fundamentally, 'extended attributes' - which tags, authors, ratings, and other data are, should be stored with the xattr mechanism. The OS is designed to deal with them.

Agreed. And the convention for extended attributes is that you prefix them with your own domain name, not someone else's.

Hoge:

I don't think anyone is worried about Apple actively breaking the "com.apple.metadata" trick, just because they're mean. However, it's possible that in the course of improving the OS they will change something that causes it to stop working. Secondly, since they never said it would work in the first place, I'm not sure I want to rely on it today. Who's to say whether I can search for a tag and actually find a particular file with that tag? The best I can do is say that I tried it with some other files and it seemed to work.

OpenMeta Is a Hack

2009-01-28T15:00:53-07:00

Michael,

I think I understand your logic. If one's subjective probability "Apple is a mean company which suddenly abandons com.apple.metadata (or, any other technologies OpenMeta relies on)", the cost for developing new search engine is negligible. On the other hand, if one's subjective probability is low, it makes sense to stick the current OpenMeta system.

I think this debate converges to a kind of subjective estimation of what can happen in future - what's your subjective probability that Apple is a mean company? I don't have sufficient amount of information for judging whose subjective probability is closer to the reality.

At least, I think conversation between you and Tom help other lay people like me to understand something important but only programmers can feel in one's bones through their experiences.

OpenMeta Is a Hack

2009-01-28T15:09:08-07:00

And the convention for extended attributes is that you prefix them with your own domain name, not someone else's.

If this is true, please point to the official documentation that supports this claim. Also, please point to the official documentation that prohibits use of com.apple.metadata.

However, it's possible that in the course of improving the OS they will change something that causes it to stop working.

Possible? This does happen. Just look at all the deprecated / changed functions in Leopard. Apple never promises to keep the status quo (and that's part of what we expect from them.) Did OS X screw up things for OS9 devs? Yes. Did going Intel screw up things for PPC only devs? You betcha! If everyone took the stance that Apple may not support a given procedure in the future nothing would ever get written.

Secondly, since they never said it would work in the first place

But they never said it wouldn't work either.

If we have the privilege of having a discussion with someone from the Spotlight / Metadata Team at Apple (which we hope and look forward to, especially outside WWDC) and they say, "Hey! You can't do that!!" or "Spotlight indexing or the filesystem is going to be changing in such a way that this is not a reliable mechanism." then we will have a nice talk about what can be done to remedy the current problems.

OpenMeta Is a Hack

2009-01-28T15:11:37-07:00

Hoge:

I don't think you got what I was saying. I was trying to show that there's a third possibility: Apple is not mean, and yet it doesn't work.

OpenMeta Is a Hack

2009-01-28T15:17:58-07:00

Michael,

It is not essential. Anyway, the biggest difference between you and Tom seems to be subjective estimation to what extent com.apple.metadata: is secure. That's it. It may be secure or may not. Basically, what you're suggesting is to avoid the worst scenario without paying attention to the benefit of taking a risk (and Tom and Jim estimate the risk is smaller than what you estimate).

OpenMeta Is a Hack

2009-01-28T16:10:50-07:00

Bluefrog:

Also, please point to the official documentation that prohibits use of com.apple.metadata.

That's not a realistic standard. If the location belongs to you, you are free to do with it as you choose. If it belongs to the system, it is automatically prohibited unless you are given permission. There is no official documentation telling you not to edit the database file for iTunes or Spotlight or Aperture directly. There doesn't need to be, because it's obvious the data is owned by those processes.

If everyone took the stance that Apple may not support a given procedure in the future nothing would ever get written.

This is why Apple tells us what they plan to support (through documentation, WWDC, etc.). Of course, there will always be new and better ways to do things, but Apple works very hard to maintain backward compatibility. Even most deprecated APIs still work--such as QuickDraw, which dates to the 80s.

The alternative, if everyone just reverse-engineers the OS and does whatever they feel like, leads to a fragile system. Remember what a mess extensions were with the original Mac OS? And it makes it hard for Apple to move the OS forward. There are behaviors that they never intended to support, but which third-party apps are relying on.

But they never said it wouldn't work either.

No. Apple does not make a list of all the things that don't work. They tell us what does work, which covers 99+% of what anyone wants to do, and the remainder is a gamble. It might do what you expect, it might do nothing, it might make your Mac overheat and catch on fire (not likely in this case, of course :-).

Hoge:

I'm suggesting that there is no reason to use a private Apple area as the primary storage location when we could use "org.openmeta" (presuming that someone purchases that domain :-). The data could also be written to "com.apple.metadata" in order to get the indexing. Then the (main) risk is that Apple changes the OS such that the indexing no longer works. But there's no risk of losing the data, because it's stored in the area that belongs to OpenMeta.

Secondly, consider that there will be lots of deployed applications using OpenMeta that want to read each other's tags. It's better if they rely on "org.openmeta" so that the sharing feature continues to work even if Apple changes the OS to restrict access to "com.apple.metadata" (as they already do for certain special xattrs). The same is true if Apple changes the format for the data in "com.apple.metadata". It would be a mess to have a mix of old and new formats. But if the data is stored under "org.openmeta" the format can be stable and whatever we want, and we can convert it to Apple's current format as needed for indexing.

OpenMeta Is a Hack

2009-01-28T17:00:56-07:00

Michael,

I think we are trapped in never-ending cycle.

Validity of your argument totally depends on how likely one thinks com.apple.metadata disappears in future. You think it likely to happen. Because of such your belief, it is rational for you to develop a system for storing and searching metadata in org.openmeta.

I (Jim and Tom) think that it is less likely. Furthermore, the benefit of using com.apple.metadata is that we can use familiar spotlight. It is really, really useful. Throwing away this mega benefit is totally ridiculous to me. There is a reason to use com.apple.metadata.

If only the current benefit of using org.openmeta is that it is less likely to disappear than com.apple.metadata, the rational solution seems to just make a backup of com.apple.metada:kOMUserTags on org.openmeta:kOMUserTags and synchronize them perfectly. That's it.If com.apple.metadata is actually deactivated, then I will wait for Tom to write sql database or whatever as he wrote in the last comment. You may say that we should not wait but should move to the new system earlier. I don't buy that argument just because spotlight is so handy and useful.

So, my final question is; is there anything you recommend other than that keeping information on both places (com.apple.metadata and org.openmeta)? Technically speaking, it seems to be a tiny change on a source code. I'm pretty confused why you need to make this problem so big. To me as an end-user, wording does not matter at all. As long as OpenMeta is more robust than spotlight comments, it is fine. What you suggest at the end is just to make a backup of com.apple.metadata on org.openmeta so that OpenMeta gets more robust. That's a simple suggestion that does not require any big story. (I'm sorry if making a backup on org.openmeta is only the technical suggestion you made from the beginning of this thread.)

OpenMeta Is a Hack

2009-01-28T18:13:10-07:00

Hoge:

I like to follow the principle of defensive coding. When building on top of someone else's software (Apple's, in this case) it's best to rely on the specification (API, documentation) rather than on incidental behaviors that you happen to observe. Liklihood doesn't really enter into it. If there's a known safe way to do something, and it's super-easy, then I would expect that to be the default choice. It's irrational to play the odds for no benefit.

So the primary tag storage should be "org.openmeta:kOMUserTags". "com.apple.metadata:kOMUserTags" should be the backup, not the other way around; otherwise that would defeat the purpose.

The developers have repeatedly said that the main point of OpenMeta is to develop a way for applications to share metadata. Tom recently wrote, "The real problem is agreeing on standard keys, and the formats for those keys. That is what OpenMeta is all about."

Yet, as it stands now, to participate in this sharing, an application is required to write to "com.apple.metadata:". This may have negative consequences today, and we don't know about the future. If you want the benefits of Spotlight, that's your choice. However, I think that:

(1) Users should be able to participate in the sharing but opt out of the unsafe stuff.

(2) The developers should be open about "com.apple.metadata:" not being supported.

These are the same suggestions that I had in the first post. I'm sorry the thread has gotten to be so long.

OpenMeta Is a Hack

2009-01-28T18:27:39-07:00

I feel that another difference between you and Tom (& Ted) may be a stance as a programmer. You try to build a software under the strict constraints of specifications. I appreciate that stance. Another stance may be to pursue practical benefit at the sake of going across the border which I also appreciate in some cases.

Anyway, as an end user, I hope that the bottom line of this controversy is to improve OpenMeta and make it more robust and useful.

OpenMeta Is a Hack

2009-01-29T08:58:20-07:00

Does anybody happen to know why Skim saves its annotations on net_sourceforge_skim-app_notes but not under com.apple.metadata? I'm just curious if they made this decision because they thought the former is safer than the latter or any other reason.

OpenMeta Is a Hack

2009-01-29T09:09:51-07:00

I can't say for sure but I would guess that they hadn't even considered using com.apple.metadata.

OpenMeta Is a Hack

2009-01-29T10:16:10-07:00

OpenMeta (now) stores backups of all metadata set with the system. So in the event of someone erasing tags, etc the system can recover them. Tagit and Deep have not yet been updated to use the new backup system, but we are working on it.

It only makes sense to use com.apple.metadata for data you want mirrored in spotlight. I assume that much of the data that skim needs is locations colours, etc. They could (should?) also enter any short text they want searchable with an OpenMeta like keyword.

--Tom

OpenMeta Is a Hack

2009-01-29T12:50:15-07:00

Thank you, Jim and Tom.

>It only makes sense to use com.apple.metadata for data you want mirrored in spotlight.

Actually, annotations can be searched by spotlight if a file is saved as "pdf bundle", a skim's original package format where annotations are saved as *.txt/rtf inside of a package. So, I had just wondered why they did make skim so that annotations can be searched only when it is saved as a pdf bundle (or, when annotaions are saved as a separate file).

Just let me allow murmuring some thoughts of a non-professional.

Honestly speaking, I still don't get the logic why we can logically conclude that org.openmeta is safer than com.apple.metadata. I had relied on -f option of mdimport on Tiger for my shell scripts. It was an official option even described on the documents. But, it is obsolete on Leopard which annoyed me for a while. This is just a tiny example. But, it seems to me, a non-professional end user, that even what is explicitly supported on the Apple's official document is mortal. I may be missing something important but can't stop thinking in this way.

OpenMeta Is a Hack

2009-01-29T13:18:21-07:00

So, I had just wondered why they did make skim so that annotations can be searched only when it is saved as a pdf bundle (or, when annotaions are saved as a separate file).

Because they either did not know about or did not want to use the undocumented "com.apple.metadata" trick.

Honestly speaking, I still don't get the logic why we can logically conclude that org.openmeta is safer than com.apple.metadata.

Because the former is owned by OpenMeta and the latter is owned by Apple. Do you think it's safer to store your data in your own file or inside com.apple.safari.plist, for example?

But, it seems to me, a non-professional end user, that even what is explicitly supported on the Apple's official document is mortal.

Your logic seems to be that because Apple doesn't always do what they say (or they change their mind), one should ignore what they say.

OpenMeta Is a Hack

2009-01-29T13:33:36-07:00

Michael,

Thanks for your reply. I wrote the last comment hoping that I can learn something new what I don't know yet.

>Because the former is owned by OpenMeta and the latter is owned by Apple. Do you think it's safer to store your data in your own file or inside com.apple.safari.plist, for example?

What does it mean com.apple.metadata is *owned* by Apple? As long as I understand it's just EA. Even I can make an arbitrary EA with com.apple.metadata prefix using xattr on terminal (thought it is useless). I understand that nobody can deny the possibility that such an arbitrary EA may be excluded from spotlight search in future. That's certainly a possibility. But, how about preservation of an arbitrary EA with com.apple.metadata prefix? If Apple moves to a new file system that does not support EA at all, it can happen that my arbitrary EA is lost together with all the rest. What kind of story can we think of in which my arbitrary com.apple.metadata is *deleted* but my org.openmeta EA is not deleted?

As an end-user, I'm confused about the meaning of *owned* by Apple. Sorry for asking question that may be too obvious to you.

OpenMeta Is a Hack

2009-01-29T14:32:55-07:00

Hoge:

The owner determines whether it's preserved or deleted, how and when the data are interpreted, and what the side effects are of modifying it, etc. Of course, files and folders can be owned, too. Apple owns /System and some other folders. Applications own their folders in Application Support and the files that they create. Rejecting the idea of ownership would mean that you think it's OK to modify some other application's database without its knowledge or permission. In fact, extended attributes are simply a database file that's managed by the filesystem. The prefix for EAs determines ownership just as the folder structure does for files. By editing Apple's EAs, you could make files display incorrectly, prevent applications from opening or perhaps make them crash if the data is no longer in the expected format, corrupt a Time Machine backup, etc. Likewise, by "just" creating a file in Terminal you could affect the behavior of other applications and even get the OS to launch new processes.

If you want to modify data that another application owns in a safe way, you should to get permission or be able to prove (usually impossible, because you don't know exactly how the other application works) that you aren't causing harm. You could instead do what you want and ask the application developer and the user for forgiveness if something goes wrong, but I think that would be rather rude.

One possible story is that Apple decides to store the metadata elsewhere. So it copies the xattrs it created and then deletes all of "com.apple.metadata:" in order to record the fact that the data has been migrated and free up disk space.

OpenMeta Is a Hack

2009-01-29T15:33:42-07:00

Michael,

Yeah, assuming that something is *owned* by somebody, s/he has a right to do anything on his/her property.

In the first comment, Tom wrote that com.apple.metada is just a prefix. When having read it, I thought com.apple.metadata is just a name that can be used for spotlight-indexing all the attributes under it. Then, you wrote;

> It should be obvious from the "com.apple" that this is Apple's area of the extended attributes. This isn't explicitly documented,

So, now I understood that why people can see different meanings in com.apple.metadata because it is not explicitly documented. You continued;

> which is why I asked Apple to clarify. And they said that they don't support this.

This sounds very critical as this seems to be only the explicit information that can be used for defining the meaning of com.apple.metadata. I'm curious what Apple told to you. What statements made you convinced of that com.apple.metadata is owned by Apple but not just a prefix?

OpenMeta Is a Hack

2009-01-29T18:06:53-07:00

mmmh ... you know, the OS could "choose to overwrite" almost anything, so that argument seems flawed.
Also, it has been my experience with OS X, the best stuff is always in the "hacks", or to phrase it differently, apps that do things slightly differently then would be possible if they only used "blessed by apple" methods.
Sad, but true.

OpenMeta Is a Hack

2009-01-29T19:48:41-07:00

Hoge:

Tom didn't say it was just a prefix. He said it gave him his desired result, so he was going to use it.

Prefixes are a common pattern in software that are used to indicate ownership. This is what's obvious and doesn't need to be documented. The question I had for Apple was not whether Apple owned it but whether they would support developers using it. For example, "com.apple.TextEncoding" is an extended attribute that Apple owns, but they document it and explain how it should be used. I was hoping that, even though "com.apple.metadata" isn't documented, maybe Apple was OK with developers using it. Maybe they would give permission and make it official in the future. This is not uncommon, to have Apple say that something is not official yet, but go ahead and do it because it's in line with what they are planning. However, that was not the case this time.

step21:

The OS can't choose to overwrite anything. If it deleted user data (that wasn't stored in one of Apple's reserved areas), that would be a bug, and the users would have a legitimate complaint. Certainly, one can do lots of cool stuff with hacks. I'm just saying, (1) don't be under the illusion that this is supported, and (2) half of OpenMeta can be easily implemented in "blessed" fashion, so I think it should be.

OpenMeta Is a Hack

2009-01-29T20:50:08-07:00

Michael:

You keep referring to this "conversation" you've had with Apple (with "Quinn" (I'm hazarding a guess that it may be the Internet Config guy) who has supposedly spoken to someone on the Spotlight Team). What would be nice would be: 1) Documentation of that discussion, including a transcript, and 2) name of contacts on the Spotlight Team that were consulted about this matter.

I understand you're passionate about your opinion on this subject but your supporting statements are purely anecdotal and without corroborating evidence. Also, as a courtesy to us, why not provide your contacts so we may make our own inquiries into these matters?

OpenMeta Is a Hack

2009-01-29T22:05:13-07:00

Let me join this discussion. I, as a potential OpenMeta developer, think the point that Michael has brought up is very important.

BLUEFROG,

The issue does not depend on what the Apple guys actually told to Michael. Of course, Michael can provide such information and it is somewhat interesting, but the issue exists without such evidence. We know that if Apple officially states that com.apple.metadata can be used, then the issue is gone. However the issue does not disappear even if Apple does not explicitly state that com.apple.metadata must not be used.

Michael's suggestion is, as far as I understand, that 1) OpenMeta should use org.openmeta (for example) for the primary storage and 2) a developer can also chose to use com.apple.metadata to receive benefit from spotlight.

I think this is a practical and reasonable solution, which gives both security and usability. A drawback is that an application that wants to utilize spotlight needs to maintain two storage. But I think it is trivial. Does anyone have a better solution? Also is there any other drawbacks with this solution?

I don't think we should not use com.apple.metadata. Personally I like hacks very much. But if OpenMeta offers a choice that does not depend on the undocumented behavior of the OS as well, then it will be more attractive to every developer.

OpenMeta Is a Hack

2009-01-30T01:24:05-07:00

shg,

You're talking about actions one should take. Such argument does make sense only when people see it is necessary. In this case, people do have different thoughts about the necessity of taking that action. Only when majority (or, those who commit modifying codes at google code) see the necessity, I think we can start fruitful discussion what action we actually need to take.

What I believe we need to discuss is what made the differences in our perception of necessity to take the action.

-------------
At the moment, the fact is that the meaning and definition of com.apple.metadata is not documented. This seems to be only the certain fact.

Some believes that it is common sense that non-documented technic is unsafe. Some believes that it is not (or safer than the former group of people think). This is just a difference in programming policy. This is a difference in interpretations of the identical fact. We can never settle such differences.

If the source of Michael's argument was only the fact that com.apple.metadata is not documented, I didn't post the last comment.
--------------

Michael,

As Jim wrote, I'm really curious what Apple told you. It does not provide definite answer but you seem to be only the person who has explicit information that does not include any guess. That would be really valuable for those who are interested in this issue if you share the fact. I know your policy. I appreciate your programming style. But, I'm much more happier if you give us a bit more clarified descriptions of what you heard from Apple.

---------------
p.s.1
Should it be explicitly noticed that com.apple.metadata is not documented? I think Michael is right in this sense (especially if it is convention among programmers). But, an interpretation of this fact is open to the developers who read that statement.

p.s.2
Should tags be saved on both com.apple.metadata and org.openmeta? I don't know but, at least, the latest OpenMeta code saves backup of com.apple.metadata in ~/Library. Is it sufficient for those who think com.apple.metadata is insecure and tags may be lost in future? I can imagine that, when com.apple.metadata is forcefully deleted in future OS X, somebody (maybe Tom and Ted?) write a program that copies tags from backup to org.openmeta and write a new search function with sql or whatever. So, this seems to be one possible answer to the argument that "com.apple.metadata may disappear. Is it too late to make org.openmeta when that time comes? What's pros and cons of keeping tags in both com.apple.metadata and org.openmeta?

OpenMeta Is a Hack

2009-01-30T06:46:48-07:00

Welcome, shg and thanks for your comments.

Personally I think the opposite is more desirable (1) com.apple.metadata as primary so Spotlight immediately indexes without intervention, and (2) org.openmeta as the secondary. As our method continues to work… fantastic. If it doesn't then it would be fairly easy to switch over to the org.openmeta setup. (Bear in mind that I am not doing the programming.)

BTW, I hope no one is taking this discussion as antagonistic or mean spirited. Passion is good and it's obvious that there are passionate people involved here. The point is a Dialogue has been started. This is all very good stuff. - Jim

OpenMeta Is a Hack

2009-01-30T09:18:00-07:00

Quinn (yes, the Internet Config guy, who now works in Apple Developer Relations) wrote:

The word from Spotlight engineering is that the "com.apple.metadata:" extended attribute prefix technique is not something that we officially support.

If you'd like to see official support for this feature added in the future, I encourage you to file a bug describing your requirements. While we may have seen similar requests before, a fresh bug report will allow you to express your needs in your own terms, and allow Spotlight engineering to gauge the level of demand.

The internal Apple follow-up number is 65169852. I agree with shg that this is not very relevant. If Apple had said that this was supported, we would have learned something new. Since they confirmed that it's not supported, we don't really have any new information.

I think it would have been courteous for OpenMeta to get feedback from the community and Apple before announcing and shipping software that was doing something questionable. Instead of telling developers that OpenMeta uses no secret API, I think the wiki should say that "com.apple.metadata" is not supported and list the Radar number of the feature request to make it supported, so that developers can express their wishes and Apple can gauge demand.

Instead, we now have people like Hoge who seem to say that, in the absence of overwhelming proof that there will be harm, we should keep the status quo. Yes, the code could be modified in the future if a problem arises, but this can get messy when there are lots of users with data in the old format and multiple applications containing different versions of the code. I think that OpenMeta is still very new, so there's still time to back up a bit, think about the design, and do something sensible, practical, and forward-looking.

Hoge:

I think there are essentially no cons to keeping the data in both "com.apple.metadata" and "org.openmeta". The code changes would be trivial.

I've not seen a description from Tom about how the ~/Library backups work and what they are intended to do. However, from skimming the code, this does not seem like an attractive approach to me. The primary storage remains the unsupported location, and it adds an extra "backup" location. For every file that has tags, there will be one or more plist files in the user's home folder to store backups of these tags. That is potentially a lot of overhead, and I don't want to create backup files in my home folder for files that are stored on other drives.

Bluefrog:

Personally I think the opposite is more desirable (1) com.apple.metadata as primary so Spotlight immediately indexes without intervention, and (2) org.openmeta as the secondary. As our method continues to work… fantastic. If it doesn't then it would be fairly easy to switch over to the org.openmeta setup.

Why do you think that? I cannot think of any benefit to making the unsupported location be primary. There are at last two benefits to making "org.openmeta" primary:

(1) No switch would be necessary. If the indexing trick stops working, the user data will be intact, and applications will still be able to share tags with no code changes and no need to update the format of the existing data.

(2) Users and/or developers could choose not to depend on the undocumented behavior at all, which would make it more likely for OpenMeta to be widely adopted.

OpenMeta Is a Hack

2009-01-30T11:07:33-07:00

Michael,

Thank you for your patient reply.

> I agree with shg that this is not very relevant.

Your new information was valuable as it tells lay people a tone of the issue.

In principle, you're right - it is hack if we make a binary judgment. However, I had imagined that you heard kind of very strong warning from Apple as you frequently used a phrase such as owned by Apple. You know that property or possession give very strong prohibitive meaning to lay people. With the reference you indicated, I think I can understand it is avoidable people keep having different attitudes to com.apple.metadata. Concerning the future of com.apple.metadata, the reference sounds rather neutral. As I can't stop recalling several instances that were on the official documents but suddenly became obsolete without warning, I don't know how much negative value this reference indicates. But, I have been sure that you indicated us, end-users, very important point that anything can happen on com.apple.metadata. I appreciate it.

-------------------------
Aside from semantics, practical solution seems to be a bit complicated.

I think that you recommend to use org.openmeta for two reasons. First, it won't disappear. Second, OpenMeta should provide developers freedom not using unsupported feature of OS. I perfectly agree that we had better to make backup of com.apple.metadata given its unclear future. I'm concerned the second reason.

Just because spotlight is useful, I think nobody wants to stop using com.apple.metadata for a while. So, if OpenMeta applies your recommendation, we will have the same information on two places. Then, my question is; who has an incentive not to access com.apple.metadata but only org.openmeta?

First case: We heard that Skim will access OpenMeta tags and show them on its information panel. What's an incentive for them to access org.openmeta but not to com.apple.metadata?

Second case: Imagine a developer like Gravity decided to develop a new tagging/searching software. If they decide to use only org.openmeta, they need to develop their own search program as spotlight is not available. Who wants to choose org.openmeta instead of com.apple.metadata in this case? Maybe those who have a strong belief that com.apple.metadata will have no future and want to avoid rewriting a software when it is abandoned. Where is such a developer?

I understand the following logic - there may appear who desires a feature X. So, we had better implement such a feature so that we can include them. This logic gets more valid as such a demand increases. I just want know how likely such a developer appears because I literally have no idea how strong such a demand is. Do you have any idea?

If such a demand is not so strong, then, the reason to have org.openmeta is mainly for backup.

In general, keeping the information in different formats (xattr & files in ~/Library) seems to be more robust than keeping the information in the same format and in close places. Given that com.apple.metadata's future is unclear and file contents are more robust than xattr, Tom's solution seems to be reasonable one.
------------------------

I think that I'm irritating you but I'm just curious. And, it is my style to take extremely opposite position when trying to understand a person who has a strong opinion. Believe it or not, I think I've never rejected an idea of org.openmeta. I just want to know how valid the argument is.

OpenMeta Is a Hack

2009-01-30T11:17:41-07:00

Hoge brings up a good point that reminds me of something I asked early on… GravityApps is using the same mechanism as we are though independently discovered. Are you having the same discussions with them??

OpenMeta Is a Hack

2009-01-30T12:25:56-07:00

Hoge:

Your new information was valuable as it tells lay people a tone of the issue.

I think, as a lay person, you don't have experience with the vocabulary that Apple and developers use. So the "tone" can be misleading. Saying that something is not supported is the normal way that Apple indicates prohibition. I would not expect them to say something stronger unless they felt like being verbose and happened to have specific knowledge about a particular problem that would be caused.

Just because spotlight is useful, I think nobody wants to stop using com.apple.metadata for a while.

Please don't speak for everybody. I don't use Spotlight, and I know I'm not the only one.

So, if OpenMeta applies your recommendation, we will have the same information on two places. Then, my question is; who has an incentive not to access com.apple.metadata but only org.openmeta?

Anyone who wants to write reliable code and not have to worry about what Apple does with "com.apple.metadata" or what the unknown side effects are of using it.

First case: We heard that Skim will access OpenMeta tags and show them on its information panel. What's an incentive for them to access org.openmeta but not to com.apple.metadata?

See above. Secondly, for accessing and showing tags in the information panel there is no benefit to using "com.apple.metadata".

Second case: Imagine a developer like Gravity decided to develop a new tagging/searching software. If they decide to use only org.openmeta, they need to develop their own search program as spotlight is not available. Who wants to choose org.openmeta instead of com.apple.metadata in this case?

A developer (such as myself) may already have their own search engine, so they don't need Spotlight. Or maybe their files aren't accessible to Spotlight, anyway (DEVONthink, for example) so there is no benefit to using "com.apple.metadata". I would like my customers to be able to choose whether they want to use both "org.openmeta" and "com.apple.metadata" and use Spotlight, or stay with what Apple supports and only use "org.openmeta". The current design of OpenMeta means that if they don't use the unsupported "com.apple.metadata", they can't share tags with other applications. And yet, nothing is gained by imposing this requirement except the exclusion of those who don't want to rely on hacks.

Honestly, I thought I made a reasonable and practical suggestion, and I don't understand why there's opposition. Would it help if I registered the domain and wrote the code?

Bluefrog:

My understanding is that Gravity has agreed to follow OpenMeta. Since Ironic owns OpenMeta, I thought it made sense to discuss it here. So, no, I am not seeking out all the developers that are using OpenMeta to repeat what I've said here. Presumably anyone who is interested in using OpenMeta is following the discussions about the project.

OpenMeta Is a Hack

2009-01-30T13:22:14-07:00

From TUAW: Tags takes organization to a new level

Tags are immediately indexed in Spotlight, allowing for searches and Smart Folders outside of Tags, as well as integration with other Spotlight-enabled applications. Its keyword storage method is directly compatible with Ironic Software's Deep, and the same method is planned for use in Leap, eventually. Ironic has actually just announced OpenMeta, an open source library for accessing and modifying this kind of metadata (more on that coming soon).

Comment from the author of the aforementioned article:

Brett Terpstra said 1:43PM on 1-19-2009

Actually, it's not using the OpenMeta tool set, yet (I did link OpenMeta above). The keys Tags uses were changed prior to release to work with the standard, and the toolkit will probably be incorporated in the future, according to the developers. …

Gravity put out the app just before we opened OpenMeta. I am also not aware of any communication between us and them concerning the extent of any cooperation (though Tom may have contact outside the channels I have access to).

OpenMeta Is a Hack

2009-01-30T14:03:49-07:00

Bluefrog:

When it says "the toolkit will probably be incorporated in the future, according to the developers," doesn't that mean that they're planning to use OpenMeta?

OpenMeta Is a Hack

2009-01-30T14:08:44-07:00

Hey, Michael. I was waiting for that words.

>Would it help if I registered the domain and wrote the code?

I feel that most of people agree your argument that com.apple.metadata is not officially supported and the data may disappear in future. However, many people who are interested in OpenMeta seem to be attracted to use spotlight. You cannot deny this fact. For these people, it is sufficient if at least one way to backup the metada is provided. Now, it is implemented in OpenMeta. No matter how it seems to be tedious to copy the data from bakcup files in ~/Library to org.openmeta, there is a backup. So, they don't need to worry about data loss.

Who else is interested in org.openmeta? Here is a developer who wants to use org.openmeta other than for just in a case. He does seem to have a concrete plan to use org.openmeta intensively for his software. He wants to make it compatible with naming rules and kinds of attributes used in the current OpenMeta system. He is a programmer who has a capacity to engage in code development. Moreover, he believes it is good for all of us.

As long as I can see, it is only you (and shg) who can (and want to) add the code for mirroing data between com.apple.metadata and org.openmeta (by the way, I thins it is a disaster if developers can choose where metadata is store. End users are usually not aware of where the data is store. What if they find that they cannot find a tag they thought they surely added?). How about Tom? I don't know but he seems to think it's sufficient to have a backup in Library.

If you show the other developers who registered google code that having org.openmeta literally does not cause any problem (I don't know but, say, size limit of EA, realibility and speed of mirroing two EAs, complexity and readbility of codes, etc.), then I think they should accept it because no harm is proved by you. If Tom still rejcts your suggestion after you proof and support from the other developers, it just means that he is stubborn! I hope not.

No matter how you argue that it is beneficial and has no cost, those who are not interested in it will use his or her extra time for working something else. It is their right.

It is you who has a vision. It is you who is ready to do it. Then, why don't you demonstrate it by yourself?

>I think, as a lay person, you don't have experience with the vocabulary that Apple and developers use. So the "tone" can be misleading.

Absolutely. We end-users need to rely on judgment of multiple developers. If you show implementing org.openmeta (aside from) com.apple.metadata does not have any side effect AND majority of developers see your argument is somehow plausible, I believe that any reasonable developers will accept your modification of the code.

OpenMeta Is a Hack

2009-01-30T14:14:50-07:00

"the toolkit will PROBABLY be incorporated in the future, according to the developers," (emphasis mine) doesn't that mean that they're planning to use OpenMeta?

I don't think this says anything concrete about their plans.

As I said previously…

I am also not aware of any communication between us and them concerning the extent of any cooperation (though Tom may have contact outside the channels I have access to).

… so I, unfortunately, cannot shed any more light on this right now.

OpenMeta Is a Hack

2009-01-30T14:16:40-07:00

Hi,

just a question: if org.openmeta would be used instead of com.apple.metadata how would spotlight search be enabled? Via a spotlight-importer (sorry about the term, I donot recall the right technical term)?

I want to point out that the ability to use Spotlight (smart searches/folders; the ability to combine a spotlight search with other search criterias via programs like the excellent HoudahSpot (HS)) is very important to me! Use of third-party-software (HS) is able to overcome limitations in Apples latest installment of Spotlight in Leopard…

As enduser I do not care so much for what is going on under the hood – as long things continue to work (or are being kept in shape through maintenance)

Greetings,
Rolf

OpenMeta Is a Hack

2009-01-30T14:55:55-07:00

Hoge:

It is you who has a vision. It is you who is ready to do it. Then, why don't you demonstrate it by yourself?

I don't think any demonstration is necessary. I've said what I want to do. shg understands it. If Tom doesn't understand it, I hope he will ask some questions. If it would help Tom to see the code, then I will write it, but I imagine that he understands what I have in mind. If Tom doesn't like the idea and is going to reject it, anyway, then I'm not going to waste my time writing the code.

Bluefrog:

I don't think this says anything concrete about their plans.

I don't understand why you are parsing the words so carefully. Gravity said they will probably do it. Nothing is for sure until they ship. Do you want them to specify a release date?

rlfsoso:

if org.openmeta would be used instead of com.apple.metadata how would spotlight search be enabled?

The user would have a choice of just "org.openmeta" (with no Spotlight search) or both (with Spotlight search working the same way it does now). If a developer doesn't want to ask the user, he could make his app always use both so that Spotlight search would always be enabled. So, if you want to use Spotlight, nothing would change for you, except that it would be a little more robust.

OpenMeta Is a Hack

2009-01-30T15:06:49-07:00

rifsoso,

As long as I understand, if a tag is stored under org.openmeta, Spotlight does not index it. That's why kOMUserTags is under com.apple.metada right now which is not officially supported. If I am correct, only the solution that can satisfy two needs (use spotlight & use org.openmeta) is to store a tag in both places. If it is also true that mirroring both EAs does not cause any problem, I think the issue is who will implement it.

OpenMeta Is a Hack

2009-01-30T15:40:59-07:00

I don't understand why you are parsing the words so carefully. Gravity said they will probably do it. Nothing is for sure until they ship. Do you want them to specify a release date?

Because, to not read carefully is to risk misunderstanding. "Probably" is not a firm commitment so I would not assume they will do it. Things change, plans change - just look at push notifications for a current example… things change.

Either way, the salient point in my previous post was: Gravity put out their application before we debuted OpenMeta. We did not give them the com.apple.metadata code - they discovered what we discovered independently and make no mention of OpenMeta on their website (as of today - 01.30.2009). Therefore, since they are using the same mechanism discovered and used by them in their applications why aren't you contacting them? And they are, according to your logic, using an unsafe application and not telling anyone that it is unsafe… and charging for it too?!? (Note: I disagree with this line of logic and in no way mean to imply anything unethical or malicious on the part of Gravity Apps, LLC. - Jim)

Your argument is against the mechanism being unsafe, not against using kOM structured metadata so you should be contacting them with your concerns as well.

We appreciate the concern and welcome all opinions and suggestions (if we didn't we would close threads we didn't agree with - which we have never done) but you seem determined to single us out and I can't understand why.

OpenMeta Is a Hack

2009-01-30T16:11:12-07:00

Bluefrog:

My understanding is that Deep was the first application to use "com.apple.metadata". I don't know how Gravity found out about it. Maybe they discovered it independently; maybe they reverse-engineered Deep. It doesn't really matter.

My goal is for all the various applications to work in the safer way I suggested above. I think the way to do this is to (a) change OpenMeta, and (b) get everyone to use OpenMeta. If Gravity is using OpenMeta, then their application will automatically conform after OpenMeta is changed. If Gravity decides not to use OpenMeta, I will ask them to reconsider. For now, Gravity said they will probably use OpenMeta, and I take them at their word.

I am singling you out because you're the ones promoting the standard. If I had a concern about WebKit, I would contact Apple (the maintainers of WebKit), not the OmniGroup (who make a browser that uses WebKit).

OpenMeta Is a Hack

2009-01-30T16:47:09-07:00

Michael,

You (and shg) think it is the best to use org.openmeta as a primarily place and let users/developers choose whether to use com.apple.metadata. You said that the cost is trivial. You also think demonstration is not necessary.

I don't think so. From the perspective of user experience, demonstration is not trivial.It is a MUST.

Imagine that X.app saves tags only on org.openmeta and Y.apps saves on both places and uses spotlight for searching tags. It is not difficult to imagine an end-user who does not know the exact mechanisms but know both apps use OpenMeta starts to wonder why s/he can't search a tag added by X with Y. S/he wonders why only the tags added by Y can be searched from both X and Y. What if he uses many more OpenMeta applications? Furthermore, what if users can choose whether to use com.apple.metadata when they first install applications? Do standard end-users remember their choice of configurations? It is almost obvious that allowing users to make a choice makes the situation much more complicated. I recommended you to write demonstration partly because I didn't have any idea how you are to dissolve this problem.

Demonstration may be trivial only when tags are forced to be saved in both places. Only in that case, an end-user is not confused. Both X and Y always return the same search results. In that case, there is no distinction which is primary or secondary. In this case, it becomes almost meaningless to ask which EA an application uses. Both contains identical information. Choice is made only by a programmer's preference what search engine s/he wants to use. But, this is not what you (and shg) argued. Then, demonstration is indispensable.

Remember that we must assume users use multiple applications. OpenMeta enables it and that is THE power of OpenMeta.

I am an end-user and I don't have sufficient ability to make a judgment whether org.openmeta should be used. I think, if it is easy and does not provide any cost, it makes sense to implement org.openmeta together with com.apple.metadata. At least, Tom did not reveal explicit opinion about whether to use org.openmeta. Then, only what is logically concluded from this situation is that it is you who has a responsibility to show us how another world looks like.

I believe you made very important point on com.apple.metadata. I really appreciate that you patiently answered to almost all my questions. I really learned lots of new things from you. But, at least, it seems to be that your suggestion is not the only solution. It may turn out to be the best but only after somebody demonstrate it in front of the other developers (or, end-users). Let me repeat one thing. I don't (and, cannot) reject the possibility that your suggestion is the best one at the moment.

OpenMeta Is a Hack

2009-01-30T17:41:42-07:00

Hoge:

An application would either always make its tags searchable with Spotlight, or it would provide an option in the preferences to turn that on or off. If you turn off Spotlight in X, an application such as Y that uses Spotlight will not be able to search the tags. However, Y would still agree with X about which tags are assigned because it reads them from the same primary location. If Z uses its own search engine instead of Spotlight, it will always produce the same search results as X (no matter how the preferences are set) because they are searching the same primary location.

So, the decision for the user is that if he uses Y, and Y requires Spotlight, then he should set X and Z to use Spotlight, too. If he is only using X and Z (or W, which reads tags but doesn't search them) then
it doesn't matter (for the user experience) how the preference is set.

Why do you think a demonstration is necessary? (Now, it sounds like you want me to write a bunch of demo apps, not just the code for OpenMeta...) Do you doubt that the above description is the way it would work? Or do you need to "feel" how it works rather than read about it?

OpenMeta Is a Hack

2009-01-30T18:10:11-07:00

Do you think that system can be readily managed by end-users?

I feel you (and shg) tend to assume enlightened users - who know the logic of system perfectly and have a capacity to make a correct choice receiving all the necessary information. Not all the users who need tagging applications are like that. A system must be designed so that end-users can use it without thinking so much. In the world you envisioned, users may be able to get used to the system at the end but learning curve may be flat.

The world of Tagit.app. Tags.app. OpenMetaWizard.app, omtool and spotlight does not require any consideration. Copy apps on the application folders and that's it. I could tag with omtool and immediately search it with Tags.app. I synchronized two Macs with rsync 3.0 that detects changes in xattr and I could immediately search tags on the second machine because tags are promptly indexed by the spotlight on the second Mac after syncing. No thinking. No configuration.

User experience is important. That determines whether the system can attract users. It is always possible to provide a complex logic for solving a particular problem but it is not necessarily be user friendly. I believe that people are so much excited at OpenMeta mainly because of its seamlessness.

As I wrote, I know logical validity of your argument after asking so many questions. I thought it now is a time to see if it actually works. Mental simulation did not seem to be sufficient as, for instance, your solution in the last comment may work for some people but may be too much for the rest. What's solution can compromise multiple demands - security, robustness, transferability and user experience? I honestly think that you may be able to provide a solution. I'm not a perfect person. I make a misjudgment. I don't want to reject the possibility that something that seem to be bad can be actually good. That's why I think you need demonstration or something what is sufficient to give a flavor of another world to end-users.

At least, I personally think, forcing syncing com.apple.metadata and org.openmeta is one possible solution for providing assurance to end-users who are informed that com.apple.metadata is not officially supported.

OpenMeta Is a Hack

2009-01-30T20:54:40-07:00

Michael wrote:

My goal is for all the various applications to work in the safer way I suggested above.

That's the impression I've had of your intentions ever since you started this thread.

hoge wrote:

User experience is important. That determines whether the system can attract users.

Using Spotlight Comments for file tagging may attract users and provide a positive experience (at least initially), yet I've never used or recommended it because of awareness of ways it can and has proven to be failure-prone.

I don't want to reject the possibility that something that seem to be bad can be actually good.

I suspect most of us don't. :)

Hopefully discussions like this will benefit people who want a better understanding of OpenMeta before they choose to use and/or recommend it. And benefit, more indirectly, those who are less compelled to understand it, e.g.:

I feel you [Michael] (and shg) tend to assume enlightened users - who know the logic of system perfectly and have a capacity to make a correct choice receiving all the necessary information. Not all the users who need tagging applications are like that. A system must be designed so that end-users can use it without thinking so much.

Even if they're not thinking much they still trust (implicitly/explicitly) that it'll work (and keep working) if they expect/need it to be reliable.

Is there anyone who doesn't want OpenMeta to be a tagging system that can accommodate a broad spectrum of usage, implemented to minimize known issues that may seem ignorable at first but could have pitfalls later (like those Michael has brought to the discussion)? For me it's not enough that OM can/will resolve (most) Spotlight Comment tagging problems if it has certain kinds/amounts of its own (both immediate and potential). Or, more accurately, I'd want to restrict OM usage to contexts where any irrecoverable failures would be acceptably inconvenient and not devastatingly catastrophic.

Michael wrote:

I am singling you [Jim] out because you're the ones promoting the standard.

I appreciate OM-related discussion being as centralized as possible on this forum so it doesn't get redundantly fragmented in too many other places. I'd be glad to see more developers participating in some of the topics here.

OpenMeta Is a Hack

2009-01-31T00:03:48-07:00

Bare minimum of what we learned from Michael is that OpenMeta needs backup of tags for assuring users that it is secure. Now, we have one backup system. I can think of another system forcefully storing tags in both org.openmeta and com.apple.metadata. Which is better? Having both may be a foolproof way. Just my guess.

OpenMeta Is a Hack

2009-01-31T13:03:06-07:00

So the concern here is that one day com.apple.metadata may be deprecated, and that it is not impossible that at that time all com.apple.metadata extended attributes on files might be deleted. To prevent this possibility from ever become actuality, Michael Tsai wants OpenMeta tags stored in org.openmeta extended attributes. It has been suggested that Michael write code to ensure that all tags are written to both org.openmeta and to com.apple.metadata, and submit it to the OpenMeta code base. Michael has thus far refused to do so on the grounds that he doesn't know if his code would be welcomed. As yet, no word of welcome or rejection has been offered.

Michael, if OpenMeta doesn't meet your needs, you are going to have to write code to do what you want anyway. So why not make the changes you need and submit them? I doubt they will be rejected if they don't break anything. If you don't want to write the code, go find another solution to your problem.

Tom, since Michael seems reluctant to do anything unless he thinks it likely that his efforts will be worthwhile, can you give him a simple indication that his code will be seriously considered for inclusion, provided it doesn't cause problems?

OpenMeta Is a Hack

2009-01-31T14:19:21-07:00

I'm also happy if we can hear at some point in near future (maybe not soon) from Tom himself how OpenMeta intends to solve the problem Michael pointed and can reassure end-users. I feel no word other than yours make end-users feel confident about OpenMeta.

OpenMeta Is a Hack

2009-02-01T22:12:51-07:00

Michael,
I appreciate your comments here. As a fan of both Yep & Eagle Filer, I'm glad to see your thoughts on Open Meta. I had not realized that Open Meta was relying on an unsupported mechanism and absolutely agree that Ironic should be upfront about that. I had already downloaded the code and started figuring it out as I was eager to help contribute. I had wondered about the use of com.apple.metadata and now I know. As an end user, I don't want to rely on something that Apple could arbitrarily and justifiably change. But I also find Spotlight, warts & all, to be indispensable so I appreciate the ease of searching using com.apple.metadata EA. Is it technically possible to create a spotlight importer to index org.openmeta attributes? You also mentioned problems with Time Machine not recognizing changes in EAs. I hadn't noticed that, but it would certainly be a problem, and one that may require Apple's intervention. Changing the modification time on files just because a tag was added is not desirable.

So, I'm a little less excited about the prospects for Open Meta but I'm glad to see developers other than Ted & Tom (who are surprisingly quiet on this thread) thinking about Open Meta and I hope that you (the collective Mac developer community) can come to some sort of agreement and that Apple will eventually support some sort of metadata standard. In the mean time, I want the ability to tag my data and I'll use Open Meta, spotlight comments, whatever else I can, in order to stay more organized. But a common standard would sure be nice!

Thanks again for your thoughtful comments.
Cheers,
Rhet

OpenMeta Is a Hack

2009-02-01T23:57:12-07:00

I remain as excited as ever about OpenMeta, just as it is.

I see the merits of also writing copies of the tags to an org.openmeta namespace, so if anyone wants to take the time to write and submit code to do so, I say by all means do so.

But regardless whether anyone does that or not, OpenMeta is great, and is perfectly reliable as it is.

I am not concerned that Apple may somehow break OpenMeta later. It isn't impossible, but the risk is negligible. They spent a long time developing the current system, after all, and there is a lot built on top of it, so it should be around for a while yet. And if they do change it one day, OpenMeta can be rewritten to use the replacement system. Having implemented a metadata system of the scale they have, Apple isn't just going to drop it. Spotlight is here to stay, so even if they change the way some parts of the underlying system work one day, OpenMeta can be updated. And for those concerned that Apple might overwrite the com.apple.metadata extended attributes of files, it is inconceivable that Apple would do anything so drastic for anything less than a full point upgrade to the OS (if even then), which means that Ironic would have plenty of opportunity to test things on developer builds and update OpenMeta long before the hypothetical problem version of OS X were ever to be released.

So the long and the short of it is, while having a redundant copy of the tags in an org.openmeta namespace isn't a bad idea per se, all the doom-and-gloom in this thread is a bunch of FUD.

OpenMeta Is a Hack

2009-02-02T11:04:23-07:00

Lots of good discussions: here is my take.

1) The only way to get spotlight to index extended attributes is by using com.apple.metadata: as a key prefix. There is no way in 10.5 to make (as far as I know) a 'trailer' importer that runs on every file - which is I think how spotmeta did it in 10.4.

2) Using another namespace like org.openmeta: and then doubling the data in it seems complicated and wasteful at best, and it also does not solve the problem of backup adequately. Also having users decide is a non starter.

3) When xattrs are set on a file, Time Machine does not back up the file again (this is how Apple designed it - it seems the right path).

4) There are certain applications (photoshop, other CS4, subversion in some circumstances, etc) that can strip off all xattr - so a backup/restore solution is needed, not only for Apple wiping the disk of xattrs, but also the much more likely scenario of hard drive disasters, etc.

5) OpenMeta now backs up all metadata set with it (all using kOM* as a key). These backups are in turn backed up by TimeMachine. This gets around having to twiddle with the mod date on a potentially HUGE file in order to back up a very small amount of metadata.

--Tom

OpenMeta Is a Hack

2009-02-02T12:10:45-07:00

Sesquipedalian:

It does me no good to have my own system that's different from what other applications are using.

Tom:

Would you care to describe how your plist backup system is meant to work? The original OpenMeta was very easy to understand. It simply wrote the metadata into the xattrs. Now, it's much more complex, with more code and disk space devoted to the backups and their maintenance than to the actual metadata. Further, this is not a backup in the traditional sense, since the act of reading the metadata accesses the backup and can result in the data from the backup replacing the data in the xattrs.

Reading the code, there seem to be quite a lot of design decisions and subtleties, e.g. the format and location of the backup, the number of backups to keep, what happens to orphaned backups for files that were moved or deleted, which metadata is automatically backed up, how to detect when to restore, what happens when the xattrs are present but don't match the backup, how to handle long filenames and moved files, backups for files not on the boot volume, race conditions between different applications reading/writing to the xattrs and backup, what happens when the user Time Machine restores the file but not the backup or vice-versa, what happens when a file is in a shared location and multiple users have differing backups of its metadata, etc.

Some more specific questions:

What does kBackupStampOM do for us--is it necessary or an optimization? If the latter, how much faster does it make the system and what are the tradeoffs?

Why one file per backup? Is it not needlessly wasteful to store multiple 4 kilobyte files to backup perhaps a couple hundred bytes of metadata?

Won't performance degrade because HFS+ gets slow when there are many files in the same folder?

There are certain applications (photoshop, other CS4, subversion in some circumstances, etc) that can strip off all xattr - so a backup/restore solution is needed, not only for Apple wiping the disk of xattrs, but also the much more likely scenario of hard drive disasters, etc.

We've gone from me saying that "com.apple.metadata" is not to be relied upon to you worrying about all the xattrs being stripped off. And I agree with you that this is a real risk, especially with older applications that don't use metadata-preserving APIs.

OpenMeta now backs up all metadata set with it (all using kOM* as a key). These backups are in turn backed up by TimeMachine. This gets around having to twiddle with the mod date on a potentially HUGE file in order to back up a very small amount of metadata.

I don't like backing up large files, either, but one side effect of this is that Spotlight searches will return incorrect results after restoring from a backup. Perhaps there's no good way around that, but in any case it should be documented that this is what happens.

OpenMeta Is a Hack

2009-02-03T19:09:16-07:00

The backup is pretty simple.

Every time you change a file's metadata, any kOM* metadata is backed up to a ~/Library/Application Support/OpenMeta/year/month/filename_hash.omback file. So if you change the tags on a file daily, the backup file will chang

The advantage of using a month by month system is that old months could be heavily optimized. Also a file by file system is friendly to Time Machine, is easily readable by simple Cocoa programs and is fairly easily understood. As for disk space, 100,000 4k attribute backups use up 400MB of disk space, which is about 6 cents worth of file space on a new desktop, or perhaps 50 cents on a laptop. The space used is trivial. That does not mean that there are no problems - all software has problems. The backups have aliases and paths embedded in them to aid in finding files.

This is a backup - if there is any OpenMeta data on a file already, then that is for the most part considered to be accurate. The backup is only used if there is no meta data found on a file. Sure - just like any data there are lots of questions you can come up with - but most of them can be solved by thinking of the data on the file as the real data, and backups being just that. A few time stamps handle most problems fairly well.

The xattr system is designed to hold meta data - old programs that use 'old apis' will almost always work properly with xattrs. In order to not work with them, you need to go out of your way. Which I guess Adobe does, and is a huge bug on their part for all sorts of reasons. (hard link failure, breaking aliases, xattr data loss, along with likely other effects).

I guess if a developer thinks that xattr should not be used to store metadata, then they should not use them. When someone writes a better backup system for extended attributes, for OpenMeta or not, then OpenMeta could use it. The present backup system is designed to be simple to change, expand or replace. Its not perfect, not ideal, but nothing is. It seems like Linux, and many Unixes, including Apple are using xattrs now.

--Tom

OpenMeta Is a Hack

2009-02-03T21:19:30-07:00

Slightly OT (but following a statement Tom made…)

I am documenting the Adobe Bug and have gotten the brush off by a Photoshop Engineer on their forums (of course, it's Apple's fault according to them). They are deleting data that is not their own (and not just OpenMeta, ANY EAs). If anyone else want to they can file a bug with Adobe at Feature Request/Bug Report Form

Jim

PS: This affects Adobe Illustrator as well as Photoshop. (Early tests show inDesign behaving as expected (another nod toward Adobe) but I haven't tested any other products yet). I encourage people to file a report for each product they're seeing the issue with.

OpenMeta Is a Hack

2009-02-04T07:41:26-07:00

Yea, this problem particularly affects me as I 'live' in Illustrator & Photoshop. I filed a bug report a few days ago, but don't really expect Adobe to do anything about it (at least not in the near future).

Is there any way of a workaround where when adding a tag to these Adobe files it also writes to Spotlight comments? (As this problem doesn't arise with text written in Spotlight comments.) So that OM would check there to see if it needs to restore tags/metadata after these files have been saved or re-saved.

OpenMeta Is a Hack

2009-02-04T09:05:24-07:00

Jono:

Tom has worked out a workaround that includes backing up OpenMeta Tags on-the-fly. We should be seeing something with it soon (updated Tagit).

Yea, this problem particularly affects me as I 'live' in Illustrator & Photoshop. I filed a bug report a few days ago, but don't really expect Adobe to do anything about it (at least not in the near future).

I understand your cynicism but this also affects users not using OpenMeta. It is a BUG in every sense of the word. I would love to start a "class action bug report" with them - shake 'em up so they recognize the seriousness of the issue. I encourage you to spread the word (and the URL) - the more we file, the harder it is for them to ignore. 8^)

OpenMeta Is a Hack

2009-02-04T09:42:05-07:00

BLURFROG:
Tom has worked out a workaround that includes backing up OpenMeta Tags on-the-fly. We should be seeing something with it soon (updated Tagit).

Oo, that sounds great. Can't wait for that then!

I'll spread the word on the bug, & hope they surprise me by fixing it!

OpenMeta Is a Hack

2009-02-05T09:26:21-07:00

Tom,

The advantage of using a month by month system is that old months could be heavily optimized. Also a file by file system is friendly to Time Machine, is easily readable by simple Cocoa programs and is fairly easily understood. As for disk space, 100,000 4k attribute backups use up 400MB of disk space, which is about 6 cents worth of file space on a new desktop, or perhaps 50 cents on a laptop. The space used is trivial.

I think it's more like 60 cents of disk space, not 6. (A bare 1 TB drive is around $100 now.) Anyway, it's not really the dollar cost that bothers me. Disk space is super valuable on a laptop because it's so limited. Also, there's the overhead of having so many extra files (especially in the same folder), which slows down the file system as well as backups and synchronizations.

It sounds like you have a plan to eventually compact or delete the old backups, although this isn't implemented yet. I wonder, though, why we need multiple backups in the first place. Under what circumstances would we not use the most recent backup?

This is a backup - if there is any OpenMeta data on a file already, then that is for the most part considered to be accurate. The backup is only used if there is no meta data found on a file.

Then it sounds like a Time Machine restore of a file could bring back old tags in the xattrs, and these would be used even though the backup has newer tags?

The xattr system is designed to hold meta data - old programs that use 'old apis' will almost always work properly with xattrs. In order to not work with them, you need to go out of your way. Which I guess Adobe does, and is a huge bug on their part for all sorts of reasons.

I agree that it's a bug if Adobe's software doesn't preserve xattrs. However, I think you and Bluefrog are mistaken about having to go out of one's way to make it not work. The problems probably don't come from deleting the metadata, but rather from not actively doing something to preseve it when saving a new file. My understanding is that an application needs to go out of its way to call an API such as FSReplaceObject (added in 10.5) in order to preserve the metadata.

When someone writes a better backup system for extended attributes, for OpenMeta or not, then OpenMeta could use it. The present backup system is designed to be simple to change, expand or replace.

I want to make sure that the backup system really is a backup that's independent from the core OpenMeta functionality. This was not totally clear as I tried to explain in the previous post. What I'm getting at is that I don't want there to be any problems caused by having multiple applications, some using the original OpenMeta code that doesn't do backups (or Gravity's similar code), some using the current code, and some using a hypothetical newer backup system.

OpenMeta Is a Hack

2009-02-05T09:59:50-07:00

The problems probably don't come from deleting the metadata, but rather from not actively doing something to preseve it when saving a new file. My understanding is that an application needs to go out of its way to call an API such as FSReplaceObject (added in 10.5) in order to preserve the metadata.

I am only an advanced layperson, not a "real" programmer (but a really good Applescripter! 8^) so I don't know how much water this argument holds but… I have documented proof that the xattrs are not removed in all circumstances when saving from either program (which makes the problem doubly frustrating. It's like "Hey, you preserve them when I do this but not when I do that?!?"). Maybe they are triggering FSReplaceObject under certain conditions but I can't see why they wouldn't always try to maintain the metadata.

And for anyone who wants to know… this is the last response I got from a Photoshop engineer:

Jim - of course the original is not retained: the apps are doing a safe save (where you save to a new filename, then replace the original with the duplicate in a semi-atomic way). This is done using standard OS APIs whenever possible, and is done in most applications. All file saves in Photoshop behave the same way, except maybe PDF (which is in some code we don't control) and SaveForWeb (also different code).

Again, the root cause is an Apple bug in the way they chose to associate metadata with the files, and how they updated existing APIs to deal with the additional metadata. We might be able to work around this using new APIs -- but should apps have to change their code every time someone decides to add new metadata invisible to the app and separate from the file? Shouldn't the design of the metadata system be more robust, and work with existing applications? Shouldn't someone have thought about this before releasing a metadata system, that, well, nobody knew about?

• How could the saves be the same when some preserve extended attributes and others dont?!!?
• "…metadata invisible to the app and separate from the file…"?!? So, Photoshop doesn't use it so it's worthless? What about Spotlight Comments - how does Photoshop use those?!? Yet they are preserved And how are extended attributes "separate from the file"?!?
• And lastly, the very cavalier "…Shouldn't someone have thought about this before releasing a metadata system, that, well, nobody knew about? …" Gee, I didn't know Apple had hidden extended attributes from everyone except for people in this discussion. ;^) Oh wait, that was most likely a dig at us - lol [rant off]

I fully admit I don't know the deep mechanics of some of this stuff but it is apparent there is a bug with the Adobe apps and this disregard is sad to see (though it won't shut me up!) I wonder what John Nack would think of this guys comments? 8^D

OpenMeta Is a Hack

2009-02-05T10:50:13-07:00

Bluefrog:

It's not a question of removal because Photoshop uses safe saving (as all apps should). The new file doesn't have any metadata, and it isn't supposed to. The metadata gets added back when the app calls the API to swap the new file with the old one. Until 10.5, the proper way to do this was using FSExchangeObjects. However, FSExchangeObjects does not handle xattrs or hard links (or, at least doesn't do so consistently--some consider this a bug in the OS). So now apps are supposed to be modified to use FSReplaceObject, but this API did not exist until 10.5, which shipped after CS 4.

I don't like the Photoshop engineer's attitude, but he or she is correct on the facts. Apple designed xattrs in such a way that applications must take positive steps to preserve them. They actually are "separate from the file," in some respects. Applications that have not been updated to use the latest APIs, or which use the normal Unix APIs, will not preserve the xattrs. (Even the Finder did not preserve them under Tiger.) Application that want to use xattrs must take this unfortunate reality into account.

OpenMeta Is a Hack

2009-02-05T11:16:13-07:00

Thanks for the clarifications, Michael. Why then if you open and save a JPEG file are the xattrs preserved? I'm really trying to understand what's going on here, BTW.

Here's my "experiment":

1. Add xattr metadata to a file (using Tagit or omtool for expedience)
2. In Terminal: mdls the file to verify xattrs are present
3. In Terminal: ls -li the file to record the inode
4. Open the file in Photoshop (or whatever app I'm testing)
5. Make a minor change that will trigger fileModified to be true and Save not Save As, etc.
6. In Terminal: ls -li the file to verify the inode
7. In Terminal: mdls the file to check for preservation of xattrs

This may be simplistic but I went with the logic of: an inode is assigned when a file is created, therefore if the inode changes, it is a new file so xattrs don't belong to it. If the inode is the same, it is the same file and xattrs should be preserved.

JPEGs saved maintain the inode and the xattr (perfect and makes logical sense to me). TIFF files do not maintain the inode (so the file has been replaced with a renamed duplicate) and xattrs are gone (again, logical from the metadata standpoint) but why the replacement? This one surprised me. PSDs getting "safe saves" seemed less surprising. So to my slightly trained eye 8^) it appears that Photoshop is not using safe saving for all file formats.

Does that make sense?

OpenMeta Is a Hack

2009-02-05T11:40:48-07:00

Bluefrog:

My guess is that Photoshop CS 4 is not doing a safe save for the JPEG in that situation, and that's why the xattrs are preserved. (I'm using Photoshop Elements 6. In this situation, it does not preserve the xattrs.)

By the way, if the inodes are different, you can conclude that the file is new. However, if the inodes are the same you cannot conclude anything, because one of the features of the metadata preservation API is that it assigns the old inode to the new file.

OpenMeta Is a Hack

2009-02-06T12:14:57-07:00

The bigger issue I see is that everyone in this discussion is being a little short-sighted. OpenMeta is a first step, but where is the journey going to end? Why restrict it to just OS X? IMHO, the end goal should be a widely supported solution for attaching meta data to files that is supported cross-platform and preserves meta data as files are moved between platforms. It's a long journey, but I think the results would be worth it, and this first step is absolutely necessary, but it needs to be in the right direction.

In order to achieve that goal, meta data needs to be stored in a portable location, i.e. not com.apple. There should also be support for namespaces (e.g. my tags should be distinct from say, corporate tags) and the user should have control over which namespaces are used on their machine. Obviously all this complexity would be abstracted away from the user by nice applications like Leap.

Anyway, that's a short explanation of my vision, which is may be a bit grander than what was envisioned when OpenMeta was launched.

OpenMeta Is a Hack

2009-02-06T13:08:02-07:00

Welcome, Colin!

Your vision is just fine, 20/10 possibly! >.<

Actually that is on the 1000 yard view for this whole thing. The ideal is dissolution of application / operating system boundaries for metadata. It is a grand vision indeed but we need to start somewhere and that's in our own neighborhood. (Think globally, act locally - right? ;^)

We welcome any and all comments (though we should have new threads going for new ideas - this one's getting a little long in the tooth).

Thanks for your comments.

OpenMeta Is a Hack

2009-02-06T15:02:11-07:00

@colin yeah, that would be very nice, but I think for storing metadata on the platform xattrs are quite o.k. Seperately from the above discussion there are other issues to consider then. No matter whether you use com.apple or org.openmeta, the programm copying the files needs to make sure xattrs get copied too, and not just as a .* file. This would be easiest to implement for linux etc. i think, because most files systems there now support xattr and in most modern installations it should be enabled by default, although I'm not aware of a "tagging" program like leap or so on that platform. On Windows however while NTFS in theory supports some kind of metadata, afaik it is much more limited and even less supported then xattrs on os x or linux.

OpenMeta Is a Hack

2009-02-09T11:56:21-07:00

Just to be clear in my suggestion, I wasn't suggesting moving the meta data anywhere other than xattr. NTFS supports attributes sufficiently to implement a Leap-type app on Windows. I agree that the best thing to do is get widespread adoption on OS X and then try to expand globally, but, and I believe I am making the same point as Mr. Tsai, although for different reasons, you don't want to start out by choosing a namespace that isn't going to be portable (to be fair, their is no reason you couldn't use a com.apple namespace on Linux, but it makes little sense for a number of reasons, the biggest being the fact that OpenMeta neither owns nor controls that domain).

As for preserving/copying attributes, it's a bit of a chicken and egg problem. Nobody is going to worry about preserving attributes across file systems until there is widespread need to do so and nobody is going to create apps with need for attribute preservation without support (witness the Adobe problems, which I do sympathize with Adobe over).

My proposal is that OpenMeta should own the OS-level meta data specification on the OS X platform, and when it has begun to really take off, start approaching the Linux community, maybe even setting up a working group to manage the specification (partnering with freedesktop.org would be a good idea here). Then you go after Windows. Contrary to popular belief, Microsoft will support community-lead initiative when it is in their interests to do see (e.g. recent integrated support for JQuery in their flagship development IDE).

Personally, as a professional software developer, I think the correct approach is to store OpenMeta tags in an OpenMeta namespace, and, to achieve the Spotlight integration, keep a shadow copy of the tags wherever Spotlight is expecting to find them. That way you have a future-proof primary storage, and you can react to whatever changes are introduced by future versions of Spotlight (e.g. if Apple were to provide a supported extension mechanism).

OpenMeta Is a Hack

2009-02-09T12:25:39-07:00

I hear you, Colin. Especially in your last posting, regarding how to approach, err, global domination.

For what it’s worth, I published a rather shallow (German) article about how I see OpenMeta on Apfelquak.

I’m posting the link in this thread as I’m referencing, well, to this thread. I’m quite interested in the technical discussion here, I may add. Even though as a writer, I’m not even a layman. But it puts things into perspective, hence, thanks!

Cheers,
-Sascha

OpenMeta Is a Hack

2009-02-09T21:22:41-07:00

Since other OSes store xattrs in different places and different ways, I think trying to solve things for the world is premature and overly ambitious, and simply won't work at this time. I think we need to solve real problems that exist now, and KISS - keep it simple. If there are compelling reasons why com.apple makes things simpler, I think we should go with that and worry about what Apple may or may not do next year when that happens. We can always write a utility to update things later. I don't think we should complicate things for things that in all likelyhood won't happen.

OpenMeta Is a Hack

2009-02-11T11:59:40-07:00

Guest:

Posted by MatthewB,

I'll have to post this as a "guest" as my application for a account on this board has not yet gone through.

I've read this entire thread and I support Michael Tsai's, shg's, and colin_young's positions. I would like to see OpenMeta use its owned named space (org.openmeta has been suggested) to store tag attributes and include the ability to mirror these attributes into the com.apple space to allow Spotlight searching (when and if desired by a particular app). It's a good suggestion.

- matthewb

OpenMeta Is a Hack

2009-02-11T12:08:38-07:00

Welcome, matthewb (your account is active now)…

Thanks for your comments.

OpenMeta Is a Hack

2009-02-13T12:09:22-07:00

Guest:

I think Michael Tsai's suggestion is very reasonable. My only comment would be that an application that *modifies* an already existing tag that is present in both org.metadata and com.apple, modifies both copies, even if the app in question has been set to not store the tag as a spotlight- indexable tag, and has thus been set to not touch com.apple. This would be a good compromise to reduce user confusion.

charles
(waiting for username approval)

OpenMeta Is a Hack

2009-02-13T12:24:05-07:00

You're all set, charles. Welcome and thanks for your comments.

OpenMeta Is a Hack

2009-02-13T12:44:09-07:00

FWIW: Daring Fireball Linked List: OpenMeta Is a Hack

OpenMeta Is a Hack

2009-02-13T13:08:48-07:00

Thanks for the heads-up, sjk.

OpenMeta Is a Hack

2009-02-13T13:14:13-07:00

Guest:

I think the problem is that there are two different things at work here. The first is the desire to create a standard place to store metadata information that any application can access. The second is the desire to add a feature that allows searching using Spotlight.

From the outside looking in it appears that the needs of the feature are overriding the needs of the standard. Perhaps the Spotlight searching was the initial goal and the standard came up later. The problem is that the whole point of creating a standard is to make something that is as bullet proof and future proof as possible. Something that you can grow, but still expect to maintain some backwards compatibility with if things change in the future. Using com.apple.metadata as the standard place to store information completely defeats this purpose. I've seen a lot of back and forth about how probably something going wrong is. Some think it's extremely unlikely. That may be true but it completely misses the point. Just because it's unlikely doesn't mean it won't happen. Any thing that relies on something that may go away at any moment isn't a standard but a future tech support nightmare.

It's early enough in the process that you can change the xattr tag to openmeta.org or what have you with out the huge amount of pain it would cause in the future. In addition it would be more attractive to developers that won't use something that hasn't been endorsed by Apple. I would expect that this would increase interest in the project and help it actually become a true standard.

The Spotlight searching really is a feature, and extension to the idea of having a standard meta data repository. Yes, to implement it now, you have to store duplicate data in two places. So what? If it's handled in the library it makes no difference to the user or the developer. What happens if Apple decides to make all xattr information indexed? Just turn off the code that puts it in com.apple.metadata. If Apple decides to store a 5MB image of a unicorn in com.apple.metadata? Spotlight searching stops working but applications can still access the meta data with out any need for an update.

It's more work now yes. But it will prevent a hell of a lot more work later on.

Michael Krzyzek

OpenMeta Is a Hack

2009-02-13T13:33:22-07:00

Guest:

I'm a late comer to this discussion (via Daring Fireball), and I am quite intrigued. The two main points of discussion seem to be: 1) using com.apple.metadata: as the xattr prefix for non-Apple xattrs; and, 2) Spotlight's (in)ability to index xattrs that are _not_ prefixed by com.apple.metadata.

I come from a background of xattr from BeOS (and its BFS, which was designed by the same person who is now employed at Apple working on filesystem and Spotlight-related work), so I have a different perspective on xattrs. HFS+ has had xattrs for quite some time, but only recently received OS level support for them. With BFS files, xattrs did not have to have any specific "prefix", they were indexed by default, and if a particular filetype had a set of xattrs associated with it, those xattr fields would be present on newly created files of that type.

Back to OpenMeta: Why do the xattr fields have to prefixed at all? Is that just for the ease of programatically reading them? If that's the case, I personally see no reason _not_ to use com.apple.metadata. Spotlight is an Apple program, and if it can only read metadata prefixed in that manner, then that's OK. It's similar to other programs that perhaps want to access to read (and possibly modify) your Safari bookmarks (say for iPhone syncing). Apple "owns" Safari's bookmarks, but access to the bookmarks file must conform to what Safari expects. In this same manner, access to indexed metadata must be what Spotlight expects.

I'm not up to par on the limitations of Spotlight plug-ins, but as I understand from this discussion, mdimporter will only recognize xattrs prefixed with com.apple.metadata. Is that correct? If that is correct, then I fully support OpenMeta using this method. However, if it is also correct, then it is a distinct limitation of Spotlight, and should be changed and filed as a bug. If Apple's search and index mechanism for their filesystem does not let me search or index the things I put there (such as custom xattrs), then this is an artificially imposed limitation. Apple has hindered the state of HFS+'s metadata on OS X for *years*, and it's time they finally let users have control of their own metadata. If OpenMeta can do this, then I fully support their effort. It does not appear they are doing anything in a "hack"-like manner, but rather working within the artificially-created limitations that Apple has imposed on developer's because of the lackluster implementation of xattr indexing.

-- R. Patrick Cameron
rpcameron at rpcameron dot net

OpenMeta Is a Hack

2009-02-13T13:50:56-07:00

Guest:

As a programmer, I think this is an interesting discussion that could go on for ever.

As an end user, however, I do not care at all about how anything works below the surface. What I do care about, is that everything keeps on working after I run a software update on my machine. And if OpenMeta writes to any com.apple.* files whatsoever, NO ONE can guarantee that.

In general, developers tend to care too much about making things technically possible, and care less about how the end user will experience their wonderful technical ideas.

I for one am going to be VERY angry as an end user when the OpenMeta systems fails to work after a simple system or security update which I always install immediately and automatically.

OpenMeta Is a Hack

2009-02-13T14:33:37-07:00

Guest:

Let me add the perspective of someone who manages application supportability and risk for a living.

Whenever you write an application that makes use of data structures, APIs, etc. that are provided by another system -- in this case the OS-X file system and Spotlight -- you are making certain choices about what to rely on. Apple provides documentation that essentially says "if you want to add xattr metadata in your own namespace, we promise to respect that." They don't make any promise to respect data other people put in com.apple.* namespaces.

OpenMeta wants to accomplish two main goals:
1. tag files in a standard way that other applications can use and manipulate
2. make those file tags searchable in Spotlight

Goal 1 can be accomplished using either an org.openmeta namespace or a com.apple namespace. All other things being equal, if this were our only goal it would be silly not to use org.metadata. After all, Apple has made an implicit promise to leave that data alone.

Goal 2 can only be accomplished if OpenMeta uses the com.apple.metadata namespace.

So here comes the first decision: do we use this item even though Apple hasn't promised that it will work in reasonable ways? If we do, we gain a great feature, but at the risk that it might break something unexpectedly either now or in the near future (not to mention that the feature might disappear entirely without any kind of warning). At least with org.openmeta.* we have some assurance that Apple won't just yank the feature without warning, and some assurance that doing this isn't supposed to break anything.

Now, OpenMeta has chosen to use com.apple.metadata and accept that risk.

**HOWEVER** the claim that OpenMeta "doesn't use any secret API", which is *technically* true, is a little misleading. There may not be a secret API in use, but something unsupported is being used, which means the user is taking a risk -- even if it's a small one -- by trusting tag data to OpenMeta. At the very least, OpenMeta should be, well, more open about that. Users can decide whether they're willing to accept that risk.

Mr. Tsai has made an additional suggestion: OpenMeta can achieve Goal 1 in a way that's supported by Apple (using the org.openmeta namespace), and supply an application *option* to the user which will achieve Goal 2 by mirroring that data to the unsupported com.apple.metadata area.

This way, the user can get a lot of good out of OpenMeta even if they don't want to accept the risk in using an unsupported file system feature. User choice is a good thing. The approach has the additional advantage that if Apple dramatically changes how the com.apple.metadata feature works (say, by altering its data format), then all users lose is the Spotlight-searching feature -- their tag data is still safe and can be accessed by OpenMeta and other applications. Resilient software is *also* a good thing.

--Darren Meyer

OpenMeta Is a Hack

2009-02-13T22:55:11-07:00

Guest:

Hi there,

I used SpotMeta in 10.4 and was devastated when my great organization of files was dead with the 10.5 Update. I took me weeks to get back to a similar system (with YEP). OpenMeta was announced as an open standard (which I understood was playing after Apple's rules), and now I feel scared that everything will be lost again.

This isn't amusing at all...

If you value you customers, please, play safe with their data, their work and their time.

So I see that Michael Tsai has asked some questions which are in dire need of an appropriate answer... OpenMeta is a great idea, but don't toy with us

Best

Bernt

OpenMeta Is a Hack

2009-02-14T02:58:46-07:00

This thread may be too long to read everything. The fact is that OpenMeta now backups all the tags in ~/Library. This is repeated in this thread.

OpenMeta Is a Hack

2009-02-14T06:31:33-07:00

Hello,

hoge, I think the backup of OpenMeta date is only a (small but important) part of addressing the questions posed and pondered again and again on this thread. Currently I think the arguments in favour of NOT using com.apple-sth. as main starting point for openmeta data but using org.openmeta-sth. prevail, problems for developers notwithstanding… If Apple choses to change the underlying basement with a systems update I and others don't want to be left standing on the side of the road.

I do think the spotmeta was suffering from a different problem though: the frontend of manipulating and displaying data applied by spotmeta was not compatible with Leopard and didn't get updated. Though ironic and gravity do things not open-source and in their freetime and thus development will probably make things work again in a similar situation, the argument to design and implement a PROPER (future-proof) standard seems a very valid one.

Rolf

Just want to add that the developers of BibDesk and Skim have voiced their concern that using com.apple instead of org.openmeta is potential dangerous, following Michael Tsai's line of thought (on the BibDesk-user-list)

OpenMeta Is a Hack

2009-02-14T10:19:15-07:00

Rolf,

My personal opinion is simple and I've already talked this issue with Michale and I think he understood my point. Using org.openmeta as a primary place and let developers/users choose to use com.apple.metadata as an option just brings chaos. I've wrote how such a world brings confusion to users. At least, I think Michael understood this point.

Of course, I know the superiority of org.openmeta. So, I personally prefer mirroring both com.apple.metadata and org.openmeta.

However, a person who is in charge of Openmeta does not seem to take this route. Instead, this person decided to store backup in ~/Library. It is not exactly what I prefer but this is much better than nothing. Furthermore, this is much better than a chaotic world where tags are primarily stored on org.openmeta and only the wishful developers use com.apple.metadata.

I strongly fear to lose tags. But, I strongly believe the solution should not destroy usability and I personally believe it is possible - just mirroring com.apple.metadata and org.openmeta. It looks just simple to me. I, an end-user, wonder why we need theological arguments.

Below is what I wrote in this thread two weeks ago. Who wants to live in such a messy world?
----
Imagine that X.app saves tags only on org.openmeta and Y.apps saves on both places and uses spotlight for searching tags. It is not difficult to imagine an end-user who does not know the exact mechanisms but know both apps use OpenMeta starts to wonder why s/he can't search a tag added by X with Y. S/he wonders why only the tags added by Y can be searched from both X and Y. What if he uses many more OpenMeta applications? Furthermore, what if users can choose whether to use com.apple.metadata when they first install applications? Do standard end-users remember their choice of configurations? It is almost obvious that allowing users to make a choice makes the situation much more complicated. I recommended you to write demonstration partly because I didn't have any idea how you are to dissolve this problem.

OpenMeta Is a Hack

2009-02-14T14:03:58-07:00

Hoge: I understand your point, but I don't agree that your chaos scenario is what's likely to happen. So I still think the primary storage should be in org.openmeta, with mirroring to com.apple.metadata. If an app wants to make a backup in ~/Library, that's fine, but (a) I don't think that backup system is a good general solution, and (b) it doesn't address the points I've raised.

OpenMeta Is a Hack

2009-02-14T14:06:56-07:00

Guest:

With regards to the 'cost' of 400MB...I pay for continuous online backup. I pay per GB uploaded, and per month it's stored. So the 'cost' is not 6 cents, nor 60 cents. If all my applications made 400MB of backups, I'd be broke.

Aside, I think Michael Tsai made an excellent point, which has yet to be refuted (except by logical fallacies; 'show me where it says we can't do this'). Rather than address what he raised we've had doubt in his credibility; 'show me the Apple email', which he disproved, to be met with silence (not even a thank-you!). We've had 'why are you attacking only us', followed by an explanation, again met with silence. We’ve had a suggestion he would contribute if his work would be accepted followed by...silence. And then we've had a 'theological' discussion as to the whys and wherefores of making a decision which made this so pointlessly academic (I use that word very loosely) it was, at times, painful.

It's your project and you can do what you want, but please just man up and say 'we're doing it this way because...'. Going for the defensive 'well Apple didn't say we can't, ok they did but forget that for a minute, blah blah blah' looks really shoddy. As does the 'secret api' rubbish. People want transparency.

Guest

OpenMeta Is a Hack

2009-02-14T14:17:59-07:00

Why do I care about this?

I to am an ex-SpotMeta user. I've been down this path before. Quite frankly the larger portion of work, and risk, is with the people managing content who trust a tagging system (because this work is continuously on-gong and is multiplied by the number of users who use the tagging software). Before I start down this route again with my 45,000 text documents I need to be sure about the system. I've been hoping for a resurrection of SpotMeta, or some other contender emerging (something more robust than Spotlight comment area tagging). I really want OpenMeta (and Leap) to be it. Unfortunately, I'm unwilling to bet on OpenMeta while it is primarily based on using the com.apple namespace.

I think OpenMeta is tremendously important but also that the hurdles it has to overcome are substantial. If it is to gain wide acceptance in the programming community, and more importantly gain the trust of the community of informed content managers, it needs to have *everything* going for it. We want, and need, a tagging system to become an integral and standard part of Macintosh file management whether Apple acknowledges it or not. If OpenMeta succeeds in this role it will become the underpinning for a great deal of functionality for many applications and fundamentally change the way many people manage files. I think it is close to being on the right path for this to happen. Adopting the org.openmeta (or some such) namespace for it's primary data storage will, I think, put it over the top and result in a broad and growing groundswell of support. (It feels close already.) But without this change I'm afraid the project will not be the answer. At least it won't be for me. And if it pulls in all the applications that are looking for an improved method for system-wide tagging it might even do harm (those programmers will no longer be looking, thinking they have found the solution). Sounds a bit corny, but there is a fair amount of responsibility in this. OpenMeta is good enough now that it may become the dominant behind-the-scenes tagging mechanism as it is. That's why it is important that this decision is right (as in, from my perspective, that compromises are made if necessary to assure, with the greatest degree of likelihood, that tagged collections continue to function without major upheaval regardless of normal operating system evolution).

I have Tags, and Leap, and omwizard, and the other Tags. I really want to begin tagging and improving access to my documents. "No secret APIs" (and my positive experience with Leap) had me believing that OpenMeta would be it. Not creating it's own namespace and writing its tag attribute data into the com.apple space is a bit contrary to what "no secret APIs" had me thinking (though I acknowledge that it is literally true). I understand why writing to com.apple is a more elegant programming solution (avoiding the need to mirror the tag attributes into the com.apple space to gain automatic Spotlight indexing). But, in this case, I think there is a need to go with more certainty, at the expense of programming elegance.

OpenMeta Is a Hack

2009-02-14T16:23:54-07:00

Michael,

>So I still think the primary storage should be in org.openmeta, with mirroring to com.apple.metadata.

If you mean that org.openmeta should be forcefully mirrored to com.apple.metadata as a default setting, I am perfectly with you. I don't care which is called primary or secondary. I believe (perfect real-time) mirroring is the best option.

I think many people concerning this problem want to achieve two goals. Use org.openmeta and allow users use spotlight by default. Only the solution I can think that can achieve both goals is mirroring. Honestly speaking, I don't understand why this option is not chosen by those who are in charge of OpenMeta. It seems to be so simple and user don't need to change anything in their daily usage.

Making backups in ~/Library is just the second best. As I asked the question in another thread, I feel that this second option may cause some problems aside from what you pointed out. However, it is much better than nothing. If the best option is implemented, we don't need to consider the necessity of this second option. I support this second best option only for assuring users who worry the tags they are currently adding may disappear. I am not perfectly happy with this backup solution. However, I think some people are getting too nervous given that the current OpenMeta system has a backup system.

OpenMeta Is a Hack

2009-02-17T08:27:21-07:00

This is an interesting discussion. I guess what I'm curious about here is what you guys think is the real likelihood that using com.apple.metadata will lead to lost tags? It sounds like this risk is simply Apple's lack of commitment to protecting that data, but is there any reason to believe (ie. some plausible reason why) Apple would "lose" these tags at some point?

Another way of looking at this: if Apple *did* guarantee that com.apple.metadata was safe to use, then even the ~/Library backup wouldn't be necessary, correct? If we are "pretty sure" that there is no reason why com.apple.metadata shouldn't be safe, then the ~/Library backup seems like a very reasonable insurance policy, ensuring a recovery if something catastrophic happens. Going all the way to full mirroring between com.apple.metadata and org.openmeta seems "dirty", and only necessary if we expect frequent problems of losing the com.apple.metadata tags, where single files would need to be frequently repaired in the background.

OpenMeta Is a Hack

2009-02-17T16:39:17-07:00

Guest:

sherkaner here is the problem, the standard is that the domains are owned by the developer, whether it's Apple or Whacky Company. So com.whackycompany is a unique identifier that they can use. Now since Apple writes the OS it's tempting to use com.apple.metadata. But Apple still owns that data space. It's not that they didn't commit to protecting the data. They said in dev speak that it should not be used. While you may think using org.openmeta and mirroring it to com.apple.metadata seems dirty it is really the safest thing to do. Apple can remove com.apple.metadata at any time whenever they do an update. Mirroring the data means that user information is still preserved even if Spotlight search is impacted if Apple changes things. From a dev perspective the fact that they said that using com.apple.metadata is unsupported means they are going to change Spotlight searching of meta data at some point. Doing things the "dirty" way means that OpenMeta can be more flexible in the future. I'm really quite stunned that there is so much push back on this.

Michael Krzyzek

OpenMeta Is a Hack

2009-02-17T22:00:41-07:00

Ok folks, where does apple say that if you don't use com.apple you are safe, and conversely not to use com.apple because it is reserved? As far as I know, and I'm happy to be corrected, Apple says no such thing. In fact it says nothing whatsoever about what names to use, and doesn't even suggest that the reverse domain name convention be used. For all we know, Apple might start using org.openmeta next year, with or without owning the corresponding domain. In fact, the openmeta.org domain is already owned. Why would owning the domain make you feel safe? What if someone in our cabal does own the domain, then it is sold, are you now unsafe?

For all we know, com.apple is Apple's preferred method for people to specify Spotlight indexing. As far as I see, you make it work now the way that makes sense now. If that changes in the future, in ways we can't predict now, THEN you build in backwards compatibility. Maybe in the future Apple gives its blessing to com.orange or com.grapefruit or com.metatags or com.apple.metatags, or who knows what. Then you're going to have to do something else again, and you'll have two legacies to clean up instead of just one.

OpenMeta Is a Hack

2009-02-18T10:01:29-07:00

Guest:

countach first of all the revers domain is a convention used so that each developer can have a unique namespace. It's the equivalent of application creator codes from back in the day. Take a look at the Preferences folder for bloody sake. No proper developer is ever going to use a domain they don't own, seriously get real. Secondly the openment.org domain was used as an example, Michael Tsai even offered to buy it so perhaps he did. Thirdly read the whole thread. Do you see the comment from Michael Tsai where he asked Apple about using com.apple.metadata and they said it wasn't supported? That means that they do not want people using it. That is their way of saying it is reserved. Lastly the idea that you should wait for something to break and then fix it is just laughable. Why not be proactive and avoid the whole problem in the first place? That is the whole point of this discussion that keeps being missed over and over again.

At this point it seems like OpenMeta is a good idea with a flawed implementation. As it stands I don't see it getting much uptake from the rest of the developer community.

Michael Krzyzek

OpenMeta Is a Hack

2009-02-18T19:27:20-07:00

I can't help suspecting that this whole controversy has arisen for largely non-technical reasons, even though the technical reasons are the ones being talked about. Maybe some readers saw the title of Michael's post and thought he was being deliberately provocative or hostile. Maybe some readers knew who he was, and thought he was coming to sow doubt to ruin OpenMeta and thus somehow benefit his own program. Maybe the developers had breathed a big sigh of relief at finishing the OpenMeta code, generating some interest from other developers in the process, and were looking forward to devoting a few months to further development of Deep and converting Leap to OpenMeta, and were taken aback at the criticism (which may well have been the first). Maybe they even thought subconsciously "Shit, he's right, but no way are we going to go back and rewrite the thing." And quite likely they simply haven't had time to give the criticisms and suggestions the attention they deserve, and have thus unwittingly given some readers the impression that they're avoiding or have something to hide. (Note that they haven't taken the easy option of moderating discussions on the forum.)

I'm not criticizing here. From where I'm sitting, Ironic Software have created the best example so far of a comprehensive metadata-based file browsing solution (Leap) that clearly shows that Apple is no longer the place to look for interface innovation, in addition to a similarly-spirited PDF management program (Yep), and now an even more imaginative application for image management (Deep). They could easily have focused only on improving their own programs, but instead opted to leverage their expertise in this field to benefit users and developers of competing software. As far as I know, they've devoted lots of time to OpenMeta and haven't really received any tangible rewards. (Perhaps this is the kind of thing that goes without saying, but I think it can still sometimes be useful to say it.) They're human beings, too, and a feeling that they've done enough, or simple fatigue, or just being really busy, are all things that we can all identify with.

Going back to the technical issues, I've seen counter-arguments to Michael's ideas to the effect that using com.apple.metadata isn't as dangerous as he suggests, but I haven't seen one easily understood argument AGAINST his proposal, i.e. that what he suggests is inordinately costly or dangerous, or whatever.

Anyway, I hope that the talented programmers in many companies who will actually be bringing OpenMeta to us users can maybe take a bit of time out and recharge their batteries while further clarifying their thoughts on this matter, focusing on the future rather than what's gone before. Who knows, maybe conversations are already taking place by email and other channels and an understanding is already being reached. And perhaps we users can remind ourselves that none of this comes to us by god-given right: it's the vision and effort of dedicated developers that gives us even the hope of a long-term solution to metadata portability.

OpenMeta Is a Hack

2009-02-20T06:22:06-07:00

Jon Gotow from St. Clair Software here - I'd like to add my two cents to the discussion. This matters quite a bit to me because I've got a build of Default Folder X sitting here with OpenMeta support built in, and don't want to release it until we get this sorted out.

First, I'd like to get past the semantic quibbling - by convention, namespaces and files tagged with a company's domain are reserved for that company. That IS the convention and I don't think we need to argue about it. Writing a com.apple.metadata prefix in a file's extended attributes is writing to a data store that you don't own, regardless of whether you're technically using a private api or not.

I'll summarize the issues that have been raised:

1. Extended attributes on the filesystem are not always preserved because some applications use old API's that are not EA-preserving. Photoshop and Illustrator's 'safe save' procedures are among these. In my mind, this is essentially an early-adoption problem in which we're relying on a filesystem service that is not fully supported yet.

2. Writing extended attributes to the com.apple.metadata space works now, but because the format and use of this data is controlled by Apple, that may change or conflict with future formats and/or uses.

3. Spotlight is hard-wired to only index extended attributes prefixed with com.apple.metadata.

Now ideally we'd like to address all of these - doing so would make things as safe as possible, though still not guaranteed since we're fiddling with features and data that Apple owns and has not documented for public use. The goal here is to make OpenMeta work for the end-user. To that end, we need to:

a. Patch up the incomplete extended attirbute support in the filesystem. If EA's worked completely, they would be the ideal way to attach metadata to files, since the data is paired explicitly with the files themselves (as opposed to keeping it in separate parallel storage, as Spotlight comments and OpenMeta backups do). Unfortunately, the data can currently be wiped out by certain apps, or by transferring files to a non-EA-aware filesystem.

b. Provide a safety net that protects user data as best we can from changes to the com.apple.metadata changes in the future, while allowing indexing in Spotlight as it functions today.

As far as proposed solutions, we have:

A. OpenMeta now creates backups to ~/Library. This takes care of issue #1 and is already a done deal (though we can still argue about the implementation - I haven't looked at the code in enough detail to do that yet).

B. Michael has proposed that OpenMeta store tags in an org.openmeta labeled extended attribute, then 'mirror' those tags into the com.apple.metadata prefixed attribute. That makes complete sense to me and is the best we can do with issue #2. We should have OpenMeta write to a data store that it owns, and then push the tags to Spotlight as a separate 'mirroring' operation. com.apple.metadata should not be the primary storage space, since Apple owns that, not OpenMeta. Ideally, tag 'mirroring' should merge OpenMeta tags with whatever already exists in the com.apple.metadata store so that we don't stomp on any other data added by Apple or others. Yes, the logic is messy and not completely deterministic (because we don't know what others are doing, we have to 'play it safe' and may end up leaving extra tags there in certain exceptional cases - if that's the price of ensuring that we don't mess up the user's data, so be it).

I personally think that BOTH of these are necessary. Just writing to org.openmeta and mirroring to com.apple.metadata does not take care of issue #1. Issue #2 cannot be solved perfectly, since we don't know what Apple will do in the future. What we do know is that com.apple.metadata is currently a property list - we can check for that and not touch the data there at all if we see an unexpected format - that will 'future proof' things to the extent that we won't be corrupting data if things change. And by storing OpenMeta tags in org.openmeta as the primary store, we're preserving user data so that a utility can be written in the future, if necessary, to deal with changes from Apple when they occur.

OpenMeta Is a Hack

2009-02-20T07:21:09-07:00

I'm not a programer so don't understand all the technicalities, but I've been following this topic for a while.

(To me) there doesn't seems to be much difference in implementation or the work involved if the the primary storage is com.apple.metadata mirroring to org.openmeta, or the other way around.

But it seems that quite a few developers are wary about adopting this as a standard with the way it works at the moment. If developers are holding back on adopting OpenMeta because of this then wouldn't it make sense (if only to instil confidence & get more developers to accept this as a standard) to make the primary storage org.openmeta, with mirroring to com.apple.metadata?

OpenMeta Is a Hack

2009-02-20T08:21:47-07:00

Pretty compelling argument Jon makes there (even more so as the developer of a utility that I really want to see OpenMeta support in :)

OpenMeta Is a Hack

2009-02-20T10:02:22-07:00

Yeah, probably the most important issue is that the fact that OpenMeta uses the unsupported place to store metadata can be solely a sufficient reason for some developers to hesitate to join OpenMeta. We should note that this is true independently from the possible bad outcomes (losing data, interferences with the system, etc.). It's not a good situation since OpenMeta has been proposed to be a standard...

Actually this point was already pointed out in the first post by the starter of this discussion.

OpenMeta Is a Hack

2009-02-20T11:59:58-07:00

Hopefully Jon and Tom will be having some discussion :)

OpenMeta Is a Hack

2009-02-21T15:15:03-07:00

Hi,

I'd like to point out that the developers of BibDesk (and Skim by the way) though having implemented basic openmeta support in BibDesk (display of openmeta tags) consider the use of apple's namespace not advisable. They have insured that openmeta-tags are read-only in Bibdesk (access via applescript - still need someone to write me such a script…).

Rolf

OpenMeta Is a Hack

2009-02-23T15:03:56-07:00

Guest:

I've spend a significant portion of my day reading through this entire thread. I work for a company that really needs a metadata solution like openmeta. That said, as an IT person, I need to make sure that our data is future proof (as much as possible). That is why I support using org.openmeta as the primary storage location and mirroring data to com.apple.metadata (I don't care if the mirroring is forced or an option the user can turn off and on). Given the current implementation, I am reluctant to adopt the standard. Please make this change for the good of this very promising standard.

- RainDelay

OpenMeta Is a Hack

2009-02-24T15:19:28-07:00

FWIW,
I'm anxious to get my hands on open meta copies of Yep and Leap. I'm not a developer or a programmer, I'm an end user. Most of these discussions are WAY over my head. I firmly believe tagging over hierarchical structure is the way to go and I also believe the powers that be at Apple know a good idea when they see one.

With that said, I encourage those at Ironic Software. And I'll support you with my wallet.

James

OpenMeta Is a Hack

2009-02-24T16:05:04-07:00

I'm glad to see a weight of opinion turning to using org.openmeta as the primary store & mirroring tags to com.apple.metadata. I think that's a cleaner solution, and less risky in the long-run.

My experience was that I went through and tagged a huge number of files in YEP. Subsequently all the tags went missing, leaving me perplexed & irritated :D
I really don't want that to happen again because my tags were slid in under the spotlight radar. I've given up using YEP for tagging my files, because I don't trust its tagging mechanism (spotlight comments). I came on here to check progress towards YEP using OpenMeta, which I thought would be more robust. Now I'm cautious about any app using OpenMeta (or any other way of writing tags exclusively to com.apple.metadata)

I'm not sure where that leaves me, but I'm currently building a file tagging and storage workflow utility, and am now less convinced of using OpenMeta.

A further concern I have, is that the ~/Library backup solution is, in my eyes stupid (sorry ;-) The bulk of my files are on an external drive, which I share between 2 Macs. What will the behaviour of the Backup folder be? I dunno! I really hope that it's not just going to force create a /Volumes/ExtDisk/Library/OpenMeta Folder, akin to what YEP did, which I hated.

My 2c...

OpenMeta Is a Hack

2009-02-24T23:07:50-07:00

In addition to stuartleith I am wondering what happens to tagged files on encrypted volumes. Do they show up in the ~/Library backup as well?

OpenMeta Is a Hack

2009-02-28T17:22:18-07:00

May I respectfully suggest that Ironic post an "official" response at about this point, summarising the issues raised in this thread and explaining their plan moving forward? Ideally, they could then close down this thread, while inviting people to continue this discussion on other threads. If we continue like this, I'm worried that this long and unsatisfactory thread will be the first thing that prospective OpenMeta users or developers see, coming here from Daring Fireball, and also the last, as they get a false impression of stubbornness and unresponsiveness.

OpenMeta Is a Hack

2009-03-01T03:31:47-07:00

Yea, at the moment it seems like the situation is in limbo. The silence from the OM developers is a little unnerving. If this was my 'baby' I'd be on here all the time responding to posts, reassuring developers etc. It doesn't strike me with confidence that the matter of bringing developers on board & making this a standard can move forward.

OpenMeta Is a Hack

2009-03-01T18:52:05-07:00

But Jono, maybe instead of replying here the guys are working hard to please everyone ;)

Who knows! In irony, uh, ironic I trust ;) ^^

OpenMeta Is a Hack

2009-03-02T15:01:50-07:00

Well the app is open source for a reason. Ironic sw is sitting around like deity in a Golden Palace.

The best solution for approaching customers with this is honesty. I would simply inform them that
Open Meta uses a portion of OS X that may not remain stable over large upgrades but shouldn't be harmful
to their computer.

The developers have to find a consensus about how to implement Open Meta in a way that shields them from
support costs should Apple do something to render tags unstable.

I don't think i'm alone in my feeling that tagging is nigh useless unless its system wide. Tagging requires that I
set up a schema or hierarchy that works for me and after that actually tag my files. I don't enough sweat equity if
this burden simply locks me into an app. I need ALL my tag aware apps to follow a system.

Apple's silence in this area is par for the course. They're not going to say anything until they're ready to ship a solution which
may or may not come. My guess is "hardened" tagging that is system-wide won't come until 10.7 at the earliest. The only way
to spur their efforts is to build a case for it.

Spotlight, in its current incarnation, is marginally useful. I'm beholden to basica metadata in my files and I frequently get too much
data making finding the right data harder.

I kind of laugh when we get this futuristic and halcyonic speculation about how we'll use computing devices and OS in the future when I don't even
think we've nailed finding a users files efficiently and Apple and Microsoft have great poker faces here because neither seem to be up challenge openly.

They seam more intent on telling me i'm going to be flipping shiza around with my fingers. Yawn.

OpenMeta Is a Hack

2009-03-05T01:02:06-07:00

Am I right in thinking that perhaps this is supposed to be the answer to the issues raised in the current thread?

OpenMeta Is a Hack

2009-03-05T09:21:21-07:00

That seemed to be the official Ironic statement on it, but a lot of the input from other developers that we've seen has happened since then, so my hope is that Tom is talking to these guys to find a solution that gets OpenMeta adopted more widely.

OpenMeta Is a Hack

2009-03-11T18:25:03-07:00

The most thorough answer seems to be here, posted by Jon of Default Folder fame. Poor Michael Tsai had to come back and ask again whether using org.openmeta wouldn't be better, and the answer is that "also storing tags in an org.openmeta:kOMUserTags xattr would result in extra data shuffling without a real benefit". If Jon's satisfied, I guess I am, too.

I hope the developers of BibDesk, Skim, DevonThink, Bookpedia (probably lots of others, too) are alerted to the new thread.

OpenMeta Is a Hack

2009-03-12T00:15:16-07:00

Hi,

send the bibdesk-list a link. Skim will not write openmeta tags because of its way of saving files (that is what I understood).

Rolf

OpenMeta Is a Hack

2009-04-08T11:24:21-07:00

I’ve posted additional thoughts in these threads: 1, 2, 3. Christiaan Hofman of Bibdesk/Skim agrees with me that Jon Gotow is mistaken. I respect Jon, but keep in mind that his main product is a haxie, so he’s probably more accustomed to relying on unsupported OS behavior.

OpenMeta Is a Hack

2009-04-09T13:29:25-07:00

Guest:

I am not Jon Gotow but, Michael, I now really hate your way of handling this issue. Do you enjoy downgrading the others? You may not agree but what you're doing is the same as a personal attack. I really, really hate discussion over OpenMeta is getting a personal attack.

OpenMeta Is a Hack

2009-04-09T14:18:05-07:00

This thread is now closed. Official discussions have been taken to a more appropriate venue. - Jim