IRC Archive for channel #xwiki on 10 August 2012

00:55 <polx> has quit
01:16 <CIA-114> Denis Gervalle master * rde3c8c0 https://github.com/xwiki/xwiki-platform/commit/de3c8c0c4949d891bd385a4fecb1408616fcd9d1 / xwiki-platform-core/xwiki-platform-oldcore/src/main/java/com/xpn/xwiki/store/migration/hibernate/R40000XWIKI6990DataMigration.java : XWIKI-8126: Better error reporting when proper hibernate mapping is not found during R40000XWIKI6990 - http://git.io/qwZ-mw
01:18 <CIA-114> Denis Gervalle stable-4.1.x * rb51311e https://github.com/xwiki/xwiki-platform/commit/b51311efea9da3feb87856b2af3582ee7d66362f / xwiki-platform-core/xwiki-platform-oldcore/src/main/java/com/xpn/xwiki/store/migration/hibernate/R40000XWIKI6990DataMigration.java : XWIKI-8126: Better error reporting when proper hibernate mapping is not found during R40000XWIKI6990 - http://git.io/EKfFOQ
02:09 <@sdumitriu> cjd: Ping again
02:09 <@cjd> pong
02:10 <@sdumitriu> Good, now let me remember what I was pinging you about...
02:10 <tekzilla> has quit
02:10 <@sdumitriu> I think it was about the attachment stuff
02:10 <@sdumitriu> From what I see in https://github.com/cjdelisle/xwiki-newstore/blob/master/xwiki-platform-store-datanucleus/xwiki-platform-store-datanucleus-blob/src/main/java/org/xwiki/store/blob/datanucleus/internal/BlobSaveTransactionRunnable.java
02:11 <@sdumitriu> I gather that you're splitting the content into small chunks (BlobChunk) that you store individually
02:12 <@cjd> yes
02:12 <@sdumitriu> I don't get who's actually going to push this stuff into the store
02:12 <@sdumitriu> I mean, what's happening behind PersistenceManager.makePersistent
02:12 <@cjd> At that point it's JDO which is equivilant to Session.save()
02:13 <@cjd> did I answer the right question?
02:13 <@sdumitriu> So when it sees
02:13 <@sdumitriu> @PersistenceCapable(table = "BlobChunk")
02:13 <@sdumitriu> It's going to call getContent() and setContent(byte[])?
02:14 <@cjd> It adds setters and getters during the build process
02:14 <@cjd> so there is something hidden in there like   jdoGetField(int fieldIndex)
02:14 <@cjd> there's a bytecode mangler which reads those annotations and adds the necessary methods
02:14 <@cjd> which I have come to really like since it keeps stuff clean on my end
02:15 <@sdumitriu> Yep
02:15 <@sdumitriu> Hibernate should also be used using annotations
02:15 <@sdumitriu> Using a mapping file is kind of deprecated
02:15 <@cjd> I think it can but it still needs the setters and getters
02:16 <@cjd> with jdo you write your class as you want it and then it fixes it so that it can handle it </ad>
02:16 <tekzilla> has joined #xwiki
02:16 <@sdumitriu> Yep
02:16 <@sdumitriu> Hibernate doesn't do that, but you can combine it with another tool that adds getters and setters
02:16 <@sdumitriu> There are several that do that
02:17 <@cjd> hmm interesting
02:17 <@sdumitriu> And you can even let Hibernate use reflection to access private fields without getters and setters
02:18 <@sdumitriu> I can't remember their names
02:18 <@cjd> yeah, the only annoying thing is it breaks any hope of running under a security manager
02:18 <@sdumitriu> Yep
02:18 <@sdumitriu> OK, let's get back to our code
02:18 <@sdumitriu> So
02:18 <@cjd> also IIRC there is one configuration which has performance problems, I can't remember what it is
02:19 <@sdumitriu> I guess that you need the output stream for this line:
02:19 <@sdumitriu> https://github.com/cjdelisle/xwiki-newstore/blob/master/xwiki-platform-store-datanucleus/xwiki-platform-store-datanucleus-blob/src/main/java/org/xwiki/store/blob/datanucleus/internal/BlobLoadTransactionRunnable.java#L55
02:19 <@cjd> https://github.com/cjdelisle/xwiki-newstore/blob/master/xwiki-platform-store-datanucleus/xwiki-platform-store-datanucleus-attachments/src/main/java/org/xwiki/store/attachments/datanucleus/internal/DataNucleusAttachmentContentStore.java <-- here's the inter-thread communication mess which makes it work, I'd like to merge the patch and scrap this since it just reaks of fragility
02:19 <@cjd> yes, that's the one
02:21 <@sdumitriu> I'm trying to find a way to make it work with an InputStream instead of an OutputStream
02:22 <@cjd> I suppose it would be possible to implement a state machine but that would be fragile and unmaintainable
02:23 <@cjd> and reading from it outside of a transaction would make it just blow up
02:23 <@sdumitriu> Nope, I'm thinking more like a funnel
02:23 <@cjd> so you'd have a magic InputStream which worked sometimes
02:23 <@sdumitriu> Like have an intermediary pipe
02:23 <@sdumitriu> You write into it from your loader code
02:23 <@sdumitriu> And you read from it in the attachment
02:24 <@sdumitriu> And it will read with hops
02:24 <@cjd> 2 threads?
02:24 <@sdumitriu> Yes
02:24 <@cjd> that's what I have now
02:24 <@cjd> https://github.com/cjdelisle/xwiki-newstore/blob/master/xwiki-platform-store-datanucleus/xwiki-platform-store-datanucleus-attachments/src/main/java/org/xwiki/store/attachments/datanucleus/internal/DataNucleusAttachmentContentStore.java
02:24 <@cjd> I don't like it, IMO it's way too fragile
02:24 <@sdumitriu> Yes, I've seen that code
02:25 <@cjd> and I don't like spawning a thread without a good reason.
02:25 <@sdumitriu> Right, I missed the attachment.setContent(pis);
02:25 <@sdumitriu> Right, now that I read it again, it's exactly what I was thinking of
02:26 <@sdumitriu> But I think I'd do the threads differently
02:27 <@sdumitriu> The part that's gathering data and is writing should be in a side thread
02:27 <@sdumitriu> And the main thread should block on reading the data
02:27 <@sdumitriu> Actually, I'm not even sure it's needed
02:27 <@sdumitriu> Is it?
02:28 <@sdumitriu> Hm...
02:28 <@cjd> Is what needed?
02:28 <@cjd> I really don't like that code, if we were to merge that into the platform, we are almost guaranteed to get funny race conditions and magic on various different systems
02:28 <@sdumitriu> Since setContent(inputstream)  doesn't actually block on setting the content, you'll actually hang when trying to read the attachment content later on
02:29 <@cjd> no, it does
02:29 <@sdumitriu> And any call on getContent() will wait until the database reading thread calls stream.close()
02:29 <@cjd> no, setContent copies the content into a FileItem
02:30 <@sdumitriu> It does?
02:30 <@sdumitriu> Let me check
02:30 <@sdumitriu> Right, it does
02:30 <@sdumitriu> So it will block until it can read all the data from the InputStream
02:31 <@sdumitriu> So, to sum up
02:32 <@sdumitriu> You have a thread that's waiting for chunks of data to come from the distributed cassandra store
02:32 <@sdumitriu> And that thread writes this data into an output stream
02:32 <@cjd> A thread loading them and handing them through the pipe and another thread waiting for them in a blocked read call
02:32 <@sdumitriu> On the other side, there's an attachment content that's trying to copy this data into a file item
02:33 <@cjd> yeap
02:33 <@sdumitriu> And it will block until it gets a close() on that stream
02:33 <@cjd> or unti it gets interrupted, or until the magic thread gremlins show up and cause problems
02:33 <@cjd> IMO my code there is not stable
02:34 <@sdumitriu> If you move the copy to a side thread, it is indeed unstable
02:34 <@sdumitriu> At least one of the two branches has to block the main call
02:35 <@cjd> A single thread doing a load/write loop is pretty stable but that requires access to the OutputStream in XAC
02:35 <@sdumitriu> You moved the copy code in a side thread, but I would do it the other way
02:35 <@sdumitriu> Since Cassandra is more asynchronous IMO than XWiki
02:35 <@cjd> PersistenceManager.getObject() is synchronous
02:36 <@cjd> sadly
02:36 <@sdumitriu> Yes, removing the pipe would remove the need for two threads
02:38 <@cjd> Actually I think if it came down to it, it would be better to read all of the content out to a temp file and write it back. It's double buffering but at least it is not threading magic.
02:38 <@cjd> But IMO the best solution it the OutputStream
02:42 <@sdumitriu> I for one don't like that you're loading the chunks one by one
02:42 <@sdumitriu> It's going to be slow
02:42 <@sdumitriu> Well, normally we won't have huge attachments
02:43 <@cjd> 1MB chunks
02:43 <@sdumitriu> But still, that's a perfect target for parallelization
02:43 <@cjd> You can only pull so much from the disk at a time
02:43 <@sdumitriu> Load all the chunks in parallel, asynchronously, and then combine them in a temporary file
02:44 <@sdumitriu> But that's outside the scope of this discussion
02:44 <@sdumitriu> OK, so let's say that I see why you need the output stream now
02:45 <@sdumitriu> There are ways around it, but it would be simpler to have it
02:45 <@cjd> That's my thinking.
02:45 <@sdumitriu> I'll send a +1 vote after supper
02:45 <@cjd> Thanks a lot
02:46 <@cjd> Note that re the slow loading, another optimization target is to cache attachments by their sha1 on the filesystem and load them from the cache if possible
02:46 <@cjd> so the slow database load is only the first load of the dat
02:46 <@cjd> *day
02:47 <@cjd> right now we do almost that well since they are cached until the XWikiDoc is garbage collected
02:55 <Denis> has quit
03:09 <ssavi> has quit
03:57 <ssavi> has joined #xwiki
05:03 <CIA-114> Caleb James DeLisle master * rf1bdd33 https://github.com/xwiki/xwiki-platform/commit/f1bdd33ee3d6f069699fd2405b77c996f668268c / (2 files in 2 dirs): XWIKI-8121: XWikiAttachmentContent should have a way to set the content by writing to a provided OutputStream. - http://git.io/yL8KxQ
05:03 <CIA-114> Caleb James DeLisle master * r9452ee7 https://github.com/xwiki/xwiki-platform/commit/9452ee7ad0a42e9036fb5e3508a0a5216ac016be / xwiki-platform-core/xwiki-platform-oldcore/src/main/java/com/xpn/xwiki/doc/XWikiAttachmentContent.java : XWIKI-8121: XWikiAttachmentContent should have a way to set the content by writing to a provided OutputStream. - http://git.io/zCaWpw
05:03 <CIA-114> Caleb James DeLisle master * r724202d https://github.com/xwiki/xwiki-platform/commit/724202d64fed533b705a57197b91064b8d9f27e6 / (2 files in 2 dirs): XWIKI-8121: XWikiAttachmentContent should have a way to set the content by writing to a provided OutputStream. - http://git.io/cpgCRw
05:06 <ssavi> has quit
05:39 <ssavi> has joined #xwiki
05:41 <ssavi_> has joined #xwiki
05:43 <ssavi> has quit
05:43 <ssavi_> is now known as <ssavi>
05:56 <ssavi_> has joined #xwiki
05:59 <ssavi> has quit
06:01 <ssavi_> has quit
07:22 <SvenDowideit> has joined #xwiki
08:00 <Denis> has joined #xwiki
08:54 <tmortagne> has joined #xwiki
09:13 <polx> has joined #xwiki
09:19 <vmassol> has joined #xwiki
09:29 <Denis> has quit
11:03 <Denis> has joined #xwiki
11:17 <vmassol> has quit
11:24 <tmortagne> has quit
11:28 <mflorea> has joined #xwiki
11:39 <mflorea> has quit
12:24 <Helge> has joined #xwiki
12:26 <Helge> Hi, short question: Change of Rights to a page is currently a minor edit. Can that be configured to be a majjor edit? Thanks for advise.
12:46 <polx> has quit
12:55 <+Denis> Helge: no, it is not. A partial solution would be to change the interface used for setting rights, but it will not capture all cases. Why do you want those edit to be major ?
12:59 <sburjan`> has quit
13:05 <CIA-114> Denis Gervalle master * rd863d0a https://github.com/xwiki/xwiki-platform/commit/d863d0a7a81b210943f733e972f97354d6c91fcd / xwiki-platform-core/xwiki-platform-oldcore/src/main/java/com/xpn/xwiki/store/migration/hibernate/R40000XWIKI6990DataMigration.java : XWIKI-8126: Better error reporting when proper hibernate mapping is not found during R40000XWIKI6990 - http://git.io/xGqxLA
13:07 <CIA-114> Denis Gervalle stable-4.1.x * r92ccbce https://github.com/xwiki/xwiki-platform/commit/92ccbcefc6f6e69c04918266df0e66cc7abbf485 / xwiki-platform-core/xwiki-platform-oldcore/src/main/java/com/xpn/xwiki/store/migration/hibernate/R40000XWIKI6990DataMigration.java : XWIKI-8126: Better error reporting when proper hibernate mapping is not found during R40000XWIKI6990 - http://git.io/isbWKw
13:15 <vmassol> has joined #xwiki
13:19 <vmassol> has quit
13:20 <mflorea> has joined #xwiki
13:29 <CIA-114> Denis Gervalle master * rde245de https://github.com/xwiki/xwiki-platform/commit/de245de5221ebd0df84636b40b163ae3a5609198 / xwiki-platform-core/xwiki-platform-oldcore/src/main/java/com/xpn/xwiki/store/migration/hibernate/R40000XWIKI6990DataMigration.java : XWIKI-8125: Add support of Microsoft SQL Server databases in R40000XWIKI6990 migration - http://git.io/qpp2QQ
13:31 <CIA-114> Denis Gervalle stable-4.1.x * rd80e96a https://github.com/xwiki/xwiki-platform/commit/d80e96a7dfd3768d54ff8ce66f8a4c79545426cd / xwiki-platform-core/xwiki-platform-oldcore/src/main/java/com/xpn/xwiki/store/migration/hibernate/R40000XWIKI6990DataMigration.java : XWIKI-8125: Add support of Microsoft SQL Server databases in R40000XWIKI6990 migration - http://git.io/VTTEcw
13:50 <vmassol> has joined #xwiki
13:53 <+Denis> vmassol: Hi, I have found an issue with custom mapped class that I am puzzle with, maybe you can provide some insight on how I should fix it ?
13:54 <vmassol> Denis: hi… Sorry I'm still on holidays, was just connecting for a few seconds to get my mails… ;) I'm back on Monday though
13:54 <+Denis> anyone else on this ?
13:54 <vmassol> has quit
14:01 <@sdumitriu> Denis: Shoot, maybe I can help
14:01 <+Denis> sdumitriu: When you made a copy of a document containing a custom mapped class, a new custom mapped class is created by the exact copy of the original document class definition, but the custom mapping for this copied class is probably incorrect or not available.
14:02 <+Denis> For exemple, if you made a copy of XWiki.XWikiPreferences, you get a new class that has an internal mapping, but since the .xbm is unchanged, there is probably no chance that the newly defined class has a mapped entity in the hibernate mapping
14:02 <+Denis> This cause the migration to fail to migrate a database that contains a copy of XWiki.XWikiPreferences
14:03 <+Denis> Note that the copied document contains an unusable class.
14:03 <@sdumitriu> OK, I see
14:05 <+Denis> I really do not know what to do with that, IMO there is 2 issue, how we handle copy of document with custom mapped classes, and how we handle the past during migration
14:06 <+Denis> Not that dynamically custom mapped class could do something even worse, since it will map both class the same way
14:07 <+Denis> WDYT ?
14:08 <@sdumitriu> IMO, the root of the problem is that when cloning BaseClass, we copy the customMapping field
14:09 <@sdumitriu> So in XWikiDocument.cloneInternal we should call doc.getXClass().setCustomMapping(false)
14:09 <@sdumitriu> (after cloning the XClass)
14:09 <@sdumitriu> WDYT?
14:09 <@sdumitriu> Hm...
14:09 <@sdumitriu> Actually that's not good
14:09 <@sdumitriu> Since we might just clone the same document
14:10 <@sdumitriu> hasCustomMapping shouldn't be something that the BaseClass stores in a field...
14:11 <@sdumitriu> That's what the storage should handle on the fly
14:12 <+Denis> hasCustomMapping is not stored
14:12 <+Denis> the custom mapping is, and could either contains "internal" or an XML mapping
14:13 <+Denis> I also see another tricky situation, the renaming of a document containing a custom mapped class
14:13 <+Denis> which is a copy/delete, and maybe this one should keep the mapping
14:15 <+Denis> I agree that generaly we should drop the custom mapping, but this will not fix existing copies
14:16 <+Denis> should I relax my current throw during migration, into a warning, and risk to reach a migration success while there is a potential failure ?
14:16 <@sdumitriu> The only one that truly knows how custom mappings should work is Ludovic...
14:17 <+Denis> I know how it works, this is more how a copied class should work that have been overlooked
14:19 <+Denis> In the case of Jeremie, who raise the issue, he made a backup copy of its preferences, and was unable to migrate its database afterwards, which is really annoying and could repeat over time. Should even a custom mapped class be copied ?
14:26 <+Denis> sdumitriu: I see your puzzle like me, I have sent a mail on the ML
14:26 <@sdumitriu> I asked Ludo
14:26 <@sdumitriu> I for one am in favor of dropping the custom mapping on clone
14:30 <mflorea> has quit
14:40 <+Denis> could we base that on cloneInternal()'s keepsIdentity arguments ?
14:41 <@sdumitriu> Yes
14:43 <mflorea> has joined #xwiki
14:43 <mflorea> has quit
14:43 <mflorea1> has joined #xwiki
14:44 <+Denis> sdumitriu: could I cleanup the xClassXML, or should I update it ?
14:44 <@sdumitriu> Just cleanup
14:53 <+Denis> sdumitriu: http://pastebin.com/uuL753Pm
14:56 <@sdumitriu> Sounds good
14:56 <@sdumitriu> com.xpn.xwiki.store.XWikiHibernateStore.saveXWikiDoc(XWikiDocument, XWikiContext, boolean) will recreate the XML
14:56 <+Denis> yes
14:58 <+Denis> however, this does not fix existing copies, wdyt ?
15:00 <@sdumitriu> Relax the precondition on the migrator
15:00 <@sdumitriu> Just warn
15:01 <@sdumitriu> For the rest...
15:01 <@sdumitriu> I don't know, let's leave it to someone else to fix it :D
15:02 <+Denis> so you think that we may risk the migrator to success in rare cases when it should have failed, and that the resulting db could be unusable
15:06 <qwebirc23723> has joined #xwiki
15:07 <Helge> has quit
15:07 <qwebirc23723> +Denis: Thanks for the reply. I have documents which shall be "approved" by a user and be read only after approval. The act of approval should be visible in the history but it disappears if the subsequent edit rights is a minor edit (the "approval" edit is onlly listed when you click "show minor edits"). It would be visible if the rights edit was major.
15:17 <qwebirc23723> +Denis: In other words, I want the version previous to the right edit to be listed in the history. That's only the case when rights edit is major. Regards.
15:36 <mflorea1> has quit
15:36 <mflorea> has joined #xwiki
15:49 <jvelo> has joined #xwiki
16:12 <polx> has joined #xwiki
16:19 <polx> has quit
16:22 <polx> has joined #xwiki
16:28 <jvelo> has quit
16:45 <polx> has quit
17:03 <jvelo> has joined #xwiki
17:15 <polx> has joined #xwiki
17:37 <CIA-114> Sergiu Dumitriu feature-portlet * r52a9ab2 https://github.com/xwiki/xwiki-platform/commit/52a9ab25294c4d405299f89f43dec27b581804b3 / pom.xml : [release] Preparing release xwiki-platform-3.5.1 - http://git.io/pp7GAw
17:37 <CIA-114> Sergiu Dumitriu feature-portlet * r3102478 https://github.com/xwiki/xwiki-platform/commit/31024788158cc45879bf15832fa38c4834d434df / (205 files in 205 dirs): [maven-release-plugin] prepare release xwiki-platform-3.5.1 - http://git.io/UNzAUw
17:38 <vdox2> has left #xwiki
17:42 <polx> has quit
17:56 <qwebirc23723> has quit
18:03 <CIA-114> Marius Dumitru Florea feature-portlet * r85d1eed https://github.com/xwiki/xwiki-platform/commit/85d1eed331c62e3efb03cdfb15b0928a45cc43e2 / (2 files in 2 dirs): Make the AJAX behaviour of comments work in portlet mode (work in progress). (+6 more commits...) - http://git.io/WDeLcQ
18:16 <jvelo> has quit
18:26 <mflorea> has quit
18:41 <jvelo> has joined #xwiki
18:48 <polx> has joined #xwiki
18:58 <Denis> has quit
19:13 <Denis> has joined #xwiki
20:04 <polx> has quit
20:17 <polx> has joined #xwiki
20:28 <pgmjsd> has quit
20:52 <Denis> has quit
21:02 <pgmjsd> has joined #xwiki
21:17 <Denis> has joined #xwiki
21:34 <abusenius> has joined #xwiki
22:12 <CIA-114> Denis Gervalle master * re01a2b2 https://github.com/xwiki/xwiki-platform/commit/e01a2b29a8469a39c86ed5f08b689974dac31786 / (2 files in 2 dirs): XWIKI-8130: copyDocument should not copy the custom mapping of the document xClass - http://git.io/MFjqTw
22:14 <CIA-114> Denis Gervalle stable-4.1.x * r1e64133 https://github.com/xwiki/xwiki-platform/commit/1e64133958914c7b70e50ce16e999dd34485a11a / (2 files in 2 dirs): XWIKI-8130: copyDocument should not copy the custom mapping of the document xClass - http://git.io/QMUKog
22:46 <abusenius> has quit
23:48 <Denis> has quit