From Guillaume.Drevon at lumension.com Mon Apr 4 11:27:15 2011 From: Guillaume.Drevon at lumension.com (Guillaume Drevon) Date: Mon, 4 Apr 2011 10:27:15 +0100 Subject: [Expat-discuss] Expat loading the entire file in memory? Message-ID: <750C20E09BB8BF458D0AC0A0CE22B98D27D70A01CD@IE-ML-01v.patchlink.com> I have been able to solve my issue. In fact there is a macro that you need to define before building expat lib: XML_MIN_SIZE With this, expat will keep the memory usage very low. In my case, it was below 3MB, which is much much less than 500MB! Expat documentation warns that this option will make expat running slower so this is why it is not the default behavior. This makes sense as it will work well except for extreme files like mine. Note that expat was not loading the entire file in memory, as else it would have needed 1GB due to the unicode conversion. On Wed, Mar 30, 2011 at 12:08 PM, Guillaume Drevon wrote: > I am trying to use Expat for parsing a 500MB file. I read the file by chunks (of 1000 byte) and send them to the parser. However, the memory used by my application goes up to 550 MB. > The memory usage is entirely done by expat as the XmlParserFree does remove almost all memory consumption. I see that the memory is mostly used by the m_buffer member of the parser. > > Am I doing something wrong? The purpose of this SAX-like parser is not memory usage? From karl at waclawek.net Mon Apr 4 14:53:00 2011 From: karl at waclawek.net (Karl Waclawek) Date: Mon, 4 Apr 2011 08:53:00 -0400 Subject: [Expat-discuss] Expat loading the entire file in memory? In-Reply-To: <750C20E09BB8BF458D0AC0A0CE22B98D27D70A01CD@IE-ML-01v.patchlink.com> References: <750C20E09BB8BF458D0AC0A0CE22B98D27D70A01CD@IE-ML-01v.patchlink.com> Message-ID: XML_MIN_SIZE only affects the size of the library. It is not meant for decreasing memory usage. Also, Expat should not use 500MB on a 1GB file, unless that file contains a 500MB long element name or such things. Maybe you should post your code. On Mon, Apr 4, 2011 at 5:27 AM, Guillaume Drevon < Guillaume.Drevon at lumension.com> wrote: > I have been able to solve my issue. In fact there is a macro that you need > to define before building expat lib: XML_MIN_SIZE > > With this, expat will keep the memory usage very low. In my case, it was > below 3MB, which is much much less than 500MB! > > Expat documentation warns that this option will make expat running slower > so this is why it is not the default behavior. This makes sense as it will > work well except for extreme files like mine. > > Note that expat was not loading the entire file in memory, as else it would > have needed 1GB due to the unicode conversion. > > > > > > On Wed, Mar 30, 2011 at 12:08 PM, Guillaume Drevon > > wrote: > > > I am trying to use Expat for parsing a 500MB file. I read the file by > chunks (of 1000 byte) and send them to the parser. However, the memory used > by my application goes up to 550 MB. > > > The memory usage is entirely done by expat as the XmlParserFree does > remove almost all memory consumption. I see that the memory is mostly used > by the m_buffer member of the parser. > > > > > > Am I doing something wrong? The purpose of this SAX-like parser is not > memory usage? > > _______________________________________________ > Expat-discuss mailing list > Expat-discuss at libexpat.org > http://mail.libexpat.org/mailman/listinfo/expat-discuss > From yanliu at cienet.com.cn Wed Apr 20 12:18:15 2011 From: yanliu at cienet.com.cn (Helen Liu (Yan)) Date: Wed, 20 Apr 2011 18:18:15 +0800 Subject: [Expat-discuss] Consult Expat ON IBM Z model -zEnterprise 196 Message-ID: Hi, Service. Our product is using your expat 1.95 version on HP UNIX. And now, we plan to move our product to IBM zEnterprise 196 HOST with RedHat Enterprise Linux or Suse Linux Enterprise Server OS. Could I know if the expat 1.95 version supports the IBM zEnterprise 196 for Linux? Or where could I get the expat 1.95 supports the IBM zEnterprise 196 for Linux? Thanks a lot. Regards, Helen From janezz55 at gmail.com Wed Apr 20 14:38:51 2011 From: janezz55 at gmail.com (Janez Zemva) Date: Wed, 20 Apr 2011 14:38:51 +0200 Subject: [Expat-discuss] Consult Expat ON IBM Z model -zEnterprise 196 In-Reply-To: References: Message-ID: I don't have access to a zEnterprise system, but I'm fairly certain expat supports it. I've compiled it for everything from mobile phones to embedded systems. As to where to get it, why don't you go to the expat homepage and compile the library for yourself? 2011/4/20 Helen Liu (Yan) : > Hi, Service. > > > > Our product is using your expat 1.95 version on HP UNIX. > > And now, we plan to move our product to IBM zEnterprise 196 HOST with RedHat > Enterprise Linux or Suse Linux Enterprise Server OS. > > Could I know if the expat 1.95 version supports the IBM zEnterprise 196 for > Linux? > > Or where could I get the expat 1.95 supports the IBM zEnterprise 196 for > Linux? > > > > > > Thanks a lot. > > Regards, Helen > > > > _______________________________________________ > Expat-discuss mailing list > Expat-discuss at libexpat.org > http://mail.libexpat.org/mailman/listinfo/expat-discuss > From nickmacd at gmail.com Fri Apr 22 03:51:25 2011 From: nickmacd at gmail.com (Nick MacDonald) Date: Thu, 21 Apr 2011 21:51:25 -0400 Subject: [Expat-discuss] Consult Expat ON IBM Z model -zEnterprise 196 In-Reply-To: References: Message-ID: Just to follow up from another responder. eXpat is a library for developers, not a product. You would need to get the source, and build it for your intended purpose... but this should not be a challenge for a competent developer. You should probably not use an old version like 1.95... the current version is 2.01 and has some bugfixes since the 1.95 release. You can find the current version at http://expat.sourceforge.net/ ... it is very portable source code... it should be compilable on an Linux derivative, such as your RedHat Enterprise Linux on the IBM. In fact, its well and truly possible that RedHat already includes eXpat... its pretty commonly used by many open source projects and so often comes as a base library on systems based on open source. Good Luck... On Wed, Apr 20, 2011 at 6:18 AM, Helen Liu (Yan) wrote: > Our product is using your expat 1.95 version on HP UNIX. > > And now, we plan to move our product to IBM zEnterprise 196 HOST with RedHat > Enterprise Linux or Suse Linux Enterprise Server OS. > > Could I know if the expat 1.95 version supports the IBM zEnterprise 196 for > Linux? > > Or where could I get the expat 1.95 supports the IBM zEnterprise 196 for > Linux? From yanliu at cienet.com.cn Fri Apr 22 04:06:51 2011 From: yanliu at cienet.com.cn (Helen Liu (Yan)) Date: Fri, 22 Apr 2011 10:06:51 +0800 Subject: [Expat-discuss] Consult Expat ON IBM Z model -zEnterprise 196 In-Reply-To: References: Message-ID: <0F437A350A5C4DD6B5005466B619B8ED@cnt01.local> Hi, Nick Thank you very much... As I know the linux on IBM zenterprise 196 is different to common linux.. So I'm afraid about whether expat lib could works well... Another: our products (low version) always use 1.95...I don't know if we could upgrade to 2.01.... Anyway, Thanks again. Regards, Helen -----Original Message----- From: Nick MacDonald [mailto:nickmacd at gmail.com] Sent: Friday, April 22, 2011 9:51 AM To: Helen Liu (Yan) Cc: expat-discuss at libexpat.org Subject: Re: [Expat-discuss] Consult Expat ON IBM Z model -zEnterprise 196 Just to follow up from another responder. eXpat is a library for developers, not a product. You would need to get the source, and build it for your intended purpose... but this should not be a challenge for a competent developer. You should probably not use an old version like 1.95... the current version is 2.01 and has some bugfixes since the 1.95 release. You can find the current version at http://expat.sourceforge.net/ ... it is very portable source code... it should be compilable on an Linux derivative, such as your RedHat Enterprise Linux on the IBM. In fact, its well and truly possible that RedHat already includes eXpat... its pretty commonly used by many open source projects and so often comes as a base library on systems based on open source. Good Luck... On Wed, Apr 20, 2011 at 6:18 AM, Helen Liu (Yan) wrote: > Our product is using your expat 1.95 version on HP UNIX. > > And now, we plan to move our product to IBM zEnterprise 196 HOST with RedHat > Enterprise Linux or Suse Linux Enterprise Server OS. > > Could I know if the expat 1.95 version supports the IBM zEnterprise 196 for > Linux? > > Or where could I get the expat 1.95 supports the IBM zEnterprise 196 for > Linux?