🔒 Locked

[java] sigh java.lang.OutOfMemoryError : Java heap space

Started by Kais0n Jul 10, 2006 at 11:01 AM 5 replies 1.3k views

Original Post

Jul 10, 2006 at 11:01 AM

Hello everybody, this isnt really a game related question but since im lurking around these forums for some years now and its about java i thought my question might be at the right place here anyways. Here it comes. I need to readout a rather large txt file containing genomic data, and convert these data into a n*m int matrix. To do so i read the file using a FileReader and a BufferedReader. Then im going to tokenize the file using the StreamTokenizer Example Data:


rs3000004 A/G chr14 19166795 - ncbi_b35 perlegen urn:lsid:perlegen.hapmap.org:Protocol:Genotyping_1.0.0:2 urn:lsid:perlegen.hapmap.org:Assay:25761.5787732:1 urn:lsid:dcc.hapmap.org:Panel:CEPH-30-trios:1 QC+ GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG NN GG GG GG GG GG GG GG NN GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG GG
rs8013379 C/G chr14 19217196 + ncbi_b35 perlegen urn:lsid:perlegen.hapmap.org:Protocol:Genotyping_1.0.0:2 urn:lsid:perlegen.hapmap.org:Assay:25760.7547817:1 urn:lsid:dcc.hapmap.org:Panel:CEPH-30-trios:1 QC+ CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC NN CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC NN CC CC CC CC CC NN CC CC CC CC CC CC CC CC CC CC CC CC NN CC CC CC CC CC CC CC CC CC CC NN

I basicly just check what kind of nucelotide combination i have found and then set my matrix to the corresponding value (int). However if my file exceeds ~7MB i encounter a java.lang.OutOfMemoryError : Java heap space. What really puzzels me is that i can read files up to 100MB without any problem if i only print them back on screen for example, aswell as manipulating a 200000 * 900 matrix isnt a problem either. Im not creating any references to the Strings returned by StreamTokenizer aswell :/ Your help will be much appreciated! -Kai Kratz p.s. sorry for any grammar/spelling errors i made, bad english mmmh kay ;)

OrangyTang

Jul 10, 2006 at 11:14 AM

This isn't a proper solution, but by default the VM starts with a maximum of only 64Mb. Theres a command line switch you can use to increase the amount available (eg. -Xmx256m for 256Mb) which might be good enough for a quick hacky workaround.

EmmetjeGee

Jul 10, 2006 at 11:16 AM

Hi,

Two remarks:
- I think some source code could help us.
- Have you checked the bug database?

Regards,
EmmetjeGee

Son of Cain

Jul 10, 2006 at 2:15 PM

Using...

java -Xmx128m MyProgram.java

...will give you more heap space. However, it seems you're holding references that you shouldn't while parsing your file. I know you mentioned you're not using any references to the captured String, but maybe one of the API classes you're using may be holding unwanted stuff.

As the above post suggests, post some code so that we can try to spot what's causing this memory abuse.

Son Of Cain

Kais0n

Jul 11, 2006 at 2:56 AM

Ok here is the sorcecode of my nasty little method.
But before you start to tear it appart i think i neet to explain how it works a little further.

As you saw in my previous post i was extracting data from txt files.
Each line started with rsxxxx then some other stuff and finaly what im really intrested in the GG AA TT etc...

However, when i translate these into my n*m matrix im assining values according to what combination came first in this line.

The first pair of identical chars gets a 0 in my int matrix.
The secong pair of identical chars gets a 1
The pairs of non-identical chars get a 2

if my sample data looks like this

GG GG AA AT
GG GG GG GG
AA TT AA AT

it will be translated to: (rows in the flatfile represent columns in the matrix)

0 0 0
0 0 1
1 0 0
2 0 2

Conversion into my matrix isnt any problem(well at least for files smaller than ~7MB) its tested and working well, but since i share the code with you i want to give you all information to fully understand wtf i am doing there ^^.

Used Methods:
setTaxaIDs <- placeholder Method does nothing yet
matrix.setStateMatrixLegendColumnInfo <- is supposed to store some information about the line found in the flatfile, currently disabled
matrix.setStateMatrixLegendColumnInfo <- disabled for debugging aswell
matrix.setState <- sets a position in my n*m matrix to a value

About GenotypeMatrix:
Well as you might see im using this line to set my int:

matrix.setState(GenotypeMatrix.Kind.genotype,currentTaxon,currentColumn,0);

The last 3 are pretty self explanatory its n and m of my matrix and what value.
The first parameter however is a enum used to determin wich matrix in Genotypematrix im accessing. So far it holds only one matrix but is planed to hold several (not yet implemented).

I tested with disabling all above methods step by step and i didnt get any increase in performance, thats why i didnt include them in the sorce code (seems like the problem is not in there)

I honestly admit my code is a bit messy in between but i didnt come up with a better way of reading my file and writing my matrix at the same time, since the coding is dependant on what pair of chars i found first in the currently read line.
My first approach was to store all information in a List but i didnt programm it out after seeing that my list might be a little longer than 200000*90 Strings :P
So i thougt a direct conversion to the corresponding int would be better.

HapMapIO.java

	public static GenotypeMatrix readHapmapFlatFile(String filename){		try{				int[] dimension = getDimension(filename);			matrix = new GenotypeMatrix(dimension[0],dimension[1]);						setTaxaIDs(filename, matrix);						fr = new FileReader(filename);			br = new BufferedReader(fr);			st = new StreamTokenizer(br);			st.ordinaryChar('/');			st.ordinaryChar(':');			st.ordinaryChar('#');						int currentColumn = -1;			int currentTaxon = 0;			String rsID = "";			char firstHomozygot = '-';			char secondHomozygot = '-';			char firstHeterozygot = '-';			char secondHeterozygot = '-';			boolean foundFirstHomozygot = false;			boolean foundSecondHomozygot = false;			boolean foundHeterozygot = false;						do{				if(st.nextToken() == StreamTokenizer.TT_WORD && st.sval.startsWith("rs") && st.lineno()>1){					if(currentColumn >= 0)						matrix.setStateMatrixLegendColumnInfo(currentColumn, rsID, firstHomozygot, secondHomozygot,															  firstHeterozygot, secondHeterozygot);					currentColumn++;					rsID = st.sval;					firstHomozygot = '-';					secondHomozygot = '-';					firstHeterozygot = '-';					secondHeterozygot = '-';					foundFirstHomozygot = false;					foundSecondHomozygot = false;					foundHeterozygot = false;					currentTaxon = 0;				}								if(st.ttype == StreamTokenizer.TT_WORD && tokenIsHomozygotGenotypeInfo(st.sval)){					if(foundFirstHomozygot && foundSecondHomozygot){						if(st.sval.charAt(0) == firstHomozygot){							matrix.setState(GenotypeMatrix.Kind.genotype,currentTaxon,currentColumn,0);							currentTaxon++;						}						if(st.sval.charAt(0) == secondHomozygot){							matrix.setState(GenotypeMatrix.Kind.genotype,currentTaxon,currentColumn,1);							currentTaxon++;						}					}else{						if(foundFirstHomozygot && !foundSecondHomozygot){							if(st.sval.charAt(0) == firstHomozygot){								matrix.setState(GenotypeMatrix.Kind.genotype,currentTaxon,currentColumn,0);								currentTaxon++;							}else{								foundSecondHomozygot = true;								secondHomozygot = st.sval.charAt(0);								matrix.setState(GenotypeMatrix.Kind.genotype,currentTaxon,currentColumn,1);								currentTaxon++;							}						}						if(!foundFirstHomozygot){							foundFirstHomozygot = true;							firstHomozygot = st.sval.charAt(0);							matrix.setState(GenotypeMatrix.Kind.genotype,currentTaxon,currentColumn,0);							currentTaxon++;						}					}					}else{									if(st.ttype == StreamTokenizer.TT_WORD && tokenIsHeterozygotGenotypeInfo(st.sval)){						if(foundHeterozygot){							if((st.sval.charAt(0) == firstHeterozygot && st.sval.charAt(1) == secondHeterozygot)||							   (st.sval.charAt(1) == firstHeterozygot && st.sval.charAt(0) == secondHeterozygot)){																matrix.setState(GenotypeMatrix.Kind.genotype,currentTaxon,currentColumn,2);								currentTaxon++;							}						}else{							firstHeterozygot = st.sval.charAt(0);							secondHeterozygot = st.sval.charAt(1);							foundHeterozygot = true;							matrix.setState(GenotypeMatrix.Kind.genotype,currentTaxon,currentColumn,2);							currentTaxon++;						}					}else{						if(st.ttype == StreamTokenizer.TT_WORD && tokenIsUnknownGenotypeInfo(st.sval)){							matrix.setState(GenotypeMatrix.Kind.genotype,currentTaxon,currentColumn,GenotypeMatrix.MISSING_DATA);							currentTaxon++;						}					}				}								if(st.ttype == StreamTokenizer.TT_EOL && st.lineno()>1)					matrix.setStateMatrixLegendColumnInfo(currentColumn, rsID, firstHomozygot, secondHomozygot, firstHeterozygot, secondHeterozygot);			}while(st.ttype != StreamTokenizer.TT_EOF);					}catch(IOException e){			// Nothing yet.		}		return matrix;	}

I did look into the StreamTokenizer sorce aswell but couldnt find any varaible holding old references.

Ah and before i forget, your help is much appreciated!

Lucidquiet

Jul 11, 2006 at 12:59 PM

Are you sure the rsIDs are not getting stored any where? 1 per line isn't really a whole lot only 9 bytes and 1000 lines would only be ~1k, and it would take over 100,000 lines of stored rsIDs even to amount to 1mb...

But a 200,000x2 int matrix >1mb.

L-

EmmetjeGee

Jul 11, 2006 at 2:16 PM

Hi,

A remark:
Different strings are always remembered/reused in java. They are stored in the string static pool. They are reused if you create the same string twice.

I just ran your program with a file of 2GB (consisting of the two lines you gave) and commented out the matrix.setState(...) and tokenIs..GenotypeInfo(...) methods.

Unfortunately I don't get a "out of memory exception" (I have 1GB). But then again, I am still missing parts of your program.

I can't see a "memory consumer" in the part given, but I can give you some advice:
- Look carefull at the stack trace when the exception occurs;
- Strip your program down to the essential that still crashes;
- Post your whole (but small) program again.

You can also try some profilers (Look at the memory footprint while your application runs).

Regards,
EmmetjeGee

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.