variable Have I reached the limits of the size of objects JavaScript in my browser can handle?




memory concept in javascript (5)

Yes, it's too much to ask of a browser.

That amount of data would be managable if it already was data, but it isn't data yet. Consider that the browser has to parse that huge block of source code while checking that the syntax adds up for it all. Once parsed into valid code, the code has to run to produce the actual array.

So, all of the data will exist in (at least) two or three versions at once, each with a certain amount of overhead. As the array literal is a single statement, each step will have to include all of the data.

Dividing the data into several smaller arrays would possibly make it easier on the browser.

I'm embedding a large array in <script> tags in my HTML, like this (nothing surprising):

<script>
    var largeArray = [/* lots of stuff in here */];
</script>

In this particular example, the array has 210,000 elements. That's well below the theoretical maximum of 231 - by 4 orders of magnitude. Here's the fun part: if I save JS source for the array to a file, that file is >44 megabytes (46,573,399 bytes, to be exact).

If you want to see for yourself, you can download it from GitHub. (All the data in there is canned, so much of it is repeated. This will not be the case in production.)

Now, I'm really not concerned about serving that much data. My server gzips its responses, so it really doesn't take all that long to get the data over the wire. However, there is a really nasty tendency for the page, once loaded, to crash the browser. I'm not testing at all in IE (this is an internal tool). My primary targets are Chrome 8 and Firefox 3.6.

In Firefox, I can see a reasonably useful error in the console:

Error: script stack space quota is exhausted

In Chrome, I simply get the sad-tab page:

Cut to the chase, already

  • Is this really too much data for our modern, "high-performance" browsers to handle?
  • Is there anything I can do* to gracefully handle this much data?

Incidentally, I was able to get this to work (read: not crash the tab) on-and-off in Chrome. I really thought that Chrome, at least, was made of tougher stuff, but apparently I was wrong...


Edit 1

@Crayon: I wasn't looking to justify why I'd like to dump this much data into the browser at once. Short version: either I solve this one (admittedly not-that-easy) problem, or I have to solve a whole slew of other problems. I'm opting for the simpler approach for now.

@various: right now, I'm not especially looking for ways to actually reduce the number of elements in the array. I know I could implement Ajax paging or what-have-you, but that introduces its own set of problems for me in other regards.

@Phrogz: each element looks something like this:

{dateTime:new Date(1296176400000),
 terminalId:'terminal999',
 'General___BuildVersion':'10.05a_V110119_Beta',
 'SSM___ExtId':26680,
 'MD_CDMA_NETLOADER_NO_BCAST___Valid':'false',
 'MD_CDMA_NETLOADER_NO_BCAST___PngAttempt':0}

@Will: but I have a computer with a 4-core processor, 6 gigabytes of RAM, over half a terabyte of disk space ...and I'm not even asking for the browser to do this quickly - I'm just asking for it to work at all!


Edit 2

Mission accomplished!

With the spot-on suggestions from Juan as well as Guffa, I was able to get this to work! It would appear that the problem was just in parsing the source code, not actually working with it in memory.

To summarize the comment quagmire on Juan's answer: I had to split up my big array into a series of smaller ones, and then Array#concat() them, but that wasn't enough. I also had to put them into separate var statements. Like this:

var arr0 = [...];
var arr1 = [...];
var arr2 = [...];
/* ... */
var bigArray = arr0.concat(arr1, arr2, ...);

To everyone who contributed to solving this: thank you. The first round is on me!


*other than the obvious: sending less data to the browser


Use lazy loading. Have pointers to the data and get it when the user asks.

This technique is used in various places to manage millions of records of data.

[Edit]

I found what I was looking for. Virtual scrolling in the jqgrid. That's 500k records being lazy loaded.


Do you really need all the data? can't you stream just the data currently needed using AJAX? Similar to Google Maps - you can't fit all the map data into browser's memory, they display just the part you are currently seeing.

Remember that 40 megs of hard data can be inflated to much more in browser's internal representation. For example the JS interpreter may use hashtable to implement the array, which would add additional memory overhead. Also, I expect that the browsers stores both source code and the JS memory, that alone doubles the amount of data.

JS is designed to provide client-side UI interaction, not handle loads of data.

EDIT:

Btw, do you really think users will like downloading 40 megabytes worth of code? There are still many users with less than broadband internet access. And execution of the script will be suspended until all the data is downloaded.

EDIT2:

I had a look at the data. That array will definitely be represented as hashtable. Also many of the items are objects, which will require reference tracking...that is additional memory.

I guess the performance would be better if it was simple vector of primitive data.

EDIT3: The data could certainly be simplified. The bulk of it are repeating strings, which could be encoded in some way as integers or something. Also, my Opera is having trouble just displaying the text, let alone interpreting it.

EDIT4: Forget the DateTime objects! Use unix era timestamps or strings, but not objects!

EDIT5: Your processor doesn't matter because JS is single-threaded. And your RAM doesn't matter either, most browsers are 32bit, so they can't use much of that memory.

EDIT6: Try changing the array indices to sequential integers (0, 1, 2, 3...). That might make the browser use more efficient array data structure. You can use constants to access the array items efficiently. This is going to cut down the array size by huge chunk.


Try retrieving the data with Ajax as an JSON page. I don't know the exact size but I've been able to pull large amounts of data into Google Chrome that way.


I would try having it as one big string with separator between each "item" then use split, something like:

var largeString = "item1,item2,.......";
var largeArray = largeString.split(",");

Hopefully string won't exhaust the stack so fast.

Edit: in order to test it I created dummy array with 200,000 simple items (each item one number) and Chrome loaded it within an instant. 2,000,000 items? Couple of seconds but no crash. 6,000,000 items array (50 MB file) made Chrome load for about 10 seconds but still, no crash in either ways.

So this leads me to believe the problem is not with the array itself but rather it's contents.. optimize the contents to simple items then parse them "on the fly" and it should work.





memory