Thursday, 12 November 2009

NotesStream to the test

At the NL LUG 2009 Martin Scheafer improvised during his shortened session Domino Globetrotting/Dev talk. Due to customer regulations he could not show all he had planned to show.

And so he did a couple of side steps into @function and lotusscript coding. NotesStream was mentioned as a performance booster when building (long) strings e.g. in webagents when building return data for a webservice. During execution of demo code the difference between concatening a string using [string = string + extrastring] and the same code but now using NotesStream to build the return string, concatenating was a factor 4 to 5 slower than NotesStream.
I immediately saw an opportunity to improve the performance of the several agents we have running every day to transport product and other data from Notes to our Websphere Commerce servers via xml files. Some of them deliver a xml file of over 100mb.
So I put it to the test running 4 agents delivering the same result and each of them differing slightly in how the data is written.

  1. agent builds string (concatenate) to 65000+ bytes and than writes to file.
  2. agent builds string (concatenate) per record and writes to file per record.
  3. agent write to file directly, line by line.
  4. agent writes to file line by line using NotesStream.
The first 3 agents write to file via LS print, agent 4 using NotesStream.

The output build has the following structure per record:

The results in performance are listed below:
  1. Concatenated string test (concat string to 65000+ bytes before write).
    [Elapsed time: 296 seconds] [processed 123838 records]
  2. Concatenated string test (concat/write per record).
    [Elapsed time: 163 seconds] [processed 123838 records]
  3. Direct print to file test.
    [Elapsed time: 169 seconds] [processed 123838 records]
  4. NotesStream string test.
    [Elapsed time: 170 seconds] [processed 123838 records]
Conclusion is, NotesStream is indeed faster, but only when building large strings. The agent (1.) that concatenates strings up to 65000 bytes proves this. When keeping the strings limited in size, agents 2, 3 & 4, NotesStream performs equal to the other string methods.
And as Martin mentioned in his session, NotesStream is certainly helpful in webagents that build return xml's on request.


Richard Hogan said...

Hi Tom, I did a bit of work on optimising this a couple of years ago. I found that the following technique gave me between 10x and 20x improvement compared to simply appending to the end of the string each time around the loop (ie. where simple appending took 10 secs this took 0.5sec)

strFinal = ""
for i = 1 to 5000
strTemp = strTemp & array(i)
if len(strTemp) > 2000 then
strFinal = strFinal & strTemp
strTemp = ""
end if
strFinal = strFinal & strTemp

Note: array() is an array of 5000 strings

I'd be interested in seeing what you think?


Jerry Glover said...

Richard, if you already have an array of strings, you can simply do a join rather than your loop of concatenation.

strFinal = join( array )

Julian Robichaux said...

A few other speed tests (and techniques) here:

Richard Hogan said...

@Jerry, thanks for the tip. I don't think I've ever used a join but a quick test shows me that it's an order of magnitude as fast as my earlier example, so very useful.
@Julian, great write up on the subject. I think my example was a variation on your r5Join() function in StringBuffer.