Tcl Library Source Code: View Ticket

Ticket UUID:	3493711
Title:	S3: Fix wasteful string conversion of large binary object
Type:	Patch	Version:	None
Submitter:	samoconnor	Created on:	2012-02-25 04:57:05
Subsystem:	amazon-s3	Assigned To:	aku
Priority:	5 Medium	Severity:	Severe
Status:	Closed	Last Modified:	2014-06-23 18:36:27
Resolution:	Fixed	Closed By:
		Closed on:
Description:	In our app S3::Get is used to download large binary objects from AWS (many hundreds of MB). The current code stores the downloaded object in "x" then does: S3::debug "Body: $x" This causes "x" to be converted to a string, and creates a second huge string "Body: .....". The patch below passes the large object directly from the binary network channel to the result dict entry. A small test program that downloads a 5MB file uses 24MB before the patch vs 9MB with the patch. --- /System/Library/Tcl/tcllib1.12/amazon-s3/S3.tcl.orig2012-02-25 15:32:45.000000000 +1100 +++ /System/Library/Tcl/tcllib1.12/amazon-s3/S3.tcl2012-02-25 15:33:25.000000000 +1100 @@ -580,9 +580,7 @@ if {[dict exists $thunk orig outchan]} { fcopy $s3 [dict get $thunk orig outchan] } else { - set x [read $s3] - dict set thunk outbody $x - S3::debug "Body: $x" + dict set thunk outbody [read $s3] } return [S3::nextdo all_done $thunk readable] } else {
User Comments:	aku added on 2014-06-23 18:36:27: Fixed contact information on closed/fixed ticket. aku added on 2014-06-20 18:36:47: Applied in revision [5e9d8b632f]. Committed. Pushed. Thank you. And apologies for the long delay. Better fixes to consider: - Modify the S3::debug to subst its arguments only when debugging is active. - Replace S3::debug with use of the 'debug' package in Tcllib (which does defer argument expansion to when it is active). These might be moot, depending on the overall fate of the S3 module and packages.