Tcl Source Code

Check-in [60cf594f82]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Revised TclFindElement() interface. The final argument had been bracePtr, the address of a boolean var, where the caller can be told whether or not the parsed list element was enclosed in braces. In practice, no callers really care about that. What the callers really want to know is whether the list element value exists as a literal substring of the string being parsed, or whether a call to TclCopyAndCollpase() is needed to produce the list element value. Now the final argument is changed to do what callers actually need. This is a better fit for the calls in tclParse.c, where now a good deal of post-processing checking for "naked backslashes" is no longer necessary. ***POTENTIAL INCOMPATIBILITY*** For any callers calling in via the internal stubs table who really do use the final argument explicitly to check for the enclosing brace scenario. Simply looking for the braces where they must be is the revision available to those callers, and it will backport cleanly. Tests for expanded literals quoting detection.
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk | potential incompatibility
Files: files | file ages | folders
SHA1: 60cf594f8284edcc7bce6e27ace97565ca19cc0f
User & Date: dgp 2011-05-02 20:17:50
Context
2011-05-02
21:12
Rewrite of parts of the switch compiler to better use the powers of TclFindElement() and do less par... check-in: 30dad692de user: dgp tags: trunk
20:17
Revised TclFindElement() interface. The final argument had been bracePtr, the address of a boolean v... check-in: 60cf594f82 user: dgp tags: trunk, potential incompatibility
20:10
Tests for expanded literals quoting detection. check-in: 52e2ce6ae1 user: dgp tags: core-8-5-branch
15:44
Replace TclCountSpaceRuns() with TclMaxListLength() which is the function we actually want. check-in: 304fa3c29b user: dgp tags: trunk
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to ChangeLog.

1
2




















3
4
5
6
7
8
9
2011-05-02  Don Porter  <[email protected]>





















	* generic/tclInt.h:	Replace TclCountSpaceRuns() with
	* generic/tclListObj.c:	TclMaxListLength() which is the function we
	* generic/tclUtil.c:	actually want.

2011-04-28  Don Porter  <[email protected]>

	* generic/tclInt.h:	New utility routines:


>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
2011-05-02  Don Porter  <[email protected]>

	* generic/tclCmdMZ.c:	Revised TclFindElement() interface.  The
	* generic/tclDictObj.c:	final argument had been bracePtr, the address
	* generic/tclListObj.c:	of a boolean var, where the caller can be told
	* generic/tclParse.c:	whether or not the parsed list element was
	* generic/tclUtil.c:	enclosed in braces.  In practice, no callers
	really care about that.  What the callers really want to know is 
	whether the list element value exists as a literal substring of the
	string being parsed, or whether a call to TclCopyAndCollpase() is
	needed to produce the list element value.  Now the final argument
	is changed to do what callers actually need.  This is a better fit
	for the calls in tclParse.c, where now a good deal of post-processing
	checking for "naked backslashes" is no longer necessary.
	***POTENTIAL INCOMPATIBILITY***
	For any callers calling in via the internal stubs table who really
	do use the final argument explicitly to check for the enclosing brace
	scenario.  Simply looking for the braces where they must be is the
	revision available to those callers, and it will backport cleanly.

	* tests/parse.test:	Tests for expanded literals quoting detection.

	* generic/tclInt.h:	Replace TclCountSpaceRuns() with
	* generic/tclListObj.c:	TclMaxListLength() which is the function we
	* generic/tclUtil.c:	actually want.

2011-04-28  Don Porter  <[email protected]>

	* generic/tclInt.h:	New utility routines:

Changes to generic/tclCmdMZ.c.

1639
1640
1641
1642
1643
1644
1645
1646
1647
1648
1649
1650
1651
1652
1653
1654
1655
1656
1657
1658
1659
1660
1661
1662
	    /*
	     * Need to figure out where the list parsing failed, which is
	     * fairly expensive. This is adapted from the core of
	     * SetListFromAny().
	     */

	    const char *elemStart, *nextElem;
	    int lenRemain, elemSize, hasBrace;
	    register const char *p;

	    string1 = TclGetStringFromObj(objPtr, &length1);
	    end = string1 + length1;
	    failat = -1;
	    for (p=string1, lenRemain=length1; lenRemain > 0;
		    p=nextElem, lenRemain=end-nextElem) {
		if (TCL_ERROR == TclFindElement(NULL, p, lenRemain,
			&elemStart, &nextElem, &elemSize, &hasBrace)) {
		    Tcl_Obj *tmpStr;

		    /*
		     * This is the simplest way of getting the number of
		     * characters parsed. Note that this is not the same as
		     * the number of bytes when parsing strings with non-ASCII
		     * characters in them.







|








|







1639
1640
1641
1642
1643
1644
1645
1646
1647
1648
1649
1650
1651
1652
1653
1654
1655
1656
1657
1658
1659
1660
1661
1662
	    /*
	     * Need to figure out where the list parsing failed, which is
	     * fairly expensive. This is adapted from the core of
	     * SetListFromAny().
	     */

	    const char *elemStart, *nextElem;
	    int lenRemain, elemSize;
	    register const char *p;

	    string1 = TclGetStringFromObj(objPtr, &length1);
	    end = string1 + length1;
	    failat = -1;
	    for (p=string1, lenRemain=length1; lenRemain > 0;
		    p=nextElem, lenRemain=end-nextElem) {
		if (TCL_ERROR == TclFindElement(NULL, p, lenRemain,
			&elemStart, &nextElem, &elemSize, NULL)) {
		    Tcl_Obj *tmpStr;

		    /*
		     * This is the simplest way of getting the number of
		     * characters parsed. Note that this is not the same as
		     * the number of bytes when parsing strings with non-ASCII
		     * characters in them.

Changes to generic/tclDictObj.c.

563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
SetDictFromAny(
    Tcl_Interp *interp,
    Tcl_Obj *objPtr)
{
    const char *string;
    char *s;
    const char *elemStart, *nextElem;
    int lenRemain, length, elemSize, hasBrace, result, isNew;
    const char *limit;	/* Points just after string's last byte. */
    register const char *p;
    register Tcl_Obj *keyPtr, *valuePtr;
    Dict *dict;
    Tcl_HashEntry *hPtr;

    /*







|







563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
SetDictFromAny(
    Tcl_Interp *interp,
    Tcl_Obj *objPtr)
{
    const char *string;
    char *s;
    const char *elemStart, *nextElem;
    int lenRemain, length, elemSize, result, isNew;
    const char *limit;	/* Points just after string's last byte. */
    register const char *p;
    register Tcl_Obj *keyPtr, *valuePtr;
    Dict *dict;
    Tcl_HashEntry *hPtr;

    /*
645
646
647
648
649
650
651


652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
     */

    dict = ckalloc(sizeof(Dict));
    InitChainTable(dict);
    for (p = string, lenRemain = length;
	    lenRemain > 0;
	    p = nextElem, lenRemain = (limit - nextElem)) {


	result = TclFindElement(interp, p, lenRemain,
		&elemStart, &nextElem, &elemSize, &hasBrace);
	if (result != TCL_OK) {
	    if (interp != NULL) {
		Tcl_SetErrorCode(interp, "TCL", "VALUE", "DICTIONARY", NULL);
	    }
	    goto errorExit;
	}
	if (elemStart >= limit) {
	    break;
	}

	/*
	 * Allocate a Tcl object for the element and initialize it from the
	 * "elemSize" bytes starting at "elemStart".
	 */

	s = ckalloc(elemSize + 1);
	if (hasBrace) {
	    memcpy(s, elemStart, (size_t) elemSize);
	    s[elemSize] = 0;
	} else {
	    elemSize = TclCopyAndCollapse(elemSize, elemStart, s);
	}

	TclNewObj(keyPtr);
	keyPtr->bytes = s;
	keyPtr->length = elemSize;

	p = nextElem;
	lenRemain = (limit - nextElem);
	if (lenRemain <= 0) {
	    goto missingKey;
	}

	result = TclFindElement(interp, p, lenRemain,
		&elemStart, &nextElem, &elemSize, &hasBrace);
	if (result != TCL_OK) {
	    if (interp != NULL) {
		Tcl_SetErrorCode(interp, "TCL", "VALUE", "DICTIONARY", NULL);
	    }
	    TclDecrRefCount(keyPtr);
	    goto errorExit;
	}
	if (elemStart >= limit) {
	    goto missingKey;
	}

	/*
	 * Allocate a Tcl object for the element and initialize it from the
	 * "elemSize" bytes starting at "elemStart".
	 */

	s = ckalloc(elemSize + 1);
	if (hasBrace) {
	    memcpy(s, elemStart, (size_t) elemSize);
	    s[elemSize] = 0;
	} else {
	    elemSize = TclCopyAndCollapse(elemSize, elemStart, s);
	}

	TclNewObj(valuePtr);







>
>

|
















|

















|

















|







645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
     */

    dict = ckalloc(sizeof(Dict));
    InitChainTable(dict);
    for (p = string, lenRemain = length;
	    lenRemain > 0;
	    p = nextElem, lenRemain = (limit - nextElem)) {
	int literal;

	result = TclFindElement(interp, p, lenRemain,
		&elemStart, &nextElem, &elemSize, &literal);
	if (result != TCL_OK) {
	    if (interp != NULL) {
		Tcl_SetErrorCode(interp, "TCL", "VALUE", "DICTIONARY", NULL);
	    }
	    goto errorExit;
	}
	if (elemStart >= limit) {
	    break;
	}

	/*
	 * Allocate a Tcl object for the element and initialize it from the
	 * "elemSize" bytes starting at "elemStart".
	 */

	s = ckalloc(elemSize + 1);
	if (literal) {
	    memcpy(s, elemStart, (size_t) elemSize);
	    s[elemSize] = 0;
	} else {
	    elemSize = TclCopyAndCollapse(elemSize, elemStart, s);
	}

	TclNewObj(keyPtr);
	keyPtr->bytes = s;
	keyPtr->length = elemSize;

	p = nextElem;
	lenRemain = (limit - nextElem);
	if (lenRemain <= 0) {
	    goto missingKey;
	}

	result = TclFindElement(interp, p, lenRemain,
		&elemStart, &nextElem, &elemSize, &literal);
	if (result != TCL_OK) {
	    if (interp != NULL) {
		Tcl_SetErrorCode(interp, "TCL", "VALUE", "DICTIONARY", NULL);
	    }
	    TclDecrRefCount(keyPtr);
	    goto errorExit;
	}
	if (elemStart >= limit) {
	    goto missingKey;
	}

	/*
	 * Allocate a Tcl object for the element and initialize it from the
	 * "elemSize" bytes starting at "elemStart".
	 */

	s = ckalloc(elemSize + 1);
	if (literal) {
	    memcpy(s, elemStart, (size_t) elemSize);
	    s[elemSize] = 0;
	} else {
	    elemSize = TclCopyAndCollapse(elemSize, elemStart, s);
	}

	TclNewObj(valuePtr);

Changes to generic/tclListObj.c.

1713
1714
1715
1716
1717
1718
1719
1720
1721
1722
1723
1724
1725
1726
1727
SetListFromAny(
    Tcl_Interp *interp,		/* Used for error reporting if not NULL. */
    Tcl_Obj *objPtr)		/* The object to convert. */
{
    const char *string;
    char *s;
    const char *elemStart, *nextElem;
    int lenRemain, length, estCount, elemSize, hasBrace, i, j, result;
    const char *limit;		/* Points just after string's last byte. */
    register const char *p;
    register Tcl_Obj **elemPtrs;
    register Tcl_Obj *elemPtr;
    List *listRepPtr;

    /*







|







1713
1714
1715
1716
1717
1718
1719
1720
1721
1722
1723
1724
1725
1726
1727
SetListFromAny(
    Tcl_Interp *interp,		/* Used for error reporting if not NULL. */
    Tcl_Obj *objPtr)		/* The object to convert. */
{
    const char *string;
    char *s;
    const char *elemStart, *nextElem;
    int lenRemain, length, estCount, elemSize, i, j, result;
    const char *limit;		/* Points just after string's last byte. */
    register const char *p;
    register Tcl_Obj **elemPtrs;
    register Tcl_Obj *elemPtr;
    List *listRepPtr;

    /*
1797
1798
1799
1800
1801
1802
1803


1804
1805
1806
1807
1808
1809
1810
1811
1812
    }
    elemPtrs = &listRepPtr->elements;

    /* Each iteration, parse and store a list element */
    for (p=string, lenRemain=length, i=0;
	    lenRemain > 0;
	    p=nextElem, lenRemain=limit-nextElem, i++) {


	result = TclFindElement(interp, p, lenRemain, &elemStart, &nextElem,
		&elemSize, &hasBrace);
	if (result != TCL_OK) {
	    for (j = 0;  j < i;  j++) {
		elemPtr = elemPtrs[j];
		Tcl_DecrRefCount(elemPtr);
	    }
	    ckfree(listRepPtr);
	    if (interp != NULL) {







>
>

|







1797
1798
1799
1800
1801
1802
1803
1804
1805
1806
1807
1808
1809
1810
1811
1812
1813
1814
    }
    elemPtrs = &listRepPtr->elements;

    /* Each iteration, parse and store a list element */
    for (p=string, lenRemain=length, i=0;
	    lenRemain > 0;
	    p=nextElem, lenRemain=limit-nextElem, i++) {
	int literal;

	result = TclFindElement(interp, p, lenRemain, &elemStart, &nextElem,
		&elemSize, &literal);
	if (result != TCL_OK) {
	    for (j = 0;  j < i;  j++) {
		elemPtr = elemPtrs[j];
		Tcl_DecrRefCount(elemPtr);
	    }
	    ckfree(listRepPtr);
	    if (interp != NULL) {
1823
1824
1825
1826
1827
1828
1829
1830
1831
1832
1833
1834
1835
1836
1837

	/*
	 * Allocate a Tcl object for the element and initialize it from the
	 * "elemSize" bytes starting at "elemStart".
	 */

	s = ckalloc(elemSize + 1);
	if (hasBrace) {
	    memcpy(s, elemStart, (size_t) elemSize);
	    s[elemSize] = 0;
	} else {
	    elemSize = TclCopyAndCollapse(elemSize, elemStart, s);
	}

	TclNewObj(elemPtr);







|







1825
1826
1827
1828
1829
1830
1831
1832
1833
1834
1835
1836
1837
1838
1839

	/*
	 * Allocate a Tcl object for the element and initialize it from the
	 * "elemSize" bytes starting at "elemStart".
	 */

	s = ckalloc(elemSize + 1);
	if (literal) {
	    memcpy(s, elemStart, (size_t) elemSize);
	    s[elemSize] = 0;
	} else {
	    elemSize = TclCopyAndCollapse(elemSize, elemStart, s);
	}

	TclNewObj(elemPtr);

Changes to generic/tclParse.c.

429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
		if (tokenPtr[i].type != TCL_TOKEN_TEXT) {
		    isLiteral = 0;
		    break;
		}
	    }

	    if (isLiteral) {
		int elemCount = 0, code = TCL_OK, nakedbs = 0;
		const char *nextElem, *listEnd, *elemStart;

		/*
		 * The word to be expanded is a literal, so determine the
		 * boundaries of the literal string to be treated as a list
		 * and expanded. That literal string starts at
		 * tokenPtr[1].start, and includes all bytes up to, but not







|







429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
		if (tokenPtr[i].type != TCL_TOKEN_TEXT) {
		    isLiteral = 0;
		    break;
		}
	    }

	    if (isLiteral) {
		int elemCount = 0, code = TCL_OK, literal = 1;
		const char *nextElem, *listEnd, *elemStart;

		/*
		 * The word to be expanded is a literal, so determine the
		 * boundaries of the literal string to be treated as a list
		 * and expanded. That literal string starts at
		 * tokenPtr[1].start, and includes all bytes up to, but not
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483

484
485
486
487
488
489
490
491
492
493

		/*
		 * Step through the literal string, parsing and counting list
		 * elements.
		 */

		while (nextElem < listEnd) {
		    int size, brace;

		    code = TclFindElement(NULL, nextElem, listEnd - nextElem,
			    &elemStart, &nextElem, &size, &brace);
		    if (code != TCL_OK) {
			break;
		    }
		    if (!brace) {
			const char *s;

			for(s=elemStart;size>0;s++,size--) {
			    if ((*s)=='\\') {
				nakedbs = 1;
				break;
			    }
			}
		    }
		    if (elemStart < listEnd) {
			elemCount++;
		    }
		}

		if ((code != TCL_OK) || nakedbs) {
		    /*
		     * Some list element could not be parsed, or contained
		     * naked backslashes. This means the literal string was

		     * not in fact a valid nor canonical list. Defer the
		     * handling of this to compile/eval time, where code is
		     * already in place to report the "attempt to expand a
		     * non-list" error or expand lists that require
		     * substitution.
		     */

		    tokenPtr->type = TCL_TOKEN_EXPAND_WORD;
		} else if (elemCount == 0) {
		    /*







|


|
|

<
<
<
<
<
<
<
<
<
<






|

|
|
>
|
|
|







451
452
453
454
455
456
457
458
459
460
461
462
463










464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484

		/*
		 * Step through the literal string, parsing and counting list
		 * elements.
		 */

		while (nextElem < listEnd) {
		    int size;

		    code = TclFindElement(NULL, nextElem, listEnd - nextElem,
			    &elemStart, &nextElem, &size, &literal);
		    if ((code != TCL_OK) || !literal) {
			break;










		    }
		    if (elemStart < listEnd) {
			elemCount++;
		    }
		}

		if ((code != TCL_OK) || !literal) {
		    /*
		     * Some list element could not be parsed, or is not
		     * present as a literal substring of the script.  The
		     * compiler cannot handle list elements that get generated
		     * by a call to TclCopyAndCollapse(). Defer  the
		     * handling of  this to  compile/eval time, where  code is
		     * already  in place to  report the  "attempt to  expand a
		     * non-list" error or expand lists that require
		     * substitution.
		     */

		    tokenPtr->type = TCL_TOKEN_EXPAND_WORD;
		} else if (elemCount == 0) {
		    /*
501
502
503
504
505
506
507

508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540

541

542
543
544
545
546
547
548
		    parsePtr->numTokens = wordIndex;
		} else {
		    /*
		     * Recalculate the number of Tcl_Tokens needed to store
		     * tokens representing the expanded list.
		     */


		    int growthNeeded = wordIndex + 2*elemCount
			    - parsePtr->numTokens;

		    parsePtr->numWords += elemCount - 1;
		    if (growthNeeded > 0) {
			TclGrowParseTokenArray(parsePtr, growthNeeded);
			tokenPtr = &parsePtr->tokenPtr[wordIndex];
		    }
		    parsePtr->numTokens = wordIndex + 2*elemCount;

		    /*
		     * Generate a TCL_TOKEN_SIMPLE_WORD token sequence for
		     * each element of the literal list we are expanding in
		     * place. Take care with the start and size fields of each
		     * token so they point to the right literal characters in
		     * the original script to represent the right expanded
		     * word value.
		     */

		    nextElem = tokenPtr[1].start;
		    while (nextElem < listEnd) {
			int quoted, brace;
	
			tokenPtr->type = TCL_TOKEN_SIMPLE_WORD;
			tokenPtr->numComponents = 1;

			tokenPtr++;
			tokenPtr->type = TCL_TOKEN_TEXT;
			tokenPtr->numComponents = 0;
			TclFindElement(NULL, nextElem, listEnd - nextElem,
				&(tokenPtr->start), &nextElem,
				&(tokenPtr->size), &brace);


			quoted = brace || tokenPtr->start[-1] == '"';

			tokenPtr[-1].start = tokenPtr->start - quoted;
			tokenPtr[-1].size = tokenPtr->start + tokenPtr->size
				- tokenPtr[-1].start + quoted;

			tokenPtr++;
		    }
		}







>



















|

|









|

>
|
>







492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
		    parsePtr->numTokens = wordIndex;
		} else {
		    /*
		     * Recalculate the number of Tcl_Tokens needed to store
		     * tokens representing the expanded list.
		     */

		    CONST char *listStart;
		    int growthNeeded = wordIndex + 2*elemCount
			    - parsePtr->numTokens;

		    parsePtr->numWords += elemCount - 1;
		    if (growthNeeded > 0) {
			TclGrowParseTokenArray(parsePtr, growthNeeded);
			tokenPtr = &parsePtr->tokenPtr[wordIndex];
		    }
		    parsePtr->numTokens = wordIndex + 2*elemCount;

		    /*
		     * Generate a TCL_TOKEN_SIMPLE_WORD token sequence for
		     * each element of the literal list we are expanding in
		     * place. Take care with the start and size fields of each
		     * token so they point to the right literal characters in
		     * the original script to represent the right expanded
		     * word value.
		     */

		    listStart = nextElem = tokenPtr[1].start;
		    while (nextElem < listEnd) {
			int quoted;
	
			tokenPtr->type = TCL_TOKEN_SIMPLE_WORD;
			tokenPtr->numComponents = 1;

			tokenPtr++;
			tokenPtr->type = TCL_TOKEN_TEXT;
			tokenPtr->numComponents = 0;
			TclFindElement(NULL, nextElem, listEnd - nextElem,
				&(tokenPtr->start), &nextElem,
				&(tokenPtr->size), NULL);

			quoted = (tokenPtr->start[-1] == '{'
				|| tokenPtr->start[-1] == '"')
				&& tokenPtr->start > listStart;
			tokenPtr[-1].start = tokenPtr->start - quoted;
			tokenPtr[-1].size = tokenPtr->start + tokenPtr->size
				- tokenPtr[-1].start + quoted;

			tokenPtr++;
		    }
		}

Changes to generic/tclUtil.c.

176
177
178
179
180
181
182
183




184

185
186
187
188
189
190
191
 *	then *nextPtr will point just after the last character in the list
 *	(i.e., at the character at list+listLength). If sizePtr is non-NULL,
 *	*sizePtr is filled in with the number of characters in the element. If
 *	the element is in braces, then *elementPtr will point to the character
 *	after the opening brace and *sizePtr will not include either of the
 *	braces. If there isn't an element in the list, *sizePtr will be zero,
 *	and both *elementPtr and *termPtr will point just after the last
 *	character in the list. Note: this function does NOT collapse backslash




 *	sequences.

 *
 * Side effects:
 *	None.
 *
 *----------------------------------------------------------------------
 */








|
>
>
>
>
|
>







176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
 *	then *nextPtr will point just after the last character in the list
 *	(i.e., at the character at list+listLength). If sizePtr is non-NULL,
 *	*sizePtr is filled in with the number of characters in the element. If
 *	the element is in braces, then *elementPtr will point to the character
 *	after the opening brace and *sizePtr will not include either of the
 *	braces. If there isn't an element in the list, *sizePtr will be zero,
 *	and both *elementPtr and *termPtr will point just after the last
 *	character in the list. If literalPtr is non-NULL, *literalPtr is set
 *	to a boolean value indicating whether the substring returned as
 *	the values of **elementPtr and *sizePtr is the literal value of
 *	a list element.  If not, a call to TclCopyAndCollapse() is needed
 *	to produce the actual value of the list element.  Note: this function
 *	does NOT collapse backslash sequences, but uses *literalPtr to tell
 * 	callers when it is required for them to do so.
 *
 * Side effects:
 *	None.
 *
 *----------------------------------------------------------------------
 */

201
202
203
204
205
206
207
208
209




210
211
212
213
214
215
216
217

218
219
220
221
222
223
224
    const char **elementPtr,	/* Where to put address of first significant
				 * character in first element of list. */
    const char **nextPtr,	/* Fill in with location of character just
				 * after all white space following end of
				 * argument (next arg or end of list). */
    int *sizePtr,		/* If non-zero, fill in with size of
				 * element. */
    int *bracePtr)		/* If non-zero, fill in with non-zero/zero to
				 * indicate that arg was/wasn't in braces. */




{
    const char *p = list;
    const char *elemStart;	/* Points to first byte of first element. */
    const char *limit;		/* Points just after list's last byte. */
    int openBraces = 0;		/* Brace nesting level during parse. */
    int inQuotes = 0;
    int size = 0;		/* lint. */
    int numChars;

    const char *p2;

    /*
     * Skim off leading white space and check for an opening brace or quote.
     * We treat embedded NULLs in the list as bytes belonging to a list
     * element.
     */







|
|
>
>
>
>








>







206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
    const char **elementPtr,	/* Where to put address of first significant
				 * character in first element of list. */
    const char **nextPtr,	/* Fill in with location of character just
				 * after all white space following end of
				 * argument (next arg or end of list). */
    int *sizePtr,		/* If non-zero, fill in with size of
				 * element. */
    int *literalPtr)		/* If non-zero, fill in with non-zero/zero to
				 * indicate that the substring of *sizePtr
				 * bytes starting at **elementPtr is/is not
				 * the literal list element and therefore
				 * does not/does require a call to 
				 * TclCopyAndCollapse() by the caller. */
{
    const char *p = list;
    const char *elemStart;	/* Points to first byte of first element. */
    const char *limit;		/* Points just after list's last byte. */
    int openBraces = 0;		/* Brace nesting level during parse. */
    int inQuotes = 0;
    int size = 0;		/* lint. */
    int numChars;
    int literal = 1;
    const char *p2;

    /*
     * Skim off leading white space and check for an opening brace or quote.
     * We treat embedded NULLs in the list as bytes belonging to a list
     * element.
     */
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
	openBraces = 1;
	p++;
    } else if (*p == '"') {
	inQuotes = 1;
	p++;
    }
    elemStart = p;
    if (bracePtr != 0) {
	*bracePtr = openBraces;
    }

    /*
     * Find element's end (a space, close brace, or the end of the string).
     */

    while (p < limit) {
	switch (*p) {







<
<
<







246
247
248
249
250
251
252



253
254
255
256
257
258
259
	openBraces = 1;
	p++;
    } else if (*p == '"') {
	inQuotes = 1;
	p++;
    }
    elemStart = p;




    /*
     * Find element's end (a space, close brace, or the end of the string).
     */

    while (p < limit) {
	switch (*p) {
298
299
300
301
302
303
304









305
306
307
308
309
310
311

	    /*
	     * Backslash: skip over everything up to the end of the backslash
	     * sequence.
	     */

	case '\\':









	    TclParseBackslash(p, limit - p, &numChars, NULL);
	    p += (numChars - 1);
	    break;

	    /*
	     * Space: ignore if element is in braces or quotes; otherwise
	     * terminate element.







>
>
>
>
>
>
>
>
>







305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327

	    /*
	     * Backslash: skip over everything up to the end of the backslash
	     * sequence.
	     */

	case '\\':
	    if (openBraces == 0) {
		/*
		 * A backslash sequence not within a brace quoted element
		 * means the value of the element is different from the
		 * substring we are parsing.  A call to TclCopyAndCollapse()
		 * is needed to produce the element value.  Inform the caller.
		 */
		literal = 0;
	    }
	    TclParseBackslash(p, limit - p, &numChars, NULL);
	    p += (numChars - 1);
	    break;

	    /*
	     * Space: ignore if element is in braces or quotes; otherwise
	     * terminate element.
387
388
389
390
391
392
393



394
395
396
397
398
399
400
    while ((p < limit) && (TclIsSpaceProc(*p))) {
	p++;
    }
    *elementPtr = elemStart;
    *nextPtr = p;
    if (sizePtr != 0) {
	*sizePtr = size;



    }
    return TCL_OK;
}

/*
 *----------------------------------------------------------------------
 *







>
>
>







403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
    while ((p < limit) && (TclIsSpaceProc(*p))) {
	p++;
    }
    *elementPtr = elemStart;
    *nextPtr = p;
    if (sizePtr != 0) {
	*sizePtr = size;
    }
    if (literalPtr != 0) {
	*literalPtr = literal;
    }
    return TCL_OK;
}

/*
 *----------------------------------------------------------------------
 *
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504

505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
    int *argcPtr,		/* Pointer to location to fill in with the
				 * number of elements in the list. */
    const char ***argvPtr)	/* Pointer to place to store pointer to array
				 * of pointers to list elements. */
{
    const char **argv, *end, *element;
    char *p;
    int length, size, i, result, elSize, brace;

    /*
     * Allocate enough space to work in. A (CONST char *) for each
     * (possible) list element plus one more for terminating NULL,
     * plus as many bytes as in the original string value, plus one
     * more for a terminating '\0'.  Space used to hold element separating
     * white space in the original string gets re-purposed to hold '\0'
     * characters in the argv array.
     */

    size = TclMaxListLength(list, -1, &end) + 1;
    length = end - list;
    argv = ckalloc((size * sizeof(char *)) + length + 1);

    for (i = 0, p = ((char *) argv) + size*sizeof(char *);
	    *list != 0;  i++) {
	const char *prevList = list;


	result = TclFindElement(interp, list, length, &element, &list,
		&elSize, &brace);
	length -= (list - prevList);
	if (result != TCL_OK) {
	    ckfree(argv);
	    return result;
	}
	if (*element == 0) {
	    break;
	}
	if (i >= size) {
	    ckfree(argv);
	    if (interp != NULL) {
		Tcl_SetResult(interp, "internal error in Tcl_SplitList",
			TCL_STATIC);
		Tcl_SetErrorCode(interp, "TCL", "INTERNAL", "Tcl_SplitList",
			NULL);
	    }
	    return TCL_ERROR;
	}
	argv[i] = p;
	if (brace) {
	    memcpy(p, element, (size_t) elSize);
	    p += elSize;
	    *p = 0;
	    p++;
	} else {
	    TclCopyAndCollapse(elSize, element, p);
	    p += elSize+1;







|

















>


|



















|







499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
    int *argcPtr,		/* Pointer to location to fill in with the
				 * number of elements in the list. */
    const char ***argvPtr)	/* Pointer to place to store pointer to array
				 * of pointers to list elements. */
{
    const char **argv, *end, *element;
    char *p;
    int length, size, i, result, elSize;

    /*
     * Allocate enough space to work in. A (CONST char *) for each
     * (possible) list element plus one more for terminating NULL,
     * plus as many bytes as in the original string value, plus one
     * more for a terminating '\0'.  Space used to hold element separating
     * white space in the original string gets re-purposed to hold '\0'
     * characters in the argv array.
     */

    size = TclMaxListLength(list, -1, &end) + 1;
    length = end - list;
    argv = ckalloc((size * sizeof(char *)) + length + 1);

    for (i = 0, p = ((char *) argv) + size*sizeof(char *);
	    *list != 0;  i++) {
	const char *prevList = list;
	int literal;

	result = TclFindElement(interp, list, length, &element, &list,
		&elSize, &literal);
	length -= (list - prevList);
	if (result != TCL_OK) {
	    ckfree(argv);
	    return result;
	}
	if (*element == 0) {
	    break;
	}
	if (i >= size) {
	    ckfree(argv);
	    if (interp != NULL) {
		Tcl_SetResult(interp, "internal error in Tcl_SplitList",
			TCL_STATIC);
		Tcl_SetErrorCode(interp, "TCL", "INTERNAL", "Tcl_SplitList",
			NULL);
	    }
	    return TCL_ERROR;
	}
	argv[i] = p;
	if (literal) {
	    memcpy(p, element, (size_t) elSize);
	    p += elSize;
	    *p = 0;
	    p++;
	} else {
	    TclCopyAndCollapse(elSize, element, p);
	    p += elSize+1;

Changes to tests/parse.test.

229
230
231
232
233
234
235






236
237
238
239
240
241
242
} {- \{*\}\\\n\ foo\ bar 3 simple {{*}} 1 text * 0 simple foo 1 text foo 0 simple bar 1 text bar 0 {}}
test parse-5.28 {Tcl_ParseCommand: {*} parsing, expanded literals} testparser {
    testparser {{*}{a b}} 0
} {- {{*}{a b}} 2 simple a 1 text a 0 simple b 1 text b 0 {}}
test parse-5.29 {Tcl_ParseCommand: {*} parsing, expanded literals, naked backslashes} testparser {
    testparser {{*}{a \n b}} 0
} {- {{*}{a \n b}} 1 expand {{*}{a \n b}} 1 text {a \n b} 0 {}}







test parse-6.1 {ParseTokens procedure, empty word} testparser {
    testparser {""} 0
} {- {""} 1 simple {""} 1 text {} 0 {}}
test parse-6.2 {ParseTokens procedure, simple range} testparser {
    testparser {"abc$x.e"} 0
} {- {"abc$x.e"} 1 word {"abc$x.e"} 4 text abc 0 variable {$x} 1 text x 0 text .e 0 {}}







>
>
>
>
>
>







229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
} {- \{*\}\\\n\ foo\ bar 3 simple {{*}} 1 text * 0 simple foo 1 text foo 0 simple bar 1 text bar 0 {}}
test parse-5.28 {Tcl_ParseCommand: {*} parsing, expanded literals} testparser {
    testparser {{*}{a b}} 0
} {- {{*}{a b}} 2 simple a 1 text a 0 simple b 1 text b 0 {}}
test parse-5.29 {Tcl_ParseCommand: {*} parsing, expanded literals, naked backslashes} testparser {
    testparser {{*}{a \n b}} 0
} {- {{*}{a \n b}} 1 expand {{*}{a \n b}} 1 text {a \n b} 0 {}}
test parse-5.30 {Tcl_ParseCommand: {*} parsing, expanded literals} testparser {
    testparser {{*}"a b"} 0
} {- {{*}"a b"} 2 simple a 1 text a 0 simple b 1 text b 0 {}}
test parse-5.31 {Tcl_ParseCommand: {*} parsing, expanded literals, naked backslashes} testparser {
    testparser {{*}"a \n b"} 0
} {- {{*}"a \n b"} 1 expand {{*}"a \n b"} 3 text {a } 0 backslash {\n} 0 text { b} 0 {}}

test parse-6.1 {ParseTokens procedure, empty word} testparser {
    testparser {""} 0
} {- {""} 1 simple {""} 1 text {} 0 {}}
test parse-6.2 {ParseTokens procedure, simple range} testparser {
    testparser {"abc$x.e"} 0
} {- {"abc$x.e"} 1 word {"abc$x.e"} 4 text abc 0 variable {$x} 1 text x 0 text .e 0 {}}