Consider the following ObjC++ snippet:

--
@protocol PA;
@protocol PB;

@class NSArray<ObjectType>;
typedef int some_t;

id<PA> FA(NSArray<id<PB>> *h, some_t group);
--

This would hit an assertion in the parser after generating an annotation token
while trying to update the token cache:

Assertion failed: (CachedTokens[CachedLexPos-1].getLastLoc() == Tok.getAnnotationEndLoc() && "The annotation should be until the most recent cached token")
...
7 clang::Preprocessor::AnnotatePreviousCachedTokens(clang::Token const&) + 494
8 clang::Parser::TryAnnotateTypeOrScopeTokenAfterScopeSpec(bool, bool, clang::CXXScopeSpec&, bool) + 1163
9 clang::Parser::TryAnnotateTypeOrScopeToken(bool, bool) + 361
10 clang::Parser::isCXXDeclarationSpecifier(clang::Parser::TPResult, bool*) + 598
...

The cached preprocessor token in this case is:

greatergreater '>>' Loc=<testcase.mm:7:24>

while the annotation ("NSArray<id<PB>>") ends at "testcase.mm:7:25", hence the
assertion.

Properly update the CachedTokens during template parsing to contain
two greater tokens instead of a greatergreater.

Differential Revision: http://reviews.llvm.org/D15173

rdar://problem/23494277

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@259311 91177308-0d34-0410-b5e6-96231b3b80d8

curlies.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@195802 91177308-0d34-0410-b5e6-96231b3b80d8

the duplicated documentation from .cpp files so that it's not processed by
Doxygen and hence doesn't generate duplicate output.


git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@195799 91177308-0d34-0410-b5e6-96231b3b80d8

cached during the non-cached lex, otherwise we are going to drop them.

Fixes a bogus "_Pragma takes a parenthesized string literal" error when
expanding consecutive _Pragmas in a macro argument.

Part of rdar://11168596

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@153994 91177308-0d34-0410-b5e6-96231b3b80d8

Don't treat 'import' as a contextual keyword when we're in a caching lexer, or when modules are disabled.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@147524 91177308-0d34-0410-b5e6-96231b3b80d8

keyword. We now handle this keyword in HandleIdentifier, making a note
for ourselves when we've seen the __import_module__ keyword so that
the next lexed token can trigger a module import (if needed). This
greatly simplifies Preprocessor::Lex(), and completely erases the 5.5%
-Eonly slowdown Argiris noted when I originally implemented
__import_module__. Big thanks to Argiris for noting that horrible
regression!



git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@139265 91177308-0d34-0410-b5e6-96231b3b80d8

totally unnecessary.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@108199 91177308-0d34-0410-b5e6-96231b3b80d8

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@108175 91177308-0d34-0410-b5e6-96231b3b80d8

In the case of backtracking, the cached token lexer will be the only 
lexer on the stack, without this the token stack will be empty and EOF
won't be returned.

This fixes PR7072.



git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@108124 91177308-0d34-0410-b5e6-96231b3b80d8

When placing an annotation token over an existing annotation token, make sure that the new token's range extends to the end of the old token. Assert that in AnnotateCachedTokens. Fixes PR6248.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@95555 91177308-0d34-0410-b5e6-96231b3b80d8

annotation token, because some of the tokens we're annotating might
not be in the set of cached tokens (we could have consumed them
unconditionally).

Also, move the tentative parsing from ParseTemplateTemplateArgument
into the one caller that needs it, improving recovery.


git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@86904 91177308-0d34-0410-b5e6-96231b3b80d8

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@81346 91177308-0d34-0410-b5e6-96231b3b80d8

fix segfault (because of erasing after the vector boundaries) when the cached token position is at the end

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@77159 91177308-0d34-0410-b5e6-96231b3b80d8

Token now has a class of kinds for "literals", which include 
numeric constants, strings, etc.  These tokens can optionally have
a pointer to the start of the token in the lexer buffer.  This 
makes it faster to get spelling and do other gymnastics, because we
don't have to go through source locations.

This change is performance neutral, but will make other changes
more feasible down the road.



git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@63028 91177308-0d34-0410-b5e6-96231b3b80d8

Remove Preprocessor::CacheTokens boolean data member. The same functionality can be provided by using Preprocessor::isBacktrackEnabled().

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59631 91177308-0d34-0410-b5e6-96231b3b80d8

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59181 91177308-0d34-0410-b5e6-96231b3b80d8

Introduce annotation tokens, a special kind of token, created and used only by the parser to replace a group of tokens with a single token encoding semantic information.
Will be fully utilized later for C++ nested-name-specifiers.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@58911 91177308-0d34-0410-b5e6-96231b3b80d8

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@55829 91177308-0d34-0410-b5e6-96231b3b80d8

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@55281 91177308-0d34-0410-b5e6-96231b3b80d8

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@55235 91177308-0d34-0410-b5e6-96231b3b80d8

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@55204 91177308-0d34-0410-b5e6-96231b3b80d8

Allow the preprocessor to cache the lexed tokens, so that we can do efficient lookahead and backtracking.

1) New public methods added:
  -EnableBacktrackAtThisPos
  -DisableBacktrack
  -Backtrack
  -isBacktrackEnabled

2) LookAhead() implementation is replaced with a more efficient one.
3) LookNext() is removed.


git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@54611 91177308-0d34-0410-b5e6-96231b3b80d8