The main motivation behind this is to cleanup the WhitespaceManager and
make it more extensible for future alignment etc. features.
Specifically, WhitespaceManager has started to copy more and more code
that is already present in FormatToken. Instead, I think it makes more
sense to actually store a reference to each FormatToken for each change.

This has as a consequence led to a change in the calculation of indent
levels. Now, we actually compute them for each Token ahead of time,
which should be more efficient as it removes an unsigned value for the
ParenState, which is used during the combinatorial exploration of the
solution space.

No functional changes intended.

Review: https://reviews.llvm.org/D29300

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@293616 91177308-0d34-0410-b5e6-96231b3b80d8

Summary:
Consider formatting the following code fragment with column limit 20:
```
{
  // line 1
  // line 2\
  // long long long line
}
```
Before this fix the output is:
```
{
  // line 1
  // line 2\
  // long long
  long line
}
```
This patch fixes a regression that breaks the last comment line without
adding the '//' prefix.

Reviewers: djasper

Reviewed By: djasper

Subscribers: cfe-commits, klimek

Differential Revision: https://reviews.llvm.org/D29298

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@293548 91177308-0d34-0410-b5e6-96231b3b80d8

Summary:
This presents a version of the comment reflowing with less mutable state inside
the comment breakable token subclasses. The state has been pushed into the
driving breakProtrudingToken method. For this, the API of BreakableToken is enriched
by the methods getSplitBefore and getLineLengthAfterSplitBefore.

Reviewers: klimek

Reviewed By: klimek

Subscribers: djasper, klimek, mgorny, cfe-commits, ioeric

Differential Revision: https://reviews.llvm.org/D28764

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@293055 91177308-0d34-0410-b5e6-96231b3b80d8

Summary:
Introduces a separate target for comment manipulation.
Currently, comment manipulation is in BreakableComment.cpp.
Towards implementing comment reflowing, we want to factor out the
comment-related functionality, so it can be reused.
Start simple by just moving out getLineCommentIndentPrefix.

Patch by Krasimir Georgiev!

Reviewers: djasper

Subscribers: klimek, beanz, mgorny, modocache

Differential Revision: https://reviews.llvm.org/D25725

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@284573 91177308-0d34-0410-b5e6-96231b3b80d8

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@239903 91177308-0d34-0410-b5e6-96231b3b80d8

Patch by Mathieu Champlon. Thank you.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@239900 91177308-0d34-0410-b5e6-96231b3b80d8

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@239404 91177308-0d34-0410-b5e6-96231b3b80d8

Splitting:
  /**
   * multiline block comment
   *
   */

Before:
  /**
   * multiline block
   *comment
   *
   */

After:
  /**
   * multiline block
   * comment
   *
   */

The reason was that the empty line inside the comment (with just the "*") was
confusing the comment breaking logic.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@236573 91177308-0d34-0410-b5e6-96231b3b80d8

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@226698 91177308-0d34-0410-b5e6-96231b3b80d8

Reviewers: djasper

Reviewed By: djasper

Subscribers: klimek, cfe-commits

Differential Revision: http://reviews.llvm.org/D6894

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@225628 91177308-0d34-0410-b5e6-96231b3b80d8

Also run clang-format over clang-format's files.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@208409 91177308-0d34-0410-b5e6-96231b3b80d8

definition below all of the header #include lines, clang edition.

If you want more details about this, you can see some of the commits to
Debug.h in LLVM recently. This is just the clang section of a cleanup
I've done for all uses of DEBUG_TYPE in LLVM.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@206849 91177308-0d34-0410-b5e6-96231b3b80d8

Summary:
This patch ensures that the lines of the block comments retain relative
column offsets. In order to do this WhitespaceManager::Changes representing
continuation of block comments keep a pointer on the change representing the
whitespace change before the block comment, and a relative column offset to this
change, so that the correct column can be reconstructed at the end of alignment
process.

Fixes http://llvm.org/PR19325

Reviewers: djasper

Reviewed By: djasper

CC: cfe-commits, klimek

Differential Revision: http://reviews.llvm.org/D3408

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@206472 91177308-0d34-0410-b5e6-96231b3b80d8

Summary:
If we need to break the second line here:
// something: aaaaa aaaaa aaaaaa aaaaa aaaaa
//            aaaaa aaaaa aaaaaa aaaaa aaaaa aaaaa

with the patch it will be turned to

// something: aaaaa aaaaa aaaaaa aaaaa aaaaa
//            aaaaa aaaaa aaaaaa aaaaa aaaaa
//            aaaaa

instead of

// something: aaaaa aaaaa aaaaaa aaaaa aaaaa
//            aaaaa aaaaa aaaaaa aaaaa aaaaa
// aaaaa

Reviewers: djasper, klimek

Reviewed By: klimek

CC: cfe-commits, klimek

Differential Revision: http://llvm-reviews.chandlerc.com/D2988

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@203458 91177308-0d34-0410-b5e6-96231b3b80d8

We cannot simply change the start column to accomodate for the @ in an
ObjC string literal as that will make clang-format happily violate the
column limit.

Use a different workaround instead. However, a better long-term
solution might be to join the @ and the rest of the literal into a
single token.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@199198 91177308-0d34-0410-b5e6-96231b3b80d8

While it is allowed to not have an @ on subsequent lines, it seems
general practice to add them. If undesired, the code author can easily
remove them again and clang-format won't re-add them.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@198871 91177308-0d34-0410-b5e6-96231b3b80d8

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@196510 91177308-0d34-0410-b5e6-96231b3b80d8

Summary:
getStringSplit used to crash, when trying to split a long string
literal containing both printable and unprintable multi-byte UTF-8 characters.

Reviewers: djasper, klimek

Reviewed By: djasper

CC: cfe-commits, klimek

Differential Revision: http://llvm-reviews.chandlerc.com/D2268

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@195728 91177308-0d34-0410-b5e6-96231b3b80d8

Summary: Solves the problem described in http://llvm.org/PR17756

Reviewers: klimek

Reviewed By: klimek

CC: cfe-commits, klimek

Differential Revision: http://llvm-reviews.chandlerc.com/D2131

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@194493 91177308-0d34-0410-b5e6-96231b3b80d8

Patch by Christopher Olsen. Thank you!

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@193678 91177308-0d34-0410-b5e6-96231b3b80d8

Summary:
Changed UseTab to be a enum with three options: Never, Always,
ForIndentation (true/false are still supported when reading .clang-format).
IndentLevel should currently be propagated correctly for all tokens, except for
block comments. Please take a look at the general idea before I start dealing
with block comments.

Reviewers: klimek, djasper

Reviewed By: klimek

CC: cfe-commits

Differential Revision: http://llvm-reviews.chandlerc.com/D1770

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@191527 91177308-0d34-0410-b5e6-96231b3b80d8

When in pre-c++11 mode, treat _T("xxx") as a single string literal, repeat the _T() part around each fragment. This addresses http://llvm.org/PR17122

Reviewers: djasper

Reviewed By: djasper

CC: cfe-commits, klimek, rsmith

Differential Revision: http://llvm-reviews.chandlerc.com/D1640

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@190804 91177308-0d34-0410-b5e6-96231b3b80d8

Summary:
This fixes various issues with mixed tabs and spaces handling, e.g.
when realigning block comments.

Reviewers: klimek, djasper

Reviewed By: djasper

CC: cfe-commits

Differential Revision: http://llvm-reviews.chandlerc.com/D1608

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@190395 91177308-0d34-0410-b5e6-96231b3b80d8

Summary:
Count column width instead of the number of code points. This also
includes correct handling of tabs inside string literals and comments (with an
exception of multiline string literals/comments, where tabs are present before
the first escaped newline).

Reviewers: djasper, klimek

Reviewed By: klimek

CC: cfe-commits, klimek

Differential Revision: http://llvm-reviews.chandlerc.com/D1601

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@190052 91177308-0d34-0410-b5e6-96231b3b80d8

It broke a public build bot.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@187957 91177308-0d34-0410-b5e6-96231b3b80d8

Summary: Only works for UTF-8-encoded files.

Reviewers: djasper

Reviewed By: djasper

CC: cfe-commits, klimek

Differential Revision: http://llvm-reviews.chandlerc.com/D1311

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@187935 91177308-0d34-0410-b5e6-96231b3b80d8

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@186314 91177308-0d34-0410-b5e6-96231b3b80d8

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@185823 91177308-0d34-0410-b5e6-96231b3b80d8

Summary:
Fixes problems that lead to incorrect formatting of these and similar snippets:
/*
 **
 */

/*
 **/

/*
 * */

/*
 *test
 */

Clang-format used to think that all the cases above use "* " decoration, and
failed to calculate insertion position properly. It also used to remove leading 
"* " in the last line.

Reviewers: klimek

Reviewed By: klimek

CC: cfe-commits

Differential Revision: http://llvm-reviews.chandlerc.com/D1113

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@185818 91177308-0d34-0410-b5e6-96231b3b80d8

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@185300 91177308-0d34-0410-b5e6-96231b3b80d8

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@185296 91177308-0d34-0410-b5e6-96231b3b80d8

Summary:
Fixes a problem where \t,\v or \f could lead to a crash when placed as
a first character in a line comment. The cause is that rtrim and ltrim handle
these characters, but our code didn't, so some invariants could be broken.

Reviewers: klimek

Reviewed By: klimek

CC: cfe-commits

Differential Revision: http://llvm-reviews.chandlerc.com/D1013

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@184425 91177308-0d34-0410-b5e6-96231b3b80d8

Summary: Split strings at word boundaries, when there are no spaces and slashes.

Reviewers: klimek

CC: cfe-commits

Differential Revision: http://llvm-reviews.chandlerc.com/D1003

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@184304 91177308-0d34-0410-b5e6-96231b3b80d8

Summary:
Don't remove backslashes from block comments. Previously this
/* \    \ \ \ \ \
*/
would be turned to this:
/*
*/
which spoils some kinds of ASCII-art, people use in their comments. The behavior
was related to handling escaped newlines in block comments inside preprocessor
directives. This patch makes handling it in a more civilized way.

Reviewers: klimek

Reviewed By: klimek

CC: cfe-commits

Differential Revision: http://llvm-reviews.chandlerc.com/D979

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@183978 91177308-0d34-0410-b5e6-96231b3b80d8

Summary:
"//Test" becomes "// Test". This change is aimed to improve code
readability and conformance to certain coding styles. If a comment starts with a
non-alphanumeric character, the space isn't added, e.g. "//-*-c++-*-" stays
unchanged.

Reviewers: klimek

Reviewed By: klimek

CC: cfe-commits

Differential Revision: http://llvm-reviews.chandlerc.com/D949

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@183750 91177308-0d34-0410-b5e6-96231b3b80d8

Summary:
Introduced two new style parameters: PenaltyBreakComment and
PenaltyBreakString. Add penalty for each character of a breakable token beyond
the column limit (this relates mainly to comments, as they are broken only on
whitespace). Tuned PenaltyBreakComment to prefer comment breaking over breaking
inside most binary expressions.
Fixed a bug that prevented *, & and && from being considered TT_BinaryOperator
in the presense of adjacent comments.

Reviewers: klimek, djasper

Reviewed By: klimek

CC: cfe-commits

Differential Revision: http://llvm-reviews.chandlerc.com/D933

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@183530 91177308-0d34-0410-b5e6-96231b3b80d8

Summary:
Detect if the file is valid UTF-8, and if this is the case, count code
points instead of just using number of bytes in all (hopefully) places, where
number of columns is needed. In particular, use the new
FormatToken.CodePointCount instead of TokenLength where appropriate.
Changed BreakableToken implementations to respect utf-8 character boundaries
when in utf-8 mode.

Reviewers: klimek, djasper

Reviewed By: djasper

CC: cfe-commits, rsmith, gribozavr

Differential Revision: http://llvm-reviews.chandlerc.com/D918

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@183312 91177308-0d34-0410-b5e6-96231b3b80d8

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@182940 91177308-0d34-0410-b5e6-96231b3b80d8

This fixes:
/*
*
* something long going over the column limit.
*/

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@182932 91177308-0d34-0410-b5e6-96231b3b80d8

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@182916 91177308-0d34-0410-b5e6-96231b3b80d8