Parse drops a tag with no error when its JSON reply line has 4,096 bytes or more #27

Problem

New reads the replies of Universal Ctags through
bufio.NewReaderSize(out, 4096)
(ctags.go line 88).
The
scanner type
skips each line that does not fit in this buffer. Its comment says that it
"skips long lines instead of returning bufio.ErrTooLong".

So Parse loses a tag when its reply line has 4,096 bytes or more, not
counting the newline. Parse returns the other tags and no error. It does not
log the skip through Options.Info.

go-ctags passes --pattern-length-limit=255 by default, so the pattern
field stays short. Universal Ctags does not cut the name field, and the C
parser does not cut the signature field. For example, a C function with 500
int parameters has a signature of 4,391 bytes, and Parse drops its tag.

The skip comes from Zoekt commit
5851b63
(2019). Before it, one long tag failed the index of a full repository. That
commit names repositories with such tags, for example corretto/corretto-8.

Zoekt pins go-ctags 349a251
(go.mod),
and its ctags.go is the same as at head. I indexed the file of the
reproduction below with zoekt-index -require_ctags at Zoekt 572f5e4. The
build returned no error. A sym: query for the long name found nothing, but a
content query found line 5.

Reproduction

go-ctags 5ec1554 (the head of main), Universal Ctags 6.2.1, Go 1.27.3,
macOS arm64:

package main

import (
	"fmt"
	"strings"

	ctags "github.com/sourcegraph/go-ctags"
)

func main() {
	long := "Long" + strings.Repeat("x", 5000)
	content := "package sample\n\nfunc Short() {}\n\nfunc " + long + "() {}\n\nfunc After() {}\n"
	parser, err := ctags.New(ctags.Options{Bin: "ctags"})
	if err != nil {
		panic(err)
	}
	defer parser.Close()
	entries, err := parser.Parse("sample.go", []byte(content))
	fmt.Println("err:", err)
	for _, entry := range entries {
		fmt.Printf("%.12s line %d\n", entry.Name, entry.Line)
	}
}

Output:

err: <nil>
sample line 1
Short line 3
After line 7

The function on line 5 is missing. Its name has 5,004 bytes, and its reply
line has 5,459 bytes. With a name of 3,004 bytes, the reply line has 3,459
bytes, and the output has all four tags.

Proposed change

Read each reply line with no size limit, for example with
bufio.Reader.ReadBytes('\n'). Then a long tag does not fail the parse, as in
5851b63, and Parse also keeps the tag.

If a limit must stay, a caller could set it in Options. Parse could then
log each skipped line through Options.Info, as it does for the files that it
skips.

I can send a pull request with a test if you agree.