Search message ids as phrases on the SQL Server persisters

[johnsimons]

Searching for a message id on the SQL Server persisters now returns only the messages that contain the whole id. Before, FREETEXT matched every message that shared any piece of the id, and ServicePulse could not open the message. A trailing * is now a prefix search on SQL Server and PostgreSQL, as it already is on RavenDB.

Analysis

• Both SQL Server dialects filtered with FREETEXT. FREETEXT splits the input at punctuation, adds other forms of each word, and returns the rows that contain any of the resulting words. A GUID or order-1234-5678 matched every message that shared one of its pieces.
• ServicePulse opens a successful message by calling messages/search/{messageId} with the default paging (50 rows, newest time_sent first) and looking for the row with the matching id. With more than 50 matches the message is usually not on that page, and ServicePulse shows "Could not find message".
• In a load test with about 72,000 audit messages, 36 of 36 sampled background-noise-NNNN-... ids failed to open, and each search returned about 15,000 rows. On a 40,000-row table, a random GUID matched 124 rows with FREETEXT and 1 with a quoted CONTAINS phrase.
• The primary instance's SQL Server persister has used the same predicate since 6.20.0 (Refactor message body full-text search configuration and tests #5708). The audit one is unreleased (Add EF Core SQL Server and PostgreSQL persisters to the audit instance #5936).
• The PostgreSQL dialects already matched ids as phrases, because websearch_to_tsquery turns a hyphenated term into a phrase. They had two other gaps: websearch_to_tsquery drops a trailing *, and it reads a leading - as NOT, so zzz OR -order matched every message without "order".

Changes

• SQL Server (audit and primary): EF.Functions.Contains replaces EF.Functions.FreeText. The dialect splits the input on spaces, wraps each term in double quotes with embedded quotes doubled, and joins the terms with OR. A quoted term is a phrase, so the words the word breaker splits it into must appear together and in order. CONTAINS reads the same full-text index and catalog, so there is no migration.
• PostgreSQL (audit and primary): to_tsquery replaces websearch_to_tsquery, with a query built from the input. Each term is quoted as a phrase, a term ending in * gets :*, and the terms are joined with |. Backslashes and quotes are escaped and a bare * is skipped, because to_tsquery raises a syntax error on them. The tsvector expression is unchanged. EXPLAIN on a scratch GIN index shows a bitmap index scan for a query with | and :*.

Behaviour changes

• SQL Server no longer matches other forms of a word, so verify does not find "verified". RavenDB (StandardAnalyzer) and PostgreSQL (simple configuration) never did.
• A term ending in * is a prefix search on SQL Server and PostgreSQL. RavenDB search already supports a trailing wildcard. FREETEXT and websearch_to_tsquery ignored the *.
• PostgreSQL no longer reads websearch syntax (-term, "...", or) in a search. Each space-separated term is a phrase, as on SQL Server.

Not changed

• Search input over 4,000 characters still fails on SQL Server. CONTAINS, like FREETEXT, does not accept an nvarchar(max) argument.
• SQL Server populates the full-text index in the background (CHANGE_TRACKING AUTO), so a new message becomes searchable 4 to 6 seconds after it appears in the list. ServicePulse can still show "Could not find message" for a message that is only seconds old.

Testing

• New shared EF tests, run on SQL Server and PostgreSQL for both instances: a search for a message id does not return a message that shares pieces of the id, a trailing * matches a prefix, and 13 inputs that are query syntax in one of the dialects (*, zarquon\, it's*, ", a & b, NEAR and others) do not throw. Against the old FREETEXT dialect, the id and prefix tests fail on both instances.
• FullTextSearchIndexTests (PostgreSQL, both instances) pin the | join and the :* suffix.
• Search and message view tests, Debug: persistence on SQL Server 38 audit and 53 primary, on PostgreSQL 42 audit and 57 primary; acceptance on SQL Server 4 audit and 27 primary, on PostgreSQL 4 audit and 27 primary. Release build of the solution: 0 warnings, 0 errors.