Showing posts with label fulltext. Show all posts
Showing posts with label fulltext. Show all posts

Monday, March 26, 2012

FTI on whole Database

I am new at FULLTEXT INDEX, I wonder if there is a possibility to index the whole DB at once and search it in the same way rather than specifying the Table and the columns?

NO, and I can't imagine why one would consider that as a possibility.

I wouldn't consider searching datetime columns, or money columns, or numeric columns for a text string. So even if it were possible, it would amount to a terrible waste of time/effort searching unnecessary columns. As well as significant resources just to create and maintain indexes on columns that may never need to be searched.

|||

Ok, Here is the scenario, We have this huge DB from a client -about 6k Tables!!!!- I am looking for the column that contains a specific string? I thought of using the FULLTEXT INDEX to get all the instances that contains that strings.
Is there a way to do that? Definitely as you mentioned it will be a terrible waste of time/effort but I don't see doing this otherwise, searching the DB table by table will be worse.

|||

No matter how you cut it, that will be a resource intensive task.

Here is a procedure that I created that would do the search. Hopefully, it will give you a good idea or two. Expect it to take quite some time with 6k tables.

--*******************************************
-- Problem: Locate data in any column, any table
-- Demonstrates:
-- WHILE looping
-- Using Dynamic SQL (sp_executesql)
-- #Temp Table vs. Table Variable

--*******************************************

SET NOCOUNT ON

DECLARE
@.TotalRows int,
@.Counter int,
@.TableName varchar(50),
@.ColumnName varchar(50),
@.FieldValue varchar(250),
@.SQLCommand nvarchar(1000),
@.ValueToFind varchar(100)

-->
SET @.ValueToFind = 'mexico'
USE Northwind -- For demo only, not needed

-->

DECLARE @.MyTable table
( RowID int IDENTITY,
TableName varchar(50),
ColumnName varchar(50)
)

CREATE TABLE #FoundTable
( RowID int IDENTITY,
Tablename varchar(100)
)

INSERT INTO @.MyTable
SELECT
TABLE_NAME,
COLUMN_NAME
FROM INFORMATION_SCHEMA.COLUMNS
WHERE DATA_TYPE IN ( 'char', 'varchar', 'nchar', 'nvarchar', 'text', 'ntext' )

SELECT
@.TotalRows = @.@.ROWCOUNT,
@.Counter = 1

WHILE ( @.Counter <= @.TotalRows )

BEGIN

SELECT
@.TableName = TableName,
@.ColumnName = ColumnName
FROM @.MyTable
WHERE RowID = @.Counter

SET @.SQLCommand = 'IF EXISTS ( '+
'SELECT 1 FROM [' + @.TableName + '] ' +

'WHERE [' + @.ColumnName + '] LIKE ''%' + @.ValueToFind + '%'' ' +
' )' +
'INSERT INTO #FoundTable ' +
' SELECT ''[' + @.TableName + ']([' + @.ColumnName + '])'''

EXECUTE sp_executesql @.SQLCommand

SET @.Counter = ( @.Counter + 1 )

END

SELECT * FROM #FoundTable

DROP TABLE #FoundTable

/*
RowID Tablename
-- --
1 Invoices(Salesperson)
2 Employees(LastName)
3 Employees(PhotoPath)
*/

|||Thanks!!!!!!!!!!! It worked, I had some issue with the variables, I was getting errors "Must declare the scalar variable "@.TotalRows". which I think it is the result of the variables scope.

|||Sorry, I had left a [GO] in the script, and it starts a new variable scope.

FT searching and weights

I have a couple questions about fulltext searching...
1) FULLTEXTTABLE appears to be my preference. I have 3 columns in my
catalog. I'm getting results, but I think the rankings should be different.
My lower rankings should be higher than the highest ranking results. Can I
give weight to specific columns with FULLTEXTTABLE? I don't see that option
available in BOL with FULLTEXTTABLE. If not, how could I skew the results by
saying one column is more important than another column?
2) Assume a scenario...
I index a table (10 columns, 10MB size, 100,000 rows)
Catalog is populated.
I empty and re-populate the table. Most of the data would be identical.
Can I get away with doing an incremental update? Would this be faster than a
re-population?
Or would it basically be the same thing as a full re-population?
Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05
Copyright (c) 1988-2003 Microsoft Corporation Developer Edition on Windows
NT 5.1 (Build 2600: Service Pack 1)
Hi Shank,
#1. You can do something like the below CONTAINSTABLE (or FREETEXTTABLE as
there is no FULLTEXTTABLE ) query below using the Northwind employee table
and it's two FT-enabled columns Notes and Title with different weights:
SELECT e.LastName, e.FirstName, e.Title, e.Notes
from Employees AS e,
containstable(Employees, Notes, 'ISABOUT (BA weight (.2) )', 10) as A,
containstable(Employees, Title, 'ISABOUT (Sales weight (.5) )', 15) as
B
where
A.[KEY] = e.EmployeeID and
B.[KEY] = e.EmployeeID
Also, you should review SQL Server 2000 BOL title "Full-text Search
Recommendations" and the last paragraph on RANK as in order to get valid
RANK values you must have statisticlly significatntly number of rows
(10,000+) in your FT-enable table.
#2. Since you're using SQL Server 2000, I'd strongly recommend that you
consider using "Change Tracking" and "Update Index in Background" as
Incremental Populations can and do take as long to complete as Full
Populations, even with no changes. If you're emptying the table and
repopulating it, I'd recommend a Full Population over CT with UIiB and over
Incremental Pop as the Incremental will have to first delete all the entries
in the FT Catalog and then re-populate it, so a Full Population is best
given your situation.
Regards,
John
"shank" <shank@.tampabay.rr.com> wrote in message
news:#jZ5A47fEHA.2524@.TK2MSFTNGP09.phx.gbl...
> I have a couple questions about fulltext searching...
> 1) FULLTEXTTABLE appears to be my preference. I have 3 columns in my
> catalog. I'm getting results, but I think the rankings should be
different.
> My lower rankings should be higher than the highest ranking results. Can I
> give weight to specific columns with FULLTEXTTABLE? I don't see that
option
> available in BOL with FULLTEXTTABLE. If not, how could I skew the results
by
> saying one column is more important than another column?
> 2) Assume a scenario...
> I index a table (10 columns, 10MB size, 100,000 rows)
> Catalog is populated.
> I empty and re-populate the table. Most of the data would be identical.
> Can I get away with doing an incremental update? Would this be faster than
a
> re-population?
> Or would it basically be the same thing as a full re-population?
>
> --
> Microsoft SQL Server 2000 - 8.00.760 (Intel X86) Dec 17 2002 14:22:05
> Copyright (c) 1988-2003 Microsoft Corporation Developer Edition on
Windows
> NT 5.1 (Build 2600: Service Pack 1)
>

Wednesday, March 7, 2012

FREETEXTTABLE returns no result for a complex freetext only on JDB

This problem sounds like a problem of fulltext module, but it's not.
I'm using SQL Server 2005 Jun CTP on Windows 2k3 server
through MS SQL Server JDBC Driver SP3
from J2EE 1.4.2_08 on Windows 2k.
I've indexed and am searching tons of Japanese news articles
to test fulltext features of 2005.
SELECT ID,RANK,FOO FROM M_HOGE AS M INNER JOIN FREETEXTTABLE
(M_HOGE,*,N'関西電力は阪神大震災で倒 した従X員向け福利厚生施Xの跡地 特別養X老人3施Xを建Xした。')
ON(ID=[KEY])
(This Japanese sentense is a little complex, saying
'The Kansai-Electric has constructed three elder care facilities at a
vacant lot where their employee welfare facilities which had given way
because of the Great Hanshin Earthquake had existed.')
This query returns no result if it was passed through JDBC,
but if I pass it to SQLServer Management Studio directly,
it returns tons of result with very good ranking.
(btw, I feel the ranking of FREETEXTTABLE was so much improved in 2005.).
But, as a strange thing, if I simplify the freetext like
SELECT ID,RANK,FOO FROM M_HOGE AS M INNER JOIN FREETEXTTABLE
(M_HOGE,*,N'関西電力は福利厚生施Xの 地に施Xを建Xした。')
ON(ID=[KEY])
(It says 'The Kansai-Electric has constructed facilities at a
vacant lot where their employee welfare facilities had existed.')
, it returns good result even through JDBC!
Our company provides a product on Java using fulltext feature of SQL Server
2000,
and is going to adapt it to 2005, expecting much improvements.
We're looking forward to update about JDBC as well as fulltext features!
Hideaki:
The SQL Server 2000 JDBC driver isn't supported against SQL Server 2005.
Please try the SQL Server 2005 driver --
http://www.microsoft.com/sql/downloads/2005/jdbc.mspx.
-shelby
Shelby Goerlitz
Microsoft SQL Server
"hideaki" <hideaki@.discussions.microsoft.com> wrote in message
news:21C664BC-0FD8-488B-BC58-0E119E58CBE0@.microsoft.com...
> This problem sounds like a problem of fulltext module, but it's not.
> I'm using SQL Server 2005 Jun CTP on Windows 2k3 server
> through MS SQL Server JDBC Driver SP3
> from J2EE 1.4.2_08 on Windows 2k.
> I've indexed and am searching tons of Japanese news articles
> to test fulltext features of 2005.
>
> SELECT ID,RANK,FOO FROM M_HOGE AS M INNER JOIN FREETEXTTABLE
> (M_HOGE,*,N'??????3? ??')
> ON(ID=[KEY])
> (This Japanese sentense is a little complex, saying
> 'The Kansai-Electric has constructed three elder care facilities at a
> vacant lot where their employee welfare facilities which had given way
> because of the Great Hanshin Earthquake had existed.')
> This query returns no result if it was passed through JDBC,
> but if I pass it to SQLServer Management Studio directly,
> it returns tons of result with very good ranking.
> (btw, I feel the ranking of FREETEXTTABLE was so much improved in 2005.).
>
> But, as a strange thing, if I simplify the freetext like
> SELECT ID,RANK,FOO FROM M_HOGE AS M INNER JOIN FREETEXTTABLE
> (M_HOGE,*,N'?????')
> ON(ID=[KEY])
> (It says 'The Kansai-Electric has constructed facilities at a
> vacant lot where their employee welfare facilities had existed.')
> , it returns good result even through JDBC!
> Our company provides a product on Java using fulltext feature of SQL
> Server
> 2000,
> and is going to adapt it to 2005, expecting much improvements.
> We're looking forward to update about JDBC as well as fulltext features!
>