Information Management Methods

Retrieved Oct. 23, 2005 from
IBM Think research

Mining Documents for Data
COVER STORY: Part 3

By Tom Halfhill

New software brings to ordinary text files the search capabilities of the most sophisticated databases -- and then some

One of the ironies of modern computing is that, as online information becomes more plentiful, it gets harder to find the information you need. But Intelligent Miner™ for Text, a new data mining tool, can change all that. Developed by an IBM Software Solutions group in Böblingen, Germany, in cooperation with several Research teams, the product began shipping in early 1998.

Unlike conventional data mining tools, which operate on structured databases -- with information organized in neatly defined fields and records -- the new product works on ordinary text files. It can search, analyze, organize and index all kinds of free-form documents: email messages, memos, reports, notes, outlines, business plans, technical papers, documentation, Web pages and more. It automatically generates metadata -- structured information about the documents -- which helps turn a mass of text files into a database of useful information. "This is very important for knowledge management because it helps people sift through what amounts to noise," says Alan Marwick, senior manager of text analysis and advanced search tools at Watson.

Although Intelligent Miner for Text can analyze documents written in a variety of languages, there are still some it cannot handle. One of the first steps, therefore, is to identify the language in which a document is written. That is done with a component called the Human Language Recognizer -- based on the work of John Prager at the Thomas J. Watson Research Center -- which can recognize a language from clues such as frequently used words and letter sequences.

Another module -- the Feature Extractor, based on the Textract tool developed by a Watson team led by Roy Byrd -- consists of two components. One, nicknamed the Nominator, identifies proper nouns, while its counterpart, the Terminator, finds multiword terms such as "laser printer." Identifying proper nouns and terms is a prerequisite to classifying a document, because it allows another module called the clustering tool to sort the documents by topic.

Clustering is the key to finding related documents in large masses of text. The clustering tool, called HiClust, was developed at the Haifa Research Laboratory and refined at Böblingen. It works by grouping documents that discuss the same topic. That not only facilitates browsing, but it can help refine search results, especially when the query is fuzzy or ambiguous. For instance, if one enters the search term "Merced," almost any Web search engine will return a list of documents, some relating to Merced County in California and others to the new Intel chip architecture. HiClust will automatically cluster the results, separating one group from the other.

Intelligent Miner for Text also includes utilities that bring knowledge management to Web browsing. Built on work by Rob Barrett, a research staff member at Almaden, they can save all the Web pages you've ever visited, for later classification and reference. The utilities can automatically customize Web pages for individual usage patterns -- say, adding links to relevant sites you've viewed before.

INTELLIGENCE TO COME
Despite its many capabilities, Intelligent Miner for Text is still developing. Future versions, says Marwick, not only will support additional natural languages but will have even more sophisticated tools for searching and organizing unstructured documents.

As part of the knowledge base on which future versions will be built, Watson researchers are pursuing many aspects of the underlying technologies. One outcome is a component that summarizes large documents by extracting the most important sentences, as determined by the frequency of certain words and other language patterns.

A new Context Thesaurus helps focus a user's search. If the term "mustang," say, is entered, the user might be prompted to choose "horse," "Ford" or "airplane," to rule out irrelevant documents.

Another development finds relationships between markedly different terms. For instance, it might figure out that "toner cartridge" is related to "laser printer" if both terms frequently occur in the same documents. Later, a search for all documents related to laser printers would also retrieve those that mention toner cartridges, even if "laser printer" never appears in those documents.
----------------------
Tom R. Halfhill, a former senior editor of Byte magazine, lives near San Francisco.

MG_web

The Baby Name Wizard's Name Voyager
http://babynamewizard.com/namevoyager/lnv0105.html

Eva Zeisel (1906- )

Hungarian-born American studio potter and designer

Eva Zeisel portrait


Classic Century


Nambe Platter


Nambe Bowl


Nambe Pouring


Salt & Pepper Shakers, Town and Country, Red Wing Pottery (Red Wing, Minnesota)

Education

看美國K25計畫
樂為良／文字工作者（台北市)
教育部給幾所大學五年五百億後，好像台灣的高等教育就會突飛猛進。沒錯，搞研究與辦教育要有錢有人，但問題是錢是不是用對了？不妨看看美國如何用錢培養人才。
美國國家衛生院為了突破科學研究，在兩千年推出一個叫K25的計畫，每年讓二十名學有專長的教授與研究員，重回學校上課學習新的領域，為期五年。現在第一批「高材生」要畢業了，這項跨學科專案的正面效應已逐漸展現，尤其是生物醫學這一塊。
四十一歲的畢比是位傑出的電機工程專家，他在麥迪遜大學講課並主持一個二十五人的微流體實驗室，專業成就備受肯定。幾年前他申請到K25獎學金重回大學主修生物。他的同學多半愛玩不好學，頗教他意外，不過很快他便想到自己當年也好不到哪，也因此更加明白修他課學生的心理，師生互動大獲改善。
更重要的是，他把微流體學帶進生物實驗室，解決了培養皿的一大難題。生物學家一直想知道幹細胞分泌的蛋白質如何控制生長，但一般的皮氏培養皿裡有數百萬個細胞彼此傳遞訊號，環境難以管控；而微流設備只容納少許的細胞，使得培養環境大獲改善，要測出細胞如何互相管控就容易多了。
根據最新一期《自然雜誌》的報導，像這種在生命科學引進新方法的例子，因為K25，已有數個成功之例。生命科學已經從實驗室中一個人獨自研究，變成一個團隊處理大型問題。馬里蘭大學流行病學與預防醫學主任摩里斯就說，他延攬進的物理學家「學生」，在建構模型上就比他們強多了。
是什麼吸引了這些事業有成的專家重做學生？一方面是尋求生命與事業的創新，另方面是聯邦政府的巨額撥款。案子獲准，申請人就有年薪十七萬美元（約五百六十萬台幣）可領，外加四萬美元的研究費，以及薪水百分之八的行政費用。也就是一年賞你七百五十萬當學生，還能不吸引人？不過彼此的權利與義務也訂得十分清楚。
要派美國的不是很容易，也不該凡事學美國；但美國之所以成為超強，機制上必有不凡之處，從這個在中文網頁查不到的K25計畫，就可管窺美國這龐大機器運作的動態。相形之下，我們政府在用錢哲學上能相似地創新、務實、高效嗎？有時看百億百億納稅人的錢被輕易揮霍，心如淌血。 【2005/10/22 聯合報】