Semantic Technology bringing Structure to Unstructured Text

Semantic Technology can be explained better with analogy comparison with RDBMS SQL instead of Search.

Semantic Technology takes a normal unorganized text and converts into a structured meaningful text (similar to RDBMS database) which can be queried using SPARQL (similar SQL queries)

In this blog, I take an example of a random text from a history book. Then, explain how it is converted to RDF and then what becomes possible on this RDF.

Article
“One of the most famous stories of decipherment comes from Egypt, a country in north Africa where there were kings and queens about 5000 years ago.
Rosetta is a town on the north coast of Egypt, and here an inscribed stone was found, which contained inscriptions in three different languages and scripts (Greek, and two forms of Egyptian). Scholars who could read Greek figured out that the names of kings and queens were enclosed in a little frame, called a cartouche. They then placed the Greek and the Egyptian signs side by side, and identified the sounds for which the Egyptian letters stood. As you can see, a lion stood for L, and a bird for A. Once they knew what the letters stood for, they could read other inscriptions as well.”

The text is converted in RDF Tuples:

<?xml version="1.0"?>
<rdf:RDF
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:owl="http://www.w3.org/2002/07/owl#"
xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
xmlns:xsd="http://www.w3.org/2001/XMLSchema#"
xmlns:skos="http://www.w3.org/2004/02/skos/core#"
xml:base="http://xmlns.temp.org/rdf/"
xmlns:prop="http://xmlns.temp.org/rdf/property/"
xmlns:report="http://xmlns.temp.org/rdf/report/"
xmlns:med="http://127.0.0.1/ontology/my.owl/"
>
<rdf:Description rdf:about="http://xmlns.temp.org/rdf/Famous stories of decipherment">
<rdf:type>#Story</rdf:type>
<prop:come_from resource="http://xmlns.temp.org/rdf/Egypt"/>
<prop:decipher resource="http://xmlns.temp.org/rdf/inscribed stone"/>
</rdf:Description>

<rdf:Description rdf:about="http://xmlns.temp.org/rdf/Egypt">
<rdf:type>#Country</rdf:type>
<prop:in>http://xmlns.temp.org/rdf/North Africa</prop:in>
<prop:where_there_were> kings and queens about 5000 years ago</prop:where_there_were>
<prop:circumtances> about 5000 years ago</prop:circumtances>
</rdf:Description>

<rdf:Description rdf:about="http://xmlns.temp.org/rdf/Rosetta">
<rdf:type>#Place</rdf:type>
<prop:north_coast resource="http://xmlns.temp.org/rdf/Egypt"/>
<prop:found rdf:resource="http://xmlns.temp.org/rdf/inscribed stone"/>
</rdf:Description>

<rdf:Description rdf:about="http://xmlns.temp.org/rdf/inscribed stone">
<prop:contained>
inscriptions in three different languages and scripts Greek and two forms of Egyptian
</prop:contained>
<prop:read_by resource="http://xmlns.temp.org/rdf/Scholars"/>
</rdf:Description>


<rdf:Description rdf:about="http://xmlns.temp.org/rdf/Scholars">
<rdf:type>#Profession</rdf:type>
<prop:qualification>read Greek</prop:qualification>
<prop:figured rdf:resource="http://xmlns.temp.org/rdf/cartouche"/>
<prop:method_of_reading resource="http://xmlns.temp.org/rdf/Decipher Method"/>

</rdf:Description>


<rdf:Description rdf:about="http://xmlns.temp.org/rdf/cartouche">
<prop:comment>
names of kings and queens were enclosed in a little frame
</prop:comment>
</rdf:Description>


<rdf:Description rdf:about="http://xmlns.temp.org/rdf/Decipher Method">
<prop:identified>
the sounds for which the Egyptian letters stood
</prop:identified >
<prop:method>
placed the Greek and the Egyptian signs side by side
</prop:method>
<prop:result>
a lion stood for L, and a bird for A
</prop:result>
<prop:coud_read resource="http://xmlns.temp.org/rdf/Other Inscriptions"/>
</rdf:Description>
<rdf:Description rdf:about="http://xmlns.temp.org/rdf/Other Inscriptions">
<prop:condition>
Once they knew what the letters stood for
</prop:condition>
</rdf:Description>

</rdf:RDF>

Now, I see it through a Jena Visualizer…

Queries


Natural language Query
What could Scholars identify?

Equivalent SPARQL:

BASE
PREFIX rdf:
PREFIX rdfs:
PREFIX prop:
PREFIX report:
SELECT $subject $relation $value
WHERE {
?subject ?property ?object .
?object ?relation ?value
FILTER (REGEX(STR(?subject), "Scholars", "i")
)
FILTER (REGEX(STR(?relation), "identified", "i")
)
}


Result

subject	relation	value
<http://xmlns.temp.org/rdf/Scholars>	<http://xmlns.temp.org/rdf/property/identified>	" the sounds for which the Egyptian letters stood"

2. Natural Query

Where did the stories of decipherment come from? Also tell about that place?

Equivalent SPARQL
BASE
PREFIX rdf:
PREFIX rdfs:
PREFIX prop:
PREFIX report:
SELECT $object $relation $value
WHERE {
?subject ?property ?object .
?object ?relation ?value
FILTER (REGEX(STR(?subject), "Famous stories of decipherment", "i")
)
FILTER (REGEX(STR(?property), "come", "i")
)
FILTER (REGEX(STR(?relation), "", "i")
)
}


Result

object	relation	value
<http://xmlns.temp.org/rdf/Egypt>	<http://xmlns.temp.org/rdf/property/circumtances>	"about 5000 years ago"
<http://xmlns.temp.org/rdf/Egypt>	<http://xmlns.temp.org/rdf/property/where_there_was>	"kings and queens about 5000 years ago"
<http://xmlns.temp.org/rdf/Egypt>	<http://xmlns.temp.org/rdf/property/in>	"http://xmlns.temp.org/rdf/North Africa"
<http://xmlns.temp.org/rdf/Egypt>	<http://www.w3.org/1999/02/22/-rdf-syntax-ns#type>	"#Country"

Now, imagine all the course books are converted into such RDF tuples. Anybody can do a natural language query on it to retrieve the answer. Now, search would not return the list of book which have mentions of the keyword searched for, But would return the actual answer.


Scale this to World Wide Web; where all sites are expressed in form of a RDF.  RDF is understandable by computers ....


Thus, it enables Tim Berners-Lee's vision of the Semantic Web.  http://articles.techrepublic.com.com/5100-10878_11-5552998.html?part=rss&tag=feed&subj=tr

"The Web was designed as an information space, with the goal that it should be useful not only for human-human communication, but also that machines would be able to participate and help. ..."


Semantic Agent could be written
- To query for “Doctor who is a lady in within a 100 miles radius of my address and has open appointment between 2:30 PM – 3:30 PM”.
- If a successful query, book an appointment and SMS me the appointment#.

OR a scenario,
- Find me a holiday with a beach theme, has budget of between $1000-1500.
- If successful, find a return flight within that range of dates in budget of $500-600.
- If successful, find a car and hotel …

Future - World of Semantic Intelligence and Automation …

Search

Information in an Enterprise exists in many places. Some places are structured like the databases. Some are unstructured like Emails, Documents, Contracts, and Circulars etc.

Search in Structured Information

The structured information space is very mature with database players like Oracle, SQL Server providing RDBMS based data infrastructure which allows application to store data according to the pre-defined schema. This schema defines the structure of that information.

This structure also defines the way to retrieve the data. SQL Query is very general way to retrieve the information from the RDBMS database. This SQL Query is generally in he format of:

SELECT (ColumnNames) From (Table) Where {Expression (= ColumnName Relation Value) }
This format describes the format to retrieve the data.

• Select: allows selecting the required information to be retrieved.
• From: allows to select the source where to retrieve.
• Where: allows to filter the information as per the requirement.

 So for a table,
     Select EmployeeAddress from Employee Where EmployeeName = “Kapil”.

Will retrieve information of employee address from Employee Table but only for employee name with provided.
This space is very mature.

However, it is limited only for information which is entered in the database as per the table or database schema. The information which cannot be/or is not entered in database cannot be retrieved.

Searching in Unstructured Information

Enterprise information however does not exist only in databases. There is lot of information almost more than 80% information which is in non-structured sources like Email, Word Documents, PDF Documents, Excel, Chat, Circulars…

• 90% of office communication and information exchange happens over email. Circulars are text or word documents which contains lot of events and decision making information.

• Minutes of Meetings which contains lots of important information exists in the word or text documents.
• Vendors Contracts and SLAs are exist as a Word Document.
• Systems and application’s user guide and user manual exist as documents.

These are typical information of an Enterprise office.

There is very specific information also for various domains Education like-
Schools Books, Courses, Technical Books and Documents, Journals all exist in the Word or PDF format in unstructured manner.

In non-connected PCs …

All this unstructured information has been traditionally stored in our machines file folders.

How do we retrieve them?
-In Windows system, using the search interface we do a search on keyword and specify other parameter like drive name, extension of the file…

 But it retrieves the list of large documents which may have the keyword present in the document. We then manually read the document to find the actual information.

-In UNIX system, we do the grep; which find the through the list of documents expression or keyword in regular expression.

Keyword Search

This searching is based on the keyword search which is based on the concept of the system indexing the information offline. This indexing actually inverses the linkage; all the words and phrases are created as index pointing to the document.

So when we search, system actually searches the index. Now when our keyword matches the word in the index, it returns back the document list that word was pointing to.

However, in interconnected world like intranet

All this information of the enterprise can be stored in the common centralized machine in the file folders. This how it still happens…

The retrieval method is almost the same. We specify the keyword and the document list containing that keyword and matching that filter expressions are listed. Then we manually can go in the access the information.

But lately, Enterprise Systems built only managing this type of content Enterprise Content Management (ECM) products have started being used. These ECM products also come in other flavours like Document Management Systems, Digital Asset Management (Digital Assets like audio, video, text document) …

They manage the entire lifecycle of these Information assets; ingest, manage and access/deliver. These products also come in with capabilities of search, policy management, preservation and storage.

So retrieve an asset from ECM, we use the ECM search with same keyword principle and it returns the list of document matching the criteria and we can retrieve the document.

Search, here works on the same principle of keyword search. ECM crawls the entire content and creates index of the content.

Metadata Search 

ECM however allows the other type of search called metadata which is based on the metadata of the content. Metadata is the information about the document; abstract, author, date etc. So we can also use the search of metadata to get the list of document we want.

However, in interconnected world like internet

Same keyword search is also the principle used for searching the document or information in larger scale of internet.

Google, the main search engine, crawls the information publish across the word using the crawlers and indexes it on its on large GFS (Google File system) clusters for managing the huge volume of index.

But huge volume of content across the world (almost more than billion distinct web sites), the keyword based search has reached its limitations. Google search has been utilized to its fullest capability to provide the information at fast pace. However, with 1000 of pages being returned for a keyword is resulting in the overload of data with no useful information.

People have to manually search through list of documents to get the information he wants…

Now, comes in the next phase of search linguistic

Linguistic search is further divided into Shallow linguistic and Deep Linguistic(Semantic).

Shallow Linguistic

Shallow linguistic does more than simple keyword search by linking the keyword with its respective lemma and synonyms.

It also retrieves the document with summarization and sentiment analysis.
This is made possible by using training corpus which contains the similar text and which maps the content with particular sentiments. The system is intelligent as it matches the set of keywords with sentiments and categorization.

So when new content is feed to this intelligent sentiment analyzer, it matches the content with its information and scoring index to return the sentiment of the information. It also is able to abstract out the content by removing the repeated sentences and frequency and some linguistic algortimn.

Search, now is narrower, however the documents are still needs to be human analyzed.

Deep Linguistic (Semantic)

Semantic Analysis includes grammatical, logical, morphological analysis and identification of the content. The analysis is as per the context of the sentence, identified with meaning and relations.

Once Semantic Analysis has been done, the words are organized in relation using the semantic technology (RDF). RDF allows defining the statement in form of tuples; Subject, Predicate and Object.

Subject is the resources about which is the subject of this informational statement.Predicate depicts the relationship of this resource to the Object. This Object could be another resource or be text value (some explanation of the subject).

Once the entire document is converted into RDF tuples, it is now in structured format. This structures can be explained in analogy to the RDBMS. Breaking the statement into 3 columns; subject, predicate and Object.

SPARQL (similar RDBMS SQL) is used to retrieve the information from the RDF tuples.

SPARQL is in same format as SQL RDBMS:

SELECT (subject,predicate,object) FROM WHERE {Expression: relation between subject and object}

This differentiates form other searches by directly answers the queries based on the semantic relations existing in the document.

More of Advantage of Semantic Search in different post….