BIGDATA

I started working in Semantic technologies in 2008 and was very intrigued with potential of Semantic technologies to create a meaning out of unstructured data. I did POCs and developed libraries using GATE and lingpipe the NLP parser to parse the free English text and breaks down the sentences in to RDF tags. I could run SPARQL queries over the entire text similar SQL queries over RDBMS Database.

Suddenly, in 2012 the concept has come back again with new name BigData.

Major software Vendors are making their products Bigdata compliant. Trend I see is more for Content Management Product companies because they hosted the unstructured text in their CMS

What is BIGData? 

Layman uptake-

1)      Does when data gets very big it becomes BIGDATA?  Keep adding disk does it not solve problem.

2)      Is it is about Hadoop? Hadoop has HDFS similar to Google File Cluster and Map-Reduce which google used for processing large files. So does Hadoop does solves my problem

Let me list few things which characterizes BiGDATA as industry is speaking about-

1)      Big data is massive scale of data which is being generated in the organization in order of petabytes and zetabytes.

a.       Humans generated data like journal, reports , text, documents (text or otherwise), pictures, videos, slideware, chat, blogs etc.

b.      Machine generated data like logs , GPS output, Sensors outputs, Output from interfaces of medical, devices, cameras etc.

c.       One file (video/audio/corpus) itself can be so big as to be of size terabytes. Hence, need the provision of splitting the file and storing.

2)      Data is not very valuable or even interesting because

a.       Data is not so critical that if we lose some of the data the system will crash or revenue loss will happen.

b.      Data starts to provide some value only by aggregation or summary of the entire data.

We would not spending million dollars to house this in traditional data tools like expensive databases.

So BigData is not transactional data.

Transactional is very critical and required ACID for marinating the integrity of the data. Any loss or corruption of such data can be huge revenue loss to the company.

Enter HADOOP

Hadoop Distributed File System (HDFS)- Hence, enters Hadoop, which is Cloud of commodity low value workstation.  Distributed architecture challenges of synchronization, network failure tolerance, redundancy of data are in-built in hadoop. It solves the problem of storing this inexpensive large data on the grid of low commodity machines.

Map-Reduce - Next, the data is so large that processing it in real-time is a very large problem. Again, Hadoop solves this problem by utilizing the famous Map-Reduce paradigm to break large data into smaller sets of data and processing to aggregate into a meaningful data.

Hadoop may be able to solve the problem of infrastructure for BigData.

Vendors are now implementing their product using Hadoop stack for developing grid using low commodity servers.

However, real problem lies in how we actually extract meaning out of unstructured free text. Is Hadoop’s Regular expression enough to extract the meaning out of this free unstructured text.

 Log files are fine; because they have well defined structure and can be extracted meaningfully by regular extraction patterns.

But free text like journals, chat, blogs, reports, books, audio files etc. are different ballgame.  

Enter Semantic technology

Analysis of aggregation–

First step towards analyzing these texts have been of using cluster analysis using corpus by way of machine learning.

Training or Machine Learning - Corpus is created by feeding the Analyzer with large text which creates a dictionary of a sort associated with a concept. This phase is called training.

The next phase is actual analysis. Analyzer is fed the free text and based on its dictionary of words with concepts predicts what is the concept of the input text. It provides the concept matching with some accuracy percentage.

Example -

 If the text contains words like football, stadium, match, score … Most probably the analyzer will predict that the text is of Category:Sports, Sub-Category:Football-Match.

Few more words are like, Obama, speech, audience, ties… Analyzer may indicate some like 15% probability of concept being politics.

Advance Phase –

The Free text is processed and analyzed and concepts (entities- place, person, thing etc.)  are extracted linked with the ton of informative data on web by linking (similar to dumb linking we have of pages in html today; but it will be intelligent linking this time).

The entire text can be broken in RDF statement of facts:- subject, predicate and objects. Well depicted by RDF statements (n-tuple of RDF/XML).

This will allow SQL like queries over RDFs and worlds’s entire information (Bigdata) will be a big meaningful database accessible to Web agents.

IT industry vendors have a unique way of converting every business concept into mundane technology problem to sell their technology products

IT industry has it own way of converting every beautiful business concept into mundane technology problem to sell their technology stack. They reduce every business problem entering market and killing it by labeling as technology problem. 

WEB 2.0

Web 2.0 is not so much of technology but new way of doing business using crowd-sourcing, social networking, collaboration across value chain. 

Strength of  - JavaScript , Making rich UI, Open source LAMP,  Open source products gave way for blogging sites, Rapid Development frameworks, AJAX ... --- thus technology was starting to overshadow Web 2.0 concept. People started claiming Web 2.0 === AJAX.    

Vendors started a war of coming up with framework of AJAX. Each came up with his own version of AJAX library selling them to customers with make-believe of Web 2.0. In this war actual meaning of Web 2.0 was lost somewhere.

Web 2.0 is very highly innovative concept. It needs a radical shift in culture and Way of Working. It may disrupt the entire traditional revenue model.  

Ex- In news industries, where there were defined reporters who could publish the news. Now in Web 2.0, every man can be a reporter and contribute to news making.

In publishing industry, there were renowned authors who could author a book or article. Now in Web 2.0, every man can be a author, publisher. Any person can become a reviewer and do proof-editing.

In music industry, only contracted bands came out with music. Now any man can try his hand at singing and upload it on Youtube.

In Retail, companies will contract person to develop the software and sell. Now any man can develop software and host it on AppStore and sell it to mass.

Telecom companies are new music sellers.  

Coffee houses are new retailers.

Retail companies are new IT Hardware Sellers. 

These concepts are business ideas requiring a deep thinking and change in which make the difference between a successful and failures.

But innovative business concepts cannot be commoditized hence cannot be sold and hence ignore by Vendors.

SOA

SOA is not so much of technology but aligning technology to business. It is paradigm of thinking in terms of Enterprise functions in terms of business process. And technology then is aligned to meets those processes. 

IT Vendors knew what will sell is not this concept but their technology products. So they came out with tons of materials equating SOA with ESB, BPM .... and SOA === ESB+BPM.

Vendors started a war of coming up with their ESBand  BPM. Each came up with his own version of ESB  product selling them to customers with make-believe of SOA. SOA was killed every project started with the customer asking which ESB, which BPM.  

Aligning technology to Business went missing. result thousands of dollars down the drain because of failure of SOA projects.

  

SOA is foremost a business concept. Demanding companies start thinking in terms of their business services they offer to the world. These services are then broken down in business processes which are broken down into granular services. Real challenge is in service identification and aligned with attributes which make them reusable in different client scenarios.

The services identifications, making service reusable, service design principles are complex concepts require deep analysis not commoditizable. Hence are ignored by vendors. These are ideas which make a SOA project successful or failure.

BIGDATA

Bigdata is a business problem of utilizing millions of unstructured information assets (text, audio, video, chats etc.) to the benefit of business.

IT Vendors knows and have converted this into a mundane technology problem. Enter problem “Of Storing of large data and processing large data”. Enters Hadoop.

Vendor has now started the race of coming up with their own version of Hadoop infrastructure.

But real problem lies in how we actually extract meaning out of unstructured free text. Is Hadoop’s Regular expression enough to extract the meaning out of this free structure text? Log may be fine; because they have structure and can be extracted meaningfully by regular extraction patterns.

BIGDATA is more a concept of semantic text meaningful web than infrastructure. It is more about converting the text of non-meaningful words, phrases into more meaningful text by extracting the concepts and entity from them and linking into existing Web of information. LinkedData, DBPedia , WikiPedia, Semantic technology, Ontology , Corpus, are more important than infrastructure. 

The extracting meaning, pattern recognition, Natural language parsing, ontology, corpus creation are complex concept not comoditizable hence ignored by most. These are ideas which will make BigData problem successful or failure.

Semantic Technology bringing Structure to Unstructured Text

Semantic Technology can be explained better with analogy comparison with RDBMS SQL instead of Search.

Semantic Technology takes a normal unorganized text and converts into a structured meaningful text (similar to RDBMS database) which can be queried using SPARQL (similar SQL queries)

In this blog, I take an example of a random text from a history book. Then, explain how it is converted to RDF and then what becomes possible on this RDF.

Article
“One of the most famous stories of decipherment comes from Egypt, a country in north Africa where there were kings and queens about 5000 years ago.
Rosetta is a town on the north coast of Egypt, and here an inscribed stone was found, which contained inscriptions in three different languages and scripts (Greek, and two forms of Egyptian). Scholars who could read Greek figured out that the names of kings and queens were enclosed in a little frame, called a cartouche. They then placed the Greek and the Egyptian signs side by side, and identified the sounds for which the Egyptian letters stood. As you can see, a lion stood for L, and a bird for A. Once they knew what the letters stood for, they could read other inscriptions as well.”

The text is converted in RDF Tuples:

<?xml version="1.0"?>
<rdf:RDF
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:owl="http://www.w3.org/2002/07/owl#"
xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
xmlns:xsd="http://www.w3.org/2001/XMLSchema#"
xmlns:skos="http://www.w3.org/2004/02/skos/core#"
xml:base="http://xmlns.temp.org/rdf/"
xmlns:prop="http://xmlns.temp.org/rdf/property/"
xmlns:report="http://xmlns.temp.org/rdf/report/"
xmlns:med="http://127.0.0.1/ontology/my.owl/"
>
<rdf:Description rdf:about="http://xmlns.temp.org/rdf/Famous stories of decipherment">
<rdf:type>#Story</rdf:type>
<prop:come_from resource="http://xmlns.temp.org/rdf/Egypt"/>
<prop:decipher resource="http://xmlns.temp.org/rdf/inscribed stone"/>
</rdf:Description>

<rdf:Description rdf:about="http://xmlns.temp.org/rdf/Egypt">
<rdf:type>#Country</rdf:type>
<prop:in>http://xmlns.temp.org/rdf/North Africa</prop:in>
<prop:where_there_were> kings and queens about 5000 years ago</prop:where_there_were>
<prop:circumtances> about 5000 years ago</prop:circumtances>
</rdf:Description>

<rdf:Description rdf:about="http://xmlns.temp.org/rdf/Rosetta">
<rdf:type>#Place</rdf:type>
<prop:north_coast resource="http://xmlns.temp.org/rdf/Egypt"/>
<prop:found rdf:resource="http://xmlns.temp.org/rdf/inscribed stone"/>
</rdf:Description>

<rdf:Description rdf:about="http://xmlns.temp.org/rdf/inscribed stone">
<prop:contained>
inscriptions in three different languages and scripts Greek and two forms of Egyptian
</prop:contained>
<prop:read_by resource="http://xmlns.temp.org/rdf/Scholars"/>
</rdf:Description>


<rdf:Description rdf:about="http://xmlns.temp.org/rdf/Scholars">
<rdf:type>#Profession</rdf:type>
<prop:qualification>read Greek</prop:qualification>
<prop:figured rdf:resource="http://xmlns.temp.org/rdf/cartouche"/>
<prop:method_of_reading resource="http://xmlns.temp.org/rdf/Decipher Method"/>

</rdf:Description>


<rdf:Description rdf:about="http://xmlns.temp.org/rdf/cartouche">
<prop:comment>
names of kings and queens were enclosed in a little frame
</prop:comment>
</rdf:Description>


<rdf:Description rdf:about="http://xmlns.temp.org/rdf/Decipher Method">
<prop:identified>
the sounds for which the Egyptian letters stood
</prop:identified >
<prop:method>
placed the Greek and the Egyptian signs side by side
</prop:method>
<prop:result>
a lion stood for L, and a bird for A
</prop:result>
<prop:coud_read resource="http://xmlns.temp.org/rdf/Other Inscriptions"/>
</rdf:Description>
<rdf:Description rdf:about="http://xmlns.temp.org/rdf/Other Inscriptions">
<prop:condition>
Once they knew what the letters stood for
</prop:condition>
</rdf:Description>

</rdf:RDF>

Now, I see it through a Jena Visualizer…

Queries


Natural language Query
What could Scholars identify?

Equivalent SPARQL:

BASE
PREFIX rdf:
PREFIX rdfs:
PREFIX prop:
PREFIX report:
SELECT $subject $relation $value
WHERE {
?subject ?property ?object .
?object ?relation ?value
FILTER (REGEX(STR(?subject), "Scholars", "i")
)
FILTER (REGEX(STR(?relation), "identified", "i")
)
}


Result

subject	relation	value
<http://xmlns.temp.org/rdf/Scholars>	<http://xmlns.temp.org/rdf/property/identified>	" the sounds for which the Egyptian letters stood"

2. Natural Query

Where did the stories of decipherment come from? Also tell about that place?

Equivalent SPARQL
BASE
PREFIX rdf:
PREFIX rdfs:
PREFIX prop:
PREFIX report:
SELECT $object $relation $value
WHERE {
?subject ?property ?object .
?object ?relation ?value
FILTER (REGEX(STR(?subject), "Famous stories of decipherment", "i")
)
FILTER (REGEX(STR(?property), "come", "i")
)
FILTER (REGEX(STR(?relation), "", "i")
)
}


Result

object	relation	value
<http://xmlns.temp.org/rdf/Egypt>	<http://xmlns.temp.org/rdf/property/circumtances>	"about 5000 years ago"
<http://xmlns.temp.org/rdf/Egypt>	<http://xmlns.temp.org/rdf/property/where_there_was>	"kings and queens about 5000 years ago"
<http://xmlns.temp.org/rdf/Egypt>	<http://xmlns.temp.org/rdf/property/in>	"http://xmlns.temp.org/rdf/North Africa"
<http://xmlns.temp.org/rdf/Egypt>	<http://www.w3.org/1999/02/22/-rdf-syntax-ns#type>	"#Country"

Now, imagine all the course books are converted into such RDF tuples. Anybody can do a natural language query on it to retrieve the answer. Now, search would not return the list of book which have mentions of the keyword searched for, But would return the actual answer.


Scale this to World Wide Web; where all sites are expressed in form of a RDF.  RDF is understandable by computers ....


Thus, it enables Tim Berners-Lee's vision of the Semantic Web.  http://articles.techrepublic.com.com/5100-10878_11-5552998.html?part=rss&tag=feed&subj=tr

"The Web was designed as an information space, with the goal that it should be useful not only for human-human communication, but also that machines would be able to participate and help. ..."


Semantic Agent could be written
- To query for “Doctor who is a lady in within a 100 miles radius of my address and has open appointment between 2:30 PM – 3:30 PM”.
- If a successful query, book an appointment and SMS me the appointment#.

OR a scenario,
- Find me a holiday with a beach theme, has budget of between $1000-1500.
- If successful, find a return flight within that range of dates in budget of $500-600.
- If successful, find a car and hotel …

Future - World of Semantic Intelligence and Automation …

Search

Information in an Enterprise exists in many places. Some places are structured like the databases. Some are unstructured like Emails, Documents, Contracts, and Circulars etc.

Search in Structured Information

The structured information space is very mature with database players like Oracle, SQL Server providing RDBMS based data infrastructure which allows application to store data according to the pre-defined schema. This schema defines the structure of that information.

This structure also defines the way to retrieve the data. SQL Query is very general way to retrieve the information from the RDBMS database. This SQL Query is generally in he format of:

SELECT (ColumnNames) From (Table) Where {Expression (= ColumnName Relation Value) }
This format describes the format to retrieve the data.

• Select: allows selecting the required information to be retrieved.
• From: allows to select the source where to retrieve.
• Where: allows to filter the information as per the requirement.

 So for a table,
     Select EmployeeAddress from Employee Where EmployeeName = “Kapil”.

Will retrieve information of employee address from Employee Table but only for employee name with provided.
This space is very mature.

However, it is limited only for information which is entered in the database as per the table or database schema. The information which cannot be/or is not entered in database cannot be retrieved.

Searching in Unstructured Information

Enterprise information however does not exist only in databases. There is lot of information almost more than 80% information which is in non-structured sources like Email, Word Documents, PDF Documents, Excel, Chat, Circulars…

• 90% of office communication and information exchange happens over email. Circulars are text or word documents which contains lot of events and decision making information.

• Minutes of Meetings which contains lots of important information exists in the word or text documents.
• Vendors Contracts and SLAs are exist as a Word Document.
• Systems and application’s user guide and user manual exist as documents.

These are typical information of an Enterprise office.

There is very specific information also for various domains Education like-
Schools Books, Courses, Technical Books and Documents, Journals all exist in the Word or PDF format in unstructured manner.

In non-connected PCs …

All this unstructured information has been traditionally stored in our machines file folders.

How do we retrieve them?
-In Windows system, using the search interface we do a search on keyword and specify other parameter like drive name, extension of the file…

 But it retrieves the list of large documents which may have the keyword present in the document. We then manually read the document to find the actual information.

-In UNIX system, we do the grep; which find the through the list of documents expression or keyword in regular expression.

Keyword Search

This searching is based on the keyword search which is based on the concept of the system indexing the information offline. This indexing actually inverses the linkage; all the words and phrases are created as index pointing to the document.

So when we search, system actually searches the index. Now when our keyword matches the word in the index, it returns back the document list that word was pointing to.

However, in interconnected world like intranet

All this information of the enterprise can be stored in the common centralized machine in the file folders. This how it still happens…

The retrieval method is almost the same. We specify the keyword and the document list containing that keyword and matching that filter expressions are listed. Then we manually can go in the access the information.

But lately, Enterprise Systems built only managing this type of content Enterprise Content Management (ECM) products have started being used. These ECM products also come in other flavours like Document Management Systems, Digital Asset Management (Digital Assets like audio, video, text document) …

They manage the entire lifecycle of these Information assets; ingest, manage and access/deliver. These products also come in with capabilities of search, policy management, preservation and storage.

So retrieve an asset from ECM, we use the ECM search with same keyword principle and it returns the list of document matching the criteria and we can retrieve the document.

Search, here works on the same principle of keyword search. ECM crawls the entire content and creates index of the content.

Metadata Search 

ECM however allows the other type of search called metadata which is based on the metadata of the content. Metadata is the information about the document; abstract, author, date etc. So we can also use the search of metadata to get the list of document we want.

However, in interconnected world like internet

Same keyword search is also the principle used for searching the document or information in larger scale of internet.

Google, the main search engine, crawls the information publish across the word using the crawlers and indexes it on its on large GFS (Google File system) clusters for managing the huge volume of index.

But huge volume of content across the world (almost more than billion distinct web sites), the keyword based search has reached its limitations. Google search has been utilized to its fullest capability to provide the information at fast pace. However, with 1000 of pages being returned for a keyword is resulting in the overload of data with no useful information.

People have to manually search through list of documents to get the information he wants…

Now, comes in the next phase of search linguistic

Linguistic search is further divided into Shallow linguistic and Deep Linguistic(Semantic).

Shallow Linguistic

Shallow linguistic does more than simple keyword search by linking the keyword with its respective lemma and synonyms.

It also retrieves the document with summarization and sentiment analysis.
This is made possible by using training corpus which contains the similar text and which maps the content with particular sentiments. The system is intelligent as it matches the set of keywords with sentiments and categorization.

So when new content is feed to this intelligent sentiment analyzer, it matches the content with its information and scoring index to return the sentiment of the information. It also is able to abstract out the content by removing the repeated sentences and frequency and some linguistic algortimn.

Search, now is narrower, however the documents are still needs to be human analyzed.

Deep Linguistic (Semantic)

Semantic Analysis includes grammatical, logical, morphological analysis and identification of the content. The analysis is as per the context of the sentence, identified with meaning and relations.

Once Semantic Analysis has been done, the words are organized in relation using the semantic technology (RDF). RDF allows defining the statement in form of tuples; Subject, Predicate and Object.

Subject is the resources about which is the subject of this informational statement.Predicate depicts the relationship of this resource to the Object. This Object could be another resource or be text value (some explanation of the subject).

Once the entire document is converted into RDF tuples, it is now in structured format. This structures can be explained in analogy to the RDBMS. Breaking the statement into 3 columns; subject, predicate and Object.

SPARQL (similar RDBMS SQL) is used to retrieve the information from the RDF tuples.

SPARQL is in same format as SQL RDBMS:

SELECT (subject,predicate,object) FROM WHERE {Expression: relation between subject and object}

This differentiates form other searches by directly answers the queries based on the semantic relations existing in the document.

More of Advantage of Semantic Search in different post….