Natural Language Processing
Natural Language Processing is an intersection between AI, machine learning and linguistics,
focusing on processing human
languages such as English, Chinese, Japanese etc. using computers. The application areas of NLP
information retrieval, text mining, sentiment analysis, dialogue systems, machine translation and
Tokenization - Breaking strings into tokens (each token is a word)
Stemming - Normalizes words into its base form or root form
Lemmatisation - Maps words into one common root (lemma) and outputs a proper
Paths of Speech Tags - Indicates how a word functions in meaning as well as
within a sentence.
Named Entity Recognition (NER) - Process of detecting the named entities such
names, company names, quantities, locations, etc.
Chunking - Picking up individual pieces of information (tokens) and grouping
into the bigger pieces (chunks).
Multiple intents in 1 question - Customer asks for different things at the same
To deal with this, AI could identify all the topics related to the customer’s question and
them separately. In this way, the conversation can continue.
Assuming it understands context and has memory - this occurs when a customer
provide enough details about what they are looking for. Like for example you would like to apply
for a boat loan, but you just say “loan”. In this case, AI is able to ask for more details and
narrow information down as a result of that.
Misspellings in entity extractions - A good way of dealing with spelling
to use machine learning and spell correction, these would help the system interpret what the
Same word, different meaning - This has to do with distinguishing the
every word in a sentence. The idea is that conversational AI is able to identify which of the
keywords in any given sentence are most relevant to a user’s query in order to deliver the
Keeping the conversation going - AI should give you the tools for deeper
extract variables, determining the flow of a conversation based on user input.
Tackling false positives
- These happen when a user requests something that the system should be able to answer
but has not yet learned how. The solution here would be similar to point 4, which means
the relevant parts of a message and providing assistance based on existing knowledge while
the gaps in its ability.