In our daily lives, we are pretty much used to ask location based, transactional or navigations questions in search engines. We are now doing the same in LLM based chats as well. It is very common for us to ask questions like “Show an Italian restaurant near me” OR “Which is the best vegan restaurant in Japan”. How do these AI agents and answer engines know our location in order to give contextually correct answers? Let’s dig in a little deeper.
How do LLMs always give contextually right answer, location wise?
LLM based answer engines always pick up our context and intent. When it comes to location based queries, their answers may not be the best at present but are surely contextually right.
For the context of this article, let us assume that I am based in Gurugram. In the question to an LLM where I ask “Show an Italian restaurant near me”. It will pick up my location Gurugram and show me the options without having to ask you where I am based out of.
Similarly for the next question “Which is the best vegan restaurant in Japan”, It will show me some options of vegan restaurants in Japan and at the end it will ask a question like “Are you planning a trip to Japan, Do you want me to build an itinerary?”.
LLMs understand your query, its context and intent, location is one important parameter for the same. Understanding how an AI knows where you are involves understanding various components within the data layer. It is a combination of system level data and conversational memory.
Lets look at the step by step logic of how an LLM triangulates your location -
1. System Generated Meta Data (Direct Inputs)
Which ever AI platform you are using, the system will attach a small packet of “Contextual Metadata”. This contextual metadata is the system generated meta data which includes information like date, time, location based on IP address or device permissions (GPS location, WiFi or Cellular Network). This by deafult information becomes a primary source of directly available and reliable information.
2. Personal Context and Memory (Identity Profiling)
AI agents also do user profiling based on their daily conversations with the users. The objectives are multifold like collecting data points, training on those data points and also providing a personalised experience for the users. Based on my previous conversations with the LLM where I might have asked about things like Gyms in Gurugram, it has added this factual data point as a part of my profiling and stored indicative location as Gurugram.
3. Linguistic Inference (AI as Detective Pikachu)
AI agents are always looking for data points which could help them profile the user. Whenever a query is run, it will look for cues within your current chat session for contextual indications of your location or any other data point about you.
For example, if I have asked about rupees to dollar conversion, it will use that information to narrow down my current region / country as India and my preferred currency as Rupees.If I have asked for temperature in Gurugram, it will pick that up to mark my location as Gurugram.
Dialect also forms an important part of user profiling, the words you use are highly indicative of your region. For example, if you ask for petrol station, then it may indicate the region as India but if you ask for gas station then it will put it as America. If you ask about regional holidays, the agent will take educated guess about where you are based out of.
4. Tool Calling (Verification)
In some cases it may also happen that LLM does not have enought data about you to make an educated guess. These could be situations like where you travel a lot or use VPN to different country locations. The first best option of verification LLM will probably use is to ask you upfront!
The next technical verification an LLM will do us to trigger a tool calling connection to say google maps to get your current location OR to pick up saved labels like home location and work location. It can also tool call a search engine and use the localised results based on your current IP address.
The above listed methods are important but not limited to various strategies which LLM will use to know a user’s current location.
Lets look at the logical flow that LLM may take to get your location -
For this lets take the example of - “Italian restaurants near me”
Step 1: LLM will first check for intent of my query. It understands I am looking for local recommendations of Italian restaurants but it needs to understand what does ‘near me’ means here.
Step 2: It tries to understand the context of my query. Does it know what location “near me” points at? Is there any system metadata available which tells my location. It may check the IP address or perform some tool calling to get the verification on address. Let’s say it now knows through these methods that location is ‘Gurugram’
Step 3: The LLM will check its memory to see if it has any records of where I live. Let’s say from it’s memory it now knows that the persion is based in Gurugram.
Step 4: Action. Since both the deciphered location and historical data points to gurugram as a location. LLM will then show results of Italian restaurants in Gurugram, without needing to ask for any clarifications.