I wouldn't just dismiss the guy the first time, even after an ID check (and filling out an FI - field interview - card).
"Why are you here? That building is closed up. Been closed for 2 years now."
"Going to meet a friend? Oh. What makes you think she's here? You sure you've got the right address?"
"Got a phone number? We can try to call her to make sure you've got the right address."
"OK, there's no answer on that line, so it's time for you to mosey."
And then if I had nothing pressing, I'd stay parked right there for a bit to make sure he leaves.
The second time I see him, I might radio in "I'll be out at Smith Brothers. Someone banging on the gate. May be that same male from two days ago."
Does the guy run as soon as I get out of the car? If I can chase him down in the first couple hundred feet, I'll do it on foot. If not, I'll chase him in the car as far as I can, yelling on the radio for backup while I do.
Depending on where the gun is, a cop may notice it before your guy is expecting him to. Is it a concealed carry state? Even so, most states require you to notify an officer that you're carrying, and if you don't, they can arrest you. If nothing else, when the cop notices it, he'll be asking for the ID and permit. Different cops will handle the gun differently. Some will prone you out at gunpoint. Some will just tell you not to touch it. Some will ask you to surrender it to them for the duration of the stop.
What do you need or want to happen?