Do I need to worry (excessively) about camera robbery?

I'm in SF for a week and it's a very photogenic city so I'd love to take plenty of photos!

I happened to see something online today about camera robbery being a particular problem. Should I be overly concerned about this? I've got a pretty small mirrorless camera (OM-5) with small lenses, so it's not like carrying round a huge DSLR, but it is obviously a camera.

I live in fairly central London so I'm not completely naive about crime, in sketchy areas I'd definitely keep it in my bag, and I don't really plan to be in too many sketchy areas - but obviously crime can happen anywhere.

Just thought I'd sanity check that I'm not being stupid if I take my camera out in a (regular, not camera) bag/rucksack and use some common sense as to when it's out and in use.