Control is so difficult that persons with ornamental olives aren't going to do it.
Very true. Now many of the landscaping olives that are planted don't even form fruits. The plants/trees that do form fruits usually drop their ripe, dark olives and form messy stained patches of dark purple flesh and pits on any concrete walks or drives beneath. If there are lots of fruits - and there usually are - it can be unpleasant to walk over them.
Coastal Southern California, zone 10, avocado belt, El Nino is coming!