It is mostly happening when you pin the object at he top, or at the bottom of the gallery. What you can so is to keep an eye on pinning, prevent pinning unless it is really necessary. To test, you can simply unpin all around the gallery (and the gallery container itself), if they are pinned, and see the pushing result.
The other possibility would be having a very large background behind the gallery, in such a case, the gallery does not push it down and you get an empty scape in the back. In general, the pushing happens when you have the very bottom border of an object upper then the very upper border of another, then the one at the bottom will be pushed, but if they overlap no pushing takes place.