(written from a Production point of view)
The fourth wall is the metaphorical barrier between the characters of the story and the audience. Generally, characters are unaware that they are being watched, and therefore never address the audience directly and faithfully act according to the rules of the world created for them. However, a few episodes in Star Trek break these rules, thus breaking the fourth wall.
The Original Series
Journey to Babel
The conclusion of "Journey to Babel" involves one of the rare times the fourth wall is broken in Star Trek. (Star Trek: The Original Series 365, p. 219) However, it is a less direct violation of the fourth wall than in "The Changeling". McCoy is looking slightly to the right of the screen and appears to be speaking to Amanda Grayson and Nurse Chapel when he expresses delight at finally having the last word.
Deep Space Nine
Rules of Engagement
For the fourth season installment "Rules of Engagement", it was Ira Steven Behr who came up with the concept of having the characters speak to-camera, effectively breaking the fourth wall. He was determined not to do "just another trial show" (like "The Measure Of A Man" or "Dax"), and while trying to decide what to do to make "Rules of Engagement" stand out, he saw the 1995 Spike Lee movie Clockers. During that film, in a flashback, a character played by Harvey Keitel speaks directly to-camera. Behr thought this was an excellent idea and suggested it to Ronald D. Moore, who concurred, and who wrote it into the DS9 teleplay. Moore describes this device as "sort of breaking the fourth wall, but sort of not, because the actors aren't talking to the audience, they're actually talking to someone in the courtroom." (Star Trek: Deep Space Nine Companion)
In the Pale Moonlight
In the sixth season episode "In the Pale Moonlight", this stylistic technique is taken even further, with Captain Sisko dictating an entire log directly to the camera and audience. However, there is no direct acknowledgment of the audience in the episode itself; instead, the viewer merely takes the perspective of the computer to which Sisko dictates his log.
"Captain's log," spoken by William Shatner, were the first two words viewers heard on the first Star Trek broadcast, "The Man Trap", on 8 September 1966. Obviously, the "real" purpose of the log is to inform the audience about key plot points.
In some of the first episodes of Star Trek: The Original Series, Kirk's log sometimes broke the fourth wall and described information that neither he nor the crew could be aware of. For example, in "The Naked Time", Kirk describes "... but unknown to us...." There clearly wasn't time to make some other log entries; for instance, in "By Any Other Name", Kirk made a log entry about his crew's skulduggery against the episode's villain while he, Kirk, is sitting next to that same villain. Presumably, these log entries were created after the events had transpired, when the captain had time to update his log, and it was either Starfleet procedure or just Kirk's personal habit to record such logs in the present tense.