William Strunk suggested that the paragraph is the unit of prose composition. It is true that it only takes a sentence to express a thought, but if you want to introduce an idea into an academic discourse, to enter a conversation between specialized knowers, you will need to embed that thought in a paragraph. It is only in the context of a paragraph that a sentence carries the weight of knowledge. To know something is not just the ability to formulate a single true sentence about it; it is the capacity to converse about it. The paragraph expresses a thought and indicates the conversation you are willing to have about it. You know you are writing prose when you are expressing yourself in well-formed paragraphs.
Tony Tost's 1001 Sentences is a good demonstration of the idea that sentences ain't necessarily prose. You can read part of it here, and watch him read a sequel here. Here sentences are clearly being used to make poetry, not prose.
Outside of poetry, Ludwig Wittgenstein's Tractatus Logico-Philosophicus may be the most famous example of a distinctly non-prose work. It opens like this:
1 The world is everything that is the case.
1.1 The world is the totality of facts, not of things.
1.2 The world divides into facts.
Wittgenstein calls these numbered units "propositions"—Sätze in German, which can also mean simply "sentences". They are, in any case, not prose paragraphs. In a way, Wittgenstein contented himself with presenting an "after-the-fact outline" of his book. Wittgenstein is presenting only the core of his thinking; he is not offering any elaboration. One does not get the sense that he is interested in having a conversation about the idea that the world is everything that is the case, does one?
One of my back-burner literary projects is to write a prose version of the Tractatus, a book that simply expands each "key sentence" into a full paragraph of academic prose. In fact, Wittgenstein famously loosened up a little in his later years. His writing (in the Philosophical Investigations) does not quite become "academic", but it does become more recognizably prosaic. One clue to why this happened is found in his drafts of his Remarks on Frazer's Golden Bough, which Rush Rhees cites in his introduction. "When I began [the Tractatus] to talk about the 'world' (and not about this tree or table), was I trying to do anything except conjure up something of a higher order by my words?" (vi). Prose does not appeal to higher orders. It does not conjure. Prose is ordinary. It is straightforward. That's what the word means.
So we can begin to imagine what a prose version of the Tractatus might look like:
The world is everything that is the case. There is a tree outside my window, for example, and its branches are swaying gently in the breeze. I am sitting at my table. On it, there are two books and this notebook, where I am writing these words. The tree is outside my window; that is the case. The notebook is on the table; that, too, is the case. The world is just everything that is the case in that way.
We can think of many variations on this way of writing. But clearly beginning a book with this paragraph, even though it begins with the same sentence as Wittgenstein's "conjuring" act, is not trying to pull off a trick of abstraction. It is explicating "everything that is the case" with examples of particular cases. It is trying to bring "the world" down to earth. It is beginning a conversation.