How can I do this?

How do people disguise the unnatural looking straight edge of a textured piece of ground? I have made a rocky area in my forest clearing, and it looks quite good except for the straight edge.

There are many ways. I usually use the land tools in Edit to raise, lower, smooth, and otherwise shape the land to create a rolling surface that covers parts of exposed rock, pavement, walkways...... Vegetation helps a lot, too. Plant a few low bushes or some flowers. Or pile some rocks randomly to hide the sharp edge. These are exactly the same things you would do in RL .... so be creative and do them in SL too.

Thanks Rolig. Unfortunately I am doing this on a platform in the sky and I don't know if it is possible to use edit terrain on a platform. I'm frightened of accidentally deleting part of it.

No, terrain is only on the ground, and can't affect a prim sky platform.

The only things you can do are to use some non-square prims to "soften" the edge. Maybe a rim made of several sculpted prims could give you a look like a torn-off hunk of soil. I saw a sky build once where the maker did some amazing effects with a layer of prim "clouds" that a sky build seemed to rest on. It wasn't the particle clouds I was seeing, but rather was a walkable surface that looked like cloud tops. And this was before sculpted prims. I never learned just how he did it. The build was deleted before I got curious enough to inspect it closely.

what about using alpha textures which create smooth transitions at the edges of your prims ? Or use sculpted prims as mentioned by Ceera ? ... or use a combination ...

If the layout will allow it, a circle looks more natural than a square for your textured ground, especially if you use a large flat sphere - e.g. the 20x20x0.5 megaprim turned into a sphere is quite a good prim for this purpose, gives you a gentle bump that fades into the surrounding prims. If you make it a sculpty by applying a boulder or rock sculpt texture it can be even better, with the edges flowing quite naturally.

Ceera --

Whoever created the sky platform for Pillow Talk did the same thing, I think. It is VERY effective. I haven't ever stopped to peek carefully at how it's done, although it may be simply particle effect clouds over transparent prims..... Nice, anyway.

Rolig? Would you be so kind as to send me a LM in-world for that "Pillow Talk" sky platform, if it is a publicly accessible area. I would like to look at it in-world.

Thank you Rolig. I checked that LM you sent me.

A very creative approach, but not what I had seen before. Pillow Talk built their platform at the top edge of the Linden particle clouds, and made the platform itself out fo multiple layers of 10 x 10 flat box prims, textured with an alpha-mapped cloud texture. One layer is phantom, and the layer below that can be walked on. And they didn't keep them level, so it creates a rather dreamy effect as you wade between waist deep and ankle deep in the cloud tops. My hat is off to the Builder! Well done.

Nice! That's more than I remembered from my last visit there and trickier than I imagined. Now I have to go back and look more carefully myself. So, there are a couple of other tools the OP might use for softening prim edges -- if he likes fog.

Thank you all so much for the suggestions. I'm a novice at all this. The idea of the circle hit me! I found an item textured to look like a round pebblefilled bubbling spring and I have sunk it into the rock so it covers the square edge that was bothering me. I softened the edges of that with some small rocks and grasses as others suggested. I am SO pleased with it. Thanks again everybody!

theres tools that convert series of photos into a spherical 360° image to put on the inside of a hollow sphere.

theres sculpted prims that make floor bumpy.